Information processing method, information processing device, and program

Abstract

The present invention is configured such that, for each section of a three-dimensional model of an object when the three-dimensional model is generated using a plurality of images that result from imaging the object with a plurality of different camera attitudes, one or a plurality of information processing devices is or are used to perform direct-facing score calculation processing to calculate a direct-facing score indicating the extent to which the camera attitudes were direct-facing at the time of imaging the images used for generation, and presentation information generation processing to generate presentation information for a user on the basis of the results of the direct-facing score calculation processing.

Description

Information processing method, information processing device, and program

[0001] The present technology relates to an information processing method, an information processing device, and a program, and relates to a technology that can be applied in the field of three-dimensional model generation, for example.

[0002] One method for 3D modeling of 3D (three-dimensional) objects having a three-dimensional shape is called photogrammetry, which captures images of the 3D object from multiple directions and generates 3D data based on the resulting captured images. Another method is called real-time 3D modeling, which generates 3D data instantly (in real time) based on captured images, posture information, depth, and other information. Furthermore, in recent years, a method collectively known as neural rendering (e.g., NeRF (Neural Radiance Fields)) has been proposed, which constructs neural fields based on the posture of captured images and generates images or 3D models from any viewpoint.

[0003] The following Patent Document 1 describes an image processing device that generates a three-dimensional image by pasting a texture image onto a three-dimensional shape model, and selects one viewpoint image that is most directly facing the patch surface that constitutes the three-dimensional shape model. The following Patent Document 2 describes adding texture information from an image in which a polygon has the largest projection area.

[0004] JP2003-337953A JP11-120372A

[0005] When performing 3D modeling based on photographed images of a 3D object, it is desirable to take as many photographs as possible with a camera pose (position and shooting direction) that faces the 3D object as directly as possible in order to increase the accuracy of the 3D model. On the other hand, to reduce the workload and processing volume, it is desirable to perform photographs efficiently with as few photographs as possible. To perform photographs efficiently, the trajectory of the camera movement and the camera pose during the movement are set, but it is difficult for the user to check whether the actual photographs were taken according to the settings during or immediately after photographing. Therefore, after photographing, the generated 3D model must be checked after a certain amount of time has passed or after leaving the shooting location, and if the accuracy of the 3D model is insufficient, re-photographing must be performed. This makes it difficult to perform the photographing work efficiently. The technologies described in Patent Documents 1 and 2 are applied to texture mapping after photographing and cannot be applied to photography support.

[0006] Therefore, the present disclosure proposes a technique that enables a user to easily and efficiently capture an image of a 3D object using an imaging device.

[0007] An information processing method according to the present technology is an information processing method that, when generating a three-dimensional model of an object using a plurality of images of the object captured at a plurality of different camera poses, executes, by one or more information processing devices, a facing score calculation process that calculates, for each part of the three-dimensional model, a facing score that indicates the degree of facing of the camera pose at the time of capturing the images used for generation, and a presentation information generation process that generates presentation information for a user based on the result of the facing score calculation process. The camera pose referred to in the present disclosure is the camera pose defined by the shooting position and shooting direction. When generating a three-dimensional model of an object from a plurality of images captured at a plurality of camera poses, it is important that each camera pose at the time of capturing is facing the object directly. Therefore, a facing score for each camera pose is calculated, and presentation to a user based on the facing score is possible.

[0008] 1 is an explanatory diagram of an overview of photogrammetry; an explanatory diagram of an overview of real-time 3D modeling; an explanatory diagram of the difference between photogrammetry and real-time 3D modeling; an explanatory diagram of an example of a device configuration according to an embodiment; an explanatory diagram of incremental 3D compositing according to an embodiment; an explanatory diagram of a 3D model generated by incremental 3D compositing according to an embodiment; a block diagram of an information processing device according to an embodiment; an explanatory diagram of an example of a system configuration according to an embodiment; an explanatory diagram of an example of a system configuration according to an embodiment; an explanatory diagram of an example of a system configuration according to an embodiment; an explanatory diagram of an example of a system configuration according to an embodiment; an explanatory diagram of an example of a system configuration according to an embodiment; an explanatory diagram of an example of a system configuration according to an embodiment; an explanatory diagram of an example of a system configuration according to an embodiment; a flowchart of incremental 3D compositing processing according to an embodiment; an explanatory diagram of an example of a display of a 3D model image and a camera attitude according to an embodiment; an explanatory diagram of an example of the functional configuration of an incremental compositing calculation system according to an embodiment; an explanatory diagram of a 3D model image according to an embodiment. 1 is an explanatory diagram of a 3D model image according to a facing score according to an embodiment. FIG. 2 is an explanatory diagram of a 3D model image according to a facing score according to an embodiment. FIG. 3 is an explanatory diagram of a 3D model image in which the facing score is expressed in a part of an embodiment. FIG. 4 is an explanatory diagram of a display of correspondence between a camera pose and a facing score according to an embodiment. FIG. 5 is an explanatory diagram of a display of correspondence between a camera pose and a facing score according to an embodiment. FIG. 6 is an explanatory diagram of a display of correspondence between a camera pose and a facing score according to an embodiment. FIG. 7 is an explanatory diagram of a display of correspondence between a camera pose and a facing score according to an embodiment. FIG. 8 is an explanatory diagram of an example of a screen in which score calculation can be switched according to an embodiment. FIG. 9 is an explanatory diagram of a first calculation example of a facing score according to an embodiment. FIG. 10 is an explanatory diagram of a first calculation example of a facing score according to an embodiment. FIG. 11 is a flowchart of incremental 3D composition processing to which facing score calculation according to the first calculation example of an embodiment is added. FIG. 12 is an explanatory diagram of a second calculation example of a facing score according to an embodiment. FIG. 13 is a flowchart of incremental 3D composition processing to which facing score calculation according to the second and third calculation examples of an embodiment is added. FIG. 14 is an explanatory diagram of a third calculation example of a facing score according to an embodiment.FIG. 10 is an explanatory diagram of a fourth calculation example of an embodiment. FIG. 11 is a flowchart of preview display control processing of an embodiment. FIG. 12 is an explanatory diagram of a preview display with a missing camera pose added according to an embodiment. FIG. 13 is an explanatory diagram of a preview display with a missing camera pose added according to an embodiment. FIG. 14 is an explanatory diagram of a preview display with a missing camera pose and a shooting trajectory added according to an embodiment. FIG. 15 is a flowchart of incremental 3D synthesis processing with facing score calculation and missing camera calculation added according to an embodiment. FIG. 16 is a flowchart of missing camera calculation of an embodiment. FIG. 17 is an explanatory diagram of missing camera calculation of an embodiment. FIG. 18 is an explanatory diagram of a preview display with a missing camera pose and a shooting trajectory added according to an embodiment.

[0009] <1. 3D Modeling> <2. Device Configuration for Incremental 3D Synthesis> <3. System Configuration Example> <4. Incremental 3D Synthesis Processing Example> <5. Presentation of Facing Score and Corresponding Camera Pose> <6. Presentation of Facing Score and Missing Camera Pose> <7. Summary and Modifications>

[0010] In this disclosure, the term "image" includes both still images and moving images. Furthermore, the term "image" refers not only to an image in a displayed state, but also to image data in a signal processing process or a transmission process.

[0011] <1. 3D Modeling> First, photogrammetry will be described. As a method for generating (reconstructing) a model of a three-dimensional shape of an object (also referred to as a 3D object), there is a method called photogrammetry, in which the 3D object is photographed from multiple directions and a 3D model (3D data) is generated based on the multiple captured images. In this disclosure, generating a model of the three-dimensional shape of a 3D object is also referred to as 3D modeling.

[0012] Photogrammetry is a technique for reconstructing a highly accurate three-dimensional model using the principles of triangulation from multiple images taken from various viewpoints. For example, a camera 150 shown in FIG. 1 captures images of a 3D object 100 from multiple viewpoints to obtain multiple captured images. These captured images are then used to perform processes called SfM (Structure from Motion) and MVS (Multi View Stereo), followed by post-processing such as meshing and texturing to generate a 3D model 105.

[0013] In this specification, the term "accuracy" for a 3D model refers not only to the reproducibility of the three-dimensional shape of the target 3D object, i.e., the accuracy and definition of the three-dimensional shape, but also to the reproducibility (accuracy, definition, etc.) of the texture applied to the surface of the 3D model.

[0014] In SfM, for example, corresponding points are searched for between captured images, the position and orientation of the camera are derived using epipolar constraints, and the positions of each corresponding point in three-dimensional space are identified by triangulation based on the camera position and orientation. In this disclosure, these points in three-dimensional space are also referred to as three-dimensional points. In other words, three-dimensional points corresponding to each corresponding point are identified. Then, the entire three-dimensional point group identified in this manner is optimized by bundle adjustment.

[0015] In MVS, for example, a 3D point group derived as described above is used to perform a more detailed search for corresponding points, and 3D points are added.

[0016] As described above, photogrammetry performs a global optimization calculation called bundle adjustment to minimize errors, which allows it to obtain very accurate results, but it also places a heavy burden on the calculations.Furthermore, because it is based on geometric calculations rather than physical measurements, in principle, the higher the resolution of the images used, the more accurate the model can be restored.

[0017] Next, real-time 3D modeling will be described. A different 3D modeling technique from the above-described photogrammetry is called real-time 3D modeling, which generates 3D data instantly (in real time) based on information such as captured images, posture information, and depth. In this technique, as shown in FIG. 2 , for example, a camera 150 captures an image of the 3D object 100 while moving around the periphery of the 3D object 100 as indicated by the dotted line 109. The camera 150 is equipped with not only an image sensor but also a Lidar (Light Detection And Ranging) scanner (dToF (Direct Time of Flight) module), which captures images and detects depth (the distance to the subject).

[0018] In recent years, dToF modules have become smaller and more sophisticated, enabling accurate depth measurement over relatively long distances (e.g., about 5 m) both indoors and outdoors. This has made it easy to experience real-time modeling and capturing at the consumer level.

[0019] Furthermore, the camera 150 is equipped with an inertial sensor that detects inertial information such as the acceleration and angular velocity of the camera 150 .

[0020] In real-time 3D modeling, a three-dimensional orientation calculation process called SLAM (Simultaneous Localization and Mapping) is performed to generate orientation information indicating the position and orientation of the camera 150. In addition, the orientation information and depth are used to update a truncated signed distance function (TSDF), and a 3D model 105 (mesh and texture) is generated by a process called MC (Marching Cubes).

[0021] In SLAM, for example, the position and orientation of the camera 150 are estimated based on captured images and inertial information (self-localization). In TSDF updates, depth is associated with voxels to detect volume. In MC, isosurfaces are calculated using adjacent voxels. With real-time orientation information from SLAM, the volume of a voxel can be detected without using a point cloud by overlaying depth (how far the light rays have reached) from multiple frames. Using a voxel representation makes it possible to estimate viewpoints that are in shadow and must be photographed (missing viewpoints). This makes it possible to detect perforated and protruding structures in the 3D object 100.

[0022] There is also a 3D modeling technique collectively known as neural rendering, which constructs neural fields based on the posture of a captured image and the captured image, and generates images and 3D models from any viewpoint. For example, NeRF (Neural Radiance Fields) has been proposed.

[0023] Each of these 3D modeling techniques has different characteristics, and no technique is superior in all respects. Figure 3 compares the characteristics of photogrammetry and real-time 3D modeling. As shown in Figure 3, photogrammetry uses SfM (including localization) and MVS, while real-time 3D modeling uses localization (SLAM) and TSDF. Furthermore, comparing the data used, photogrammetry uses only image data, while real-time 3D modeling uses depth and orientation data in addition to image data. Furthermore, comparing processing times, photogrammetry requires a long time, from several minutes to several tens of hours, while real-time 3D modeling allows for nearly instantaneous processing at 30 fps (frames per second).

[0024] In addition, when comparing the required computing power, photogrammetry requires computing power at the level of a high-end CPU (Central Processing Unit) and GPU (Graphics Processing Unit), while real-time 3D modeling requires computing power at the level of a mobile AP (Application Processor).In addition, when comparing the resolution of the generated models, photogrammetry is relatively high-resolution, depending on the resolution, number, and shooting method of the captured images, while real-time 3D modeling is relatively low-resolution, depending on the depth and self-position estimation accuracy, etc.

[0025] Furthermore, the internal representation of the generated 3D data is point cloud-based in photogrammetry, whereas it is voxel-based in real-time 3D modeling. Furthermore, while there are no restrictions on subject size or resolution in photogrammetry, real-time 3D modeling is sensor-dependent. Furthermore, when comparing the absolute accuracy of the model, photogrammetry is relatively high because it is optimized using bundle adjustment, while real-time 3D modeling is relatively low, depending on the sensor and self-localization accuracy. Furthermore, when comparing scale, photogrammetry is indefinite (the size is unknown), whereas real-time 3D modeling is unique (the absolute size is known).

[0026] Photogrammetry and real-time 3D modeling have the following differences in characteristics. In other words, applying real-time 3D modeling can reduce the workload and processing volume of 3D modeling compared to applying photogrammetry or neural rendering. However, applying photogrammetry or neural rendering can generate a more accurate 3D model than applying real-time 3D modeling. The technology described in this embodiment can be applied when photographing a 3D object 100 using these various 3D modeling methods.

[0027] 2. Device Configuration for Incremental 3D Composition> Figure 4 shows an example of various device configurations used for 3D modeling according to the embodiment. Incremental 3D composition as referred to in the present disclosure, which will be described in detail later, is a technique in which, in the process of photographing a 3D object 100, each time an image is acquired, the image is additionally input into a 3D modeling process to generate a 3D model.

[0028] In this embodiment, the device configuration for 3D modeling includes a photographing device 1, a UI (User Interface) device 2, and a 3D model generating device 3, as shown in FIG.

[0029] The photographing device 1 is a device that can photograph a 3D object 100 and output an image. A specific example of the photographing device 1 is a mobile body such as a drone 10 equipped with a camera 11. The mobile body equipped with the camera 11 is not limited to the drone 10, and may be other flying bodies, vehicles, ships, submarines, etc. Further, the photographing device 1 may also be a camera 11 that is handheld by a person to take pictures, or an information processing device such as a smartphone 12 equipped with a photographing function.

[0030] The UI device 2 is a device that provides various interfaces to the user. In this case, the interfaces include an input interface for operating the imaging device and an output interface for presenting a 3D model and imaging posture. The UI device 2 may be an information processing device such as a smartphone 12, a personal computer 13, or a tablet device (not shown). The UI device 2 may also be a control device for a moving object such as the drone 10.

[0031] The 3D model generation device 3 is a device that generates a 3D model using captured images. For example, in this embodiment, a device that performs incremental 3D composition is taken as an example. The 3D model generation device 3 can be any of various information processing devices, such as a server device 14 such as a cloud server, a personal computer 13, or a smartphone 12.

[0032] In such a configuration, the photographing device 1 photographs the 3D object 100, and the photographed images are transferred, for example, one by one, sequentially to the 3D model generation device 3. The transfer of images to the 3D model generation device 3 may be performed directly by the photographing device 1, or the photographing device 1 may transfer the images to the UI device 2, and the UI device 2 may transfer the images to the 3D model generation device 3.

[0033] The 3D model generation device 3 generates a 3D model based on the acquired images. In this case, the 3D model generation device 3 performs incremental 3D composition. The 3D model generation device 3 then transmits the 3D model generated at each point in time to the UI device 2. The UI device 2 can display the received 3D model. This allows the user of the UI device 2 to check the 3D model as it gradually grows during shooting.

[0034] 5 shows the difference between normal 3D composition and incremental 3D composition. In normal 3D composition, a large number of images G1, G2, ... G(n) are first taken as a photography task. After the photography is completed, the images G1, G2, ... G(n) are input into a 3D model generation process in a 3D model generation device 3. Depending on the size of the 3D object 100 and the required accuracy, this 3D model generation process takes about half a day to a day.

[0035] Thereafter, a 3D model Mc is generated based on the images G1, G2, ... G(n), and the user can visually confirm a display image dM of the 3D model Mc. In this case, the user cannot determine whether the photographing work was appropriate until checking the display image dM of the 3D model Mc. For example, if the user views the display image dM and finds that some parts are insufficient in accuracy, they may have to take the photograph again.

[0036] On the other hand, incremental 3D synthesis performs 3D synthesis sequentially in parallel with the shooting work. For example, as shown in the figure, images G1, G2, ... G(n) obtained during the shooting work are sequentially transferred to the 3D model generation device 3 and input into the 3D model generation process. This allows the 3D model generation device 3 to perform 3D modeling based on the multiple images acquired at each time point, and to generate 3D models M2, M3, ... M(n) at each time point. For example, as schematically shown in Figure 6, the 3D models M2, M3, M4, ... M(n) gradually grow as the 3D model grows.

[0037] Then, when the 3D model generation device 3 generates a 3D model M2, it transfers it to the UI device 2, allowing the user to check the 3D model M2 at that time using the display image dM2. Also, when the 3D model generation device 3 generates a 3D model M3, it transfers it to the UI device 2, allowing the user to check the 3D model M3 at that time using the display image dM3. After the shooting work is completed, a 3D model M(n) can be generated based on the images G1, G2, ... G(n), allowing the user to check the 3D model M(n) in its state after shooting is complete using the display image dM(n). Therefore, the user can check the progress and final state of the 3D model from during shooting to immediately after shooting. This provides information to determine whether the shooting up to that point was appropriate during or after shooting is completed.

[0038] The 3D modeling process as incremental 3D compositing may be performed in a somewhat simplified manner if it is only for confirmation during shooting, because the high-precision 3D model as the actual finished product can be created after shooting is completed in the same way as normal 3D compositing.

[0039] Incremental 3D compositing allows a user to check a 3D model during or immediately after shooting, allowing the user to immediately evaluate the shooting work and, if insufficient, to reshoot, thereby promoting the efficiency of the overall 3D model production work. However, even if the user actually checks the 3D model and determines that the accuracy is insufficient, it is difficult for the user to know what camera pose or shooting trajectory to use for reshooting. Therefore, in this embodiment, in addition to monitoring the 3D model using incremental 3D compositing, inappropriate camera poses among the camera poses captured are presented to the user as abnormal camera poses, and camera poses that the user wants to capture an image are presented as insufficient camera poses.

[0040] For example, the 3D model generation device 3 can generate information about the camera position and transfer it to the UI device 2, making it easier for the user to understand how to retake the image. This will be described in detail later.

[0041] Up to this point, incremental 3D composition has been described, but the actual device forms of the photographing device 1, UI device 2, and 3D model generation device 3 used to generate the 3D model illustrated in Figure 4 are diverse. The photographing device 1, UI device 2, and 3D model generation device 3 may each be separate devices, or a single device may serve as both. For example, the smartphone 12 may function as both the photographing device 1 and the UI device 2. Furthermore, the UI device 2 and the 3D model generation device 3 may be implemented as a single device, such as the smartphone 12 or personal computer 13. Furthermore, the photographing device 1, UI device 2, and 3D model generation device 3 may be implemented as a single device, such as a single smartphone 12.

[0042] That is, the photographing device 1 may be realized by a device having at least a photographing function, and the UI device 2 and the 3D model generation device 3 may be realized by an information processing device. The 3D model generation device 3 may be an information processing device such as a cloud server, or may be an information processing device capable of communicating with the UI device 2 or the photographing device 1 via a network such as a LAN.

[0043] Here, we will explain information processing devices that can be configured as the 3D model generation device 3 and the UI device 2. For example, various information processing devices such as a server device 14, a smartphone 12, a personal computer 13, or a tablet terminal or workstation (not shown) can be used as the 3D model generation device 3 or the UI device 2. The configuration of an information processing device 70 in these cases is shown in FIG.

[0044] 7 executes various processes in accordance with programs stored in a nonvolatile memory unit 74 such as a read-only memory (ROM) 72 or an electrically erasable programmable read-only memory (EEP-ROM), or programs loaded from a storage unit 79 to a random access memory (RAM) 73. The RAM 73 also stores data necessary for the CPU 71 to execute various processes, as appropriate.

[0045] The image processing unit 85 is configured as a processor that performs various types of image processing, such as 3D modeling processing, rendering, database processing, image processing including color and brightness adjustment processing and color conversion processing for captured images, encoding processing, decoding processing, image editing processing, and image analysis and detection processing, or a combination of these processes.

[0046] The image processing unit 85 can be realized by, for example, a CPU separate from the CPU 71, a graphics processing unit (GPU), a general-purpose computing on graphics processing units (GPGPU), an artificial intelligence (AI) processor, etc. The image processing unit 85 may be provided as a function within the CPU 71.

[0047] The CPU 71, ROM 72, RAM 73, nonvolatile memory unit 74, and image processing unit 85 are interconnected via a bus 83. To this bus 83, an input / output interface 75 is also connected.

[0048] An input unit 76 consisting of operators and operation devices is connected to the input / output interface 75. For example, the input unit 76 may be various operators and operation devices such as a keyboard, a mouse, keys, a trackball, a dial, a touch panel, a touchpad, or a remote controller. The input unit 76 detects a user operation, and a signal corresponding to the input operation is interpreted by the CPU 71. A microphone may also be used as the input unit 76. Voice uttered by the user may also be input as operation information.

[0049] A display unit 77, which may be an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, and an audio output unit 78, which may be a speaker, are connected to the input / output interface 75 either integrally or separately. The display unit 77 displays various types of information. The display unit 77 may be, for example, a display device provided in the housing of the information processing device 70 or a separate display device connected to the information processing device 70. The display unit 77 displays various images, operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), on the display screen based on instructions from the CPU 71.

[0050] The input / output interface 75 may be connected to a storage unit 79 and a communication unit 80 configured from a hard disk drive (HDD) or a solid state drive (SSD).

[0051] The storage unit 79 can store various data and programs. A database can also be configured in the storage unit 79.

[0052] The communication unit 80 performs communication processing via a transmission path such as the Internet, and communication with various devices such as external DBs, editing devices, and information processing devices via wired / wireless communication, bus communication, and the like.

[0053] A drive 81 is also connected to the input / output interface 75 as needed, and a removable recording medium 82 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately attached. The drive 81 allows video data, various computer programs, and the like to be read from the removable recording medium 82. The read data is stored in the storage unit 79, and the video and audio contained in the data are output on the display unit 77 and the audio output unit 78. In addition, the computer programs and the like read from the removable recording medium 82 are installed in the storage unit 79 as needed.

[0054] In this information processing device 70, for example, software for the processing of this embodiment can be installed via network communication by the communication unit 80 or via a removable recording medium 82. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, etc.

[0055] 3. System Configuration Examples In the present embodiment, various system configuration examples capable of performing incremental 3D compositing will be described. Note that in the configuration examples described below in FIGS. 8 to 19, the "system" shown in each block is not necessarily limited to a single device. The "system" refers to hardware, software, or an operation or processing function realized by a combination of these. For example, it is to be considered that the "systems" shown in FIGS. 8 to 19 are provided as each functional unit within any of the hardware devices such as the image capture device 1, UI device 2, or 3D model generation device 3 shown in FIG. 4.

[0056] 8 shows an example of a configuration in which incremental 3D compositing is performed on a cloud or on-premise server device 14, separate from the shooting site. The figure shows a moving system 40, a shooting system 41, a control system 42, an image transmission system 43, an image reception system 44, an incremental compositing calculation system 45, an information transmission system 46, an information reception system 47, and a display system 48.

[0057] The moving system 40 is a functional unit that moves the camera 11 during shooting, and is realized by a moving body such as the drone 10. The shooting system 41 is a functional unit that shoots the 3D object 100, and is realized by the camera 11, etc.

[0058] The control system 42 is a functional unit that controls the movement of the mobile system 40 and the photography of the photography system 41. The control system 42 transmits control signals to the mobile system 40 to instruct the trajectory to move for photography (photography trajectory) and to instruct movement, causing the mobile system 40 to perform the movement required for photography. For example, the control system 42 sets a flight plan as a photography trajectory for the drone 10 and controls flight according to the flight plan. The control system 42 also sequentially sends photography triggers to the photography system 41 in accordance with flight according to the flight plan, causing the photography system 41 to perform image photography.

[0059] The control system 42 may be provided in a smartphone 12 or the like as a UI device 2, or may be provided in a control device of the drone 10, or the like.

[0060] The photographing trigger may be given from the moving system 40 to the photographing system 41. Alternatively, the photographing system 41 may generate a photographing trigger internally.

[0061] The photographing system 41 photographs in response to a photographing trigger. The photographed image is transmitted and output by the image transmission system 43. At this time, the photographing system 41 or the image transmission system 43 may perform image reduction processing to reduce the amount of data to be transmitted or to simplify 3D modeling. The image transmission system 43 may be provided in a device present at the photographing site, such as the photographing device 1 or the UI device 2.

[0062] The captured images are sequentially transmitted from the shooting site to, for example, a cloud server by the image transmission system 43. Note that instead of transmitting the images, the images may be sent to the cloud or an on-premise information processing device by transferring a physical recording medium, such as a memory card, on which the images are recorded.

[0063] The image receiving system 44, the incremental synthesis calculation system 45, and the information transmitting system 46 are provided in the 3D model generating device 3, for example.

[0064] When the captured images are received by the image transmission system 43, the incremental synthesis calculation system 45 performs processing as incremental 3D synthesis, generating point cloud data and camera posture information as the 3D model up to that point. The camera posture information includes an estimated camera posture estimated as the camera posture at the time of shooting, an abnormal camera posture that is an inappropriate posture among the estimated camera postures, and an insufficient camera posture that is a camera posture that recommends additional shooting to improve the accuracy of the 3D model. In this embodiment, for example, the incremental synthesis calculation system 45 also calculates, for each part of the 3D model, information on a facing score that indicates the degree of facing of the camera posture at the time of shooting the image used for generation.

[0065] For example, the facing score is calculated by calculating the degree of facing between a point and the camera pose using input information such as normal information and position information for each point in the point cloud, the pose (position and shooting direction) of the camera that contributed to the generation of the point, and optical parameters.

[0066] The point cloud data, camera posture information, facing scores, etc. generated by the incremental synthesis calculation system 45 are transmitted by an information transmission system 46 and received by an information receiving system 47. Then, a display system 48 performs display based on the point cloud data, camera posture information, facing scores, etc., i.e., display of a 3D model, display related to the camera posture, or display reflecting the facing scores thereon. The information receiving system 47 and the display system 48 are provided in the UI device 2. The above is one example of a system configuration.

[0067] Differences from the previously described system configurations will be described from Figure 9 onwards. Like Figure 8, Figure 9 also shows an example configuration in which incremental 3D composition is performed on a cloud or on-premise server device 14, etc., separate from the shooting site. What differs from Figure 8 is the mobile system 40A. The mobile system 40A shows a configuration in which a person manually moves or changes the posture of the camera 11, or a configuration in which a person holds the camera 11 and takes pictures.

[0068] For example, a person holds the camera 11 in his / her hand and takes pictures while moving around the periphery of the 3D object 100. The rest is the same as in Fig. 8, and the taken images are transmitted by the image transmission system 43 and provided to the incremental synthesis calculation system 45 via the image receiving system 44.

[0069] 10 shows an example in which display is performed by a display system 48 on-premise. A moving system 40, a photographing system 41, a control system 42, and an image transmission system 43 are provided in the equipment at the photographing site, and photographed images are sequentially transmitted. An image receiving system 44, an incremental synthesis calculation system 45, and a display system 48 are provided on-premise. As a result, a display based on the point cloud data, camera attitude information, and facing score calculated by the incremental synthesis calculation system 45 is displayed on the display system 48 provided in the on-premise UI device 2.

[0070] 11 shows an example in which, similar to FIG. 10 , when the camera 11 is manually moved as in FIG. 9 , display is performed, for example, by a display system 48 on-premise. A movement system 40A, an image capture system 41, and an image transmission system 43 are provided in equipment at the capture site, and captured images are sequentially transmitted. An image receiving system 44, an incremental synthesis calculation system 45, and a display system 48 are provided on-premise. As a result, a display based on the point cloud data, camera posture information, and facing score calculated by the incremental synthesis calculation system 45 is displayed on the display system 48 provided in the on-premise UI device 2.

[0071] 12 shows an example in which the incremental composition calculation system 45 is provided in an edge terminal, that is, an information processing device such as a terminal device provided at a photography site. Images captured by a moving system 40, a photography system 41, and a control system 42 are sequentially input to the incremental composition calculation system 45 of the on-site equipment. Note that either the photography system 41 or the incremental composition calculation system 45 may perform image reduction processing to simplify 3D modeling.

[0072] The incremental synthesis calculation system 45 is provided with a calculation unit 50 that performs processes such as incremental synthesis, camera posture determination, and facing score calculation, and a display unit 60 that displays as a user interface. The calculation unit 50 performs incremental 3D synthesis and the like to obtain point cloud data, camera posture information, and facing scores, and the display unit 60 displays the data accordingly. This allows the 3D model, camera posture, and facing scores to be checked during shooting at a shooting location, etc.

[0073] 13 shows an example in which, similar to FIG. 12, an incremental composition calculation system 45 is provided in an information processing device at the shooting site as an edge terminal when the camera 11 is manually moved as in FIG. 9. Images captured by a moving system 40A and a shooting system 41 are sequentially input to the incremental composition calculation system 45 of the on-site equipment. In the incremental composition calculation system 45, a calculation unit 50 performs incremental 3D composition and the like to obtain point cloud data, camera posture information, and facing scores, and the corresponding displays are performed on a display unit 60.

[0074] 14 shows an example in which an incremental synthesis calculation system 45 is provided on an edge terminal and a display system 48 is provided separately. Images captured by a mobile system 40, an imaging system 41, and a control system 42 are sequentially input to the incremental synthesis calculation system 45 of the field equipment. The incremental synthesis calculation system 45 generates point cloud data, camera posture information, and a facing score. This information is transmitted by an information transmission system 46, received by an information receiving system 47, and transferred to a display system 48. The display system 48 displays the point cloud data, camera posture information, and facing score according to the data.

[0075] 15 shows an example in which, in a case where the camera 11 is moved manually as in FIG. 9 , the incremental synthesis calculation system 45 is provided in an information processing device as an edge terminal, as in FIG. 14 , and a display system 48 is provided separately. Images captured by the moving system 40A and the imaging system 41 are sequentially input to the incremental synthesis calculation system 45 of the field equipment, and point cloud data, camera posture information, and facing scores are generated. This information is transmitted by an information transmission system 46, received by an information receiving system 47, and transferred to a display system 48. The display system 48 displays information according to the point cloud data, camera posture information, and facing scores.

[0076] 16 shows an example in which an incremental compositing calculation system 45 is provided in the cloud and linked to the edge side. In this case, an incremental compositing pre-processing calculation system 49 is provided in an information processing device on the edge side, for example, on the shooting site. The incremental compositing pre-processing calculation system 49 performs processing to extract information necessary for 3D modeling, such as feature amounts, from captured images.

[0077] In this case, the incremental synthesis preprocessing calculation system 49 sequentially calculates feature quantities and the like for the captured images. The calculation results are transferred by the information transmission system 58 to the incremental synthesis calculation system 45 via the information reception system 59 on the cloud side.

[0078] The incremental composition calculation system 45 performs incremental 3D composition based on the calculation results of the preprocessing, generating point cloud data, camera posture information, and a facing score. This information is transmitted by an information transmission system 46, received by an information receiving system 47, and transferred to an edge-side display system 48. The display system 48 displays the point cloud data, camera posture information, and facing score according to the data. By transmitting the calculation results of the preprocessing performed by the incremental composition preprocessing calculation system 49 to the cloud side, the amount of data transferred can be reduced compared to when transmitting images.

[0079] 17 shows an example in which, in a case where the camera 11 is moved manually as in Fig. 9, an incremental composition calculation system 45 is provided in the cloud and cooperates with the edge side, similar to Fig. 16. The processing after the incremental composition pre-processing calculation system 49 is the same as in Fig. 16.

[0080] 18 is an example of a system in which the edge side and the cloud side cooperate with each other, similar to FIG. 16 , except that an information transmission system 58 from the edge side transmits both the calculation results of the incremental combining preprocessing calculation system 49 and the captured image to the cloud side. FIG. 19 is an example of a system in which the edge side and the cloud side cooperate with each other, similar to FIG. 17 , except that an information transmission system 58 from the edge side transmits both the calculation results of the incremental combining preprocessing calculation system 49 and the captured image to the cloud side. The information transmission system 58 may transmit the input calculation results and images after performing a process such as selecting and rejecting them based on predetermined criteria.

[0081] In this embodiment, various system configuration examples such as those described above are conceivable, and various system configurations other than those described above are also conceivable.

[0082] 4. Example of Incremental 3D Compositing Processing An example of incremental 3D compositing processing executed in the system of the embodiment will be described with reference to Fig. 20. Note that Fig. 20 shows the processing divided into phases PH1, PH2, PH3, and PH4.

[0083] The processing example in FIG. 20 is an example in accordance with the system configuration in FIG. 8, where phase PH1 is processing by the imaging system 41 provided in the imaging device 1, phase PH2 is processing by the image transmission system 43, phase PH3 is processing by the 3D model generation device 3 (image receiving system 44, incremental synthesis calculation system 45, information transmission system 46), and phase PH4 is processing by the UI device 2 (information receiving system 47 and display system 48).

[0084] In step S1, photography is performed by the photography system 41. During photography, the process of Fig. 20 is performed each time a photographed image is obtained. The photographed image is subjected to reduction processing, for example, by the image transmission system 43 in step S11, and then transmitted to the 3D model generation device 3 in step S12.

[0085] On the 3D model generating device 3 side, when the image receiving system 44 receives an image in step S20, processing is performed by the incremental synthesis calculation system 45. The incremental synthesis calculation system 45 extracts feature points from the image in step S21, and matches the feature points between the currently received image and images previously received in step S22.

[0086] In step S23, the incremental synthesis calculation system 45 estimates the camera orientation at the time of capturing the currently received image based on the matching results of the multiple images, and stores the estimated camera orientation at step S24. The estimated camera orientation is stored as the orientation of the camera 11 at the time of capturing the currently received image.

[0087] In step S25, the incremental synthesis calculation system 45 performs stereo matching of the multiple images including the current new image to estimate a depth image. In step S26, the incremental synthesis calculation system 45 performs integration processing of the depth values, and based on the integration processing, generates point cloud data having three-dimensional position information and pixel information, which is stored in step S27. This point cloud data becomes information for forming a 3D model based on the multiple images up to the currently received image.

[0088] The estimated camera attitude and point cloud data generated and stored as described above are transmitted to the UI device 2 by the information transmission system 46 in step S28.

[0089] On the UI device 2 side, when the estimated camera attitude and point cloud data are received by the information transmission system 46 in step S51, the display system 48 displays a 3D model and estimated camera attitude according to them in step S52.

[0090] Note that steps S25, S26, and S27 enclosed by dashed lines in the figure may not be executed in some cases, such as when the UI device 2 has low processing power and displays only the estimated camera orientation and does not display the 3D model.

[0091] 21 shows an example of a screen display by the display system 48 in the UI device 2. For example, the UI device 2 displays a live view of the image captured by the camera 11 during shooting, but the screen display can be switched to a preview screen 130 of the 3D model by a user operation or the like.

[0092] The preview screen 130 displays a 3D model image 131 based on the point cloud data generated by incremental 3D synthesis, as well as a frustum, which is a posture presentation image serving as the estimated camera posture 20. The estimated camera posture 20 is, for example, a quadrangular pyramid shape, and represents the camera position (three-dimensional position) and shooting direction when capturing the image. This preview screen 130 allows the user to check the camera postures captured so far and the generated 3D model.

[0093] Furthermore, although the figure is shown using shading in dots, the preview screen 130 can also express the facing score by coloring the surface of the 3D model image 131. For example, the darker the green, the higher the score for image quality or information volume, and the darker the red, the lower the score. The facing score has a relatively high correlation with the accuracy of that area. Therefore, an area with a high facing score can be determined to be an area with a high accuracy of the 3D model.

[0094] Furthermore, in this embodiment, not only is the estimated camera pose 20 displayed, but camera poses that deviate from the original desirable pose are also displayed as abnormal camera poses 21. Furthermore, camera poses that are recommended for photographing in order to improve the accuracy of the 3D model may be displayed as insufficient camera poses 23, and poses that are not the original poses but are acceptable may be displayed as acceptable camera poses 22. It is conceivable that the estimated camera pose 20, abnormal camera pose 21, acceptable camera pose 22, and insufficient camera pose 23 can be recognized by the user by changing the display mode, for example, the color, brightness, size, or blinking of the frustum.

[0095] In the processing example of FIG. 20, the incremental 3D composition processing has been described without mentioning the processing for display according to the facing score, but hereinafter, the processing for displaying according to the facing score will be described.

[0096] <5. Presentation of Facing Score and Corresponding Camera Pose> Incremental 3D synthesis allows a 3D model image 131 such as that shown in FIG. 21 to be displayed during or immediately after shooting. However, since the user can recognize the facing score for the estimated camera pose at an early stage, reshooting as additional shooting can also be performed efficiently.

[0097] 22 shows processing functions provided in the system of this embodiment, for example, in the incremental synthesis calculation system 45 and the display system 48. One or both of the incremental synthesis calculation system 45 and the display system 48 constitute an incremental synthesis calculation unit 51, a presentation information generation unit 53, a storage unit 54, a facing score calculation unit 55, and a missing camera posture calculation unit 56.

[0098] The incremental synthesis calculation unit 51 is a function that performs processing as the incremental 3D synthesis described in FIG. 20 . The presentation information generation unit 53 is a function that sets and transmits presentation information to the user based on the facing score, estimated camera pose 20, missing camera pose 23, etc. The memory unit 54 is a function that stores various information during various processes of the incremental synthesis calculation system 45. The facing score calculation unit 55 is a function that calculates a facing score for each part of the 3D model. Each part of the 3D model may be, for example, a part of a single point in a point cloud, or a part as a region of a certain number of points on the 3D model. The facing score is a value that indicates the degree of facing of the camera pose at the time of capturing the image used to generate that part of the 3D model. The missing camera pose calculation unit 56 is a function that generates a missing camera pose.

[0099] In terms of specific devices, the configuration of FIG. 22 may be provided in the 3D model generation device 3 of FIG. 4, or, for example, the 3D model generation device 3 and the UI device 2 may each have a part of the functions of FIG. 22.

[0100] A display example that reflects the facing scores will be described below. Fig. 23 shows an example of a normal 3D model image 131. This is an example that does not express the facing scores of each region.

[0101] In contrast, FIG. 24 shows an example of a 3D model image 131 that reflects the facing scores calculated for each region of the 3D model. Regions with high facing scores are designated as high score regions 28, and regions with low facing scores are designated as low score regions 29, and are indicated in the figure by pointillist shading. For example, in an actual color image, the level of the facing score may be indicated by differences in color and its shading. In addition, for example, a score bar 35 may be displayed, and the level of the facing score (the accuracy of the 3D model) may be presented so that the user can grasp for each region the level of the facing score (the accuracy of the 3D model) by the color and density on the 3D model image 131. By indicating the facing score value by color and its shading, facing scores with a certain numerical range can be presented in an easy-to-understand manner.

[0102] 25 shows an example in which the facing score is expressed by two values ​​rather than by color shading. For example, a high score area 28 is purple, and a low score area 29 is yellow. This is suitable for simply presenting areas with high and low accuracy.

[0103] FIG. 26 shows an example in which a part of the range of the 3D model image 131 is set as the score presentation area 36, ​​and the facing score is presented only in that range by using different colors or shades of color.

[0104] There are other possible examples of displaying the facing score, but as in the above examples, by displaying the 3D model image 131 with a display mode according to the facing score for each region, the quality of each part of the 3D model can be presented.

[0105] 27 , the preview screen 130 displays the estimated camera pose 20 as the camera pose at which the image was captured, in addition to the 3D model image 131 representing the facing score. At this time, the estimated camera pose 20 is displayed in such a way that the facing score is reflected.

[0106] For example, the example in FIG. 27 shows an example in which an estimated camera pose 20L (hereinafter referred to as a "low-score camera pose 20L") corresponding to an area in the 3D model image 131 where the facing score is lower than a predetermined value is highlighted with a highlight frame 30. The highlight frame 30 is an example of highlighting a low-score camera pose 20L. For example, the low-score camera pose 20L may be made clear by flashing or displaying it in high brightness. In this way, camera poses with low facing scores can be presented to the user.

[0107] 28, camera poses with low facing scores can be presented by displaying the low-score camera poses 20L in a color different from that of the other estimated camera poses 20. Alternatively, the low-score camera poses 20L may be displayed in the same color as the color of the corresponding area on the 3D model image 131 (i.e., the color according to the facing score). These methods also make it possible to present camera poses with low facing scores to the user.

[0108] 29 shows an example in which a low-score camera pose 20L and its corresponding area on the 3D model image 131, that is, an area within the angle of view of the image captured in the low-score camera pose 20L, are linked by a connecting line 31. Linking the area with a low facing score to the camera pose makes it easier for the user to understand the correspondence.

[0109] 30 shows an example of displaying a low-score frame 32 that groups together a plurality of low-score camera poses 20L and a low-score frame 33 that indicates a corresponding area on the 3D model image 131 where the facing scores are low, linked by a connecting line 31. This makes it possible to present to the user which camera poses the areas with low facing scores are associated with.

[0110] As in the above examples, by presenting a posture presentation image that represents the camera posture (estimated camera posture 20) at which the image was captured, for example, using a frustum, and by highlighting the posture presentation image corresponding to the portion where the facing score is less than a predetermined value, the user can easily understand which estimated camera posture 20 had a poor facing situation. Furthermore, by clearly indicating the correspondence between the low-score camera posture 20L and the region of the 3D model image 131, the user can easily recognize which low-score camera posture 20L should be re-captured in order to improve the accuracy of each region.

[0111] As will be described below, there are multiple possible methods for calculating the facing score. The score calculation method may differ depending on the purpose or use of the 3D modeling. A score calculation switch button 34 may also be displayed on the preview screen 130 as shown in FIG. 31 so that the user can operate it. For example, when a display is being displayed according to the facing score in a first calculation example described below, if the user operates the score calculation switch button 34, the display may be switched to a display according to the facing score in a second calculation example.

[0112] Next, calculation examples of the facing score will be described. First, a first calculation example will be described with reference to Figs. 32 and 33. The first calculation example is an example that is suitable for use when determining the level of detail with which the 3D object 100 was photographed. In this first calculation example, the angle with respect to the surface of the 3D object 100 and the distance between the 3D object 100 and the camera 11 are reflected as scores.

[0113] 32 shows a specific point 101 as the 3D shape of a certain region of a 3D object 100, along with camera views showing the optical center 200 and the imaging angle of view 201 for capturing two images. The two illustrated camera views are examples of camera views that contribute to generating the 3D shape of the specific point 101. For each camera view, on the image (or image sensor), a pixel through which a ray 211 connecting the optical center 200 and the specific point 101 passes becomes a corresponding pixel 202 that corresponds to the specific point 101.

[0114] Here, for each camera view, a rectangular corresponding pixel 202 is projected onto a vertical plane 102 as shown in Fig. 33. The vertical plane 102 is a plane perpendicular to a normal 103 of a specific point 101. The normal 103 is found by fitting a plane to a local point group using, for example, the least squares method, and taking the vertical direction of the plane as the normal.

[0115] The four corners of the corresponding pixel 202 are projected onto the vertical plane 102 within the range of points P1, P2, P3, and P4. The area of ​​the rectangular projection area formed by points P1, P2, P3, and P4 is calculated. The greater the deviation from the facing state, the larger the area becomes. In other words, the facing score can be calculated based on the size of the area.

[0116] A processing example of the 3D model generation device 3 (incremental synthesis calculation system 45, etc.) to which such a first calculation example is applied is shown in Fig. 34. Fig. 34 shows a processing example of the incremental synthesis calculation system 45, etc., as phase PH3 in Fig. 20. The processing from step S20 to step S27 in Fig. 20 can be said to be mainly processing by the incremental synthesis calculation unit 51 in Fig. 22, but the processing example in Fig. 34 is obtained by adding processing by the facing score calculation unit 55 and the presentation information generation unit 53 to this.

[0117] The processes of steps S20 to S27 are as described above. In step S31, the incremental synthesis calculation system 45 generates and stores point cloud data during camera posture estimation. The point cloud data in this case is point cloud data having three-dimensional position information including distance information measured based on matching of feature points such as edge portions within the image. The point cloud data of step S27 described above is a point cloud for each pixel of the image, but the point cloud data of step S31 is a point cloud of feature points within the image, and can be said to be point cloud data that partially covers the image plane.

[0118] In step S32, the incremental synthesis calculation system 45 temporarily stores the data obtained in steps S25, S26, and S27 as intermediate data, that is, depth information within the image and point cloud data for each pixel.

[0119] In step S40, the incremental synthesis calculation system 45 performs processing to generate shape data for a 3D model based on the point cloud data generated and stored in step S27. Specifically, it performs noise reduction, mesh generation, and the like.

[0120] The point cloud data, estimated camera posture, intermediate data, and shape data obtained during camera posture estimation are input to the process of step S41. Steps S41, S42, and S43 constitute the facing score calculation process.

[0121] In step S41, the incremental synthesis calculation system 45 extracts a camera view (estimated camera pose) that contributes to generating the 3D shape of a certain region to be processed on the 3D model (the specific point 101 described above). In step S42, the incremental synthesis calculation system 45 calculates the area of ​​a rectangular projection region of the corresponding pixel 202 in the extracted camera view onto the object surface (the vertical plane 102 at the specific point 101). In other words, the area of ​​a rectangle formed by points P1, P2, P3, and P4 is calculated.

[0122] In step S43, the incremental synthesis calculation system 45 calculates the area of ​​the rectangular projection region for each corresponding pixel 202, and then calculates statistics of the calculated areas to obtain a facing score for the region. The statistics may be a maximum value, a minimum value, an average value, or the like. The above-described processing of steps S41, S42, and S43 is performed for each portion of the 3D model (each region of a predetermined size), thereby calculating a facing score for each portion.

[0123] The facing score calculated as described above, the estimated camera posture obtained in step S24, and the point cloud data in step S27 are transmitted to the UI device 2 by the information transmission system 46 in step S28 as information to be presented to the user.

[0124] On the UI device 2 side, in steps S51 and S52 shown in FIG. 20, the display system 48 displays a corresponding 3D model and estimated camera posture, and displays them in a mode corresponding to their facing scores.

[0125] A second calculation example of the facing score will be explained with reference to Figure 35. Figure 35 shows the image sensor plane 221 and optical center 200 of the camera 11 for a certain camera view. Also shown is a vertical plane 102 and a normal 103 for a specific point 101 on the 3D object 100, passing from a certain pixel on the image sensor plane 221 through the optical center 200. Here, "Z" is the depth value of the specific point 101 from the camera, "f" is the focal length, and "θ" is the angle between the vector of the normal 103 and the vector 222 of the orientation of the camera 11.

[0126] In this case, the facing score (Score) is Score = (Z / f × 1 / cosθ)^2. "^2" represents squaring. In this calculation, "Z / f" indicates the length of one side of a pixel on the vertical plane 102, and "1 / cosθ" is a term that changes depending on the degree of facing. In other words, this is a formula that increases or decreases the length of one side of a pixel depending on the degree of facing. Since pixels are one-dimensional values, they are squared. This simply determines the projected area of ​​the pixel's rectangle. As with the first calculation example, the facing score can be calculated based on the size of the area. This second calculation example is a simplified version of the first calculation example and is useful when you want a simple implementation.

[0127] Figure 36 shows an example of processing by the incremental synthesis calculation system 45 in the second calculation example. Figure 36 is an example in which step S42 in Figure 34 is replaced with step S45, and the rest is the same as Figure 34. In step S45 in Figure 36, the incremental synthesis calculation system 45 calculates the facing score using a predetermined calculation formula. This predetermined calculation formula is the above-mentioned Score = (Z / f × 1 / cos θ)^2.

[0128] A third calculation example of the facing score will be described with reference to FIG. 37. In this example, the angle θ formed by the vector of the normal line 103 and the vector 222 of the direction of the camera 11 is used as the facing score. In other words, Score = θ. This is an example used when the shooting distance is almost constant or uncontrollable, so it is ignored, and only the angle of the direction of the camera 11 relative to the surface of the 3D object 100 needs to be known. The processing example is an example in which Score = θ is used as the predetermined calculation formula in step S45 of FIG.

[0129] Although this is not the facing score, an example of a score to be used when you want to know only the distance to the surface of the 3D object 100 and want to ignore the angle of the camera's orientation toward the surface of the 3D object 100 because the target shape is complex is shown in FIG. 38 as a fourth calculation example. The score Score' is set as follows: Score' = Z / f. Z / f is information about the distance from the camera 11 to the 3D object 100. For example, this is an evaluation value that can be used instead when the degree of facing is not very meaningful due to large variations in normals within a small area.

[0130] Next, a description will be given of a processing example of the display system 48 that has received the point cloud, camera posture, facing score, etc. from the incremental synthesis calculation system 45. For example, a processing example of the display system 48 in the UI device 2 for the UIs such as those shown in FIGS.

[0131] 39 shows an example of a process for controlling the display of a preview screen in the UI device 2. In the UI device 2, in addition to a live view screen that displays an image of a subject during shooting, for example, a 3D model image 131 generated by the incremental composition calculation system 45 at that time can be displayed as a preview screen 130. The UI device 2 can also display the 3D model image 131 as the preview screen 130 after shooting. FIG. 39 shows an example of display control of the display system 48 for such a preview screen 130 in response to information received from the incremental composition calculation system 45.

[0132] In step S501, the display system 48 acquires point cloud data, camera pose, facing score, etc. from the incremental synthesis calculation system 45. Then, in step S502, the display system 48 performs a drawing process for a 3D model image 131 to be displayed based on the point cloud data, and a drawing process for a posture presentation image (e.g., a frustum) showing the estimated camera pose 20 at each time point. At this time, for each region of the 3D model image 131, a color or shade corresponding to the facing score of the estimated camera pose 20 is set, and the facing score is expressed by the color or shade.

[0133] In step S503, the display system 48 performs processing for representing correspondence. For example, the display system 48 compares the facing score of each region of the 3D model with a predetermined threshold to determine a region with a facing score below the threshold. The display system 48 also determines the estimated camera pose 20 corresponding to that region as a low-score camera pose 20L. The display system 48 then performs processing for highlighting the pose presentation image representing the low-score camera pose 20L, or for changing the color or brightness of the pose presentation image from that of the normal estimated camera pose 20, as shown in FIGS. 27 and 28 . Alternatively, the display system 48 may perform processing for drawing a connecting line 31 connecting the low-score camera pose 20L to a corresponding region on the 3D model image 131, as shown in FIG. 29 . Furthermore, the display system 48 may perform processing for drawing a connecting line 31 connecting a low-score frame 32 on the low-score camera pose 20L side to the low-score frame 32 on the 3D model image 131, as shown in FIG. 30 .

[0134] In step S504, the display system 48 controls the display of the preview screen 130 based on the above-described drawing process, thereby allowing the user to view displays such as those shown in the examples of Figs.

[0135] In step S505, the display system 48 monitors the score calculation switching operation. Here, as shown in FIG. 31 , it is assumed that the score calculation switching button 34 is displayed on the preview screen 130 and can be operated by the user. If operation of the score calculation switching button 34 is detected, the display system 48 proceeds to step S508 and recalculates the facing scores for each region. Alternatively, the display system 48 may request the recalculated facing scores from the incremental synthesis calculation system 45 and receive and acquire them.

[0136] Switching the calculation here means switching from the current calculation example to another calculation example, for example, among the first to fourth calculation examples described above. The display system 48 then returns to step S502, performs drawing processing that reflects the facing score based on the new calculation example, performs correspondence display processing in step S503, and performs display control in step S504. This allows the user to view the preview screen 130 to which the desired calculation example has been applied, depending on the circumstances.

[0137] In step S506, the display system 48 monitors whether the user has performed another operation on the preview screen 130, such as a rotation operation of the 3D model image 131, another prepared operation, or an operation to return to the live view screen. If a rotation operation of the 3D model image 131 or another prepared operation is detected, the display system 48 proceeds to step S509 and performs processing according to the operation.

[0138] In step S507, the display system 48 monitors the end of the preview screen 130. In response to an end operation or some other end trigger, the display system 48 ends display control of the preview screen 130 from step S507. For example, if the user performs an operation to return to the live view screen, the display system 48 ends the preview screen 130 and transitions to control of the live view screen.

[0139] 6. Presentation of facing score and insufficient camera pose> Next, an example of presenting an insufficient camera pose will be described. Fig. 40 shows an example in which the display of the insufficient camera pose 23 is added to the preview screen 130 shown in Fig. 28, for example.

[0140] As described above, the user can easily understand the accuracy of the 3D model being created by expressing the facing score for each region in the 3D model image 131 and clearly indicating the low-scoring camera pose 20L. However, if additional shooting is considered to improve the accuracy of the 3D model, it is difficult for the user to know where to start taking additional photos.

[0141] Therefore, a more upright shooting pose is calculated and presented based on an index of how upright each region was photographed. For example, the insufficient camera pose 23 in Figure 40 represents a more upright shooting pose obtained for the region on the roof side of the house with a low score.

[0142] Although only one missing camera pose 23 is shown in the figure, for example, a missing camera pose 23 may be calculated and displayed for each area corresponding to each low-score camera pose 20L. By displaying such missing camera pose 23, the user can know what camera pose is best for additional shooting. Note that the missing camera pose 23 can be recognized by the user by displaying it in a different display mode from the estimated camera pose 20L and the low-score camera pose 20L, for example, in a different color or brightness.

[0143] Figure 41 is an example in which the missing camera pose 23 is displayed and the improvement effect of additional shooting at the missing camera pose 23 is clearly shown. As can be seen by comparing with Figure 40, a high score area 28A is shown by using an image captured at the missing camera pose 23. In other words, the score is displayed by color and density on the 3D model image 131, taking into account the additional shooting at the missing camera pose 23.

[0144] Specifically, for example, while the display in Fig. 40 is being displayed, the user can perform an operation to specify a certain missing camera posture 23, and the score display in the 3D model image 131 can be changed to anticipate shooting the specified missing camera posture 23, as shown in Fig. 41. This provides the user with information to determine whether to perform additional shooting, which missing camera posture 23 to prioritize, and so on.

[0145] FIG. 42 shows an example of further displaying a shooting trajectory 26. For example, a shooting trajectory 26 that connects multiple missing camera poses 23 is displayed, and the user is recommended to take such additional shots. This allows the user to know the desired trajectory for additional shots. It is also advisable to display a score that takes into account taking additional shots of this shooting trajectory 26. For example, high score areas 28A and 28B indicate that the score for each area increases. For example, when the user specifies a shooting trajectory 26, a score that takes into account taking additional shots of that shooting trajectory 26 is displayed on the 3D model image 131.

[0146] Furthermore, although not shown, multiple candidates for the shooting trajectory 26 may be displayed. In this case, when the user specifies a shooting trajectory 26, a score that takes into account the additional shooting of that shooting trajectory 26 may be displayed on the 3D model image 131. This allows the user to decide which shooting trajectory 26 to adopt while checking the accuracy improvement effect of each.

[0147] As for the UI, for example, when the preview screen 130 displays the images shown in Figures 27 to 30, buttons such as "Display missing camera posture" and "Display trajectory" may be provided, and in response to operation of the "Display missing camera posture" button, the missing camera posture 23 may be displayed as shown in Figure 40, and in response to operation of the "Display trajectory" button, the shooting trajectory 26 may be displayed as shown in Figure 42.

[0148] Fig. 43 shows an example of processing by the incremental synthesis calculation system 45, which adds calculation of the missing camera pose 23. This is the example of Fig. 34 to which step S60 has been added. Note that, although score calculation is shown as step S50, this means either step S42 in Fig. 34 or step S45 in Fig. 36.

[0149] In the missing camera pose calculation in step S60, the incremental synthesis calculation system 45 calculates the missing camera pose using the point cloud data, the estimated camera pose, the intermediate data, and the facing score for each region calculated in step S43. In step S28, the calculated missing camera pose is transmitted to the display system 48 together with the point cloud data, the camera pose, and the facing score.

[0150] In the calculation of the missing camera pose in step S60, the missing shooting pose may be calculated for each point of the point cloud of the 3D model, but this makes it difficult for the user to understand from where to actually shoot in order to efficiently improve the facing score. Therefore, point clouds with low facing scores are clustered based on spatial continuity and normal similarity, and the missing camera poses calculated for each point in the cluster point cloud are subjected to statistical processing such as averaging to calculate a missing camera pose that is representative of the cluster point cloud.

[0151] An example of the processing in step S60 is shown in Figure 44. In step S601, the incremental synthesis calculation system 45 calculates the missing camera pose for each minimum unit in the area with a low facing score. As shown schematically in Figure 45A, as an example of calculation, a position facing the normal 103 of a specific point 101 on the 3D model with respect to a certain estimated camera pose 20 and at a desired distance is set as the missing camera pose 23.

[0152] In step S602, the incremental synthesis calculation system 45 clusters areas of the 3D model with low facing scores and calculates the missing camera pose 23 for each cluster. As an example of this calculation, as shown in FIG. 45B , the missing camera pose 23 calculated for each minimum unit is averaged within the cluster. For example, the solid line represents the missing camera pose 23 for the cluster.

[0153] For example, clustering may be based on the following criteria: low score, spatial continuity, and similar normal vectors. Any one of these criteria or a combination of them may also be used.

[0154] In this way, the missing camera pose 23 per cluster is calculated and transferred to the display system 48 .

[0155] An example of display control processing of the preview screen 130 by the display system 48 is shown in Fig. 46. Note that the same processing as in Fig. 39 is assigned the same step number to avoid redundant explanation.

[0156] While the preview screen 130 is being displayed in the processes of steps S501 to S504, the display system 48 monitors the operation for requesting display of the missing camera attitude 23 in step S520.

[0157] For example, if a "Display Missing Camera Posture" button is provided on the preview screen 130 and the user operates this button, the display system 48 proceeds from step S520 to step S523, and acquires the missing camera posture information as a processing target. For example, the display system 48 reads out the missing camera posture information that has been transmitted from the incremental synthesis calculation system 45 up to that point and that has been stored internally. Then, in step S502, the display system 48 performs a drawing process that adds the missing camera posture 23. As a result, the missing camera posture 23 is displayed as shown in FIG. 40.

[0158] Furthermore, in step S521, the display system 48 monitors an operation requesting display of the shooting trajectory. For example, if a "Display shooting trajectory" button is provided on the preview screen 130 and the user operates this button, the display system 48 proceeds from step S521 to step S522 and sets the shooting trajectory. For example, based on the information on the missing camera pose, an appropriate trajectory including each shooting viewpoint is calculated and set as the shooting trajectory. Then, in step S502, the display system 48 performs a drawing process adding the shooting trajectory 26. As a result, the missing camera pose 23 is displayed as shown in FIG.

[0159] Furthermore, the display system 48 monitors an operation requesting display of the predicted score in step S524. For example, an operation by the user specifying the missing camera pose 23 or the shooting trajectory 26 is regarded as an operation requesting display of the predicted score. When an operation specifying the missing camera pose 23 or the shooting trajectory 26 is detected, the display system 48 proceeds to step S525 and recalculates the facing score of each region taking into account the shooting of the missing camera pose 23 or the shooting trajectory. Alternatively, the display system 48 may request the recalculated facing score from the incremental synthesis calculation system 45 and receive and acquire it. Then, in step S502, the display system 48 performs a drawing process that adds the shooting trajectory 26.

[0160] 41 and 42, the facing score that changes as a result of additional shooting is expressed on the 3D model image 131. Although not shown in Fig. 46, the display of the predicted score may be returned to the display of the current score as shown in Fig. 40 in response to an operation to cancel the designation of the missing camera posture 23 or the designation of the shooting trajectory 26.

[0161] Steps S506, S507, and S509 are the same as those in Fig. 39. Also, a processing example in which steps S505 and S508 in Fig. 39 are added to the example in Fig. 46 is also conceivable.

[0162] In the example of Figure 46 above, the missing camera posture 23 and shooting trajectory 26 are displayed in response to user operation, but a display control example is also conceivable in which information on the missing camera posture is also acquired at step S501, and the missing camera posture 23 and shooting trajectory 26 are displayed from the initial drawing process of step S502.

[0163] 7. Summary and Modifications According to the above embodiment, the following effects can be obtained.

[0164] In the embodiment, when a 3D model is generated using multiple images of a 3D object 100 captured at multiple different camera poses, one or more information processing devices execute a facing score calculation process (see FIGS. 34 , 36 , and 43 ) that calculates a facing score indicating the degree of facing of the camera pose at the time the images were captured for each part of the 3D model, and a presentation information generation process (see FIGS. 39 and 46 ) that generates presentation information for the user based on the results of the facing score calculation process. After capturing an image, the user wants to confirm whether the camera 11 was facing the 3D object 100 at the time of capture. However, even by referring to the camera position information output during incremental composition, for example, it is not easy for the user to determine whether the image was captured at an appropriate facing pose. Therefore, the degree of facing of the camera pose at the time of actual capture is scored, and information corresponding to the score is presented. This allows the user to easily recognize whether the image was facing the camera pose at the time of actual capture, providing useful information presentation as a means of assisting 3D modeling.

[0165] In the embodiment, an example has been given in which a 3D model is generated by sequentially and incrementally processing captured images of the 3D object 100 during shooting, thereby forming the 3D model portion by portion during shooting. By inputting sequentially captured images into the incremental synthesis calculation system 45 during shooting and synthesizing the 3D model, it is possible to generate a 3D model that grows after shooting begins, as described in Fig. 6. This is also a process in which the entire 3D model is generated when shooting is completed.

[0166] In the embodiment, an example has been given in which processing is performed to present a generated 3D model to a user while shooting the 3D object 100. When a 3D model is generated during shooting using incremental synthesis, the growing 3D model is presented to the user as a 3D model image 131 on a preview screen 130. This allows the user to check the progress of shooting and whether any shots have been missed while on location, etc.

[0167] In the presentation information generation process of the embodiment, an example has been given in which presentation information including a 3D model image 131 is generated to which a display mode according to the facing score has been added for each portion (e.g., a certain number of regions of a point cloud) (see step S502 in FIGS. 39 and 46). The 3D model image 131 expresses the degree of facing in a display mode that uses different colors or densities, such as high score regions 28 and low score regions 29. This allows the user to intuitively recognize which parts of the 3D model have low accuracy.

[0168] In the presentation information generation process of the embodiment, an example has been given in which a posture presentation image that represents the camera posture at which the image was captured is presented, and presentation information is generated that highlights posture presentation images that correspond to portions of the 3D model whose facing scores are less than a predetermined value (see step S503 in FIGS. 27, 39, and 46). For example, by highlighting frustums (posture presentation images) representing low-score camera postures 20L with low scores, the user can easily recognize images captured with a low degree of facing.

[0169] In the presentation information generation process of the embodiment, an example has been given in which a posture presentation image that represents the camera posture at which the image was captured is presented, and presentation information is generated in which posture presentation images corresponding to portions of the 3D model whose facing score is less than a predetermined value are displayed in a different manner from posture presentation images whose facing score is equal to or greater than a predetermined value (see step S503 in FIGS. 28 , 39 , and 46 ). For example, by displaying frustum as a low-score camera posture 20L with a low score in a different color or brightness from frustum with a high score, the user can easily recognize images captured with a low degree of facing.

[0170] In the presentation information generation process of the embodiment, an example has been given in which presentation information is generated that includes a 3D model image 131 to which a display mode according to the facing score has been added for each part, a posture presenting image that represents the camera posture at which the image was captured, and an image that links a part in the 3D model image 131 where the facing score is below a predetermined value with a posture presenting image corresponding to that part (low-score camera posture 20L) (see step S503 in FIGS. 29 , 30 , 39 , and 46 ). For example, low-score camera postures 20L and low-score frames 32 with low scores are presented by linking them with corresponding areas on the 3D model image 131 by connecting lines 31. This allows the user to easily recognize the low-score camera postures 20L that correspond to the low-score areas 29.

[0171] In the presentation information generation process of the embodiment, an example has been given in which presentation information is generated based on facing scores calculated using different score calculation methods in response to a score calculation switching operation (see steps S505, S508, and S502 in FIGS. 31 and 39). This allows the user to switch between and check facing scores from various perspectives. The user can also select a facing score to use as a reference depending on the purpose.

[0172] In the presentation information generation process of the embodiment, an example has been given in which presentation information including a posture presentation image representing the insufficient camera posture 23, which is a recommended camera posture for photographing, is generated (see FIGS. 40 to 46 ). This allows the user to easily recognize what camera posture should be used for additional photographing in order to improve the accuracy of the 3D model.

[0173] In the presentation information generation process of the embodiment, an example has been given in which the presentation information is generated, including a first posture presentation image representing the camera posture (estimated camera posture 20) at which the image was captured, and a second posture presentation image, which has a display form different from that of the first posture presentation image and represents a missing camera posture 23 that is a camera posture recommended for image capture (see FIGS. 40 to 46 ). This allows the user to check the missing camera posture 23 displayed together with the many estimated camera postures 20, and makes it easy to recognize what camera posture should be used to capture the image in order to improve the accuracy of the 3D model.

[0174] In the presentation information generation process of the embodiment, an example has been given in which presentation information is generated that includes a 3D model image 131 to which a display mode corresponding to a predicted facing score is added when photographing is performed in the camera posture shown in the second posture presentation image (see steps S524, S525, and S502 in FIGS. 41 and 46). This allows the user to intuitively recognize the effect of improving accuracy by photographing in the insufficient camera posture 23. This is also useful as a factor for the user in deciding whether or not to reshoot.

[0175] In the presentation information generation process of the embodiment, an example has been given in which the presentation information including a posture presentation image representing the camera posture (estimated camera posture 20) at which photography was performed and a recommended photography trajectory 26 is generated (see FIG. 42 ). This allows the user to easily recognize what photography trajectory 26 is best for photography work.

[0176] In the presentation information generation process of the embodiment, an example has been given in which presentation information is generated including a 3D model image 131 to which a display mode according to a predicted facing score is added when photographing is performed along the photographing trajectory 26 (see steps S524, S525, and S502 in FIGS. 42 and 46). This allows the user to intuitively recognize the effect of improving accuracy when photographing along the recommended photographing trajectory 26. This is also useful as information for the user to decide whether or not to reshoot.

[0177] In the facing score calculation process of the embodiment, an example has been given in which the facing score is calculated based on the area of ​​a corresponding pixel 202 corresponding to a specific point 101 of the 3D object 100 in a captured image when the pixel 202 is projected onto a plane 102 perpendicular to a normal 103 of the specific point 101 (see FIGS. 32 to 34 ). This makes it possible to obtain a facing score that reflects the angle with respect to the surface of the 3D object 100 and the distance from the 3D object 100. This is a facing score that is suitable for knowing the level of detail at which the 3D object 100 was photographed.

[0178] In the facing score calculation process of the embodiment, an example has been given in which the facing score is calculated based on the one-dimensional pixel length of the normal 103 of the specific point 101 of the 3D object 100 projected onto the vertical plane 102, and the angle θ between the normal 103 and the camera direction (see FIGS. 35 and 36 ). This allows the facing score to be calculated with a relatively simple calculation. This calculation process is suitable for implementation in an information processing device 70 with a relatively low processing power.

[0179] In the embodiment, the facing score calculation process has been described as an example in which the facing score is calculated based on the angle between the normal 103 of the surface of the 3D object 100 in the captured image and the shooting direction of the camera (see FIGS. 37 and 36 ). This also makes it possible to calculate the facing score with a relatively simple calculation. This is suitable for cases in which only the angle of the orientation of the camera 11 relative to the surface of the 3D object 100 needs to be known.

[0180] In the embodiment, the 3D model generation device 3 is exemplified as the information processing device 70 that implements the incremental synthesis calculation system 45, and the UI device 2 is exemplified as the information processing device 70 that implements the display system 48. The incremental synthesis calculation system 45 and the display system 48 may be implemented in separate devices or may be implemented in a single device.

[0181] The one or more information processing devices 70 that make up the incremental synthesis calculation system 45 and the display system 48 include a control unit (CPU 71) that executes a facing score calculation process that calculates a facing score indicating the degree of facing of the camera posture at the time the images used for generation were captured for each part of the 3D model when the 3D model is generated using a plurality of images of the 3D object 100 captured at a plurality of different camera postures, and a presentation information generation process that generates presentation information for the user based on the results of the facing score calculation process. This allows the user to easily check the degree of facing of the image during or after capture, and thereby the accuracy of the 3D model, and can efficiently perform reshoots based on the results.

[0182] Note that the display control of the preview screen 130 in Figures 39 and 46 may be performed by, for example, a 3D model generation device 3 equipped with an incremental synthesis calculation system 45 and a display system 48, and the UI device 2 may simply control the screen display of image data after it has been drawn.

[0183] Furthermore, although the preview screen 130 is configured to display the estimated camera pose 20 and the like together with the 3D model image 131, the estimated camera pose 20, the low-score camera pose 20L, or the insufficient camera pose 23 may be displayed without displaying the 3D model image 131. For this purpose, steps S25, S26, and S27 in FIG. 34 may not be executed during or immediately after shooting. This reduces the processing load. For example, assuming a system configuration such as that shown in FIGS. 12 and 13 , even if the device equipped with the incremental synthesis calculation system 45 is an information processing device 70 with relatively low processing power, such as a smartphone 12, it becomes easy to allow the user to check the facing score during shooting, etc.

[0184] Although the embodiment has been described with reference to a case where incremental 3D compositing is performed, the technology of the present disclosure can also be applied when incremental 3D compositing is not performed. For example, even when the captured images are collectively transmitted to the server device 14 after capture and 3D modeling is performed, the facing score calculation process and the presentation information generation process can be performed in parallel, and the facing score can be presented to the user. Even with this process, the user can check the accuracy of each region of the 3D model at an early stage after capture.

[0185] Furthermore, after photographing and 3D modeling has been completed, it is possible to calculate a facing score for each region of the 3D model and present the facing score and the insufficient camera pose 23 to the user. Such a display allows the user to analyze, for example, the relationship between the accuracy of the 3D model and the camera pose.

[0186] Furthermore, the technology disclosed herein is applicable to various 3D modeling techniques, not limited to photogrammetry.

[0187] The program according to the embodiment is a program that causes a processor such as a CPU or a DSP, or a device including these, to execute the processes shown in Figures 34, 36, 39, 43, and 46. That is, the program according to the embodiment is a program that causes the information processing device 70 to execute a facing score calculation process that calculates, for each part of a 3D model when a plurality of images of the 3D object 100 taken at a plurality of different camera attitudes are used to generate the 3D model, a facing score that indicates the degree of facing of the camera attitude at the time the images used for generation were captured, and a presentation information generation process that generates presentation information for the user based on the result of the facing score calculation process.

[0188] By using such a program, the 3D model generating device 3 and the UI device 2 according to the embodiment can be realized by various information processing devices 70.

[0189] The programs of the above-described embodiments can be pre-recorded on a hard disk drive (HDD) or a ROM in a microcomputer having a CPU, which serves as a recording medium built into a computer or other device. Such programs can also be temporarily or permanently stored (recorded) on removable recording media such as flexible disks, CD-ROMs (Compact Disc Read Only Memory), MO (Magneto Optical) discs, DVDs (Digital Versatile Discs), Blu-ray Discs (registered trademark), magnetic disks, semiconductor memories, and memory cards. Such removable recording media can be provided as so-called packaged software. Such programs can be installed on a personal computer or the like from removable recording media, or downloaded from a download site via a network such as a local area network (LAN) or the Internet.

[0190] Furthermore, such a program is suitable for widely providing the 3D model generation device 3 and UI device 2 of the embodiment. For example, by downloading the program to a personal computer, a communication device, a mobile terminal device such as a smartphone or tablet, a mobile phone, a game device, a video device, a PDA (Personal Digital Assistant), or the like, these devices can function as the 3D model generation device 3 and UI device 2 of the present disclosure.

[0191] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0192] The present technology may also be configured as follows. (1) An information processing method, performed by one or more information processing devices, comprising: a facing score calculation process for calculating, for each part of a three-dimensional model of an object using a plurality of images of the object photographed at a plurality of different camera postures, a facing score indicating the degree of facing of the camera posture at the time of photographing the image used for generation; and a presentation information generation process for generating presentation information for a user based on a result of the facing score calculation process. (2) The information processing method described in (1) above, wherein, during photographing of the object, a three-dimensional model is generated by sequentially and additionally processing photographed images to form the three-dimensional model for each part during photographing. (3) The information processing method described in (2) above, wherein, during photographing of the object, a process for presenting the generated three-dimensional model to a user is performed. (4) The information processing method described in any of (1) to (3) above, wherein, in the presentation information generation process, the presentation information is generated including a three-dimensional model to which a display aspect corresponding to the facing score has been added for each part. (5) The information processing method according to any one of (1) to (4), wherein the presentation information generation process presents a posture presentation image representing the posture of the camera at which the image was taken, and generates the presentation information that emphasizes posture presentation images corresponding to portions of the three-dimensional model where the facing score is less than a predetermined value. (6) The information processing method according to any one of (1) to (5), wherein the presentation information generation process presents a posture presentation image representing the posture of the camera at which the image was taken, and generates the presentation information that highlights posture presentation images corresponding to portions of the three-dimensional model where the facing score is less than a predetermined value in a display mode different from that of posture presentation images corresponding to portions where the facing score is equal to or greater than a predetermined value. (7) The information processing method according to any one of (1) to (6), wherein the presentation information generation process generates the presentation information including: a three-dimensional model image to which a display mode according to the facing score has been added for each portion, posture presentation images representing the posture of the camera at which the image was taken, and an image that links portions of the three-dimensional model image where the facing score is less than a predetermined value with the posture presentation images corresponding to those portions.(8) The information processing method according to any one of (1) to (7), wherein the presentation information generation process generates the presentation information based on the facing score calculated by a different score calculation method in response to a score calculation switching operation. (9) The information processing method according to any one of (1) to (8), wherein the presentation information generation process generates the presentation information including a posture presenting image that represents a camera posture that is a recommended camera posture for photographing. (10) The information processing method according to any one of (1) to (9), wherein the presentation information generation process generates the presentation information including: a first posture presenting image that represents a camera posture at which photographing was performed; and a second posture presenting image that has a display mode different from that of the first posture presenting image and represents a missing camera posture that is a recommended camera posture for photographing. (11) The information processing method according to (10), wherein the presentation information generation process generates the presentation information including a three-dimensional model to which a display mode corresponding to a prediction of the facing score when photographing is performed with the camera posture represented by the second posture presenting image is added. (12) The information processing method according to any one of (1) to (11) above, wherein the presentation information generation process generates the presentation information including: a posture presentation image representing the camera posture at which shooting was performed; and a recommended shooting trajectory. (13) The information processing method according to (12) above, wherein the presentation information generation process generates the presentation information including a three-dimensional model to which a display mode according to a prediction of the facing score when shooting is performed along the shooting trajectory is added. (14) The information processing method according to any one of (1) to (13) above, wherein the facing score calculation process calculates the facing score based on an area when a pixel corresponding to a specific point of an object in the captured image is projected onto a plane perpendicular to a normal to the specific point. (15) The information processing method according to any one of (1) to (13) above, wherein the facing score calculation process calculates the facing score based on a one-dimensional pixel length projected onto a plane perpendicular to the normal to the specific point of the object, and an angle between the normal and the orientation of the camera.(16) The information processing method according to any one of (1) to (13), wherein the facing score calculation process calculates the facing score based on an angle between a normal to a surface of the object in the captured image and a shooting direction of the camera. (17) An information processing device comprising: a control unit that executes: a facing score calculation process that calculates, for each part of a three-dimensional model of an object when a plurality of images of the object photographed at a plurality of different camera attitudes are used to generate the three-dimensional model of the object, a facing score that indicates a degree of facing of the camera attitude at the time the images used for generation were captured, and a presentation information generation process that generates presentation information for a user based on a result of the facing score calculation process. (18) A program that causes an information processing device to execute: a facing score calculation process that calculates, for each part of a three-dimensional model of an object when a plurality of images of the object photographed at a plurality of different camera attitudes are used to generate the three-dimensional model of the object, a facing score that indicates a degree of facing of the camera attitude at the time the images used for generation were captured, and a presentation information generation process that generates presentation information for a user based on a result of the facing score calculation process.

[0193] REFERENCE SIGNS LIST 1 Photographing device 2 UI device 3 3D model generating device 20 Estimated camera pose 20L Low score camera pose 23 Deficient camera pose 26 Photographing trajectory 28 High score area 29 Low score area 45 Incremental synthesis calculation system 48 Display system 53 Transmission information generating unit 54 Storage unit 55 Facing score calculation unit 56 Deficient camera pose calculation unit 70 Information processing device 71 CPU 82 130 Preview screen 131 3D model image

Claims

1. An information processing method, executed by one or more information processing devices, comprising: a facing score calculation process for calculating, for each part of a three-dimensional model of an object when the three-dimensional model is generated using a plurality of images of the object photographed at a plurality of different camera poses; and a presentation information generation process for generating presentation information for a user based on the result of the facing score calculation process.

2. The information processing method according to claim 1, further comprising the step of generating a three-dimensional model by sequentially and incrementally processing the captured images during the photographing of an object, thereby forming a three-dimensional model portion by portion during the photographing.

3. The information processing method according to claim 2, further comprising the step of: performing processing for presenting a generated three-dimensional model to a user while photographing an object.

4. The information processing method according to claim 1, wherein the presentation information generating process generates the presentation information including a three-dimensional model to which a display mode corresponding to the facing score is added for each part.

5. The information processing method according to claim 1, wherein the presentation information generation process presents a posture presentation image that represents the camera posture at which the image was captured, and generates the presentation information that emphasizes the posture presentation image corresponding to a portion of the three-dimensional model where the facing score is less than a predetermined value.

6. The information processing method according to claim 1, wherein the presentation information generation process presents a posture presentation image that represents the posture of the camera at which the image was captured, and generates the presentation information in such a manner that posture presentation images corresponding to portions of the three-dimensional model where the facing score is less than a predetermined value are displayed in a different manner from posture presentation images corresponding to portions where the facing score is equal to or greater than a predetermined value.

7. The information processing method according to claim 1, wherein the presentation information generation process generates the presentation information including: a three-dimensional model image to which a display mode corresponding to the facing score has been added for each part; a posture presentation image expressing the camera posture at which the photograph was taken; and an image linking a part in the three-dimensional model image where the facing score is less than a predetermined value with the posture presentation image corresponding to that part.

8. The information processing method according to claim 1, wherein in the presentation information generation process, the presentation information is generated based on the facing scores calculated using different score calculation methods in response to a score calculation switching operation.

9. The information processing method according to claim 1, wherein the presentation information generation process generates the presentation information including a posture presentation image that represents a missing camera posture, which is a camera posture that is recommended for shooting.

10. The information processing method according to claim 1, wherein the presentation information generation process generates the presentation information including: a first posture presentation image representing the camera posture at which the photograph was taken; and a second posture presentation image, which has a display form different from that of the first posture presentation image and represents a missing camera posture that is a camera posture recommended for photographing.

11. The information processing method according to claim 10, wherein the presentation information generation process generates the presentation information including a three-dimensional model to which a display mode corresponding to a prediction of the facing score when photographing is performed with the camera posture shown in the second posture presentation image is added.

12. The information processing method according to claim 1, wherein the presentation information generation process generates the presentation information including a posture presentation image representing the camera posture at which the photograph was taken and a recommended photographing trajectory.

13. The information processing method according to claim 12, wherein the presentation information generation process generates the presentation information including a three-dimensional model to which a display mode corresponding to a prediction of the facing score when shooting is performed along the shooting trajectory is added.

14. The information processing method according to claim 1, wherein, in the facing score calculation process, the facing score is calculated based on an area when a pixel corresponding to a specific point of an object in a captured image is projected onto a plane perpendicular to a normal to the specific point.

15. The information processing method according to claim 1, wherein the facing score calculation process calculates the facing score based on a one-dimensional pixel length projected onto a plane perpendicular to a normal of a specific point of an object, and an angle between the normal and a camera direction.

16. The information processing method according to claim 1, wherein in the facing score calculation process, the facing score is calculated based on an angle between a normal to the surface of the object in the captured image and a shooting direction of the camera.

17. An information processing device having a control unit that executes a facing score calculation process that calculates, for each part of a three-dimensional model of an object when a three-dimensional model is generated using a plurality of images of the object photographed from a plurality of different camera poses, a facing score that indicates the degree of facing of the camera pose when the images used for generation were taken, and a presentation information generation process that generates presentation information for a user based on the result of the facing score calculation process.

18. A program that causes an information processing device to execute a facing score calculation process that calculates, for each part of a three-dimensional model of an object when a three-dimensional model is generated using a plurality of images of the object photographed at a plurality of different camera postures, a facing score that indicates the degree of facing of the camera posture when the images used for generation were taken, and a presentation information generation process that generates presentation information for a user based on the result of the facing score calculation process.

Images