Imaging device, imaging method, and program

Abstract

The present disclosure relates to an imaging device, an imaging method, and a program that enables imaging at an appropriate depth of field. This imaging device includes: a camera-information acquisition unit that acquires camera information of a first imaging unit, when a subject to be 3D modeled is imaged; a depth-of-field calculation unit that calculates a depth of field by using the camera information; and a display control unit that generates a display image in which a depth-of-field region in which the depth of field is visualized is superimposed on a picked-up image acquired by imaging the subject. The art of this disclosure can be applied to, for example, an imaging device that performs 3D modeling.

Description

Imaging device, imaging method, and program

[0001] The present disclosure relates to an imaging device, an imaging method, and a program, and more particularly to an imaging device, an imaging method, and a program that enable imaging with an appropriate depth of field.

[0002] One method of 3D modeling a 3D object having a three-dimensional shape is called photogrammetry, in which the 3D object is photographed from multiple directions and 3D model data (3D data) of the 3D object is generated based on the multiple photographed images.

[0003] There is also a technique called real-time 3D modeling, which generates 3D data instantly (in real time) based on information such as captured images, posture information, and depth. Furthermore, in recent years, a technique collectively called neural rendering (e.g., NeRF (Neural Radiance Fields)) has also been proposed, which configures neural fields based on the posture of a captured image and the captured image, and generates images and 3D models from any viewpoint.

[0004] In capturing 3D models, increasing the depth of field is important for creating higher-quality 3D models because it facilitates matching between viewpoint images in multi-view geometry. Interchangeable-lens cameras, in particular, tend to have a shallow depth of field due to the large image sensors they use, so narrowing the aperture is commonly used to increase the depth of field. However, simply narrowing the aperture too much increases the depth of field but also reduces the amount of light that can be captured. Therefore, increasing the sensitivity as a countermeasure raises concerns about image quality degradation due to noise. Because the depth of field constantly changes depending on the focal distance and aperture size, it can be very confusing for users.

[0005] Japanese Patent Application Laid-Open No. 2006-129999 discloses an imaging device that calculates the depth of field on the near side and the far side, generates a focus frame, and displays it.

[0006] JP 2013-242378 A

[0007] When capturing images for generating a 3D model, it is necessary to capture the subject and the depth of field three-dimensionally, but the technology of Patent Document 1 does not take into consideration capturing images for generating a 3D model. Because it is difficult to change the depth of field after capturing images, it is necessary to set the depth of field appropriately when capturing images.

[0008] The present disclosure has been made in consideration of such circumstances, and makes it possible to capture an image with an appropriate depth of field.

[0009] An imaging device according to one aspect of the present disclosure includes an acquisition unit that acquires camera information of a first imaging unit when an image of a subject to be 3D modeled is captured; a calculation unit that calculates a depth of field using the camera information; and a display control unit that generates a display image in which a depth of field region that visualizes the depth of field is superimposed on an image obtained by capturing an image of the subject.

[0010] In an imaging method according to one aspect of the present disclosure, an imaging device acquires camera information from a first imaging unit when an object is imaged, calculates a depth of field using the camera information, and generates a display image in which a depth of field region that visualizes the depth of field is superimposed on an image obtained by imaging the object to be 3D modeled.

[0011] A program according to one aspect of the present disclosure causes a computer to execute the following processes: acquire camera information from a first imaging unit when an object is imaged; calculate a depth of field using the camera information; and generate a display image in which a depth of field region that visualizes the depth of field is superimposed on an image obtained by imaging the object to be 3D modeled.

[0012] In one aspect of the present disclosure, camera information from a first imaging unit when an object is imaged is acquired, the depth of field is calculated using the camera information, and a display image is generated in which a depth of field region that visualizes the depth of field is superimposed on an image obtained by imaging the object to be 3D modeled.

[0013] The program to be executed by the computer can be provided by transmitting it via a transmission medium or by recording it on a recording medium.

[0014] The imaging device may be an independent device or an internal block constituting a single device.

[0015] 1 is a diagram illustrating an overview of photogrammetry. FIG. 2 is a diagram illustrating an overview of real-time 3D modeling. FIG. 3 is a diagram illustrating a difference between photogrammetry and real-time 3D modeling. FIG. 4 is a perspective view illustrating an example of the exterior of an imaging device according to an embodiment of the present disclosure. FIG. 5 is a perspective view illustrating an example of the exterior of an imaging device according to an embodiment of the present disclosure. FIG. 6 is a diagram illustrating a depth of field presentation screen by an imaging device. FIG. 7 is a diagram illustrating a depth of field presentation screen by an imaging device. FIG. 8 is a diagram illustrating a depth of field presentation screen by an imaging device. FIG. 9 is a diagram illustrating a depth of field presentation screen by an imaging device. FIG. 10 is a diagram illustrating a depth of field presentation screen by an imaging device. FIG. 11 is a diagram illustrating a depth of field presentation screen by an imaging device. FIG. 12 is a block diagram illustrating an example configuration of an imaging device. FIG. 13 is a block diagram illustrating an example configuration of a server device. FIG. 14 is a block diagram illustrating an example configuration of an information processing device that can be configured as an imaging device or a server device. FIG. 15 is a flowchart illustrating a depth of field display control process. FIG. 16 is a flowchart illustrating an imaging control process. FIG. 17 is a diagram illustrating a guide display showing a depth of field determination result. FIG. 18 is a diagram illustrating a guide display showing a depth of field determination result. 10A and 10B are diagrams illustrating an example of a manual AF operation screen, and an example of a monitor image in focus stacking mode.

[0016] Hereinafter, with reference to the accompanying drawings, a description will be given of a mode for carrying out the technology of the present disclosure (hereinafter referred to as an embodiment). The description will be given in the following order: 1. 3D modeling 2. Example of appearance of imaging device 3. Example of depth of field presentation screen 4. Example of configuration of imaging device 5. Processing flow of depth of field display control processing 6. Processing flow of imaging control processing 7. Guide display showing depth of field determination result 8. Manual AF operation screen 9. Focus stacking mode 10. Depth of field information output 11. Summary

[0017] In this disclosure, the term "image acquisition processing" refers to capturing an image using an imaging device equipped with an image sensor (image pickup element) to obtain image data to be temporarily or permanently recorded on a recording medium. The recording medium may be internal to the imaging device or external to the imaging device. For example, this term includes not only immediate recording on a recording medium in the imaging device, but also transmission to another device or network transmission (uploading, etc.) and recording on a recording medium in an external device. The term "image acquisition processing" is used to distinguish it from capturing an image of a subject solely for display on a monitor.

[0018] Furthermore, the term "image" includes both still images and moving images. The term "captured image" refers to moving images or still images captured by an image sensor, or any of these images. In other words, it also includes images recorded on a recording medium and images displayed on a monitor as through images, etc. The captured image may be the data itself (so-called RAW data) obtained by an image sensor, etc. The captured image may also be an image that has undergone color separation processing or color conversion processing. The captured image may also be an image that has undergone signal processing such as defect correction, noise reduction, AWB (Automatic White Balance), or gamma correction. Furthermore, the captured image may also be an image that has undergone other image processing.

[0019] <1. 3D Modeling> First, photogrammetry will be explained.

[0020] One method for generating (reconstructing) a model of a three-dimensional shape of an object (also referred to as a 3D object) is called photogrammetry, which captures images of the 3D object from multiple directions and generates a 3D model (3D data of the 3D model) based on the captured images. In this disclosure, generating a model of the three-dimensional shape of a 3D object is also referred to as 3D modeling.

[0021] Photogrammetry is a technique for reconstructing a highly accurate three-dimensional model using the principles of triangulation from multiple images taken from various viewpoints. For example, multiple captured images of a 3D object 100 are obtained by capturing images using a camera 50 as shown in FIG. 1. These captured images are then subjected to a process called SfM (Structure from Motion) and a process called MVS (Multiview Stereo), followed by post-processing such as meshing and texturing, to generate a 3D model 105.

[0022] In this specification, the term "accuracy" for a 3D model refers not only to the reproducibility of the three-dimensional shape of the target 3D object, i.e., the accuracy and definition of the three-dimensional shape, but also to the reproducibility (accuracy, definition, etc.) of the texture applied to the surface of the 3D model.

[0023] In SfM, for example, corresponding points are searched for between captured images, the position and orientation of the camera are derived using epipolar constraints, and the positions of each corresponding point in three-dimensional space are identified by triangulation based on the camera position and orientation. In this disclosure, these points in three-dimensional space are also referred to as three-dimensional points. In other words, three-dimensional points corresponding to each corresponding point are identified by SfM. Then, the entire three-dimensional point cloud identified in this manner is optimized by bundle adjustment.

[0024] In MVS, for example, a three-dimensional point group derived as described above is used to perform a more detailed corresponding point search, thereby adding three-dimensional points.

[0025] As described above, photogrammetry performs a global optimization calculation called bundle adjustment to minimize errors, which allows it to obtain very accurate results, but it also places a heavy burden on the calculations.Furthermore, because it is based on geometric calculations rather than physical measurements, in principle, the higher the resolution of the images used, the more accurate the model can be restored.

[0026] Next, real-time 3D modeling will be described.

[0027] A different 3D modeling technique from the above-described photogrammetry is called real-time 3D modeling, which generates 3D data instantly (in real time) based on captured images, posture information, depth, and other information. In this technique, as shown in FIG. 2 , for example, a camera 50 captures an image of the 3D object 100 while moving around the periphery of the 3D object 100 as indicated by the dotted line 101. The camera 50 is equipped with not only an image sensor but also a Lidar (Light Detection And Ranging) scanner (ToF (Time of Flight) module), which obtains captured images and detects the depth (distance to the subject).

[0028] In recent years, ToF modules have become smaller and more sophisticated, enabling accurate depth measurement over relatively long distances (e.g., about 5 m) both indoors and outdoors. This has made it easy to experience real-time modeling and capturing at the consumer level.

[0029] Furthermore, the camera 50 is equipped with an inertial sensor, which can detect inertial information such as the acceleration and angular velocity of the camera 50 .

[0030] In real-time 3D modeling, a three-dimensional orientation calculation process called SLAM (Simultaneous Localization and Mapping) is performed to generate orientation information indicating the position and orientation of the camera 50. In addition, the orientation information and depth are used to update a TSDF (Truncated Signed Distance Function), and a 3D model 105 (mesh and texture) is generated by a process called MC (Marching Cubes).

[0031] In SLAM, for example, the position and orientation of the camera 50 are estimated based on captured images and inertial information (self-localization). In TSDF updates, depth is associated with voxels to detect volume. In MC, isosurfaces are calculated using adjacent voxels. With real-time orientation information from SLAM, the volume of a voxel can be detected without using a point cloud by overlaying depth (how far the light rays have reached) from multiple frames. Using voxel representation makes it possible to estimate viewpoints that are in shadow and must be photographed (missing viewpoints). This makes it possible to detect perforated and protruding structures in 3D objects.

[0032] There is also a 3D modeling technique collectively known as neural rendering, which constructs neural fields based on the posture of a captured image and the captured image, and generates images and 3D models from any viewpoint. For example, NeRF (Neural Radiance Fields) has been proposed.

[0033] Each of these 3D modeling techniques has different characteristics, and no technique is superior in all respects. Figure 3 compares the characteristics of photogrammetry and real-time 3D modeling. As shown in Figure 3, photogrammetry uses SfM (including localization) and MVS, while real-time 3D modeling uses localization (SLAM) and TSDF. Furthermore, comparing the data used, photogrammetry uses only image data, while real-time 3D modeling uses depth and orientation data in addition to image data. Furthermore, comparing processing times, photogrammetry requires a long time, from several minutes to several tens of hours, while real-time 3D modeling allows for nearly instantaneous processing at 30 fps (frames per second).

[0034] In addition, when comparing the required computing power, photogrammetry requires computing power at the level of a high-end CPU (Central Processing Unit) and GPU (Graphics Processing Unit), while real-time 3D modeling requires computing power at the level of a mobile AP (Application Processor).In addition, when comparing the resolution of the generated models, photogrammetry is relatively high-resolution, depending on the resolution, number, and shooting method of the captured images, while real-time 3D modeling is relatively low-resolution, depending on the depth and self-position estimation accuracy, etc.

[0035] Furthermore, the internal representation of the generated 3D data is point cloud-based in photogrammetry, whereas it is voxel-based in real-time 3D modeling. Furthermore, while there are no restrictions on subject size or resolution in photogrammetry, real-time 3D modeling is sensor-dependent. Furthermore, when comparing the absolute accuracy of the model, photogrammetry is relatively high because it is optimized using bundle adjustment, while real-time 3D modeling is relatively low, depending on the sensor and self-localization accuracy. Furthermore, when comparing scale, photogrammetry is indefinite (the size is unknown), whereas real-time 3D modeling is unique (the absolute size is known).

[0036] For example, the following differences exist between photogrammetry and real-time 3D modeling. In other words, applying real-time 3D modeling can reduce the workload and processing required for 3D modeling compared to applying photogrammetry or neural rendering. However, applying photogrammetry or neural rendering can generate 3D models with higher accuracy than applying real-time 3D modeling.

[0037] Therefore, to obtain more accurate 3D data, photogrammetry or neural rendering can be applied. However, even in such cases, it is desirable to reduce the workload and processing volume of 3D modeling. To reduce the workload and processing volume of 3D modeling, it is necessary to generate a 3D model with as few images as possible and as high accuracy as possible.

[0038] For example, if the captured images necessary for 3D modeling are not available, the accuracy of the generated 3D model may be reduced. Conversely, if an excessive number of captured images are obtained to compensate for the shortage, the number of captured images increases unnecessarily, increasing the workload of the user. Furthermore, since 3D modeling is performed using unnecessary captured images, the amount of processing required increases unnecessarily.

[0039] In other words, in order to obtain more accurate 3D data with less work and processing, it is necessary to capture the 3D object at a more appropriate position and orientation. However, with each of the above-mentioned 3D modeling methods, it is difficult for the photographer to determine the appropriate position and orientation for capturing the image.

[0040] For example, in the case of photogrammetry, the 3D modeling process requires time, making it difficult for the photographer to immediately check the 3D modeling results when capturing images. Therefore, it is difficult for the photographer to determine the appropriate position and posture for capturing images. As a result, for example, there is a risk of a shortage of images captured at appropriate positions and postures, reducing the accuracy of the 3D data obtained by photogrammetry. Furthermore, if excessive images are captured at various positions and postures without planning in order to avoid a shortage of images captured at appropriate positions and postures, not only does this increase the user's workload, but it also unnecessarily increases the number of captured images, potentially increasing the load of the 3D modeling process (e.g., processing volume and processing time).

[0041] Therefore, in the imaging device of this embodiment, 3D modeling is performed twice, and the results of the first 3D modeling are used to provide guidance and control regarding imaging for the second 3D modeling.

[0042] The imaging device performs first 3D modeling, for example, using real-time 3D modeling, to generate a first 3D model that represents the three-dimensional shape of the 3D object. Information from the real-time 3D modeling is then used to guide the user who is the photographer and to control imaging for second 3D modeling, such as photogrammetry or neural rendering. That is, the imaging device uses information from the first 3D modeling (real-time 3D modeling) to provide guidance so that imaging for the second 3D modeling process can be performed at a more appropriate position and orientation. Furthermore, in this embodiment, the user does not need to operate the shutter, and shutter control, i.e., image acquisition processing, is performed automatically at the optimal timing. Imaging for the first 3D modeling and imaging for the second 3D modeling (image acquisition processing) can be performed simultaneously, or imaging for the first 3D modeling can be performed in advance, followed by imaging for the second 3D modeling (image acquisition processing).

[0043] 2. Exemplary Appearance of Imaging Device FIGS. 4 and 5 show exemplary appearances of an imaging device according to an embodiment of the present disclosure.

[0044] 4 and 5 is a camera having a lens barrel 58 attached to a main body 57. The lens barrel 58 may be an interchangeable lens type, or may be a non-detachable type that is provided integrally with the main body 57.

[0045] Light incident through the lens barrel 58 is captured, i.e., photoelectrically converted, by the image sensor disposed in the main body 57 and processed as a captured image. The image sensor here corresponds to the imaging unit 20 (FIG. 14) described below.

[0046] The main body 57 and the lens barrel 58 are provided with various controls for user operations, a display 55 for displaying captured images, various icons, menus, etc., and a viewfinder 56, etc.

[0047] A sensor unit 52 is attached to the main body 57. The sensor unit 52 is a unit provided with, for example, a depth sensor 31 constituting a distance measurement unit, an image sensor 32 constituting an imaging unit, and an IMU (Inertial Measurement Unit) 33 ( FIG. 14 ) constituting an inertial information detection unit. With this sensor unit 52, the image sensor 32 obtains a captured image, the depth sensor 31 obtains distance information relative to the subject, and the IMU 33 obtains inertial information (acceleration and angular velocity) of the imaging device 51.

[0048] An interface device 53 is also attached to the main body 57. The interface device 53 may be a device dedicated to the imaging device 51, or may be a device that can be linked to the imaging device 51 using an information processing device such as a smartphone. A display 54 in the interface device 53 displays various information to the user. For example, a captured image (through image) of a subject is displayed on the display 54. In addition, the display 54 also displays information to guide the user through the imaging operation. The display 54 is also a touch panel, and functions as an operation input unit that accepts operation inputs from the user.

[0049] In addition, although Figures 4 and 5 show an example in which a smartphone or the like is used as the interface device 53, the display by the interface device 53 and the detection of operation input may be performed by the display 55 of the main body 57, making a separate interface device 53 unnecessary.

[0050] Also, an example may be considered in which a camera having the illustrated main body 57 and lens barrel 58 is not used. For example, information processing devices such as smartphones often have an imaging function, and also often have a distance measurement function and an inertial sensor function, so that a smartphone, tablet device, or the like can be used alone as the imaging device 51 in this embodiment.

[0051] The depth sensor 31, image sensor 32, IMU 33, and the like provided in the sensor unit 52 may be built into the main body 57 or may be built into the interface device 53. When an information processing device such as a smartphone is used as the imaging device 51, the depth sensor 31, image sensor 32, and IMU 33 may be built into the information processing device.

[0052] The display device for displaying images may be an external device. For example, display data may be transmitted from the imaging device 51 via short-range wireless communication or wired communication, and displayed on a monitor display device, a head-mounted display, or the display of another information processing device.

[0053] In other words, the imaging device 51 of the embodiment is a device that can capture images regardless of its form, and is equipped with a control function that allows the user to execute an interface such as image display on an integrated or separate screen.

[0054] 3. Example of Depth of Field Presentation Screen> A user captures an image of a subject to be 3D modeled using imaging device 51, and while imaging for the first 3D modeling, imaging device 51 displays a captured image (through image) of the subject and a guide for the user (photographer) on display 54 of interface device 53. In this embodiment, imaging device 51 superimposes the depth of field on the captured image of the subject and displays it on display 54 as a guide for the user.

[0055] In capturing 3D models, increasing the depth of field is important for creating higher-quality 3D models because it facilitates matching between viewpoint images in multi-view geometry. In particular, interchangeable-lens cameras, which use large image sensors and tend to have a shallow depth of field, often achieve a deeper depth of field by narrowing the aperture. However, simply narrowing the aperture too much increases the depth of field but reduces the amount of light that can be captured. Increasing the sensitivity as a countermeasure raises concerns about image quality degradation due to noise. Depth of field constantly changes depending on the focal distance and aperture size, making it very difficult for users to understand. The imaging device 51 has a function that superimposes the depth of field on the captured image of the subject as a user guide. By visualizing the depth of field, users can easily confirm which area of ​​the subject being 3D modeled is in focus while capturing the image, thereby enabling the creation of high-quality 3D models.

[0056] A presentation screen (user interface screen) that presents the depth of field of the image capture device 51 will be described with reference to FIGS. 6 to 13. FIG.

[0057] For example, as shown in Fig. 6, a user uses an imaging device 51 to capture an image of a stepped object 131 as a subject to be 3D modeled. While checking the monitor image displayed on the display 54, the user moves around the subject object 131 as indicated by arrow 141 to capture the image of the object 131. Frustums 142A and 142B indicate the position of the imaging device 51 capturing the image of the subject (hereinafter referred to as the camera position). Here, the camera position includes both the position and orientation (imaging direction) of the imaging device 51 in three-dimensional space.

[0058] FIG. 7 shows an example of a monitor image displayed on the display 54 of the interface device 53 at the camera position of the frustum 142A in FIG. 6 during imaging for the first 3D modeling.

[0059] In the monitor image 150 of FIG. 7 , a three-dimensional mesh 162 of the object 131 generated by first 3D modeling (real-time 3D modeling) is superimposed on a captured image 161 captured by the image sensor 32. In FIG. 7 , the dashed cross-hatched pattern shown as the three-dimensional mesh 162 represents the polygon mesh generated by the first 3D modeling. This is also true in FIG. 8 and subsequent drawings. A depth-of-field region 163, indicated by a predetermined color or pattern, is also superimposed on the monitor image 150. In the example of FIG. 7 , for example, the depth-of-field region 163 is displayed in a distinguishable manner from other regions by applying a predetermined color to the color captured image 161. However, conversely, the depth-of-field region 163 may be displayed by highlighting the depth-of-field region 163 by, for example, applying gray to regions other than the depth-of-field region 163. In other words, the depth of field region 163 may display either the depth of field region or a region other than the depth of field, as long as it visualizes the depth of field.

[0060] FIG. 8 is a diagram showing the depth of field at the camera position of frustum 142A as viewed from the front of object 131.

[0061] The depth of field is the depth of field within the range of focus, and is defined by the distance (in the imaging direction) from a front depth of field 172 at the front of the focal plane (focus position) 171 to a rear depth of field 173 at the rear of the focal plane (focus position). The depth of field is determined by the focal length of the lens, the aperture value (F-number), the subject distance (the distance between the subject and the camera), and the allowable circle of confusion. The focal length, aperture value, and subject distance can be acquired as parameters during imaging. The allowable circle of confusion can be set to, for example, the pixel pitch of the image sensor, or can be set to any value depending on the degree of blur acceptable to the user. Therefore, if the allowable circle of confusion is set to a predetermined value on a settings screen, the depth of field can be calculated by acquiring the parameters of the focal length, aperture value, and subject distance during imaging. As a result, the imaging device 51 can present the depth of field to the user by superimposing a three-dimensional mesh 162 and a depth of field region 163 on an imaged image 161 during imaging of the first 3D modeling, as in the monitor image 150 in Fig. 7. By checking the superimposed depth of field region 163, the user can capture an image while checking whether the part they want to capture is in focus.

[0062] FIG. 9 shows another example of a presentation screen that presents the depth of field.

[0063] 7 , the monitor image 150 in Fig. 9 has a three-dimensional mesh 162 of the object 131 superimposed on a captured image 161 captured by the image sensor 32. On the other hand, with regard to the depth of field, a front depth of field plane 164 representing the plane of the front depth of field and a rear depth of field plane 165 representing the plane of the rear depth of field are superimposed on the captured image 161. In this way, the depth of field may be represented by the front depth of field plane 164 and the rear depth of field plane 165.

[0064] FIG. 10 shows another example of a presentation screen that presents the depth of field.

[0065] 7, in the monitor image 150 in Fig. 10, a three-dimensional mesh 162 of the object 131 is superimposed on a captured image 161 captured by the image sensor 32. Meanwhile, for the depth of field, a depth of field region 163 and a rear depth of field plane 165 are superimposed and displayed on the captured image 161. In this way, the depth of field may be expressed by the depth of field region 163 and the rear depth of field plane 165 to make it easier to see what is ahead of the subject.

[0066] The presentation screen for presenting the depth of field can be modified in various ways. For example, as shown in Fig. 11 , the depth of field region 163 shown in Fig. 7 and the front depth of field plane 164 and the rear depth of field plane 165 shown in Fig. 9 may be superimposed on the captured image 161 and the three-dimensional mesh 162.

[0067] By superimposing the depth of field region 163, front depth of field plane 164, and rear depth of field plane 165 on the three-dimensional mesh 162 of the object 131 calculated using the depth information from the depth sensor 31, the depth of field can be visualized and presented over a range with a greater three-dimensional width, making it easier for the user to perceive the subject in three dimensions and determine which area in space is in focus.

[0068] As shown on the left side of FIG. 12 , the depth of field is calculated by a means other than the imaging unit (image sensor) and focus control provided in the main body 57 of the imaging device 51, such as the distance to the subject measured by the depth sensor 31. Therefore, the depth of field can be measured and displayed regardless of the subject's conditions, such as the subject's contrast and uneven shape. This allows the user to confirm the in-focus area even when the image of the imaging unit provided in the main body 57 is out of focus. Furthermore, if the user attempts to maximize the depth of field displayed on the monitor image 150, the imaging device 51 will be oriented so as to face the subject, as shown on the right side of FIG. 12 . FIG. 13 shows an example of a depth of field area 163 when the object 131 is viewed from the front. In other words, using the depth of field displayed on the monitor image 150 as a guide can guide the user to view the subject head-on, allowing for the acquisition of a high-quality captured image with a wide in-focus area. Depth of field is depicted in three-dimensional space, not two-dimensional, so it can be seen from angles other than the one at which the shutter was released.

[0069] 4. Configuration Example of Imaging Device FIG. 14 is a block diagram showing a configuration example of an imaging device 51. As shown in FIG.

[0070] In this embodiment, the server device 40 is also shown in the figure, as it is assumed that photogrammetry, for example, as the second 3D modeling, is performed by the server device 40 rather than the imaging device 51. The server device 40 is configured, for example, as a cloud server, and the imaging device 51 is connected to the server device 40 via a predetermined network such as the Internet or a LAN.

[0071] 4 and 5, each part of the imaging device 51 is provided in, for example, the sensor unit 52, the main body 57, or the interface device 53, but may be provided in an appropriate location as a whole for the imaging device 51. Furthermore, the imaging device 51 may have devices and processing units not shown as blocks in Fig. 14, and there may be data flows and processes not shown as arrows or the like.

[0072] The imaging device 51 has, for example, a configuration for performing real-time 3D modeling as the first 3D modeling, including a depth sensor 31, an image sensor 32, an IMU 33, a SLAM 34, a TSDF update unit 35, and a mesh generation unit 36.

[0073] Furthermore, the imaging device 51 has components for performing image acquisition processing for the second 3D modeling, such as an imaging unit 20, an operation unit 21, an image processing unit 22, a storage unit 23, and a communication unit 29. The calculation for displaying the depth of field described above is performed by the image processing unit 22. The image processing unit 22 has a camera information acquisition unit 61, a depth of field calculation unit 62, and a depth of field drawing unit 63.

[0074] The imaging device 51 also has an imaging control unit 24, a scoring processing unit 25, a display control unit 26, and an output unit 30 as components for various controls and user guidance. The display control unit 26 has a superimposed image generation unit 27 and a display image generation unit 28. The output unit 30 has an image output unit 65 and a sound / vibration output unit 66.

[0075] In this imaging device 51, the depth sensor 31 includes a Lidar sensor (ToF module) or the like, detects depth information indicating the depth to the subject, and supplies the depth information to the TSDF update unit 35. The image sensor 32 captures an image of the subject to generate a captured image, which is then supplied to the display image generation unit 28 and the SLAM 34. The image captured by the image sensor 32 is used for the first real-time 3D modeling. The IMU 33 detects inertial information (acceleration and angular velocity) of the imaging device 51 and supplies the information to the SLAM 34.

[0076] The SLAM 34 estimates its own position based on the captured image and inertial information provided, and generates attitude information indicating the position and attitude of the image capture device 51. The SLAM 34 provides the generated attitude information to the TSDF update unit 35, the image processing unit 22, the image capture control unit 24, and the display control unit 26.

[0077] The TSDF update unit 35 updates the TSDF based on the attitude information of the imaging device 51 and the depth information detected by the depth sensor 31, and supplies the updated TSDF to the mesh generation unit 36. The mesh generation unit 36 ​​generates a mesh and a texture using the updated TSDF. The mesh generation unit 36 ​​supplies the mesh and texture data to the scoring processing unit 25 as 3D data (first 3D data) of the first 3D model (first three-dimensional shape information). The mesh generation unit 36 ​​also supplies data of the generated mesh (mesh data) to the image processing unit 22.

[0078] The scoring processing unit 25 performs scoring processing based on the supplied 3D data of the first 3D model and the imaging viewpoint information supplied from the imaging control unit 24, i.e., information indicating the position and orientation at which imaging (image acquisition processing) for second 3D modeling was performed. This scoring processing is processing in which the accuracy of the second 3D model generated by the server device 40 using the captured images captured by the imaging unit 20 so far is evaluated using three-dimensional shape information from the first 3D model, and a scoring result (score) is generated.

[0079] For example, the scoring processing unit 25 generates a scoring result for each local area of ​​the first three-dimensional shape information based on the first three-dimensional shape information and the position and orientation of the image capturing performed so far by the image capturing unit 20. For example, the scoring processing unit 25 may generate a scoring result for each polygon of a mesh. The scoring processing unit 25 may acquire camera information related to the image capturing unit 20 and generate a scoring result based on the camera information. The scoring processing unit 25 supplies the scoring result to the image capturing control unit 24 and the superimposed image generating unit 27.

[0080] The imaging control unit 24 controls imaging as the image acquisition process of the imaging unit 20. For example, the imaging control unit 24 sets a location (hereinafter referred to as a recommended imaging position) where image acquisition processing should be performed for the 3D object serving as the subject, based on the first three-dimensional shape information of the first 3D model supplied via the scoring processing unit 25. To perform 3D modeling using photogrammetry with consistent quality, images must be captured from a certain distance facing the imaging plane of the subject. Furthermore, the imaging interval must be changed depending on the size of the common field of view (front and back) due to the surface irregularities. Therefore, the imaging control unit 24 sets a recommended imaging position for the 3D object so that each captured image can be acquired with an appropriate overlap rate according to the shape, irregularities, etc. of each part of the 3D object. An appropriate overlap rate is set, for example, between approximately 25% and 75%. The higher the overlap rate, the greater the amount of information. The imaging control unit 24 may use the scoring results to set an appropriate recommended imaging position. For example, recommended imaging positions may be set with a higher density for areas with low evaluation scores.

[0081] In this way, the imaging control unit 24 sets multiple recommended imaging positions (points where image acquisition processing should be performed) for the 3D object that is the subject, but this alone does not allow the user to know where to capture the image. Therefore, the imaging control unit 24 can supply information on the set recommended imaging positions to the display control unit 26 and execute a guide display that guides the user to the recommended imaging positions.

[0082] The imaging control unit 24 supplies the display control unit 26 with information necessary to guide the user to move the imaging device 51 to the recommended imaging position. The imaging control unit 24 then determines the timing to capture an image by determining whether the imaging device 51 has been moved to the recommended imaging position based on the posture information of the imaging device 51. If the imaging control unit 24 determines that it is the timing to capture an image, it transmits imaging control information for controlling the shutter to the imaging unit 20, causing the imaging unit 20 to perform imaging as an image acquisition process. The imaging control unit 24 also supplies the scoring processing unit 25 with imaging viewpoint information indicating the position and posture of the imaging performed by the imaging unit 20.

[0083] The imaging unit 20 has an image sensor and captures an image of a subject to generate a captured image. In other words, the imaging unit 20 captures an image as an image acquisition process under shutter control by the imaging control unit 24, generates a captured image for second 3D modeling, and supplies the generated image to the image processing unit 22. The imaging unit 20 also supplies the focal length, aperture value (F-number), and subject distance, which are parameters for calculating the depth of field, to the image processing unit 22 as camera information.

[0084] Furthermore, the imaging unit 20 may supply, for example, internal parameters, external parameters, angle of view information, etc. of the imaging unit 20 as camera information to the scoring processing unit 25. Furthermore, the imaging unit 20 may supply, to the imaging control unit 24, imaging timing information indicating the timing of imaging that is not based on the shutter control of the imaging control unit 24, such as a user's shutter operation using the operation unit 21.

[0085] The operation unit 21 is a group of operators that can be operated by a user, and includes operators for various camera operations such as a shutter operator, a zoom operator, a focus operator, etc. The imaging unit 20 may capture an image in accordance with a shutter operation instruction from the user supplied from the operation unit 21, and generate a captured image.

[0086] The camera information acquisition unit 61 of the image processing unit 22 acquires, as camera information, parameters for calculating the depth of field, such as the focal length, the aperture value (F-number), and the subject distance. The camera information acquisition unit 61 also acquires values ​​to be used as the permissible circle of confusion, such as the pixel pitch of the image sensor or values ​​set by the user, from the storage unit 23. The camera information acquisition unit 61 supplies the parameters of the focal length, the aperture value (F-number), the subject distance, and the permissible circle of confusion acquired as the camera information to the depth of field calculation unit 62.

[0087] The depth of field calculation unit 62 calculates the depth of field based on the parameters supplied from the camera information acquisition unit 61, and supplies depth of field information indicating the depth of field to the imaging control unit 24 and the depth of field drawing unit 63. The depth of field information includes information on the focus position, the front depth of field, and the rear depth of field.

[0088] The depth of field rendering unit 63 receives depth of field information from the depth of field calculation unit 62 and orientation information of the image capture device 51 from the SLAM 34. The mesh generation unit 36 ​​also receives at least mesh data (mesh data) from the first 3D data of the first 3D model. The depth of field rendering unit 63 generates a depth of field image that visualizes an area indicating the depth of field using the depth of field information, orientation information of the image capture device 51, and the mesh data. The depth of field image is, for example, an image in which a mesh corresponding to the depth of field is colored, such as the depth of field area 163 shown in FIG. 7 , from the three-dimensional shape (three-dimensional mesh shape) of the first 3D model represented by a mesh. The depth of field rendering unit 63 supplies the generated depth of field image to the superimposed image generation unit 27 of the display control unit 26.

[0089] The image processing unit 22 also performs predetermined image processing on the captured image for second 3D modeling generated by the imaging unit 20. The content of this image processing is arbitrary. The image processing unit 22 performs signal processing on the captured image, such as color separation processing, color conversion processing, defect correction, noise reduction, AWB, gamma correction, etc. The image processing unit 22 also performs encoding processing on the captured image. The encoded data obtained by encoding the captured image is supplied to the communication unit 29 and the storage unit 23.

[0090] The storage unit 23 records the encoded data supplied from the image processing unit 22 on a recording medium. The communication unit 29 transmits the encoded data supplied from the image processing unit 22 to another information processing device (e.g., a server, etc.). For example, the communication unit 29 sequentially uploads encoded data obtained by encoding captured images for the second 3D modeling to the server device 40.

[0091] 15 shows an example of the configuration of a server device 40 that performs data communication with the imaging device 51 via the network 107. Encoded data of a captured image obtained by the imaging unit 20 performing imaging as an image acquisition process is uploaded to the server device 40, and second 3D modeling is performed in the server device 40.

[0092] The server device 40 includes a communication unit 41 , a photogrammetry processing unit 42 , and a storage unit 43 .

[0093] The communication unit 41 sequentially receives encoded data of captured images uploaded from the imaging device 51. The communication unit 41 also transmits information about the 3D model generated by the photogrammetry processing unit 42 to the imaging device 51.

[0094] The storage unit 43 stores the encoded data of the captured images uploaded from the imaging device 51, and stores the 3D data generated by the photogrammetry processing unit 42, etc.

[0095] The photogrammetry processing unit 42 decodes the uploaded encoded data and performs 3D modeling on the decoded captured image. For example, the photogrammetry processing unit 42 performs SfM or MVS processing based on the captured image generated by the imaging unit 20, and generates a second 3D model (second three-dimensional shape information).

[0096] In the SfM process, the photogrammetry processing unit 42, for example, searches for corresponding points between captured images, derives the position and orientation of the image capture device 51 using epipolar constraints, identifies the positions of each corresponding point in three-dimensional space by triangulation based on the position and orientation of the image capture device 51, and optimizes the entire identified three-dimensional point cloud using bundle adjustment.In the MVS process, the photogrammetry processing unit 42, for example, uses the three-dimensional point cloud to further perform a dense search for corresponding points, add three-dimensional points, and further perform meshing and texturing as post-processing to generate a second 3D model.

[0097] The photogrammetry processing unit 42 encodes the data of the 3D model (3D data) generated in this manner and supplies it to the storage unit 43, which records it on a recording medium. The photogrammetry processing unit 42 also supplies the encoded data of the second 3D model to the communication unit 41, which transmits it to the imaging device 51.

[0098] The photogrammetry processing unit 42 performs photogrammetry processing to generate a 3D model after obtaining the required number of captured images, but performs simple 3D modeling while the imaging device 51 is capturing images to generate a 3D model as a preview image. For example, the server device 40 performs 3D modeling for preview each time an image is uploaded from the imaging device 51, and generates and updates the 3D model. The 3D model at each point in time is transmitted to the imaging device 51.

[0099] Although the server device 40 is described as including the photogrammetry processing unit 42, it may also be configured to include a neural rendering processing unit and generate the second 3D model by neural rendering.

[0100] 14 , in the imaging device 51, the communication unit 29 receives a 3D model for preview during imaging. The received 3D model for preview is supplied to the scoring processing unit 25 and the display control unit 26.

[0101] The display control unit 26 controls the display of the captured image and the display including guidance regarding the user's imaging operation. The superimposed image generation unit 27 acquires an image to be superimposed on the captured image of the subject, for example, a depth of field image generated as a superimposed image by the depth of field drawing unit 63, and supplies the image to the display image generation unit 28. The superimposed image generation unit 27 also generates an image showing, as a guide, recommended imaging positions, imaging timing, etc., and supplies the image to the display image generation unit 28 as a superimposed image.

[0102] The display image generation unit 28 generates a display image by combining the superimposed image supplied from the superimposed image generation unit 27 so as to superimpose it on the captured image supplied from the image sensor 32. For example, the display image generation unit 28 generates the monitor image 150 shown in Fig. 7, 9, or 10 as a display image that guides the depth of field. The display image generation unit 28 can also include a 3D model or thumbnail data thereof from the server device 40 in the display image.

[0103] The image output unit 65 of the output unit 30 is configured with a display device such as the displays 54, 55, for example. The image output unit 65 displays captured images of the subject and various guide images under the control of the display control unit 26. The sound / vibration output unit 66 of the output unit 30 has a vibration element such as a piezoelectric actuator, a linear resonant actuator (LRA), a voice coil motor, or an eccentric motor, and provides guidance to the user by generating vibrations at predetermined timings under the control of the imaging control unit 24. The sound / vibration output unit 66 also has a speaker, for example, and provides guidance to the user by generating sounds (including voice) at predetermined timings under the control of the imaging control unit 24.

[0104] The imaging device 51 having such a configuration presents depth-of-field information to the user, thereby controlling the distance from the subject to a more appropriate distance, allowing the user to capture a 3D object, and enables 3D modeling to be performed on the server device 40 using the captured image. In particular, the imaging device 51 outputs guide displays and audio so that the user can perform imaging work at a more appropriate subject distance. Therefore, by using the imaging device 51, it is possible to generate more accurate 3D data while suppressing an increase in the load of 3D modeling.

[0105] In the above example, the second 3D modeling is performed by the server device 40, but the imaging device 51 may be provided with a function equivalent to, for example, the photogrammetry processing unit 42, and the second 3D modeling may be performed.

[0106] <Configuration Example of Information Processing Device> Here, a description will be given of an information processing device that can be configured as the imaging device 51 or the server device 40. For example, a portable information processing device such as a smartphone or a tablet terminal can also be used as the imaging device 51. The server device 40 is also configured by an information processing device.

[0107] FIG. 16 is a block diagram illustrating an example of the configuration of an information processing device.

[0108] The information processing device 70 in Fig. 16 is a device capable of information processing, particularly image processing, such as a computer. Specific examples of the information processing device 70 include personal computers, workstations, mobile terminal devices such as smartphones and tablets, and video editing devices. The information processing device 70 may also be a computer configured as a server device or a computing device in cloud computing.

[0109] 16 executes various processes in accordance with programs stored in a nonvolatile memory unit 74 such as a read-only memory (ROM) 72 or an electrically erasable programmable read-only memory (EEP-ROM), or programs loaded from a storage unit 79 to a random access memory (RAM) 73. The RAM 73 also stores data necessary for the CPU 71 to execute various processes, as appropriate.

[0110] The image processing unit 85 is configured as a processor that performs various types of video processing, such as 3D modeling, rendering, database processing, image processing including color and brightness adjustment and color conversion processing for captured images, encoding, decoding, image editing, and image analysis and detection processing, or a combination of these processes.

[0111] The image processing unit 85 can be realized by, for example, a CPU separate from the CPU 71, a graphics processing unit (GPU), a general-purpose computing on graphics processing units (GPGPU), an artificial intelligence (AI) processor, etc. The image processing unit 85 may be provided as a function within the CPU 71.

[0112] The CPU 71, ROM 72, RAM 73, nonvolatile memory unit 74, and image processing unit 85 are interconnected via a bus 83. To this bus 83, an input / output interface 75 is also connected.

[0113] An input unit 76 consisting of operators and operation devices is connected to the input / output interface 75. For example, the input unit 76 may be various operators and operation devices such as a keyboard, a mouse, keys, a trackball, a dial, a touch panel, a touchpad, or a remote controller. The input unit 76 detects a user operation, and a signal corresponding to the input operation is interpreted by the CPU 71. A microphone may also be used as the input unit 76. Voice uttered by the user may also be input as operation information.

[0114] A display unit 77, which may be an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, and an audio output unit 78, which may be a speaker, are connected to the input / output interface 75 either integrally or separately. The display unit 77 displays various types of information. The display unit 77 may be, for example, a display device provided in the housing of the information processing device 70 or a separate display device connected to the information processing device 70. The display unit 77 displays various images, operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), on the display screen based on instructions from the CPU 71.

[0115] The input / output interface 75 may be connected to a storage unit 79 and a communication unit 80 configured from a hard disk drive (HDD) or a solid state drive (SSD).

[0116] The storage unit 79 can store various data and programs. A database can also be configured in the storage unit 79.

[0117] The communication unit 80 performs communication processing via a transmission path such as the Internet, and communication with various devices such as external DBs, editing devices, and information processing devices via wired / wireless communication, bus communication, and the like.

[0118] A drive 81 is also connected to the input / output interface 75 as needed, and a removable recording medium 82 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately attached. The drive 81 allows video data, various computer programs, and the like to be read from the removable recording medium 82. The read data is stored in the storage unit 79, and the video and audio contained in the data are output on the display unit 77 and the audio output unit 78. In addition, the computer programs and the like read from the removable recording medium 82 are installed in the storage unit 79 as needed.

[0119] In the case of an information processing device 70 used as the imaging device 51, the information processing device 70 has a configuration including a sensor unit 84. The sensor unit 84 shows the sensor configuration required for the depth sensor 31, image sensor 32, IMU 33, and imaging unit 20 described in Fig. 14. For example, the sensor unit 84 has a Lidar sensor (ToF sensor), multiple image sensors, an angular velocity sensor, an acceleration sensor, etc.

[0120] In this information processing device 70, for example, software for the processing of this embodiment can be installed via network communication by the communication unit 80 or via a removable recording medium 82. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, etc.

[0121] 5. Processing Flow of Depth of Field Display Control Processing Next, the depth of field display control processing for calculating the depth of field and performing a superimposed display will be described with reference to the flowchart of Fig. 17. This processing is started, for example, when sensor information of the subject is acquired by the depth sensor 31, image sensor 32, etc. of the sensor unit 52. In the depth of field display control processing of Fig. 17, an example will be described in which the monitor image 150 shown in Fig. 11 is generated and displayed.

[0122] First, in step S11, the SLAM 34 estimates its own position based on the captured image supplied from the image sensor 32 and the inertial information supplied from the IMU 33, and generates attitude information indicating the position and attitude of the image capture device 51. The SLAM 34 supplies the generated attitude information to the TSDF update unit 35, the image processing unit 22, the image capture control unit 24, and the display control unit 26.

[0123] In step S12, the TSDF update unit 35 updates the TSDF based on the attitude information of the imaging device 51 supplied from the SLAM 34 and the depth information detected by the depth sensor 31, and supplies the updated TSDF to the mesh generation unit 36.

[0124] In step S13, the mesh generation unit 36 ​​generates a mesh and a texture using the TSDF updated by the TSDF update unit 35. The mesh generation unit 36 ​​supplies the generated mesh and texture data to the scoring processing unit 25 as 3D data (first 3D data) of the first 3D model (first three-dimensional shape information). The mesh generation unit 36 ​​also supplies the generated mesh data to the image processing unit 22.

[0125] In step S14, the imaging unit 20 performs focus control to adjust the focus (focus position) to a region of interest of the subject in the captured image. For the focus control, for example, a phase difference signal from a phase difference pixel included in the image sensor or contrast AF that searches for a focus position from the contrast of the captured image captured by the image sensor can be used. Note that the user may manually adjust the focus (focus position) to a region of interest of the subject.

[0126] In step S15, the camera information acquisition unit 61 of the image processing unit 22 acquires camera information for calculating the depth of field. Specifically, the camera information acquisition unit 61 acquires the focal length, the aperture value (F-number), and the subject distance as camera information from the imaging unit 20. The camera information acquisition unit 61 also acquires values ​​such as the pixel pitch of the image sensor to be used as the permissible circle of confusion from the storage unit 23. The value as the permissible circle of confusion may be stored in advance in the camera information acquisition unit 61.

[0127] In step S16, the depth-of-field calculation unit 62 calculates the depth of field based on each parameter of the camera information supplied from the camera information acquisition unit 61, and supplies depth-of-field information indicating the depth of field to the imaging control unit 24 and the depth-of-field rendering unit 63. The depth-of-field information includes information on the focus position, the front depth of field, and the rear depth of field.

[0128] In step S17, the depth of field rendering unit 63 generates an image that projects a front depth of field plane 164 indicating the front depth of field and a rear depth of field plane 165 indicating the rear depth of field onto three-dimensional space based on the camera position of the imaging unit 20, based on the attitude information of the imaging device 51 supplied from SLAM 34 and the depth of field information supplied from the depth of field calculation unit 62.

[0129] In step S18, the depth of field drawing unit 63 generates an image representing a depth of field region 163 in which a predetermined color or pattern is applied to a portion of the three-dimensional mesh 162 representing the three-dimensional shape of the subject supplied from the mesh generation unit 36, based on the depth of field information supplied from the depth of field calculation unit 62.

[0130] In step S19, the depth of field rendering unit 63 generates a composite image as a depth of field image by combining an image in which the front depth of field plane 164 and the rear depth of field plane 165 are projected onto three-dimensional space with an image in which the depth of field region 163 is colored, and supplies the composite image to the superimposed image generation unit 27 of the display image generation unit 28. If there are other images to be superimposed on the captured image of the subject (for example, a guide image indicating the capture timing, etc.), the superimposed image generation unit 27 also composites these images and supplies the composite image to the display image generation unit 28 as a superimposed image.

[0131] In step S20, the display image generation unit 28 generates a display image by combining the superimposed image supplied from the superimposed image generation unit 27 so as to superimpose it on the captured image supplied from the image sensor 32. The generated display image is supplied to the image output unit 65 of the output unit 30.

[0132] In step S21, the image output unit 65 displays the display image supplied from the display image generation unit 28. As a result, the monitor image 150 shown in FIG.

[0133] The above-described processing of steps S1 to S21 is repeatedly executed until an instruction to end the 3D modeling processing is given, for example, by performing an operation to end imaging. In the monitor image 150 displayed on the display 54, the depth of field region 163, the front depth of field plane 164, and the rear depth of field plane 165 are updated and displayed according to the posture information of the imaging device 51, the captured image, etc.

[0134] 6. Processing flow of imaging control processing The imaging device 51 can automatically perform shutter control (image acquisition processing) at optimal timing without requiring a user to operate the shutter. Referring to the flowchart in FIG. 18 , imaging control processing that combines automatic image acquisition processing and depth of field control will be described. This processing is started, for example, when sensor information of a subject is acquired by the depth sensor 31, image sensor 32, etc. of the sensor unit 52.

[0135] First, in step S40, the imaging control unit 24 sets a region of interest for the subject included in the captured image to be focused on as an imaging target. The imaging control unit 24 sets multiple recommended imaging positions for the subject based on, for example, the preset resolution of the 3D model to be generated and the spacing between imaging positions that will provide an appropriate overlap rate for the captured images. Then, the imaging control unit 24 sets a predetermined one of the multiple recommended imaging positions as the region of interest. The captured image here may be an image captured by the image sensor 32 or an image captured by the imaging unit 20.

[0136] The processes in steps S41 to S47 in FIG. 18 are similar to steps S11 to S17 in FIG. 17, respectively, and therefore will not be described further.

[0137] In step S48, the imaging control unit 24 calculates the distance to the area of ​​interest of the subject based on the posture information of the imaging device 51 supplied from the SLAM 34 and the first three-dimensional shape information of the first 3D model supplied via the scoring processing unit 25.

[0138] In step S49, the imaging control unit 24 determines whether the camera position (the position of the imaging device 51) matches the region of interest. The process of step S49 is repeatedly executed until it is determined that the camera position matches the region of interest. If it is determined that the camera position matches the region of interest, the process proceeds to step S50.

[0139] In step S50, the imaging control unit 24 determines whether the distance to the area of ​​interest of the subject is within the depth of field based on the distance to the area of ​​interest of the subject and the depth of field information supplied from the depth of field calculation unit 62.

[0140] If it is determined in step S50 that the distance to the area of ​​interest of the subject is not within the depth of field (outside the depth of field), the process proceeds to step S51, where the imaging control unit 24 supplies a predetermined control signal to the output unit 30 to notify the user that the shutter cannot be released because the depth of field is exceeded. The notification to the user may be an image displayed by the image output unit 65, or a sound or vibration from the image output unit 65. After step S51, the process returns to step S44, and the processes from step S44 onwards are repeated.

[0141] On the other hand, if it is determined in step S50 that the distance to the region of interest of the subject is within the depth of field, the process proceeds to step S52, where the imaging control unit 24 transmits imaging control information for turning on the shutter to the imaging unit 20, causing the imaging unit 20 to perform imaging as an image acquisition process. Having acquired the imaging control information, the imaging unit 20 performs imaging.

[0142] In step S53, the imaging control unit 24 sets the next region of interest. That is, the imaging control unit 24 sets one of the recommended imaging positions at which imaging has not yet been completed as the next region of interest. After step S53, the process returns to step S41, and the processes of steps S41 to S53 are executed again.

[0143] In the imaging control process of Fig. 18, the imaging control unit 24 determines whether the region of interest is within the depth of field, so the user does not need to determine whether it is within the depth of field. Therefore, the processes of steps S18 and S19 of Fig. 17, i.e., the processes of generating an image showing the depth of field region 163 and superimposing it on the captured image, are not performed, but the depth of field region 163 may be displayed superimposed on the captured image as in Fig. 17.

[0144] 7. Guide Display Indicating Depth of Field Determination Result> With reference to FIGS. 19 and 20 , a guide display that indicates the determination result of whether the distance to the region of interest is within the depth of field, which is executed in the imaging control process of FIG. 18 , will be described.

[0145] FIG. 19 shows an example of a navigation screen displayed on the display 54 of the interface device 53 during imaging for the first 3D modeling.

[0146] The navigation screen 201 in FIG. 19 displays a target indicator 211 and a self-indicator 212 superimposed on the captured image 161 captured by the image sensor 32. The target indicator 211 is a guide mark indicating a recommended imaging position set for the subject based on the resolution, overlap rate, orientation (facing degree) of the 3D model relative to the subject, and the like. The self-indicator 212 is a guide mark indicating the position of the imaging device 51 (camera position). Here, an example is shown in which the target indicator 211 and the self-indicator 212 are each hexagonal, but the shapes of these indicators are not limited to hexagonal. For example, they may be rectangular or circular. The target indicator 211 and the self-indicator 212 are displayed in different forms, such as by changing the color or brightness of their frames or the type of line of their frames, so that the user can distinguish them. For example, the target indicator 211 is white, and the self-indicator 212 is yellow. The self-indicator 212 is positioned approximately in the center of the angle of view, and the position of the target indicator 211 moves as the user (the camera position) moves.

[0147] In the navigation screen 201 of FIG. 19 , after completing imaging of the target indicator 211P indicated by the dashed line among the multiple target indicators 211, the user moves leftward and is currently moving to the position of the target indicator 211N. The target indicator 211P indicated by the dashed line indicates the position where imaging has been completed. The size of the hexagonal shape changes depending on the distance between the subject and the imaging device 51. The user can maintain an appropriate distance from the subject by adjusting (moving) the distance from the subject so that the size of the target indicator 211 approximately matches the size of the self indicator 212. Then, when the target indicator 211N and the self indicator 212 are positioned so that they can be considered to be the same (or approximately the same), it is determined in step S49 of FIG. 18 that the camera position matches the region of interest.

[0148] If it is determined in step S49 of FIG. 18 that the camera position matches the area of ​​interest, the imaging control unit 24 determines whether the distance to the area of ​​interest of the subject is within the depth of field (step S50). If it is determined that the distance to the area of ​​interest of the subject is within the depth of field, the imaging control unit 24 controls to turn on the shutter and executes imaging as an image acquisition process. If imaging is executed, the imaging control unit 24 notifies the user via the output unit 30 that imaging is successful (imaging completion). On the other hand, if it is determined that the distance to the area of ​​interest of the subject is outside the depth of field, the imaging control unit 24 notifies the user via the output unit 30 that imaging is incomplete, i.e., that the shutter cannot be released because the image is outside the depth of field.

[0149] FIG. 20 shows an example of a display image (animation) that displays whether imaging is successful or not on the navigation screen 201.

[0150] The imaging success animation 231A is an example of an animation that the image output unit 65 uses to notify the user that imaging has been successful.

[0151] In the successful imaging animation 231A, several hexagonal frames 221 appear around the original target indicator 211, each expanding and disappearing when it reaches a certain size. After displaying such successful imaging animation 231A, the display control unit 26 changes the display form of the target indicator 211. For example, the target indicator 211, which was previously white, is changed to gray. In Figure 19, the hatching indicates that the color of the target indicator 211 has changed after the successful imaging animation 231A has ended.

[0152] The imaging success animation 231B is an example of an animation that the image output unit 65 uses to notify the user that imaging is not yet complete.

[0153] The successful capture animation 231B is a less flashy animation than the successful capture animation 231A. For example, one frame 222 disappears immediately after trying to expand it. Alternatively, the color of the target indicator 211 may be changed.

[0154] 20 shows an example in which the user is notified of successful or incomplete imaging by displaying an image (animation), but the user may also be notified of successful or incomplete imaging by generating sound or vibration from the sound / vibration output unit 66. Alternatively, the user may be notified of successful or incomplete imaging by simultaneously outputting all or any two of the image, sound, and vibration.

[0155] In the imaging control process of FIG. 18, if it is determined that the distance to the area of ​​interest is outside the depth of field, the user is notified using an image, sound, or vibration without releasing the shutter. However, after releasing the shutter, i.e., after performing imaging as an image acquisition process, the user may be notified again to urge them to retake the image.

[0156] Alternatively, the imaging control unit 24 may notify the imaging unit 20 or the like that the distance to the area of ​​interest is outside the depth of field, without notifying the user to re-image, and may continue focus control and automatically control so that the distance to the area of ​​interest is within the depth of field. The user may be notified that "focus control is in progress" or the like.

[0157] The imaging control unit 24 may also control camera information such as the aperture and focus position (subject distance) based on the three-dimensional shape of the subject, and adjust the depth of field. That is, because the imaging control unit 24 is supplied with first three-dimensional shape information of the first 3D model via the scoring processing unit 25, the imaging control unit 24 can grasp the flatness and complexity of the subject shape. When the imaging control unit 24 determines that the three-dimensional shape of the subject is relatively flat, it controls the imaging control unit 24 to set the aperture and other parameters so that the depth of field is shallow, while focusing on the entire subject. Conversely, when the imaging control unit 24 determines that the three-dimensional shape of the subject is complex and has a wide depth range, it controls the imaging control unit 24 to set the aperture and other parameters so that the depth of field is deep, while focusing on the entire subject as much as possible.

[0158] <8. Manual AF Operation Screen> In the above example, the depth of field is calculated based on the imaging parameters of the imaging device 51, specifically the focal length, the aperture value (F-number), the subject distance, and the allowable circle of confusion, and at least one of the depth of field region 163, the front depth of field plane 164, and the rear depth of field plane 165 is superimposed on the captured image 161 and displayed. However, the user may specify a focus position, and the depth of field region 163, etc. may be superimposed and displayed in accordance with the focus position specified by the user.

[0159] FIG. 21 shows an example of a screen displaying the depth of field area according to the focus position specified by the user.

[0160] 21 shows a monitor image 150 in which a three-dimensional mesh 162 of an object 131 generated by first 3D modeling (real-time 3D modeling) is superimposed on a captured image 161 captured by the image sensor 32. Also displayed on the monitor image 150 are a focal plane (a plane indicating the focal position) 166, a depth of field region 163 centered on the focal plane 166, and a slider 251 for moving the focal plane 171.

[0161] The user can move the focal plane 171 back and forth by moving the slider 251 or the dial. The imaging control unit 24 adjusts the focal length, aperture value, etc. to obtain the focal plane 171 specified by the user, calculates the depth of field based on the adjusted parameters, and displays at least one of the depth of field region 163, the front depth of field plane 164, and the rear depth of field plane 165 superimposed on the captured image 161. Such a display is useful when the user uses manual focus, for example. In addition to moving the focal plane 171 back and forth using the slider 251, a user interface may be provided that allows the user to adjust the depth of field region 163, for example, by moving the front depth of field plane 164 and the rear depth of field plane 165 with a slider, and the imaging control unit 24 may control camera information such as the focal length and aperture value according to the depth of field region 163 specified by the user.

[0162] 9. Depth Stacking Mode The imaging device 51 has a depth stacking mode as an operating mode. The depth stacking mode is a mode in which imaging is performed multiple times using different focus positions, and the in-focus areas (depth of field areas) of the multiple captured images are combined to generate an image (depth stacked image) with a virtually wide depth of field without narrowing the aperture.

[0163] FIG. 22 shows an example of a monitor image (display image) displayed on the display 54 in focus stacking mode.

[0164] Monitor image 271A is an image obtained after the first image capture in depth stacking mode. In monitor image 271A, a depth of field region 281A in the first image capture is visualized by adding, for example, a color or a pattern.

[0165] Monitor image 271B is an image obtained after the second imaging in depth stacking mode. In monitor image 271B, a depth of field region 281B is visualized, which is wider in the depth direction than depth of field region 281A obtained in the first imaging. Depth of field region 281B is made up of the first depth of field region and the second depth of field region.

[0166] Monitor image 271C is an image obtained after the third imaging in depth stacking mode. In monitor image 271C, a depth of field region 281C is visualized, which is wider in the depth direction than depth of field region 281B obtained in the second imaging. Depth of field region 281C is made up of the depth of field region obtained in the first imaging, the depth of field region obtained in the second imaging, and the depth of field region obtained in the third imaging.

[0167] The user can ultimately generate and record a single depth-stitched image by moving the focus position so that the visualized depth-of-field region 281 covers the entire image. Since the imaging device 51 can visualize and present the depth-of-field region, it can efficiently generate a depth-stitched image.

[0168] 10. Depth of Field Information Output In the above-described examples, a depth of field region indicating the depth of field is displayed on a display during image capture to guide the generation of a captured image with higher accuracy. However, depth of field region information indicating the depth of field region of the captured image can be output as data along with the captured image. For example, a depth of field image in which the depth of field region is represented by "1" and regions other than the depth of field region are represented by "0" for each pixel of the captured image can be output as depth of field information as a file separate from the image data of the captured image. Data in which the depth of field region is represented by "1" and regions other than the depth of field region are represented by "0" may be generated and output as metadata for the image data of the captured image. The depth of field region information may be generated by the image processing unit 22 or the image capture control unit 24. The depth of field region information may be output from the output unit 30, transmitted (output) to the server device 40 via the communication unit 29, or stored (recorded) in the storage unit 23. The captured image for which the depth of field information is output may be an image of the first 3D modeling or an image of the second 3D modeling. For example, by outputting the depth of field information as data along with the captured image of the second 3D modeling, when performing photogrammetry or neural rendering, pixels in areas outside the depth of field region can be masked as invalid pixels, thereby preventing the inclusion of areas that are significantly blurred and prone to loss of accuracy when reconstructing a three-dimensional shape.

[0169] <11. Summary> The imaging device 51 includes a camera information acquisition unit 61 that acquires camera information from the first imaging unit (imaging unit 20) when capturing an image of a subject to be 3D modeled; a depth-of-field calculation unit 62 that calculates a depth of field using the camera information; and a display control unit 26 that generates a display image (monitor image 150) in which a depth-of-field region 163 that visualizes the depth of field is superimposed on a captured image 161 obtained by capturing an image of the subject. This allows the user to capture an image while checking the superimposed depth-of-field region 163 to confirm whether the desired portion is in focus. In other words, the image can be captured with an appropriate depth of field. The imaging device 51 also includes a depth-of-field rendering unit 63 that generates a depth-of-field image in which the depth-of-field region 163 is superimposed on the three-dimensional shape (three-dimensional mesh 162) of the subject. This allows the three-dimensional shape of the subject to be captured stereoscopically and the depth of field to be understood.

[0170] In the above example, the first 3D modeling is performed using real-time 3D modeling, and the second 3D modeling is performed using photogrammetry or neural rendering. However, the present invention can be applied to any method for reconstructing (restoring) a three-dimensional shape.

[0171] In this specification, the steps described in the flowcharts may be performed in chronological order in the order described, but they do not necessarily have to be processed in chronological order, and may be performed in parallel or at any necessary timing, such as when a call is made.

[0172] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0173] The embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the technology of the present disclosure.

[0174] For example, it is possible to adopt a configuration in which all or part of the above-described embodiments are combined.

[0175] For example, the technology disclosed herein can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network. Each step described in the flowchart above can be executed by a single device or shared among multiple devices. Furthermore, when a single step includes multiple processes, the multiple processes included in that single step can be executed by a single device or shared among multiple devices.

[0176] The effects described in this specification are merely examples and are not limiting, and there may be effects other than those described in this specification.

[0177] The technology disclosed herein may employ the following configurations. (1) An imaging device including: an acquisition unit that acquires camera information of a first imaging unit when an object to be 3D modeled is captured; a calculation unit that calculates a depth of field using the camera information; and a display control unit that generates a display image in which a depth of field region that visualizes the depth of field is superimposed on an image obtained by capturing an image of the object. (2) The imaging device described in (1), further including a rendering unit that generates a depth of field image in which the depth of field region is superimposed on a three-dimensional shape of the object, wherein the display control unit generates a display image in which the depth of field image is superimposed on the captured image. (3) The imaging device described in (1) or (2), in which the three-dimensional shape of the object is generated using a distance to the object measured by a distance measuring unit. (4) The imaging device described in any of (1) to (3), in which the captured image on which the depth of field region is superimposed is an image of a second imaging unit that is different from the first imaging unit from which the camera information is acquired. (5) The imaging device according to any one of (1) to (4), wherein the captured image on which the depth of field region is superimposed is an image of the first imaging unit from which the camera information is acquired. (6) The imaging device according to any one of (1) to (5), wherein the display control unit generates a display image on which the depth of field region, in which a predetermined color or pattern is applied to the depth of field portion of the three-dimensional shape of the subject, is superimposed. (7) The imaging device according to any one of (1) to (6), wherein the display control unit generates a display image on which the depth of field region, in which the depth of field is visualized by a front depth of field plane and a rear depth of field plane, is superimposed on the captured image. (8) The imaging device according to any one of (1) to (7), wherein the display control unit generates a display image on which a focal plane is superimposed on the captured image. (9) The imaging device according to any one of (1) to (8), wherein the display control unit generates a display image in which a focal plane is superimposed on the captured image, and the focal plane is movable according to a user's designation.(10) The imaging device according to any of (1) to (9), further comprising: an operation input unit that accepts a user's operation input to adjust the depth of field region of the display image; and an imaging control unit that controls the camera information according to the depth of field region specified by the user. (11) The imaging device according to any of (1) to (10), further comprising: an imaging control unit that controls the camera information based on a three-dimensional shape of the subject and adjusts the depth of field. (12) The imaging device according to any of (1) to (11), further comprising: an imaging control unit that determines whether the distance to the subject is within the depth of field, and causes the first imaging unit to perform imaging if the distance is within the depth of field. (13) The imaging device according to (12), further comprising: an output unit that notifies the user that the distance to the subject is outside the depth of field. (14) The imaging device according to (13), wherein the output unit notifies the user that the distance to the subject is outside the depth of field by an image, sound, or vibration. (15) The imaging device according to any of (12) to (14), further comprising an output unit that notifies a user that imaging has been performed by the first imaging unit. (16) The imaging device according to any of (1) to (15), wherein the display control unit has a mode for generating a display image that combines the depth of field regions of multiple images. (17) The imaging device according to any of (1) to (16), further comprising an image processing unit that generates depth of field information that indicates the depth of field region. (18) An imaging method, wherein the imaging device acquires camera information of the first imaging unit when imaging a subject, calculates a depth of field using the camera information, and generates a display image in which a depth of field region that visualizes the depth of field is superimposed on an imaged image obtained by imaging a subject to be 3D modeled. (19) A program for causing a computer to execute the following process: acquire camera information of a first imaging unit when an object is imaged; calculate a depth of field using the camera information; and generate a display image in which a depth of field region that visualizes the depth of field is superimposed on an image obtained by imaging the object to be 3D modeled.

[0178] 20 Imaging unit, 21 Operation unit, 22 Image processing unit, 23 Memory unit, 24 Imaging control unit, 25 Scoring processing unit, 26 Display control unit, 27 Superimposed image generation unit, 28 Display image generation unit, 29 Communication unit, 30 Output unit, 31 Depth sensor, 32 Image sensor, 35 TSDF update unit, 36 Mesh generation unit, 40 Server device, 41 Communication unit, 42 Photogrammetry processing unit, 43 Memory unit, 50 Camera, 51 Imaging device, 52 Sensor unit, 53 Interface device, 54 Display, 55 Display, 56 Viewfinder, 57 Main body unit, 58 Lens barrel, 61 Camera information acquisition unit, 62 Depth of field calculation unit, 63 Depth of field drawing unit, 65 Image output unit, 66 vibration output unit, 70 information processing device, 71 CPU, 72 ROM, 73 RAM, 74 non-volatile memory unit, 75 input / output interface, 76 input unit, 77 display unit, 78 audio output unit, 79 storage unit, 80 communication unit, 81 drive, 82 removable recording medium, 83 bus, 84 sensor unit, 85 image processing unit, 150 monitor image, 161 captured image, 162 three-dimensional mesh, 163 depth of field region, 164 front depth of field plane, 165 rear depth of field plane, 171 focus plane, 172 front depth of field, 173 rear depth of field

Claims

1. An imaging device comprising: an acquisition unit that acquires camera information of a first imaging unit when an image of a subject to be 3D modeled is captured; a calculation unit that calculates a depth of field using the camera information; and a display control unit that generates a display image in which a depth of field region that visualizes the depth of field is superimposed on an image obtained by capturing an image of the subject.

2. The imaging device of claim 1, further comprising a drawing unit that generates a depth of field image in which the depth of field region is superimposed on the three-dimensional shape of the subject, and the display control unit generates a display image in which the depth of field image is superimposed on the captured image.

3. The imaging device according to claim 2, wherein the three-dimensional shape of the subject is generated using the distance to the subject measured by a distance measuring unit.

4. The imaging device according to claim 1, wherein the captured image onto which the depth of field region is superimposed is an image of a second imaging unit different from the first imaging unit from which the camera information is acquired.

5. The imaging device according to claim 1, wherein the captured image onto which the depth of field region is superimposed is an image of the first imaging section from which the camera information is acquired.

6. The imaging device according to claim 1, wherein the display control unit generates a display image in which the depth of field region having a predetermined color or pattern applied thereto is superimposed on the depth of field portion of the three-dimensional shape of the subject.

7. The imaging device according to claim 1, wherein the display control unit generates a display image in which the depth of field area, in which the depth of field is visualized using a front depth of field plane and a rear depth of field plane, is superimposed on the captured image.

8. The imaging device according to claim 1, wherein the display control unit generates a display image in which a focal plane is superimposed on the captured image.

9. The imaging device according to claim 1, wherein the display control unit generates a display image in which a focal plane is superimposed on the captured image, and the focal plane is movable according to a user's designation.

10. The imaging device according to claim 1, further comprising: an operation input unit that accepts a user's operation input to adjust the depth of field area of ​​the displayed image; and an imaging control unit that controls the camera information according to the depth of field area specified by the user.

11. The imaging device according to claim 1, further comprising an imaging control unit that controls the camera information based on the three-dimensional shape of the subject and adjusts the depth of field.

12. The imaging device according to claim 1, further comprising an imaging control unit that determines whether the distance to the subject is within the depth of field, and, if it is within the depth of field, causes imaging by the first imaging unit to be performed.

13. The imaging device according to claim 12, further comprising an output unit that notifies a user that the distance to the subject is outside the depth of field.

14. The imaging device according to claim 13, wherein the output unit notifies the user that the distance to the subject is outside the depth of field by an image, sound, or vibration.

15. The imaging device according to claim 12, further comprising an output section for notifying a user that imaging has been performed by said first imaging section.

16. The imaging device according to claim 1, wherein the display control unit has a mode for generating a display image that combines the depth of field regions of multiple captured images.

17. The imaging device according to claim 1, further comprising an image processing unit that generates depth of field information indicating the depth of field region.

18. An imaging method in which an imaging device acquires camera information from a first imaging unit when an object is imaged, calculates a depth of field using the camera information, and generates a display image in which a depth of field region that visualizes the depth of field is superimposed on an image obtained by imaging the object to be 3D modeled.

19. A program for causing a computer to execute a process of acquiring camera information from a first imaging unit when an object is imaged, calculating a depth of field using the camera information, and generating a display image in which a depth of field region that visualizes the depth of field is superimposed on an image obtained by imaging a object to be 3D modeled.

Images