Imaging device, imaging method, and program

Abstract

The image forming apparatus includes a display control unit that performs a process of displaying an image on the image forming apparatus, processing for displaying a first indicator superimposed on the imaging monitor image to indicate a recommended imaging position and a second indicator indicating a current imaging position and changing the display of the first indicator according to an image obtaining process for imaging the image data; and an imaging control unit that causes an image obtaining process to be performed in accordance with a result of the coincidence determination between the first indicator and the second indicator on the screen.

Description

Technical Field

[0001] The present technology relates to an imaging apparatus, an imaging method, and a program, and more particularly, to a technology that enables imaging operations for three-dimensional (3D) modeling to be performed more easily. Background Art

[0002] As a method for 3D modeling a 3D object having a three-dimensional shape, there is a method called photogrammetry, in which multiple images of the 3D object are captured from multiple directions, and 3D data is generated based on the multiple captured images. For example, Patent Document 1 discloses this method.

[0003] In addition, there is a method called real-time 3D modeling in which 3D data is generated instantaneously (in real time) based on information such as a captured image, posture information, and depth.

[0004] In addition, in recent years, methods collectively referred to as neural rendering (e.g., Neural Radiance Field (NeRF) etc.) have been proposed, in which a neural field is configured based on the pose of a captured image and the captured image to generate an image from a desired viewpoint or three-dimensional model.

[0005] Citation List

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Laid-Open No. 2018-63693 Summary of the Invention

[0008] Problems to be solved by the present invention

[0009] When performing 3D modeling based on captured images of a 3D object, as in these methods, it is desirable to generate as accurate 3D data as possible with as few imaging passes as possible in order to reduce workload and processing. To this end, it is necessary to capture images of the 3D object in a more appropriate position and orientation.

[0010] However, it is difficult for an imaging operator to identify an appropriate position and posture for imaging.

[0011] Therefore, the present disclosure proposes a technology for facilitating an operation performed in a case where a user captures an image of a 3D object using an imaging device and improving the efficiency of the operation.

[0012] Solution to the problem

[0013] An imaging device according to the present technology includes: a display control unit configured to display a first indicator indicating a recommended imaging position and a second indicator indicating a current imaging position in a superimposed manner on an imaging monitor image, and to perform a process of changing the display of the first indicator according to an image acquisition process for obtaining image data; and an imaging control unit configured to perform an image acquisition process based on a result determined by matching between the first indicator and the second indicator on the screen.

[0014] A first indicator is superimposed and displayed on the imaging monitor image, and the user is guided to set the recommended imaging position in the direction of the subject. The user adjusts the imaging position and posture (orientation) of the imaging device so that the second indicator indicating its imaging position substantially matches the first indicator. For example, when image acquisition processing is automatically executed as a result of a matching determination (the matching determination is used to determine a substantially matching state in which the positions of the first indicator and the second indicator substantially match in the image), the display of the first indicator is changed to indicate that an image has been captured as part of the image acquisition processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This diagram explains the overview of photogrammetry.

[0016] Figure 2 This diagram provides an overview of real-time 3D modeling.

[0017] Figure 3 An illustration of the difference between photogrammetry and real-time 3D modeling.

[0018] Figure 4 It is a perspective view of an example of the appearance of an imaging device according to an embodiment of the present technology.

[0019] Figure 5 is another perspective view of an example of the appearance of the imaging device according to the embodiment.

[0020] Figure 6 is a block diagram showing an example of an internal configuration of an imaging device according to an embodiment.

[0021] Figure 7 It is an explanatory diagram of the functional configuration of the server device according to the embodiment.

[0022] Figure 8 This is a block diagram of an information processing device applicable to the embodiment.

[0023] Figure 9 It is an explanatory diagram of the ROI setting screen according to the embodiment.

[0024] Figure 10is another explanatory diagram of the ROI setting screen according to the embodiment.

[0025] Figure 11 This is another explanatory diagram of the ROI setting screen according to the embodiment.

[0026] Figure 12 is another explanatory diagram of the ROI setting screen according to the embodiment.

[0027] Figure 13 It is an explanatory diagram of a navigation screen according to an embodiment.

[0028] Figure 14 is an explanatory diagram of a target indicator according to an embodiment.

[0029] Figure 15 It is an explanatory diagram of changes in the navigation screen according to the embodiment.

[0030] Figure 16 It is an explanatory diagram of successful imaging on a navigation screen according to an embodiment.

[0031] Figure 17 It is an explanatory diagram of an animation on a navigation screen according to an embodiment.

[0032] Figure 18 It is an explanatory diagram of changes in the navigation screen according to the embodiment.

[0033] Figure 19 It is an explanatory diagram of changes in the navigation screen according to the embodiment.

[0034] Figure 20 It is an explanatory diagram of inappropriate imaging on a navigation screen according to an embodiment.

[0035] Figure 21 It is an explanatory diagram of re-imaging on a navigation screen according to an embodiment.

[0036] Figure 22 FIG. 1 is another explanatory diagram of re-imaging on a navigation screen according to an embodiment.

[0037] Figure 23 It is an explanatory diagram of changes in the navigation screen according to the embodiment.

[0038] Figure 24 It is an explanatory diagram of changes in the navigation screen according to the embodiment.

[0039] Figure 25 It is an explanatory diagram of a preview screen according to an embodiment.

[0040] Figure 26 It is an explanatory diagram of displaying recommended imaging positions on a preview screen according to an embodiment.

[0041] Figure 27 It is an explanatory diagram of a display based on image data evaluation on a preview screen according to an embodiment.

[0042] Figure 28 is a flow chart of a display algorithm in the case where scanning is performed in real time according to an embodiment.

[0043] Figure 29 It is an explanatory diagram of the arrangement of the display unit of the target indicator according to the embodiment.

[0044] Figure 30 This is another explanatory diagram of the arrangement of the display portion of the target indicator according to the embodiment.

[0045] Figure 31 This is another explanatory diagram of the arrangement of the display portion of the target indicator according to the embodiment.

[0046] Figure 32 This is another explanatory diagram of the arrangement of the display portion of the target indicator according to the embodiment.

[0047] Figure 33 It is an explanatory diagram of a matching determination algorithm according to an embodiment.

[0048] Figure 34 is a flowchart of a display algorithm in the case of performing a pre-scan according to an embodiment.

[0049] Figure 35 It is an explanatory diagram of a display example of a direction presentation portion of a target indicator according to an embodiment.

[0050] Figure 36 is an explanatory diagram of an example of a target indicator according to an embodiment.

[0051] Figure 37 is another explanatory diagram of an example of a target indicator according to an embodiment.

[0052] Figure 38 is another explanatory diagram of an example of a target indicator according to an embodiment.

[0053] Figure 39 It is an explanatory diagram of a display example of a navigation screen according to an embodiment.

[0054] Figure 40 This is another explanatory diagram of a display example of a navigation screen according to the embodiment.

[0055] Figure 41 This is another explanatory diagram of a display example of a navigation screen according to the embodiment.

[0056] Figure 42It is an explanatory diagram of a change of a target indicator according to an embodiment.

[0057] Figure 43 It is an explanatory diagram of a warning display according to an embodiment.

[0058] Figure 44 It is an explanatory diagram of speed display according to an embodiment.

[0059] Figure 45 It is an explanatory diagram of a model display according to an embodiment.

[0060] Figure 46 is an explanatory diagram of a display for prompting a user to perform re-imaging according to an embodiment.

[0061] Figure 47 It is an explanatory diagram of a status display according to an embodiment.

[0062] Figure 48 It is an explanatory diagram of the definition setting according to the embodiment.

[0063] Figure 49 It is an explanatory diagram of an example of quality display on a preview screen according to an embodiment.

[0064] Figure 50 It is an explanatory diagram of a display example of an insufficiently imaged portion according to an embodiment.

[0065] Figure 51 It is an explanatory diagram of the display of symbols on the preview screen and the navigation screen according to the embodiment.

[0066] Figure 52 It is an explanatory diagram of an example of the self-position display on the preview screen according to the embodiment.

[0067] Figure 53 is an explanatory diagram of an example of ROI setting according to an embodiment.

[0068] Figure 54 It is an explanatory diagram of a display example during a pre-scan according to an embodiment.

[0069] Figure 55 It is an explanatory diagram of another display example during the pre-scan according to the embodiment. DETAILED DESCRIPTION

[0070] Hereinafter, the embodiments will be described in the following order.

[0071] 1.3D Modeling

[0072] <2. Device Configuration>

[0073] <3. User Interface Screen>

[0074] <4. Processing Example>

[0075] <5. Examples of various screen displays>

[0076] <6. Conclusion and Modifications>

[0077] Note that in the present disclosure, "image acquisition processing" refers to capturing an image using an imaging device including an image sensor (imaging element) to obtain image data to be temporarily or permanently stored in a storage medium. The storage medium in which the image data is to be stored may be inside or outside the imaging device. For example, the image data may be immediately stored in the storage medium of the imaging device, but alternatively, it may be transmitted to another device or transmitted (uploaded, etc.) over a network and stored in a storage medium in an external device. The term "image acquisition processing" is used to distinguish it from imaging that is only used to display an image of a subject on a monitor.

[0078] In addition, images include both still images and moving images.

[0079] The captured image refers to a moving image or a still image captured by an image sensor or an image thereof. That is, the captured image includes an image to be stored in a storage medium or an image to be displayed on a monitor as a through image.

[0080] The captured image may be data obtained by an image sensor or the like (so-called raw data). Alternatively, the captured image may be an image that has undergone color separation processing or color conversion processing. Alternatively, the captured image may be an image that has undergone signal processing such as defect correction, noise reduction, automatic white balance (AWB), or gamma correction. Furthermore, the captured image may be subjected to other types of image processing.

[0081] 1.3D Modeling

[0082] First, photogrammetry will be described.

[0083] As a method for generating (reconfiguring) a three-dimensional shape model of an object (also referred to as a 3D object), there is a method called photogrammetry, in which images of the 3D object are captured from multiple directions, and a 3D model (3D data) is generated based on the multiple captured images. Note that in this disclosure, generating a three-dimensional shape model of a 3D object will also be referred to as 3D modeling.

[0084] Photogrammetry is a method for reconfiguring a highly accurate three-dimensional model from multiple images captured from various viewpoints using the principle of triangulation. Figure 1The camera 50 shown in FIG captures images of the 3D object 100 from multiple viewpoints, and obtains a plurality of captured images. Then, a process called structure from motion (SfM) and a process called multi-view stereo (MVS) are performed using these captured images, and meshing and texturing are further performed as post-processing to generate a 3D model 105.

[0085] Note that in this article, the "accuracy" of a 3D model may include not only the reproducibility of the three-dimensional shape of the target 3D object (i.e., the accuracy, clarity, etc. of the three-dimensional shape), but also the reproducibility (accuracy, clarity, etc.) of the texture applied to the surface of the 3D model.

[0086] In SfM, for example, corresponding points are searched between captured images, the camera position and pose are derived using epipolar constraints, and the position of each corresponding point in three-dimensional space is determined through triangulation based on the camera position and pose. In this disclosure, these points in three-dimensional space will also be referred to as three-dimensional points. That is, the three-dimensional point corresponding to each corresponding point is determined. The entire three-dimensional point cloud determined as described above is then optimized through bundle adjustment.

[0087] In MVS, for example, the three-dimensional point cloud derived as described above is used to search for denser corresponding points, and the three-dimensional points are added.

[0088] As described above, photogrammetry uses global optimization calculations (i.e., bundle adjustment) to minimize errors, resulting in highly accurate results. However, this involves a high computational load. Furthermore, since photogrammetry is based on geometric calculations rather than physical measurements, it is possible to recover a more accurate model using higher-resolution images.

[0089] Next, real-time 3D modeling will be described.

[0090] As a 3D modeling method different from photogrammetry, there is a method called real-time 3D modeling in which 3D data is generated instantaneously (in real time) based on information such as captured images, posture information, and depth. In the case of this method, for example, Figure 2 As shown in FIG, as indicated by a dotted line 101, an image of a 3D object 100 is captured while the camera 50 is moved around the 3D object 100. The camera 50 includes not only an image sensor but also a light detection and ranging (lidar) scanner (direct time of flight (dToF) module), and obtains a captured image while detecting a depth (distance to a subject).

[0091] In recent years, the miniaturization and functional enhancement of dToF modules have made progress, and they can accurately measure the depth of relatively long distances (e.g., about 5 meters) both indoors and outdoors. As a result, it has become easy to experience real-time modeling and capture at the consumer level.

[0092] The camera 50 also includes an inertial sensor, and detects inertial information including acceleration and angular velocity of the camera 50 .

[0093] In real-time 3D modeling, a process for calculating a three-dimensional pose, called simultaneous localization and mapping (SLAM), is performed to generate pose information indicating the position and pose of the camera 50. In addition, the pose information and depth are used to update the truncated signed distance function (TSDF), and a 3D model 105 (mesh and texture) is generated through a process called marching cubes (MC).

[0094] In SLAM, for example, the position and orientation of the camera 50 are estimated based on captured images and inertial information (self-positioning).

[0095] In the update of TSDF, depth and voxel are associated with each other, and the volume is detected.

[0096] In MC, isosurfaces are calculated using neighboring voxels. Leveraging real-time pose information from SLAM, the volume of a voxel can be detected by overlaying multiple depth (how far the beam has reached) frames on top of each other, rather than using a point cloud. This voxel representation allows estimation of viewpoints (or missing viewpoints) that are obscured and from which images need to be captured. This makes it possible to detect holes or protruding structures in 3D objects.

[0097] Another 3D modeling method is also known as neural rendering. This method configures a neural field based on the pose of a captured image and the captured image to generate an image from a desired viewpoint or three-dimensional model, and examples include Neural Radiance Field (NeRF).

[0098] Each of these 3D modeling methods has different characteristics, and no single method excels in all aspects. Figure 3 A comparison between the characteristics of photogrammetry and real-time 3D modeling is shown. Figure 3As can be seen from the comparison of the methods shown in , photogrammetry uses SfM (including self-localization) and MVS, while real-time 3D modeling uses self-localization (SLAM) and TSDF. In addition, as can be seen from the comparison of the data to be used, photogrammetry uses only image data, while real-time 3D modeling uses depth and pose data in addition to image data. In addition, as can be seen from the comparison of processing time, photogrammetry requires a long time ranging from several minutes to tens of hours, while real-time 3D modeling allows near-instantaneous processing of, for example, 30 fps (frames per second).

[0099] In addition, as can be seen from the comparison of required computing power, photogrammetry requires computing power at the level of a high-end central processing unit (CPU) and a graphics processing unit (GPU), while real-time 3D modeling requires computing power at the level of a mobile application processor (AP). In addition, as can be seen from the comparison of the clarity of the model to be generated, photogrammetry results in relatively high clarity, however, the clarity depends on the imaging method, number and resolution of the captured images, etc., while real-time 3D modeling results in relatively low clarity, however, the clarity depends on the depth and accuracy of the self-localization.

[0100] In addition, regarding the internal representation of the three-dimensional data to be generated, photogrammetry is based on point clouds, while real-time 3D modeling is based on voxels. In addition, photogrammetry has no limitations in terms of subject size and resolution, while real-time 3D modeling relies on sensors. In addition, as can be seen from the comparison of the absolute accuracy of the models, photogrammetry results in relatively high absolute accuracy due to optimization through bundle adjustment, while real-time 3D modeling results in relatively low absolute accuracy, however, the absolute accuracy depends on the accuracy of the sensor and self-positioning. In addition, as can be seen from the comparison of scales, photogrammetry has an inconstant scale (unknown size), while real-time 3D modeling has a unique scale (absolute size is known).

[0101] For example, there are differences in characteristics between photogrammetry and real-time 3D modeling. Specifically, the workload and processing required for real-time 3D modeling can be reduced compared to using photogrammetry or neural rendering. However, a more accurate 3D model can be generated using photogrammetry or neural rendering than using real-time 3D modeling.

[0102] Therefore, to obtain more accurate 3D data, photogrammetry or neural rendering can be used. However, in this case, it is also desirable to minimize the workload and processing of 3D modeling. To reduce the workload and processing of 3D modeling, it is necessary to generate the most accurate 3D model possible while minimizing the number of imaging times.

[0103] For example, if the captured images required for 3D modeling are unavailable, the accuracy of the generated 3D model may decrease. Conversely, if an excessive number of captured images are acquired to avoid deficiencies, the imaging frequency increases unnecessarily, and the user's workload increases accordingly. Furthermore, in this case, because 3D modeling is performed using unnecessary captured images, the processing volume increases unnecessarily.

[0104] That is, in order to obtain more accurate 3D data with less operation and processing, it is necessary to capture images of 3D objects in a more appropriate position and posture. However, in each of the 3D modeling methods described above, it is difficult for the imaging operator to identify the appropriate position and posture for imaging.

[0105] For example, since photogrammetry requires time for 3D modeling processing, it is difficult for the imaging operator to instantly view the 3D modeling results during imaging. Consequently, it is difficult for the imaging operator to identify the appropriate position and posture for imaging during imaging. Consequently, for example, a sufficient number of images may not be captured at the appropriate position and posture, and the accuracy of the 3D data obtained through photogrammetry may decrease. Furthermore, if imaging is performed excessively and arbitrarily at many positions and postures in order to avoid insufficient images captured at the appropriate position and posture, there is the possibility that not only the user's operation load increases, but also the number of captured images increases unnecessarily, and the load of the 3D modeling processing (processing volume, processing time, etc.) also increases unnecessarily.

[0106] Thus, in this embodiment, 3D modeling is performed twice, and the results of the first 3D modeling are used to guide and control the imaging for the second 3D modeling.

[0107] For example, the first 3D modeling is performed as real-time 3D modeling, and a first 3D model representing the three-dimensional shape of the 3D object is generated. Information based on the real-time 3D modeling is used to guide the user (the user is an imaging operator) and to control imaging for the second 3D modeling (i.e., for example, photogrammetry or neural rendering). That is, the user is guided so that imaging for the second 3D modeling process can be performed at a more appropriate position and posture. In addition, in the case of this embodiment, it is not necessary for the user to perform a shutter operation, and imaging recording (image acquisition processing) is automatically performed at the optimal timing.

[0108] Imaging for the first 3D modeling and imaging for the second 3D modeling (image acquisition processing) may be performed simultaneously, or imaging for the first 3D modeling may be performed in advance and then imaging for the second 3D modeling (image acquisition processing) may be performed.

[0109] <2. Device Configuration>

[0110] An imaging device 51 according to an embodiment will be described.

[0111] Figure 4 and Figure 5 An example of the appearance of the imaging device 51 is shown. In this example, it is assumed that the imaging device 51 is an image pickup device in which a lens barrel 58 is attached to a main body 57. The lens barrel 58 may be an interchangeable lens type, or may be a non-detachable lens provided integrally with the main body 57.

[0112] The light incident through the lens barrel 58 forms an image in the image sensor provided in the main body 57 (ie, the light is converted into electricity), and is processed as a captured image. Here, the image sensor constitutes a device described later. Figure 6 The imaging unit 20 in FIG.

[0113] The main body 57 and the lens barrel 58 are provided with: various controllers for the user to perform operations; a display 55 that displays an imaging monitor image, various icons, menus, etc.; a viewfinder 56, etc.

[0114] The sensor unit 52 is attached to the main body 57. The sensor unit 52 is a unit provided with, for example, Figure 6 The imaging unit 32 shown in FIG. 3 includes an image sensor; a depth sensor 31; an inertial measurement unit (IMU) 33; and the like.

[0115] With the sensor unit 52 , a captured image is obtained by the image sensor, distance information on a subject is obtained by the depth sensor 31 , and inertial information on the imaging device 51 is obtained by the IMU 33 .

[0116] In addition, the interface device 53 is attached to the main body 57. For example, an information processing device such as a smartphone can be used as the interface device 53 to cooperate with the imaging device 51. The display 54 of the interface device 53 displays various types of information to the user. For example, it displays an imaging monitor image (through image) of the subject. In addition, the display 54 also displays guidance for imaging operations to the user. In addition, the display 54 is a touch panel and receives user operation input.

[0117] Note that although Figure 4 and Figure 5 In the embodiment, a smartphone or the like is used as the interface device 53, but the display 55 of the main body 57 can replace the interface device 53 to display various types of information and detect operation input, and a separate interface device 53 can be omitted.

[0118] In addition, an example can be conceived in which the imaging device including the main body 57 and the lens barrel 58 shown in the drawings is not used. For example, since information processing devices such as smartphones generally have an imaging function and also generally have a distance measurement function and an inertial sensor function, in this embodiment, a smartphone, a tablet device, or the like can be used alone as the imaging device 51.

[0119] In addition, although the sensor unit 52 is shown, the image sensor, depth sensor 31, IMU 33, and the like constituting the imaging unit 32 may be built into the main body 57 or may be built into the interface device 53. In addition, when an information processing device such as a smartphone is the imaging device 51, the image sensor, depth sensor 31, and IMU 33 constituting the imaging unit 32 may be built into the information processing device.

[0120] Alternatively, the display device for displaying images, etc. may be an external device. For example, display data may be transmitted from the imaging device 51 via short-range wireless communication, wired communication, etc., and displayed on a monitor display device, a head-mounted display, a display of another information processing device, etc.

[0121] That is, the imaging device 51 according to the embodiment is a device capable of capturing images in any form, and only needs to have a control function of executing an interface based on image display or the like on an integrated screen or a separate screen or the like.

[0122] Figure 6 An example of the internal configuration of the imaging device 51 is shown. Figure 5 In the case of the example in FIG. 5 , for example, the illustrated components are provided in any one of the sensor unit 52 , the main body 57 , and the interface device 53 , but may be provided at appropriate positions in the entire imaging device 51 .

[0123] In addition, the imaging device 51 may include Figure 6 There may be devices or processing units not shown as blocks, or there may be data or processing flows not shown as arrows, etc.

[0124] The imaging device 51 includes, for example, a depth sensor 31 , an imaging unit 32 , an IMU 33 , a SLAM 34 , a TSDF updating unit 35 , and a mesh generating unit 36 ​​, as components for performing real-time 3D modeling as first 3D modeling.

[0125] The imaging device 51 further includes an imaging unit 20 , an operation unit 21 , an image processing unit 22 , a storage unit 23 , and a communication unit 29 as components for performing image obtaining processing for the second 3D modeling.

[0126] Note that in this example, the server device 40 performs photogrammetry as the second 3D modeling, not the imaging device 51. The server device 40 is, for example, a cloud server. The imaging device 51 uploads the captured image data obtained by the imaging unit 20 to the server device 40 via network communication.

[0127] The imaging device 51 includes a scoring processing unit 25 , an imaging control unit 24 , a display control unit 30 , and an output unit 26 as components for user guidance and various types of control. The display control unit 30 includes a superimposed image generation section 27 and a display image generation section 28 .

[0128] In the imaging device 51 , the depth sensor 31 includes a lidar sensor (dToF module) or the like, detects the depth of the subject, and supplies the depth to the TSDF updating unit 35 .

[0129] The imaging unit 32 includes an image sensor and captures an image of a subject to generate a captured image. The image captured by the imaging unit 32 is used for the first real-time 3D modeling. Therefore, the imaging unit 32 provides the captured image data to the SLAM 34.

[0130] The IMU 33 detects inertial information (acceleration and angular velocity) about the imaging device 51 , and supplies the inertial information to the SLAM 34 .

[0131] The SLAM 34 performs self-positioning based on the supplied captured images and inertial information, and generates attitude information indicating the position and attitude of the imaging device 51. The SLAM 34 supplies the generated attitude information to the TSDF updating unit 35, the imaging control unit 24, and the superimposed image generation section 27 in the display control unit 30.

[0132] The TSDF updating unit 35 updates the TSDF based on the posture information and the depth, and supplies the updated TSDF to the mesh generating unit 36 ​​.

[0133] The mesh generation unit 36 ​​generates a mesh and a texture using the updated TSDF. The mesh generation unit 36 ​​supplies the mesh and the texture to the scoring processing unit 25 as first 3D data (first three-dimensional shape information).

[0134] The scoring processing unit 25 performs scoring processing based on the supplied first 3D data and imaging viewpoint information supplied from the imaging control unit 24 (ie, information indicating the position and orientation at which imaging (image obtaining processing) for second 3D modeling has been performed).

[0135] This scoring process is a process for evaluating the accuracy of the second 3D model generated by the server device 40 using the images captured so far by the imaging unit 20 using the three-dimensional shape information based on the first 3D model and generating a scoring result.

[0136] For example, the scoring processing unit 25 may generate a scoring result for each of the local portions of the first three-dimensional shape information based on the position and posture used for imaging by the imaging unit 20 up to that time and the first three-dimensional shape information. For example, the scoring processing unit 25 may generate a scoring result for each of the polygons of the mesh. The scoring processing unit 25 provides the scoring result to the imaging control unit 24 and the superimposed image generation unit 27.

[0137] Note that the scoring processing unit 25 may obtain camera information about the imaging unit 20 and generate a scoring result based on the camera information.

[0138] The imaging control unit 24 executes imaging control processing as image obtaining processing performed by the imaging unit 20 .

[0139] For example, the imaging control unit 24 sets a portion of the 3D object as the subject to be imaged for the image acquisition process based on the first three-dimensional shape information. Then, the imaging unit 20 is used to determine the timing at which it can appropriately capture an image of the portion to be imaged based on the posture information of the imaging device 51. At this timing, the imaging unit 20 is controlled by shutter to perform imaging as the image acquisition process.

[0140] In addition, the imaging control unit 24 supplies imaging viewpoint information indicating the position and posture for the performed imaging to the scoring processing unit 25 .

[0141] In addition, the imaging control unit 24 provides the display control unit 30 with information necessary for guiding the user to move the imaging device 51 to a position where imaging is to be performed.

[0142] To achieve high-quality 3D modeling using photogrammetry, imaging must be performed while facing the imaging plane of the subject from a certain distance. Furthermore, due to surface unevenness, the imaging interval must be adjusted according to the size of the common field of view.

[0143] Therefore, the imaging control unit 24 sets recommended imaging positions for the 3D object based on the shape, unevenness, and other aspects of each part of the 3D object, so that captured images can be obtained in an overlapping state using an appropriate overlap ratio. A suitable overlap ratio is set, for example, to approximately 25% to 75%. As the overlap ratio increases, the amount of information also increases. Furthermore, the imaging control unit 24 can also use the scoring results to set appropriate recommended imaging positions. For example, imaging positions can be densely arranged at locations with low evaluation scores.

[0144] Therefore, the imaging control unit 24 sets a plurality of recommended imaging positions (points at which image acquisition processing is to be performed) for the 3D object as the subject, but the user cannot recognize where imaging is to be performed based solely on this setting. Therefore, the imaging control unit 24 provides information about the set recommended imaging positions and the like to the display control unit 30 so that the display control unit 30 can display guidance and the like for the user.

[0145] The imaging unit 20 includes an image sensor and captures an image of a subject to generate a captured image. As described above, the imaging unit 20 captures an image through shutter control by the imaging control unit 24 as an image acquisition process and supplies image data to the image processing unit 22.

[0146] The operation unit 21 is a controller that can be operated by the user and includes a shutter controller, a zoom controller, a focus controller, and other controllers for various camera operations. The imaging unit 20 can perform imaging according to a shutter operation instruction from the user provided from the operation unit 21 to generate a captured image.

[0147] In addition, the imaging unit 20 may provide camera information (ie, internal parameters, external parameters, field of view information, etc. regarding the imaging unit 20 ) to the scoring processing unit 25 .

[0148] In addition, the imaging unit 20 may provide the imaging control unit 24 with imaging timing information indicating that the imaging timing is not based on shutter control performed by the imaging control unit 24 , such as a shutter operation performed by the user using the operation unit 21 .

[0149] The image processing unit 22 performs predetermined image processing on the captured image data generated by the imaging unit 20. This image processing may have any content. The image processing unit 22 performs, for example, signal processing such as color separation processing, color conversion processing, defect correction, noise reduction, AWB, and gamma correction on the captured image data. In addition, the image processing unit 22 performs encoding processing on the captured image data.

[0150] The storage unit 23 stores the supplied encoded data in a storage medium.

[0151] The communication unit 29 transmits the supplied encoded data to other information processing devices (eg, a server, etc.) In this example, it is assumed that the communication unit 29 sequentially uploads captured image data to the server device 40 .

[0152] As in Figure 7 , the imaging device 51 and the server device 40 perform data communication with each other through the network 107. Captured image data obtained by imaging performed as an image obtaining process by the imaging unit 20 is uploaded to the server device 40, and the server device 40 performs second 3D modeling.

[0153] Therefore, the server device 40 includes components as a communication unit 41 , a photogrammetry processing unit 42 , and a storage unit 43 .

[0154] The communication unit 41 sequentially receives captured image data uploaded from the imaging device 51. In addition, the communication unit 41 transmits information on the 3D model generated by the photogrammetry processing unit 42 to the imaging device 51.

[0155] For example, the storage unit 43 stores captured image data uploaded from the imaging device 51 , and stores a 3D model generated by the photogrammetry processing unit 42 .

[0156] The photogrammetry processing unit 42 performs 3D modeling on the uploaded captured image data. For example, the photogrammetry processing unit 42 generates a second 3D model (second three-dimensional shape information) by performing processing such as SfM or MVS based on the captured image generated by the imaging unit 20.

[0157] In the SfM processing, for example, corresponding points are searched between the captured images, the position and orientation of the imaging device 51 are derived using epipolar constraints, the position of each corresponding point in three-dimensional space is determined by triangulation based on the position and orientation of the imaging device 51, and the entire determined three-dimensional point cloud is optimized by bundling adjustment.

[0158] In the processing of MVS, for example, dense corresponding points are further searched using the three-dimensional point cloud, three-dimensional points are added, and meshing and texturing are further performed as post-processing to generate a second 3D model.

[0159] The photogrammetry processing unit 42 encodes the 3D model generated in this manner, supplies the 3D model to the storage unit 43 , and stores the 3D model in a storage medium.

[0160] In addition, the photogrammetry processing unit 42 supplies the encoded data regarding the 3D model to the communication unit 41 to transmit the encoded data to the imaging device 51 .

[0161] Note that the photogrammetry processing unit 42 performs photogrammetry processing to generate a 3D model after obtaining a required number of captured images, but performs simple 3D modeling to generate a 3D model as a preview image while the imaging device 51 is performing an imaging operation.

[0162] For example, the server device 40 performs 3D modeling for preview each time a captured image is uploaded from the imaging device 51 to generate and update a 3D model. The 3D model at each time point is transmitted to the imaging device 51.

[0163] Note that, although the server device 40 includes the photogrammetry processing unit 42 , the server device 40 may include a neural rendering processing unit and generate the second 3D model through neural rendering.

[0164] exist Figure 6 In the imaging device 51 of FIG. 5 , the communication unit 29 receives the 3D model for preview during the imaging operation. The received 3D model for preview is provided to the scoring processing unit 25 and the display control unit 30.

[0165] The display control unit 30 controls the display of an imaging monitor image and a display including guidance regarding a user's imaging operation.

[0166] The superimposed image generation section 27 in the display control unit 30 generates an image to be superimposed and displayed on the imaging monitor image of the subject, that is, an image of, for example, the target indicator 1, the self-indicator 2, etc., which will be described later. In this case, the superimposed image generation section 27 also sets the mode (size, shape, and color) of the image to be superimposed on the imaging monitor image, while referring to the recommended imaging position set by the imaging control unit 24, information such as the posture information of the imaging device 51 and the imaging position, the shutter control timing of the imaging control unit 24, etc., and generates image data to be superimposed and displayed.

[0167] The display image generation unit 28 in the display control unit 30 generates display image data by combining the image generated by the superimposed image generation unit 27 with the imaging monitor image of the subject. For example, the display image generation unit 28 generates display image data for displaying the target indicator 1, which will be described later, corresponding to the recommended imaging position set by the imaging control unit 24. Furthermore, the display image generation unit 28 may include a 3D model from the server device 40 or thumbnail data regarding the 3D model in the display image data. Furthermore, the display image generation unit 28 may set the display mode of the superimposed image based on the scoring result from the scoring processing unit 25.

[0168] The output unit 26 includes, for example, a display device such as a display 54 and a display 55 and an audio output device. The display device of the output unit 26 displays an imaging monitor image of a subject and images for various guides under the control of the display control unit 30. Specifically, the display Figure 9 and the screens shown in the subsequent figures.

[0169] In addition, the audio output device in the output unit 26 outputs predetermined sounds according to the shutter control timing or the like of the imaging control unit 24 .

[0170] The imaging device 51, with this configuration, captures images of 3D objects at a more appropriate position and orientation, enabling the server device 40 to perform 3D modeling using the captured images. Specifically, the imaging device 51 outputs guidance displays and sounds, allowing the user to perform imaging operations at a more appropriate position and orientation. Therefore, using the imaging device 51 allows for the generation of more accurate 3D data while suppressing any increase in the burden of 3D modeling.

[0171] Note that, although in the above example, the server device 40 performs the second 3D modeling, a function equivalent to, for example, the photogrammetry processing unit 42 may be provided inside the imaging device 51 and the second 3D modeling may be performed instead.

[0172] Here, description will be made of an information processing apparatus that can be configured as the imaging apparatus 51 or the server apparatus 40. For example, a portable information processing apparatus such as a smartphone or a tablet terminal can be used as the imaging apparatus 51. The server apparatus 40 is also implemented as an information processing apparatus. Figure 8 The configuration of the information processing apparatus in these cases is shown.

[0173] Information processing device 70 is a device capable of performing information processing (particularly image processing), such as a computer. Specifically, a personal computer, a workstation, a portable terminal device (such as a smartphone or tablet), a video editing device, etc. are assumed to be information processing device 70. Alternatively, information processing device 70 may be a computer device configured as a computing device or server device in cloud computing.

[0174] Figure 8 A central processing unit (CPU) 71 of an information processing apparatus 70 shown in FIG. 1 executes various types of processing according to a program stored in a nonvolatile memory unit 74 such as a read-only memory (ROM) 72 or, for example, an electrically erasable programmable read-only memory (EEP-ROM) or a program loaded from a storage unit 79 into a random access memory (RAM) 73. The RAM 73 also appropriately stores data and the like required for the CPU 71 to execute various types of processing.

[0175] The image processing unit 85 is a processor configured to perform various types of image processing. For example, the image processing unit 85 is a processor capable of performing any one or more of the following processes: 3D modeling processing; rendering; database processing; image processing including color / brightness adjustment processing of captured images; encoding processing; decoding processing; image editing processing; image analysis / detection processing, etc.

[0176] The image processing unit 85 may be implemented as, for example, a CPU separate from the CPU 71 , a graphics processing unit (GPU), general purpose computing on a graphics processing unit (GPGPU), an artificial intelligence (AI) processor, or the like.

[0177] Note that the image processing unit 85 may be provided as a function within the CPU 71 .

[0178] The CPU 71, the ROM 72, the RAM 73, the nonvolatile memory unit 74, and the image processing unit 85 are connected to one another via a bus 83. Furthermore, to the bus 83, an input / output interface 75 is also connected.

[0179] An input unit 76 including a controller or an operating device is connected to the input / output interface 75. For example, various controllers and operating devices such as a keyboard, mouse, keys, trackball, dial, touch panel, touch pad, and remote controller are assumed as the input unit 76.

[0180] A user operation is detected by the input unit 76 , and a signal corresponding to the input operation is interpreted by the CPU 71 .

[0181] A microphone is also assumed as the input unit 76. Voice uttered by the user can also be input as operation information.

[0182] Furthermore, a display unit 77 including a liquid crystal display (LCD), an organic electroluminescence (EL) panel, or the like, and an audio output unit 78 including a speaker or the like are connected to the input / output interface 75 integrally or separately.

[0183] The display unit 77 performs various types of displays. The display unit 77 is realized as, for example, a display device provided in the housing of the information processing device 70, a separate display device connected to the information processing device 70, or the like.

[0184] The display unit 77 displays various images, operation menus, icons, messages, and the like, that is, a graphical user interface (GUI), on a display screen based on instructions from the CPU 71 .

[0185] A storage unit 79 including a hard disk drive (HDD), a solid state drive (SSD), or the like or a communication unit 80 may be connected to the input / output interface 75 .

[0186] The storage unit 79 can store various types of data and programs. A database can be configured in the storage unit 79.

[0187] The communication unit 80 performs communication processing via a transmission path such as the Internet, wired / wireless communication with various devices including an external DB, an editing device, and an information processing device, bus-based communication, and the like.

[0188] A drive 81 is also connected to the input / output interface 75 as needed, and a removable storage medium 82 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately inserted.

[0189] The drive 81 enables reading of image data, various computer programs, and the like from the removable storage medium 82. The read data is stored in the storage unit 79, and images and sounds included in the data are output from the display unit 77 and the audio output unit 78. In addition, the computer programs and the like read from the removable storage medium 82 are installed in the storage unit 79 as needed.

[0190] In the case where the information processing device 70 is used as the imaging device 51, the sensor unit 84 is included. The sensor unit 84 has a Figure 6 The depth sensor 31, imaging unit 32, IMU 33 and imaging unit 20 are described as required sensor configurations. For example, the sensor unit 84 includes a lidar sensor, multiple image sensors, an angular velocity sensor, an acceleration sensor, and the like.

[0191] In the information processing apparatus 70, for example, software for the processing in the present embodiment can be installed through network communication by the communication unit 80 or through the removable storage medium 82. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, or the like.

[0192] <3. User Interface Screen>

[0193] User interface screens capable of guiding a user through imaging operations for 3D modeling will be described below while taking specific examples.

[0194] As described above, achieving high-quality imaging in photogrammetry requires imaging while maintaining a clear view of the subject's imaging plane from a certain distance. Furthermore, surface unevenness requires varying the imaging interval based on the size of the common field of view. However, for users operating the imaging device, understanding this at the imaging site can be difficult. Therefore, this embodiment provides a user interface that guides the user, allowing them to smoothly perform imaging operations while intuitively maintaining an appropriate positional relationship with the subject.

[0195] Furthermore, in the imaging device 51 according to this embodiment, the user only needs to move the imaging device 51 around the 3D object without performing a shutter operation, and the shutter process, which is the image acquisition process performed by the imaging unit 20, is automatically executed under the control of the imaging control unit 24. In this case, if the user cannot recognize the imaging timing, the user may feel uncomfortable during the operation or feel anxious about whether he / she has successfully captured the image. Therefore, the shutter timing performed by the imaging unit 20 is also presented to the user on the user interface.

[0196] An example of a screen to be described below is, for example, Figure 5 The image displayed on the display 54 in the interface device 53 is displayed. The user performs an imaging operation of moving around the 3D object (which is the subject) while directing the imaging direction of the imaging device 51 toward the 3D object. At this time, the user can continue the operation while viewing the following screen displayed on the display 54 of the interface device 53.

[0197] First, refer to Figure 9 、 Figure 10 、 Figure 11 as well as Figure 12 The region of interest (ROI) setting screen 110 will be described.

[0198] ROI is a subject region to be processed, that is, a focus region. By setting the ROI in advance, the region to be processed can be specified in the subject of the imaging device 51.

[0199] There is no need to perform this ROI setting. By setting the ROI in advance, there is an advantage in that the total number of images to be captured, etc. can be predicted and presented to the user, but pre-scanning is required.

[0200] Description will be made taking as an example a case where 3D modeling is performed on a vehicle (truck). Figure 9 The imaging monitor image 150 is shown when the imaging device 51 sets a truck as a subject. For example, the image captured by the imaging unit 32 is displayed and output on the output unit 26 (display 54, etc.). Note that in the following description, the imaging monitor image 150 will be simply referred to as "monitor image 150."

[0201] The user captures a subject using the imaging device 51, views the monitor image 150, and walks around a truck (an image of which is to be captured) as a subject, as indicated by an arrow 111. Note that the arrow 111 is added for the sake of description and is not displayed as an image, but such a display for guiding the user's movement may be superimposed and displayed on the monitor image 150.

[0202] When the user performs a start operation for setting an ROI and moves, the imaging device 51 performs real-time 3D modeling while capturing an image of the subject roughly based on information from the depth sensor 31. At this point in time, it is sufficient that the user can perceive the size of the subject.

[0203] Figure 10 The state in which the area surrounding the periphery of the subject is set based on the result of the pre-scan is shown. For example, through processing by the display control unit 30, a cube 112 or the like that can surround the real-time 3D modeling range is displayed to present the ROI to the user.

[0204] The wall surface of the cube 112 can be moved by a user operation. Thus, the user adjusts the range to become the ROI. Figure 11 This is an example of minimization of the cube 112 surrounding the real-time 3D modeling range. For example, the range of the cube 112 is set as ROI.

[0205] After setting the ROI in this manner, the imaging control unit 24 uses the results of the real-time 3D modeling to set a recommended imaging position. Figure 12 As shown in FIG, the display control unit 30 superimposes and displays a target indicator 1 indicating a recommended imaging position on the monitor image 150. Therefore, the user can check the imaging position and the total number of images to be captured.

[0206] Note that, although the recommended imaging position is set in the pre-scan in the above description, instead, the recommended imaging position may be set simultaneously when actual imaging as the image obtaining process is performed.

[0207] Will refer to Figure 28 and Figure 34 Processing examples for a case where pre-scanning is performed and a case where pre-scanning is not performed are described.

[0208] Next, we will refer to Figures 13 to 24 Description will be given of the navigation screen 120. The navigation screen 120 is a screen displayed on the output unit 26 (display 54 or the like) by processing performed by the display control unit 30 when the imaging unit 20 performs imaging as an image obtaining process.

[0209] In this embodiment, by grasping the object shape, for example, Figure 13 The target indicator 1 in FIG. 1 is arranged as an indicator whose size changes in a manner following the surface of the object, and reaches a predetermined size at a position at a certain distance from the surface of the object, at which position imaging can be performed with desired quality.

[0210] Furthermore, by arranging the indicators that are candidates for the next imaging position around one indicator and similarly changing their sizes, the user can smoothly move the imaging device 51 to the next imaging position. Furthermore, by rendering the indicators in three dimensions, the shutter process is executed only when the imaging device 51 is facing the subject, thereby enabling stable and appropriate imaging.

[0211] In addition, by arranging the indicators according to the shape of the subject surface, an appropriate imaging interval is presented to the user, and the position where the next shutter process is to be performed is clearly displayed.

[0212] Figure 13 An example of the navigation screen 120 is shown. In the navigation screen 120, a monitor image 150 of a subject is displayed, and various images are superimposed and displayed on the monitor image 150. The monitor image 150 is an image captured by the imaging unit 32.

[0213] In the navigation screen 120 , a target pointer 1 and a self pointer 2 are displayed for guiding the user to an imaging position. Here, an example is shown in which the target pointer 1 and the self pointer 2 each have a hexagonal shape.

[0214] The target indicator 1 and the self-indicator 2 are displayed in different forms (i.e., different colors and brightness levels of the frames or different types of frame lines) so that the user can distinguish between the target indicator 1 and the self-indicator 2. For example, the target indicator 1 is white and the self-indicator 2 is yellow.

[0215] The target indicator 1 is a guide mark indicating a recommended imaging position.

[0216] The self-pointer 2 is a guide mark indicating the imaging position of the imaging device 51. In this case, the self-pointer 2 indicates the center of the imaging field of view of the imaging unit 20. The displayed monitor image 150 is an image captured by the imaging unit 32, and the field of view of the imaging unit 32 is wider than the field of view of the imaging unit 20. The field of view of the imaging unit 20 performing imaging as the image acquisition process is indicated by the field of view presentation frame 6 on the monitor image 150. The self-pointer 2 is displayed substantially at the center of the field of view presentation frame 6.

[0217] Note that, in the case where the imaging device 51 is an image pickup device using an interchangeable lens as the lens barrel 58 , the field of view indicated by the field of view presentation frame 6 varies depending on the model of the interchangeable lens.

[0218] Since the imaging directions of the two imaging units 20 and 32 mounted on the imaging device 51 change in the same manner, when the user moves and the range of the subject displayed as the monitor image 150 changes, the range captured by the imaging unit 20 also changes in a similar manner. Therefore, normally, the user visually recognizes that the positions of the field of view presentation frame 6 and the self-pointer 2 are fixed at the center on the screen of the display 54.

[0219] Based on the set recommended imaging position, the target indicator 1 is displayed at the corresponding position. If the recommended imaging position is set by pre-scanning, the target indicator 1 is displayed at that position. If the pre-scan is not performed, real-time 3D modeling is performed simultaneously and continuously within the range measured by the depth sensor 31, the recommended imaging positions are set sequentially, and the target indicator 1 is displayed at the corresponding position.

[0220] exist Figure 14 Target indicator 1 is shown in an enlarged manner in FIG.

[0221] For example, each target indicator 1 includes a navigation frame 3 and a direction presentation part 4. Here, the navigation frame 3 has a hexagonal shape, but this is an example.

[0222] The direction presentation section 4 has a needle shape at the center of the navigation frame 3, and indicates a direct facing direction of the imaging device 51 and the subject. The direction of the needle indicates the facing direction.

[0223] Figure 14 The right side of shows a state where the tip of the needle is displayed as a dot, that is, the state indicates that the imaging device 51 and the recommended imaging position of the subject are facing each other. Figure 14 In the left side of , the needle indicates the facing direction. That is, it indicates that the imaging device 51 is not currently facing the recommended imaging position.

[0224] The user recognizes that the facing direction is not established through the needle-shaped direction presenting portion 4 , and realizes the facing state by moving the imaging device 51 so that the user sees only the tip of the needle.

[0225] Note that the above is an example of the mode of the direction presentation section 4 , and other modes may be used instead as long as the user can be guided to the facing direction.

[0226] like Figure 13 As shown in , the target indicator 1 is displayed for the corresponding recommended imaging position, but its size is not constant. The size of each target indicator 1 varies depending on the distance to the imaging device 51 and the like.

[0227] That is, the target indicator 1 changes its size so as to achieve a predetermined size at a position at a certain distance from the object surface where imaging can be performed with desired quality. The target indicator 1 is then displayed so as to follow the object surface.

[0228] Assume that the depth distance (depth) is a distance for changing the size. In addition, the size of the target pointer 1 can be changed according to the horizontal distance from the current self pointer 2.

[0229] Note that by making the size change greater than the actual distance change, users can easily understand the actual distance relationship. For example, if the imaging position changes from the first position to the second position and the distance to the optimal imaging position becomes halved, the size of the target indicator 1 is increased by more than twice. Conversely, if the distance to the optimal imaging position doubles, the size of the target indicator 1 is decreased by less than halved. By making the size change greater than the distance ratio in this way, user recognition is improved.

[0230] For example, when the imaging device 51 reaches the optimal imaging position, the size of the target indicator 1 is set to be substantially equal to the size of the self-indicator 2. The optimal imaging position is relative to the corresponding recommended imaging position and refers to the depth position (depth distance) and the positions in the horizontal and vertical directions for each recommended imaging position, that is, the position in three-dimensional space.

[0231] When the imaging device 51 is too far from the portion of the subject corresponding to a certain target indicator 1 and is not at the optimal imaging position, the size of the target indicator 1 is smaller than the size of the self-indicator 2. When the imaging device 51 is closer than the optimal imaging position, for example, it may be considered to increase the size of the target indicator 1 or to divide the target indicator 1.

[0232] exist Figure 13 , the target indicator 1 is relatively small. This is a case where the imaging device 51 is positioned farther from the subject than the optimal distance.

[0233] In the navigation screen 120, a start button 7 is displayed in a state before the imaging operation starts. When the user operates the start button 7, imaging (image acquisition processing) for 3D modeling starts.

[0234] A simple preview window 10 is displayed on the navigation screen 120. The simple preview window 10 displays the number of images 11, a reduced preview 12, an operation presentation section 13, and the like.

[0235] For example, the number of images captured in the image acquisition process, the number of images uploaded to the server device 40, etc. are displayed as the number of images 11. Examples of display content and display mode will be described later. Figure 13In the example, the number of images 11 is displayed as the number of uploaded images, but since imaging has not yet started, the number of images 11 is "0".

[0236] In the zoomed out preview 12 (see Figure 16 In the preview, a thumbnail image of the 3D model generated in the server device 40 at that time is displayed. Figure 13 At the time of , the reduced preview 12 is not displayed because the upload is not performed and the 3D model for preview is not generated.

[0237] The operation presentation section 13 performs a display in which, for example, several points are moved. This display indicates that the imaging device 51 is performing imaging and uploading processing as image acquisition processing, 3D modeling in the server device 40, and the like.

[0238] The imaging process starts when the start button 7 is operated. The user can move around the subject while directing the imaging device 51 toward the subject.

[0239] Figure 15 This is a screen image where the user moves the imaging device 51 in the depth direction or horizontally and vertically to align the self-pointer 2 with a target indicator 1T. The size of the target indicator 1T increases as it approaches the optimal imaging position relative to the recommended imaging position indicated by the target indicator 1T. Furthermore, the sizes of other surrounding target indicators 1 also increase according to the distance.

[0240] In addition, in the drawing, the direction presentation portion 4 of the target pointer 1T becomes almost a dot, indicating that the imaging device 51 is substantially facing the recommended imaging position of the target pointer 1T.

[0241] like Figure 15 As shown in FIG, the distances between the target indicator 1T and the surrounding target indicators 1 are set to distances ds1, ds2, ds3, and ds4. These distances are set so that the overlap ratio of the captured images to be obtained at each recommended imaging position is optimized. Furthermore, the distances between the recommended imaging positions are not constant but increase or decrease depending on the shape of the subject, etc. For example, the distances are short for parts with complex shapes, while they are long for parts with flat shapes. In other words, the density of the recommended imaging positions is made different for each part of the subject.

[0242] Note that, in the navigation screen 120, a stop button 8 is displayed after imaging starts instead of the start button 7. When the stop button 8 is operated, imaging as an image obtaining process stops.

[0243] exist Figure 15After the state is reached, the user adjusts the imaging position so that the size of the target indicator 1T becomes equal to the size of the self-pointer 2 and the self-pointer 2 overlaps the target indicator 1T. Although described in detail later, it is sufficient that the self-pointer 2 and the target indicator 1T roughly overlap rather than completely match.

[0244] The imaging control unit 24 can determine whether the imaging device 51 has reached the optimal imaging position based on the relationship between the position and posture information of the imaging device 51 and the set recommended imaging position. If it is determined that the imaging device 51 has reached the optimal imaging position, the imaging control unit 24 causes the imaging unit 20 to perform shutter processing. The captured image then undergoes predetermined processing and is uploaded to the server device 40 from the communication unit 29.

[0245] At the timing of shutter control, such as Figure 16 As shown in , the display control unit 30 performs display control for notifying the shutter process. For example, the display control unit 30 displays an imaging success animation 5A in which the navigation frame 3 of the target indicator 1T is expanded in several layers. Figure 17 An example of an imaging success animation 5A is shown. This is an example of an animation in which several hexagonal frames are generated around the original navigation frame 3, and each of the hexagonal frames is enlarged and disappears when reaching a certain size.

[0246] Note that at this time, the imaging control unit 24 performs control for causing the output unit 26 to output a sound (electronic sound or the like) indicating successful imaging as well as the imaging success animation 5A.

[0247] The user can recognize that the shutter process has been performed based on the imaging success animation 5A and the electronic sound, and can switch the target to another target indicator 1 .

[0248] In addition, after the imaging success animation 5A is displayed as described above, the display mode of the target indicator 1T is changed. For example, the white target indicator 1 is changed to gray. Figure 17 , the shading indicates that the color of the navigation frame 3 has changed after the imaging success animation 5A ends.

[0249] In addition to the color, the display mode may be changed in other ways such as reducing the brightness of the navigation frame 3 or making the navigation frame 3 a dotted line.

[0250] Through such a change in the display mode, the user can grasp whether imaging has been performed or has not been performed for each target indicator 1 .

[0251] In addition, after successful imaging, the imaging range becomes the colored area 9 in the monitor image 150. Figure 16Although shown as a stippled portion in FIG, green, for example, is superimposed in the monitor image 150 to form a semi-transparent green colored area 9. The user can recognize that at least one or more captured images have been obtained in the greenish area in the monitor image 150.

[0252] Figure 18 The figure shows a state where the user, after completing a single imaging operation, moves the imaging position toward another target indicator 1N. The self-pointer 2 approaches the target indicator 1N on the display. However, since the self-pointer 2 is essentially fixed at the center of the screen, the range of the subject displayed on the monitor actually moves due to the movement of the imaging position of the imaging device 51, and the target indicator 1 moves along with the subject. Regarding the user's perception, moving the imaging device 51 in their hand creates the sensation of moving the self-pointer 2, and the user perceives the self-pointer 2 as sequentially approaching a large number of target indicators 1 on the subject.

[0253] In addition, Figure 18 In the picture, the colored area 9 and the field of view presentation frame 6 where imaging has been performed are shifted due to the movement of the imaging position.

[0254] In addition, as described above, the color or the like of the target pointer 1T for which imaging has been completed changes (indicated by shading in the drawing).

[0255] Note that as a result of imaging at the point of the target indicator 1T, the captured image is uploaded, and the server device 40 performs 3D modeling for preview. Figure 18 At the time of , in the simple preview window 10 , the number of images 11 is displayed as “1”, and a reduced preview 12 of the incomplete 3D model at that time is displayed.

[0256] Figure 19 The self-indicator 2 and the target indicator 1N are closer to each other. Figure 16 、 Figure 18 and Figure 19 It can be seen that the size of the navigation frame 3 and the state of the direction presentation part 4 of each target indicator 1 change in association with the imaging position at each time point.

[0257] Thereafter, when the user adjusts the imaging position so that the self-pointer 2 overlaps the target pointer 1N, the imaging unit 20 performs shutter processing under the control of the imaging control unit 24 at that point in time.

[0258] However, it is assumed that the quality of the captured image obtained at this time is low. For example, it is assumed that defocusing, camera shake, exposure failure, etc. are determined to have occurred. In this case, the image is displayed. Figure 20 Imaging inappropriate in Animation 5B. Figure 17 An example of an imaging inappropriate animation 5B is shown, which is an image with less flickering than the imaging successful animation 5 A. For example, the example is a box that disappears immediately when an attempt is made to expand it. Figure 20 A state in which such an imaging-inappropriate animation 5B is displayed for the target indicator 1N is shown.

[0259] In addition, in order to indicate poor imaging quality, the color of the navigation frame 3 of the target indicator 1N may be changed. The figure shows a state where the color of the navigation frame 3 has been changed. The color in this case is different from the color of the target indicator 1T (for which imaging was successful).

[0260] In addition, at this time, the imaging control unit 24 performs control for causing the output unit 26 to output a sound (electronic sound or the like) indicating inappropriate imaging and the imaging inappropriate animation 5B.

[0261] Due to the imaging-inappropriate animation 5B and the electronic sound, the user can recognize that the captured image is not obtained with appropriate quality and re-imaging should be performed.

[0262] exist Figure 16 、 Figure 18 、 Figure 19 and Figure 20 At each time point in , the user moves from right to left in a manner following each target indicator 1. For example, when the user recognizes that imaging has failed based on the imaging inappropriate animation 5B or electronic sound, as in Figure 21 As shown in FIG, it is generally assumed that the self-pointer 2 has passed a position substantially identical to the target pointer 1N in the monitor image and is moving toward the next target pointer 1S. Here, if the user recognizes that imaging has failed and considers re-imaging, the user can return the imaging position in the direction of the arrow in the figure, that is, return the self-pointer 2 to the target pointer 1N as the user feels.

[0263] Figure 22 The case where the imaging control unit 24 has caused the imaging unit 20 to perform imaging as a result of the user again matching the self pointer 2 with the target pointer 1N is shown. In this case, it is assumed that the quality of the captured image is sufficient and an example of the imaging success animation 5A is displayed.

[0264] As a result of successful imaging, the captured image area at this time is set as a colored area 9, i.e., a semi-transparent green area, for example. The portion that overlaps with the previously captured image of the target indicator 1T becomes a relatively dark green due to the overlapping green. In the drawings, this portion is shown as having a high point density for illustrative purposes.

[0265] Therefore, the imaged area on the subject is displayed as if light green thin layers overlap one another sequentially and the green becomes darker in the area where more images are taken in an overlapping manner. The dark green area can be identified as an area in which a large amount of information for 3D modeling has been obtained.

[0266] As described above, the user moves the imaging position of the imaging device 51 so that the self-pointer 2 follows each target pointer 1. Thereafter, by substantially matching the self-pointer 2 with each target pointer 1, a captured image at the corresponding recommended imaging position is obtained and uploaded to the server device 40, and 3D modeling is performed.

[0267] The user can follow each target indicator 1 like a game in which the user paints a subject with green and advance the imaging operation.

[0268] Figure 23 The display state is shown at a time when the user has moved the imaging position to near the driver's seat of the truck. The displayed shapes of the navigation frame 3 and direction presentation section 4 of the target indicator 1, corresponding to the recommended imaging position on the front side of the truck, change according to the position and orientation of the imaging device 51. Specifically, because the facing direction relative to the front side of the truck deviates significantly from the current orientation of the imaging device 51, the navigation frame 3 adopts a shape viewed from an oblique direction in accordance with this deviation, and the needle-shaped portion of the direction presentation section 4 also elongates in the lateral direction.

[0269] Figure 24 The display state at the time when the user has moved the imaging position to a position diagonally in front of the driver's seat of the truck is shown. Figure 23 By comparison, it can be seen that the shapes of the navigation frame 3 and the direction presentation part 4 of the target indicator 1 are changed due to the change of the imaging position.

[0270] That is, each navigation frame 3 looks like a regular hexagon when facing the corresponding recommended imaging position. However, when the imaging direction of the imaging device 51 is not facing the recommended imaging position, the navigation frame 3 is drawn in three dimensions on the screen plane so that the regular hexagon is viewed at an angle (i.e., in a non-regular hexagonal state). Similarly, each direction presentation portion 4 is drawn in three dimensions so that the appearance changes between the facing state and the non-facing state.

[0271] Note that since the navigation frame 3 is displayed in three dimensions, the target indicator 1 does not become a regular hexagon unless the facing state is established. This means not only that the regular hexagon indicates the facing direction, but also that the self-indicator 2, which is another regular hexagon, does not match the target indicator 1 unless it is in the facing state. In other words, there is also the effect of preventing the imaging unit 20 from performing imaging unless the facing state is established.

[0272] exist Figure 23 and Figure 24 images, such as by comparing these images with Figure 21 As can be seen by comparing the images of the truck and the driver's seat, the density of target indicators 1 is high. The shape near the driver's seat is more complex than that of the truck's cargo bed. Therefore, many recommended imaging positions are set. In areas with such complex shapes, densely arranging recommended imaging positions improves 3D modeling accuracy.

[0273] On the other hand, in a flat portion or the like, 3D modeling can be accurately performed with a small number of captured images, and by sparsely arranging recommended imaging positions on the cargo bed side or the like, the number of images to be captured can be reduced.

[0274] Regarding the simple preview window 10, for example, as shown in Figure 21 、 Figure 23 and Figure 24 As shown in , the display content changes according to the progress of 3D modeling and image uploading in the server device 40. Thus, the user can grasp the progress in a simplified manner.

[0275] As described above, in the navigation screen 120, the target indicator 1 is displayed as a guide based on the shape of the subject and the imaging position (self-position) of the imaging device 51, and the shape of the target indicator 1 changes in a manner to encourage the user to approach a point where imaging can be performed with the desired quality.

[0276] At the same time, a surrounding target indicator 1 is also displayed to indicate the point for which imaging should be performed next.

[0277] These target indicators 1 change their appearance depending on the direction and distance from the object facing the object, so that the positional relationship between the imaging device 51 and the object can be easily understood.

[0278] In addition, with respect to the target pointer 1 to be matched with the self-pointer 2 at the center of the screen, the surrounding target pointers 1 change overlap and direction based on the shape of the subject so that the shape can be recognized more accurately.

[0279] In addition, in the navigation screen 120 , the manner in which each target indicator 1 is displayed is changed so that the user can recognize the timing of the shutter process.

[0280] In addition, at the timing of the shutter process or after the timing of the shutter process, the display mode of each target indicator 1 is changed so that the user can recognize whether imaging has been performed for the target indicator 1 .

[0281] In addition, although each target indicator 1 has a three-dimensional appearance, by changing not only the appearance but also the size in an exaggerated manner, it is easy for the user to understand that he / she is approaching the correct imaging position.

[0282] Next, we will refer to Figure 25 、 Figure 26 and Figure 27 The preview screen 130 will be described.

[0283] For example, when the simple preview window 10 in the navigation screen 120 is operated, the navigation screen 120 switches to Figure 25 The preview screen 130 shown in FIG.

[0284] On the preview screen 130 , a 3D model 131 for preview generated in the server device 40 at the display time is displayed.

[0285] The 3D model 131 displayed for preview can be rotated up, down, left, and right to change the viewpoint by, for example, a drag operation, etc. Through the 3D model 131, the user can determine how much the imaging operation has progressed so far, in which part the image is insufficient, etc. during operation.

[0286] In addition, on the preview screen 130 , a score switch 132 , a frustum-shaped switch 133 , a close button 134 , icons for various other operations, and the like are displayed.

[0287] When the user turns on the frustum switch 133, the frustum 18 is displayed for the 3D model 131, as shown in FIG. Figure 26 Each frustum 18 indicates a portion where imaging as an image acquisition process has been performed.

[0288] Note that a recommended imaging position for which an image has not been captured may also be indicated by the frustum 18. In this case, the display mode of the frustum 18 indicating that imaging has been performed is made different from the display mode of the frustum 18 indicating that imaging has not been performed, so that the user can determine whether imaging has been performed. For example, different colors or brightness levels or different shapes of frames may be used.

[0289] When the user turns on the score switch 132, as shown in FIG. Figure 27 As shown in , a state in which the surface of the 3D model 131 is color-coded by the score of each of the parts is displayed.

[0290] In the drawings, the stippled portion is, for example, an area colored green, and the shaded portion is, for example, an area colored red. It is assumed that a darker green area has a higher score in terms of image quality and information volume, and a darker red area has a lower score.

[0291] In the green portion, for example, similar to the colored area 9 in the navigation screen 120, as the amount of information in the imaging so far increases, the green becomes darker. Portions where imaging has not yet been performed and portions having complex shapes for which a sufficient number of images have not been captured or a sufficient amount of information has not been obtained are colored with red having different densities depending on the degree of deficiency.

[0292] Note that when both the frustum switch 133 and the fraction switch 132 are on, for example, the frustum 18 is displayed on Figure 27 On the image, such as Figure 26 Like in.

[0293] When the close button 134 is operated, the screen returns to the navigation screen 120 .

[0294] As described above, the display in the preview screen 130 enables the user to check the progress of the operation in the middle of the imaging operation. In addition, it is possible to learn the part where imaging has been performed sufficiently or the part where imaging is insufficient, and to appropriately determine the position where imaging is to be performed thereafter.

[0295] Note that a function of displaying a thumbnail image list may be provided in the simple preview window 10 or the preview screen 130 for images captured so far.

[0296] <4. Processing Example>

[0297] The following describes a processing example of the imaging device 51 for displaying the navigation screen 120 and the preview screen 130 described above. The following processing example includes a process of real-time 3D modeling (first 3D modeling) and a control process performed by the imaging control unit 24 and the display control unit 30 in an actual imaging operation.

[0298] Figure 28 This example illustrates processing without pre-scanning. Specifically, while the imaging unit 20 is actually performing imaging as part of the image acquisition process, the imaging device 51 performs real-time 3D modeling based on detection information from the depth sensor 31 and other sensors. Based on this real-time 3D modeling, recommended imaging positions are sequentially set near the current imaging position, and target indicator 1 is displayed. Then, when target indicator 1 and self-indicator 2 substantially match, the imaging unit 20 automatically performs shutter processing, and the captured image data is uploaded. Figure 28 An example of a process of displaying the navigation screen 120 and the preview screen 130 in the case where such real-time scanning is performed is shown.

[0299] In step S101, the imaging device 51 sets the imaging clarity based on user input. This allows the user to select the clarity level at which the 3D model is to be generated, and the user sets the clarity level according to their purpose. If a highly accurate 3D model is required, the number of images to be captured increases, and the imaging workload increases. On the other hand, if the imaging workload decreases, improving the accuracy of the 3D model becomes difficult. Therefore, by allowing the user to pre-set the clarity level, the imaging device 51 can set the number of recommended imaging positions appropriate for that clarity level.

[0300] In step S102, the imaging device 51 calculates the optimal imaging distance. The range of the subject covered by a single captured image is determined based on the required level of clarity, and the imaging position relative to the subject in the depth direction is set based on this range. Therefore, the depth for imaging the appropriate range is set as the optimal imaging distance. In other words, the depth serving as the optimal imaging distance is set so that the range of the subject can be imaged according to the clarity level set for the imaging field of view of the imaging unit 20.

[0301] After the user operates the start button 7 and starts the actual imaging operation, the processing in and after step S103 is executed. That is, real-time 3D modeling starts.

[0302] In step S103 , the imaging device 51 generates a mesh using the result of the detection performed by the depth sensor 31 .

[0303] In step S104, the imaging device 51 calculates the facing vector in the detection area, that is, calculates the facing direction for each area in the grid.

[0304] In step S105 , the imaging device 51 sets the center reference point and direction of the target pointer 1 .

[0305] In step S106 , the imaging device 51 sets the display points of the surrounding target pointer 1 so that the overlapping ratio becomes constant.

[0306] In step S107 , the imaging device 51 sets or updates the drawn size and shape (direction) of the target pointer 1 based on the result of the self-positioning.

[0307] Will refer to Figures 29 to 32 A description will be given of a process for setting the display point of the target pointer 1 in the above description.

[0308] Figure 29The upper portion shows the grid generated in step S103, a target indicator 1, and a center reference point 1c. For example, the center reference point 1c is set based on the target indicator 1 near the current imaging position of the imaging device 51, the imaging position used for the previous image acquisition process, and the like.

[0309] Then, as in Figure 29 As shown in the lower part of , equidistant candidates Pc are calculated based on the central reference point 1c. The equidistant candidates Pc are candidates for recommended imaging positions around the central reference point 1c.

[0310] Next, as in Figure 30 As shown in the upper part of , the equidistant candidate Pc in the direction in which the user is moving is detected, and as shown in Figure 30 As shown in the middle of , the distance d and the normal vector v between the imaging device 51 and the polygon are calculated based on the grid.

[0311] Then, as in Figure 30 As shown in the lower part of , the overlap ratio at the position of the ideal normal vector v is calculated. This is the overlap ratio between the first imaging range F1 and the imaging range F2 at the position matching the normal vector v in the equidistant candidate Pc.

[0312] However, since the appropriate overlap ratio (e.g., 70%) is not necessarily achieved, as in Figure 31 As shown in the upper part of , the positions (intervals) of the equidistant candidates Pc are adjusted so that the overlap ratio becomes constant.

[0313] Then, as in Figure 31 As shown in the lower part of , re-detection is performed on each isometric candidate Pc, and the distance d (not shown) and the normal vector v between the imaging device 51 and the polygon are calculated.

[0314] Then, as in Figure 32 As shown in the upper part of , the size and direction are set according to the current position of the imaging device 51 , and rendering is performed so that each target indicator 1 is arranged around the position of the corresponding equidistant candidate Pc.

[0315] Furthermore, equidistant candidates Pc are similarly set for the surrounding target pointers 1 , but the size of the target pointer 1 decreases as the distance increases.

[0316] Through the above algorithm, the target indicator 1 is displayed within the field of view based on the current imaging position of the imaging device 51 .

[0317] In step S120, the imaging device 51 determines whether the self-position has moved to the range of one of the surrounding target indicators 1. For example, when a portion of the self-pointer 2 contacts a certain target indicator 1, the imaging device 51 determines that the self-pointer 2 has moved to the range of the new target indicator 1. Alternatively, when the center position of the self-pointer 2 becomes closer to one of the equidistant candidates Pc than the center reference point 1c at the time of drawing in step S107, it can be determined that the self-position has moved to the range of the corresponding surrounding target indicator 1.

[0318] If it is determined that the self position has moved into the range of one of the surrounding target indicators 1 , the imaging device 51 returns from step S120 to step S103 , and performs the processing up to step S107 .

[0319] That is, when the user moves the imaging device 51 and moves from the pointer 2 to the vicinity of the adjacent target pointer 1 displayed on the equidistant candidate Pc, the center position of the target pointer 1 (i.e., the position of the equidistant candidate Pc in the previous calculation) is set as the new center reference point 1c, and the processing from step S103 to step S107 is performed. Thus, the target pointer 1 is displayed within the field of view according to the current imaging position.

[0320] By repeating the drawing in step S107 according to the movement of the imaging position in this manner, even if pre-scanning is not performed, the recommended imaging positions within the current field of view range are sequentially set according to the movement, and the target indicator 1 is displayed.

[0321] When the imaging position is within the range of a certain target indicator 1, in step S110, the imaging device 51 determines whether the self-pointer 2 matches the target indicator 1. Here, "matching" does not only mean a complete match, but also can be established when a basic match (i.e., a rough match) is achieved. This is because it is difficult for the user to completely align the self-pointer 2 with the target indicator 1.

[0322] Therefore, “match determination” in the present disclosure is a process in which a substantially matched state can be regarded as determined to be a match.

[0323] For example, the imaging device 51 uses a reference Figure 33 Describes the algorithm to perform match determination.

[0324] As in Figure 33 As shown in the upper part of FIG, a hit determination area Ar of a sphere, for example, is virtually set at each of the vertices of the self-pointer 2. When the user attempts to align the self-pointer 2 with the target pointer 1, as shown in FIG. Figure 33 As shown in the middle of , the determination areas Ar each enter the vertex of the target indicator 1. Figure 33As shown in the lower part of , when all the hit determination areas Ar enter the apex of the target pointer 1 , it is determined that the self-pointer 2 matches (overlaps) the target pointer 1 .

[0325] By performing the determination in this manner, a “match” can be determined with a certain degree of tolerance, and usability for the user can be improved.

[0326] If it is determined that the indicator is in the matching state, the imaging device 51 proceeds from step S110 to step S111 and performs shutter processing of the imaging unit 20. That is, after imaging as an image acquisition process is performed in the image processing unit 22 and predetermined signal processing is performed, uploading from the communication unit 29 to the server device 40 is performed.

[0327] At this time, regarding display, in step S112, the imaging device 51 performs processing such as display of image quality based on the imaging result. That is, the imaging device 51 determines whether the quality of the captured image obtained by the shutter processing is good based on the focus state, camera shake state, exposure state, etc.

[0328] If it is determined that there is no quality problem, the imaging device 51 then displays the imaging success animation 5A and outputs an electronic sound indicating that the imaging is successful. In addition, after the imaging success animation 5A is displayed, a process is performed to change the display mode of the target indicator 1 to a display mode indicating that imaging has been completed. Then, the process returns to step S107.

[0329] If the imaging device 51 determines that there is a problem with the quality of the captured image, the imaging device 51 displays the imaging inappropriate animation 5B and outputs an electronic sound or the like for prompting re-imaging. The process then returns to step S107. Note that in this case, the display mode of the target indicator 1 may be the same as the display mode before imaging, but may alternatively be changed to a different display mode that indicates re-imaging, for example.

[0330] There are cases where the user performs an operation using the preview screen 130 (for example, touching the simple preview window 10). In this case, the imaging device 51 proceeds from step S150 to step S151, displays the preview screen 130, and in step S151, performs one of the various types of display processing described above according to the user operation on the preview screen 130. If the user performs an operation to end the preview screen 130 using the close button 134 or the like, the process proceeds from step S152 to step S153, returns to the navigation screen 120, and the process returns to step S107.

[0331] The user can be guided to the imaging position by performing scanning in real time and generating the target indicator 1 through the above process.

[0332] Figure 34 The following illustrates an example of processing performed when performing a pre-scan. That is, before the imaging unit 20 actually performs imaging as an image acquisition process, the imaging device 51 performs real-time 3D modeling based on detection information from the depth sensor 31 and the like, sets a recommended imaging position based on the real-time 3D modeling, and displays the target indicator 1.

[0333] Therefore, in step S101 , the imaging device 51 sets imaging clarity according to an input from the user, and performs processing from step S102 to step S106 based on the result of the pre-scan.

[0334] Since the processing of steps S102 to S106 is the same as Figure 28 The processing in is similar, so redundant description is avoided, but in Figure 34 In this case, the recommended imaging position (ie, the display position of the target indicator 1 ) may be set in advance in the entire area of ​​the subject based on the setting of the imaging clarity obtained by the pre-scan and the 3D model.

[0335] Therefore, the matching determination may be performed in step S110 while the drawing of the target indicator 1 is sequentially updated in step S107 .

[0336] That is to say, as Figure 28 As a result of the determination in step S120 in , the process returns to step S103, and there is no need to sequentially execute the processes of step S103 to step S106.

[0337] <5. Examples of various screen displays>

[0338] Various examples related to screen display such as the ROI setting screen 110 on the navigation screen 120 and the preview screen 130 will be described below.

[0339] The shape of the target indicator 1 is not limited to the hexagon described above, and may instead be a circle, a triangle, a quadrangle, or another shape. The self-indicator 2 has the same shape as the target indicator 1 .

[0340] In addition, the target indicator 1 may be displayed as, for example, a hexagonal pyramid, a hexagonal prism, or the like, so that the facing direction can be recognized.

[0341] In addition, the shape of the direction presenting portion 4 is not limited to a needle, and for example, Figure 35 As shown in , a display in which a line extends from the user's hand in a direction facing the subject can also be imagined.

[0342] Figures 36 to 38An example of a quadrilateral target pointer 1 and a self pointer 2 is shown.

[0343] Figure 36 The state in which the imaging device 51 is substantially facing the subject is shown. The target indicator 1 is drawn in a substantially square shape. Figure 37 1 and 2. The state in which the imaging device 51 obliquely observes the surface of the object is shown. The target pointer 1 has a shape that is observed obliquely.

[0344] The subject is modeled by the depth sensor 31 , the normal line of the surface is calculated, and the frame of the target pointer 1 is displayed in such a manner as to be oriented in the normal line direction to realize the above state.

[0345] In addition, Figure 36 In the example, since the distance of the imaging device 51 is relatively long, the target indicator 1 is small. On the other hand, in the example Figure 38 In the example, the distance is optimized so that the size of the target indicator 1 is close to the size of the self-indicator 2.

[0346] Even in the case of a quadrilateral as described above, by changing the size and orientation of the target indicator 1 according to the position and orientation of the imaging device 51, appropriate guidance can be provided to the user, i.e., guidance for overlapping the self-indicator 2 on the target indicator 1 in a facing state.

[0347] in addition, Figure 38 The distances ds1 , ds2 , ds3 , and ds4 between the target indicators 1 are shown, and in this case, the target indicators 1 are displayed by setting the recommended imaging positions in such a manner that an optimal overlap ratio is achieved.

[0348] Regarding the expression of the shutter process when the user overlaps the self-pointer 2 on one of the target indicators 1, various examples other than the imaging success animation 5A described above can be conceived. Since the navigation screen 120 is constantly updated, it is preferable to notify the user of the timing at which the shutter process for capturing a still image has been executed, and therefore, any notification mode can be used to express the shutter process.

[0349] For example, it is conceivable to temporarily white out or black out the screen at the shutter processing timing.

[0350] Alternatively, only vibration and sound may be presented to the user at the shutter processing timing.

[0351] Alternatively, the imaging range of the subject may be colored at the shutter processing timing.

[0352] In addition, Figure 38In the image, the colored area 9 is indicated by stippling, and the density of the stippling indicates the color density. For example, the progress of imaging can be identified by overlaying the subject range imaged at each shutter timing in green. The darker the color, the more information is obtained, allowing the user to continue the operation by painting the subject darker.

[0353] exist Figure 36 、 Figure 37 and Figure 38 In the example of , for example, a frame 121 in red or the like is displayed around the navigation screen 120. For example, the frame 121 in which imaging as the image obtaining process has started is displayed using the start button 7 to indicate that imaging is being executed.

[0354] Figure 39 、 Figure 40 and Figure 41 is an example in which the target indicator 1 and the self-indicator 2 are circular. Figure 39 As shown in FIG, the user places the entire subject whose image is to be captured in the self-pointer 2 and operates the start button 7 to start imaging. Thus, the imaging device 51 specifies the target 3D object and generates the target pointer 1 according to the object, as shown in FIG. Figure 40 As shown in Figure 41 As shown in , the user adjusts the imaging position so that the self-pointer 2 substantially matches a certain target pointer 1 to automatically perform the shutter process.

[0355] Note that in Figure 39 、 Figure 40 and Figure 41 In the navigation screen 120 , a navigation button 126 , a score button 127 , and a thumbnail button 128 are provided.

[0356] The navigation button 126 is a button for setting a mode for displaying the target indicator 1 and the self indicator 2 as shown in the drawing. The score button 127 is a button for setting a mode for presenting image quality with, for example, a green overlay. The thumbnail button 128 is a button for displaying a thumbnail of a 3D model for preview.

[0357] As an example of dynamic display of the target indicator 1 , there is an example in which the target indicator 1 is divided and merged.

[0358] For example, if Figure 42 In the state of the left figure, if the imaging position is close to the subject, the target indicator 1 is divided as shown in the right figure. Conversely, if the imaging position moves away from the subject, the target indicator 1 is merged in the state of the right figure to establish the state of the left figure.

[0359] After the segmentation and merging, information indicating whether imaging has been performed on each target indicator 1 is taken over. For example, a plurality of target indicators 1 obtained by segmenting the target indicator 1 for which imaging has been performed and the color of the target indicator 1 has been changed are displayed in a color indicating that imaging has been performed.

[0360] In addition, in the case where imaging has been performed on a subset of the segmented target indicators 1 and then the segmented target indicators 1 are merged, the imaging state may be indicated in a color of density corresponding to the ratio of the target indicators 1 on which imaging has been performed.

[0361] In addition, the target indicators 1 do not necessarily need to be arranged at equal intervals.

[0362] By changing the arrangement interval based on the shape obtained through real-time 3D modeling, more detailed portions can be imaged finely, and flat portions can be imaged efficiently.

[0363] Regarding updating the displayed target indicator 1, the user can explicitly press an update button to perform the update. When the update button is pressed, the imaging device 51 recalculates the recommended imaging position at which the target indicator 1 is to be displayed based on the changed setting value. For the portion for which imaging has already been performed, the information is taken over at intervals of the new target indicator 1.

[0364] The navigation screen 120 may provide guidance on how the user should move the imaging device 51. The user carries the imaging device 51 and moves around the 3D object, and if the user's movement is too fast or too slow, it will be inappropriate. Therefore, for example, a user interface (UI) can be conceived to limit the speed of moving the imaging position and a UI can be conceived to notify that imaging cannot be performed properly if the speed is too high.

[0365] For example, Figure 43 As shown in , a warning indicating that the movement is too fast or too slow may be displayed as a character string 141 .

[0366] As in Figure 44 As shown in , a speed bar or speedometer may be displayed to inform the user whether his / her movements are appropriate.

[0367] In addition, blurring may be applied to the emphasized image according to the speed of movement to notify that imaging was not successfully performed.

[0368] In addition, the screen may be whitened, blackened, or faded out according to the speed of movement or at a certain threshold value. Thus, the user may be informed that imaging cannot be performed because the user's movement is too fast, and the user may be urged to move slowly.

[0369] In addition, if the movement is too fast or too slow, the self-pointer 2 may be faded out to inform the user.

[0370] Alternatively, sound or vibration may be used to inform the user that his / her movement is too fast or too slow.

[0371] As a UI for guiding movement speed, such as Figure 45 As shown in FIG, the guide 123 first moves at a desired speed. The user moves the imaging device 51 so that the pointer 2 follows the guide 123. The direction in which the guide 123 is moving may be estimated based on the direction the user has moved so far, and the guide 123 may move in the estimated direction, or the guide 123 may move along a pre-calculated designated trajectory so that the user can follow.

[0372] Alternatively, the guide 123 follows the user's movement (from the pointer 2) at an appropriate speed. The user moves the imaging device 51 at a speed at which the guide 123 follows at a constant interval.

[0373] Furthermore, it is also conceivable that guidance is provided by sound or vibration with a rhythm corresponding to the speed of movement of the target pointer 1 .

[0374] In the navigation screen 120, when re-imaging is desired due to inappropriate imaging, a UI for guiding the user may be displayed. That is, this is a UI for notifying the user of the intention to return the imaging position to the target indicator 1 matched immediately before.

[0375] As in Figure 46 As shown in Figure A, when the self-pointer 2 is moved away from the target indicator 1 for which imaging failed, the target indicator 1 may appear to be pulled like rubber by the self-pointer 2. The user notices this and returns the self-pointer 2 for re-imaging. However, if the user ignores this and moves the self-pointer 2 to the range of the adjacent target indicator 1, the self-pointer 2 can return from the stretched state to its original state.

[0376] In addition, as in Figure 46 B and Figure 46 As shown in C, for example, triangles may be added to the sides of the hexagon of the self-pointer 2 to inform the user of the direction in which the self-pointer 2 is to return.

[0377] In addition, as in Figure 46 As shown in FIG. 2 , when the self-pointer 2 is moved from the target pointer 1 for which imaging has failed, the self-pointer 2 may be displayed in such a manner as to be adhered to the target pointer 1 and not separated from the target pointer 1 .

[0378] Note that, when the user ignores this situation and further moves the self-pointer 2 away, the self-pointer 2 may be separated from the target pointer 1, as in Figure 46 As shown in E.

[0379] In addition, the shape and size of the target indicator 1 for which re-imaging is to be performed may be changed as it is separated from the indicator 2 .

[0380] In addition, as the self-pointer 2 moves away from the target pointer 1 for which re-imaging is to be performed, the surrounding target pointers 1 may disappear, and the user may not be able to recognize the next target pointer 1. Thus, the self-pointer 2 returns to the target pointer 1 for which re-imaging is to be performed.

[0381] For example, as a UI for notifying that the quality of a captured image is not good from the perspective of focus, camera shake, exposure, etc., it is conceivable to indicate the corresponding portion on the subject. For example, an example in which the imaged portion is superimposed in green has been described, but in the case of low quality, red or the like can be superimposed.

[0382] Additionally, the pattern of the corresponding target indicator 1 may be set to a specific pattern (color or shape) to indicate that the quality of the captured image is insufficient.

[0383] In addition, different displays can be performed for each of the causes of quality deterioration, so that a portion where the focus state is not good, a portion where camera shake occurs, a portion where exposure is inappropriate, and the like can be distinguished.

[0384] Examples of information to be displayed in the navigation screen 120 include the following information.

[0385] -The number of images uploaded to the server device 40

[0386] - Number of images captured

[0387] - The number of images remaining to be captured

[0388] - Number of thumbnails displayed in the simple preview window 10, etc.

[0389] - Number of captured images processed for preview in 3D modeling

[0390] -Ratio of the number of images remaining to be captured

[0391] - Estimated remaining scan time

[0392] -Ratio of the number of images remaining to be captured

[0393] - The predicted total number of images to be captured

[0394] - The ratio of the number of images captured to the number of images planned to be captured

[0395] As described above, it is preferable to display information on the number of images that have been captured and the progress of 3D modeling for imaging or previewing.

[0396] Although Figure 13 etc. show an example in which the number of uploaded images is displayed as the number of images 11 in the simple preview window 10 , instead, for example, the number of captured images or the like may be displayed in the simple preview window 10 .

[0397] Figure 47 The upper portion of FIG shows an example in which the simple preview window 10 is not provided in the navigation screen 120. For example, a preview button 124 is provided, and by operating the preview button 124, the screen transitions to the preview screen 130. In the case of such a navigation screen 120, the number of images already captured is indicated as the number of images in the screen 11. Of course, instead of or in addition to the number of images already captured, the number of images already uploaded, the number of images remaining to be captured, the total number of images predicted to be captured, etc. may be displayed.

[0398] In addition, multiple numerical values ​​can be displayed as the number of images 11. For example, in Figures 35 to 38 In the example of , the number of processed images (Processed) and the total number of captured images (Total) are indicated as the number of images 11. Figures 39 to 41 In the example of , the number of thumbnails (Thumbnail) and the total number of captured images (Total) are indicated as 11, which is the number of images.

[0399] In the case of indicating a ratio (such as the total remaining scan time or the ratio of the number of images remaining to be captured or the ratio of the number of images already captured), its numerical value may be shown, but it may be as in Figure 47 1. The progress bar 125 is shown as shown in the lower part of . In addition, in this example, a text display is also added, and the remaining ratio and time are clearly indicated.

[0400] Note that in the display of estimated remaining scan time, the ratio of the number of images remaining to be captured, the total number of images predicted to be captured, the ratio of the number of images captured to the number of images planned to be captured, etc., the total number of images predicted to be captured and the total operation time are calculated, but these can be corrected with a certain degree of tolerance according to the operating conditions.

[0401] It is also assumed that the overlap ratio is information displayed in the navigation screen 120. That is, the overlap ratio is the degree of overlap between the current field of view and the field of view of the frame captured immediately before. By displaying the overlap ratio, the user can easily recognize the appropriate imaging position.

[0402] It is also conceivable that the tracking status of the estimated position of the imaging device 51 is displayed as information on the navigation screen 120. For example, it may be a tracking status, a lost status, an initial status, a SLAM status, etc. The imaging device 51 tracks its own position through self-positioning processing and presents this status to the user.

[0403] The moving speed (for example, m / sec) of the imaging device 51 may be displayed in the navigation screen 120. Figure 44 It is also useful to display the speed of movement together with a speed bar or speedometer as shown in FIG.

[0404] Since the preview reconstruction process in the server device 40 takes some time, it is preferable to be able to recognize movement in the background on the navigation screen 120 .

[0405] Therefore, it is useful to display the upload process to the server device 40 and the number of uploaded images. In addition, it is also conceivable to display the number of images for which the 3D model generation process for preview has been completed in the server device 40, the ratio of the number of images, etc.

[0406] The speed of uploading to the server device 40 (MB / s, images / minute, etc.) can be displayed. In particular, the communication speed is important during operation. The user can also be made aware of poor radio wave conditions.

[0407] In addition, in the case where multiple people perform imaging, the following display is also useful.

[0408] -Other people's progress display (%)

[0409] - Total number of images captured by all persons

[0410] -Total progress display for everyone (%)

[0411] Next, changes in the sharpness setting will be described.

[0412] exist Figure 28 and Figure 34 In step S101, the definition is set based on the user's operation. The user sets the definition, plans the required number of images to be captured or the required amount of imaging based on the definition, and then performs navigation suitable for the user's desired definition. This is an example of a user interface for this definition setting.

[0413] For example, Figure 48 is an example of displaying the slide bar 129. When the user sets the fineness using the slide bar 129, the imaging device 51 calculates an appropriate distance to the subject and reflects the calculated distance in the setting of the recommended imaging position.

[0414] The setting value based on the slide bar 129 or the like may be continuous or discontinuous.

[0415] When you select the distance to the subject you want to image, you can set the sharpness automatically.

[0416] When you select the maximum number of images to capture, the resolution can be set automatically.

[0417] When you select the imaging operation time, the clarity can be set automatically.

[0418] When you set a target number of vertices or polygons to achieve, you can set the sharpness automatically.

[0419] When setting the weight of the completed data (photo material and 3D model viewpoint), the resolution can be automatically set. In this case, the number of storage media (memory cards, etc.) required can be recommended.

[0420] You can also set the definition while viewing the preview of the quality corresponding to the number of images you set. Figure 48 The preview of the target indicator 1 is displayed according to the position of the slide bar 129 in the image. Alternatively, a preview of the completed 3D model can be displayed according to the position of the slide bar 129.

[0421] The set number of people may be displayed assuming an operation to be performed by a plurality of people or suggesting how many people are required.

[0422] Note that even if the user does not set the clarity, the number of recommended imaging positions (target indicators 1 displayed at the recommended imaging positions) can be adaptively changed. For example, when capturing an image at a closer position, the recommended imaging positions are arranged more densely, and when capturing an image at a farther position, the recommended imaging positions are arranged more sparsely.

[0423] Alternatively, when capturing images at a farther position, the number of images to be captured may increase.

[0424] The navigation screen 120 may be displayed so that the 3D model is completed from a black screen. That is, the portion where the 3D model is not formed is black, and the 3D model gradually appears according to the progress of the imaging operation.

[0425] In the case where multiple people perform imaging, the color and shape of the target indicator 1 in a portion where imaging has been completed may be changed depending on the person, so that who performed imaging in the portion can be identified.

[0426] Next, an example of the preview function on the preview screen 130 will be described.

[0427] The imaging device 51 uploads the captured image data to the server device 40, reconstructs the 3D model using the resources of the server device 40, and receives the 3D model for preview. By displaying the 3D model for preview or its thumbnail in the preview screen 130, the simple preview window 10, etc., the user can accurately determine how much imaging has been completed during the imaging operation.

[0428] In this case, for example, a portion with poor quality may be displayed in the preview screen 130. For example, a portion with a poor focus state, a portion where camera shake occurs, an exposure failure portion, and the like are clearly indicated. Figure 49 The example in which the defective portion guide 135 is displayed on the preview screen 130 is shown. For example, the imaging portion is indicated by the frustum 18, and the defective portion guide 135 is displayed so that a red frame surrounds the frustum 18 of the portion where low-quality imaging has been performed, and the defective portion guide 135 is presented to the user. This allows the user to determine whether re-imaging is necessary.

[0429] In addition, an option of individually excluding the imaging viewpoint at which the defective portion guide 135 is displayed and reconstructing the 3D model may be displayed.

[0430] In addition, it is desirable to collaboratively present the defective portion presented in the preview screen 130 as described above and, for example, the score display of the preview screen 130 (see FIG. 1 ). Figure 27 ) in areas with insufficient information.

[0431] For example, in Figure 50 In the present invention, based on the preview processing, for a portion where imaging is insufficient, the target indicator 1 is displayed in a mode (color, shape, etc.) different from that of other target indicators 1, and the user is prompted to perform re-imaging, that is, the user is prompted to move the imaging position to that portion.

[0432] As in Figure 51 As shown in the upper portion of FIG. 1 , a mark 136 may be displayed in three dimensions for a portion of the captured image having poor quality or insufficient information in the preview screen 130 .

[0433] In addition, as in Figure 51 As shown in the lower portion of FIG. 1 , a mark 137 is correspondingly displayed near the target indicator 1 at a position corresponding to the mark 136 in the navigation screen 120 .

[0434] However, in the preview screen 130, since the viewpoint direction can be arbitrarily changed by operation, even if the user recognizes the mark 136, when the user returns to the navigation screen 120, it may be difficult for the user to recognize the direction corresponding to the portion where the mark 136 is located from the current imaging position. There are also cases where the portion corresponding to the mark 136 is not visible from the current imaging position. Therefore, a travel guide 138 such as an arrow image is displayed to prompt the user to move the imaging position. Figure 51 In the lower example, the marker 137 is visible in the current field of view, but in the case where the marker 137 is not visible in the current field of view, displaying the travel guide 138, for example in the left or right corner of the navigation screen 120, makes it very easy to understand the direction to move.

[0435] Through the link preview screen 130 and the navigation screen 120 as described above, the user can continue the imaging operation while visually checking the portion on which imaging is not performed and the portion on which re-imaging is to be performed.

[0436] In the preview screen 130, a current position mark 139 indicating the current position of the imaging device 51 may be displayed together with the frustum 18 indicating the imaging position, as shown in FIG. Figure 52 Thus, the user can understand the relationship between the 3D model 131 displayed in the preview screen 130 and the current position.

[0437] In cases where multiple people perform imaging, such as in Figure 52 As shown in the lower part of , current position marks 139a, 139b, and 139c having different colors and shapes may be displayed so that the position of the imaging device 51 of the person who cooperates in performing imaging can be grasped.

[0438] Furthermore, an image like a figure of a user who is performing imaging may be displayed on the 3D model 131 for preview.

[0439] Next, an example of the ROI setting screen 110 will be described.

[0440] As described above, the ROI does not need to be set, but if it is set in advance, there is an advantage in that the total number of images to be captured and the like can be predicted and presented to the user.

[0441] As a UI for ROI setting, in addition to referring to Figures 9 to 12 In addition to the UI described, there are also Figure 53 114 to make it easy to set the range. The position of the belt 114 can be specified by the user while viewing the scale 115.

[0442] In addition, in reference Figures 9 to 12 At the time of the described pre-scan, it is also conceivable to display Figure 54 The rectangular or circular guide frame 116 shown in the upper or lower part of the image is displayed, and a message such as "Start imaging so that the content you want to capture is completely displayed in the frame" or "Start imaging from all directions around the subject" is displayed to the user.

[0443] In addition, there are parts where depth information cannot be obtained normally (such as the transparent part of the glass in the subject). The missing information can be supplemented with the surrounding depth information for such parts. Figure 55 The portion corresponding to the frame 117 in the image may be displayed, and the target indicator 1 may be displayed so that the user's movement is not restricted.

[0444] The correlation between the pre-scan and the target indicator 1 is as follows.

[0445] At the time of ROI setting, the user can capture a rough image at a certain distance from the subject so that the number of points for recommended imaging positions is small. Since information becomes more detailed after two or three rounds, it is preferable to perform a display corresponding to this.

[0446] The size and display method of the target pointer 1 can be changed between the ROI setting screen 110 and the navigation screen 120 .

[0447] <6. Conclusion and Modifications>

[0448] According to the embodiment described above, the following effects can be produced.

[0449] The imaging device 51 according to the embodiment includes a display control unit 30 configured to display a target indicator 1 (first indicator) indicating a recommended imaging position and a self-indicator 2 (second indicator) indicating a current imaging position in a superimposed manner on a monitor image 150 of a 3D object to be subjected to 3D modeling, and to perform a process of changing the display of the target indicator 1 according to an image acquisition process for acquiring image data used for 3D modeling. In addition, the imaging device 51 includes an imaging control unit 24 configured to perform an image acquisition process according to a result of a matching determination between the target indicator 1 and the self-indicator 2 on the screen (see Figure 6 ).

[0450] For example, the imaging device 51 superimposes the target indicator 1 on the imaging monitor image 150 in the navigation screen 120 and guides the user so that the position of the target indicator 1 (recommended imaging position) is set in the direction of the subject. The user adjusts the imaging position and direction so that the self-indicator 2 indicating its imaging position substantially matches the target indicator 1. Thereafter, the imaging device 51 automatically performs image acquisition processing based on the matching determination and changes the display mode of the target indicator 1 to indicate that an image has been captured (image acquisition processing).

[0451] Therefore, the user can cause the imaging device 51 to acquire a large amount of image data for 3D modeling by performing an operation of moving around the object while directing the imaging device 51 toward the object (image acquisition processing). In particular, since the user does not need to perform a shutter operation and only needs to move while carrying the imaging device 51 and aligning the frame of the self-pointer 2 with the frame of the target pointer 1, the operation is very easy.

[0452] In addition, for example, due to Figure 17 As shown in the imaging success animation 5A, the display of the target indicator 1 is temporarily changed according to the timing of the automatic image acquisition process, so the user can recognize that the image acquisition process has been automatically executed, and the user can be prevented from feeling uncomfortable or uneasy. In addition, the user can advance the imaging operation while feeling the progress of the operation through the imaging success animation 5A.

[0453] Note that the user may be notified of the execution of the image obtaining process by outputting a notification sound or vibration together with, for example, a display change of the imaging success animation 5A.

[0454] In addition, although the case of imaging a 3D object to be subjected to 3D modeling has been described as an example, the imaging timing is not limited to this, and the technology of the present disclosure can be applied to various imaging timings. For example, the target indicator 1 indicating the recommended imaging position can be displayed at the timing of imaging a certain subject not for the purpose of 3D modeling, and image acquisition processing can be performed as a result of matching determination with the self-indicator 2.

[0455] In the embodiment, the example has been described in which the display control unit 30 changes the display change mode of the target pointer 1 according to the image obtaining process based on the quality evaluation result of the image captured in the image obtaining process.

[0456] For example, when the quality of the image data obtained in the image obtaining process is good, the display control unit 30 performs the following operations: Figure 17 In the case where the quality is not good, a temporary display change like the imaging successful animation 5A is performed, and in the case where the quality is not good, a temporary display change like the imaging inappropriate animation 5B is performed.

[0457] Thus, the user can recognize the quality of the image data obtained and can return to the target indicator 1 of inadequate quality and perform re-imaging. This also improves the quality of the image data collected for 3D modeling and also improves the accuracy of the resulting 3D model.

[0458] In an embodiment, the target indicator 1 comprises a geometric figure.

[0459] The target indicator 1 includes, for example, a geometric figure such as a circle, triangle, square, rectangle, or polygon with five or more sides as a navigation frame 3. By matching the self-indicator 2, which is a similar geometric figure, with this figure, the user can easily understand how to perform imaging operations for 3D modeling. In addition, the user can perform operations as if they were playing a game similar to a figure matching game.

[0460] In the embodiment, description has been made on the example in which the target indicator 1 includes the hexagonal shape.

[0461] By providing the target pointer 1 with the hexagonal navigation frame 3 , the direction relative to the self-pointer 2 can be easily understood, and the user can easily perform an operation of matching the self-pointer 2 with the target pointer 1 .

[0462] In the embodiment, the target indicator 1 indicates a direction facing the subject.

[0463] For example, the target indicator 1 indicates the direction facing the subject via the direction display unit 4. Furthermore, for example, the hexagonal shape of the navigation frame 3 also becomes a regular hexagon in the facing direction, and the navigation frame 3 at a position not facing the subject becomes a non-regular hexagon based on the direction difference. This allows the user to easily identify the facing direction relative to each target indicator 1 and appropriately and efficiently proceed with the imaging operation.

[0464] In the embodiment, the example has been described in which the display control unit 30 performs processing for changing the mode of the first pointer 1 according to the positional relationship between the imaging position and the recommended imaging position.

[0465] For example, regarding the size of the target pointer 1 , the display control unit 30 displays the target pointer 1 such that, for example, the target pointer 1 close to the imaging device 51 is large and the target pointer 1 far from the imaging device 51 is small.

[0466] In addition, with respect to the size of the target indicator 1, the display control unit 30 sets the size to be the same as the size of the self-indicator 2 when the imaging device 51 is located at an optimal distance in the depth direction to the 3D object serving as the subject, sets the size to be smaller when the imaging device 51 is away from the 3D object, and sets the size to be larger when the imaging device 51 is too close to the 3D object.

[0467] By changing the mode of the target indicator 1 according to the positional relationship between the imaging position and the recommended imaging position in this manner, the user can be guided so that a positional relationship appropriate for the image acquisition process is achieved. The user can intuitively adjust the position of the imaging device 51 according to this guidance.

[0468] Note that examples of changing the mode of the target indicator 1 according to the positional relationship are not limited to size change, and the shape may be changed, the color or brightness may be changed, or the size, shape, color, and brightness may be changed in a complex manner.

[0469] In the embodiment, the example has been described in which the display control unit 30 performs processing for changing the mode of the first pointer 1 according to the relationship between the facing direction of the recommended imaging position and the imaging direction.

[0470] For example, the display control unit 30 sets the direction presentation unit 4 to a needle shape and the navigation frame 3 to a hexagon as a display for indicating that the target indicator 1 at the position where the imaging device 51 is facing the 3D object is facing the 3D object. When facing, the navigation frame 3 is a regular hexagon, and the tip of the needle of the direction presentation unit 4 at the center looks like a dot. The target indicator 1 in a non-facing position performs a display indicating a non-facing state, that is, for example, a display indicating a facing direction using the direction of the needle of the direction presentation unit 4, a display making the navigation frame 3 a non-regular hexagon, etc. By changing the mode of the target indicator 1 according to the relationship between the facing direction of the recommended imaging position and the imaging direction, the user can be guided to achieve an appropriate direction (imaging direction) relative to the subject. The user can intuitively adjust the direction of the imaging device 51 according to the guidance.

[0471] Note that examples of changing the mode of the target indicator 1 according to the relationship between the imaging direction and the facing direction are not limited to shape change, and the size may be changed, the color or brightness may be changed, or the size, shape, color and brightness may be changed in a complex manner.

[0472] In the embodiment, an example has been described in which the display control unit 30 performs processing for displaying the target indicator 1 in a mode different from the mode before performing the image acquisition processing at a position where the image acquisition processing is performed as a result of matching determination with the self-indicator 2.

[0473] For example, in Figure 17 After the image acquisition process is completed, such as by presenting a successful imaging animation in the display, the display control unit 30 displays the target indicator 1 in a different mode from its original mode. For example, the white color before imaging may be changed to gray. The size or shape may also be changed. This allows the user to identify whether each of the numerous target indicators 1 is a target indicator 1 before or after the image acquisition process. Thus, the user can effectively advance the imaging operation by moving toward a target indicator 1 for which the image acquisition process has not yet been performed.

[0474] In the embodiment, the example has been described in which the display control unit 30 performs processing for simultaneously displaying a plurality of target indicators 1 corresponding to a plurality of recommended imaging positions in the monitor image 150 .

[0475] The display control unit 30 displays a target pointer 1 at each of the positions where it is recommended to perform still image capturing. By recognizing a plurality of target pointers 1, the user can perform an imaging operation of sequentially navigating the plurality of target pointers 1.

[0476] Note that only one target indicator 1 to which the user is to move may be displayed. For example, when the user performs imaging using the imaging unit 20 by matching the self-indicator 2 with the displayed target indicator 1, the next target indicator 1 is displayed. An example in which target indicators 1 to be targeted are sequentially displayed in this manner is also conceivable.

[0477] In an embodiment, an example has been described in which the display control unit 30 performs processing for simultaneously displaying a plurality of target indicators 1 corresponding to a plurality of recommended imaging positions in the monitor image 150, and makes the display density of the target indicators 1 different depending on the shape of each of the parts of the subject or the distance between the current imaging position and each part of the subject.

[0478] For 3D modeling, for example, only a relatively small amount of information is required in a portion of the subject having a flat, relatively monotonous shape, and therefore, the recommended imaging positions are roughly set and the target indicator 1 is sparsely displayed. On the other hand, in a portion where it is desired to obtain a relatively large amount of information (e.g., a portion having a complex shape or a curved surface), the recommended imaging positions are densely set and the target indicator 1 is densely displayed. By executing image acquisition processing based on the target indicator 1, image acquisition processing can be performed for an appropriate number of images (the number of images is neither too large nor too small), and the operation becomes efficient.

[0479] Note that the density of the target indicator 1 is made different based on the set imaging clarity. Thus, the image acquisition process can be performed with an appropriate operation amount corresponding to the required accuracy of the 3D model.

[0480] In the embodiment, the example has been described in which the display control unit 30 performs display indicating the field of view of the image data obtained in the image obtaining process on the monitor image 150 .

[0481] Monitor image 150 is an image of a subject captured by imaging unit 32. Meanwhile, image data collected for 3D modeling is obtained by imaging unit 20. Therefore, the field of view of imaging unit 20 is indicated by field of view presentation frame 6. Thus, the user can recognize the actual imaging range on monitor image 150.

[0482] In the embodiment, the example has been described in which the display control unit 30 performs display indicating the area on the subject for which the image obtaining process has been completed on the monitor image 150 .

[0483] For example, green is superimposed in a portion of the target subject where the image obtaining process has been performed to clearly indicate that image data has been obtained in that portion.

[0484] Thereby, the user can perform the imaging operation while recognizing the progress status.

[0485] Note that when multiple camera operators are performing operations, the color of the portion where image acquisition processing has been performed can be changed for each camera operator. This allows each camera operator to continue operating while recognizing his / her progress and the areas where other operators have completed image acquisition processing.

[0486] In the embodiment, the example has been described in which the display control unit 30 displays information indicating the progress status of the operation related to 3D modeling in the monitor image 150 .

[0487] For example, a reduced preview 12 is displayed in the simple preview window 10 on the navigation screen 120, or the number of images captured during the image acquisition process, the number of images already uploaded, etc. is displayed as the number of images 11. This allows the user to perform the imaging operation while recognizing the progress status. Alternatively, information indicating the progress status of the imaging operation may be displayed on the navigation screen 120 without providing the simple preview window 10.

[0488] Examples of operations related to 3D modeling include: information indicating the progress status of an imaging operation; information indicating the progress status of a 3D model configuration process based on obtained image data; and information about imaging operations being performed by multiple people.

[0489] Specific examples of the information indicating the progress status of the imaging operation include the number of currently captured images in the image acquisition process, the total number of images predicted to be captured, the estimated scan time, the number of images remaining to be captured, the remaining ratio, and the like.

[0490] Examples of information indicating the progress status of the 3D model configuration process based on the obtained image data on the server device 40 side include the number of images being uploaded, the number of images uploaded, and the number / ratio of images for which preview processing has been completed.

[0491] Examples of the information on imaging operations performed by a plurality of persons include progress displays of other persons, the total number of images captured in the image obtaining process by all persons, and a total progress display of all persons.

[0492] Since the information indicating the progress status of the operation related to 3D modeling varies according to the operation situation, the display is sequentially changed.

[0493] In the embodiment, the example has been described in which the display control unit 30 performs processing for displaying a preview image of a 3D model generated based on image data obtained in the image obtaining processing.

[0494] For example, the display control unit 30 may switch from the navigation screen 120 to display the preview screen 130. Thus, even during the imaging operation, the user can check how much of the 3D model 131 has been generated at the current time.

[0495] Note that since the 3D model 131 can be rotated arbitrarily on the preview screen 130, the user may not be able to recognize his or her own position. Figure 52 As shown in , the current position mark 139 can be used to display the own position.

[0496] Note that the display of the state for guiding the rotation operation to the viewpoint at the own position may be performed by displaying an arrow, etc. Thus, the user can easily return to the preview screen 130 with his / her own viewpoint, and the usability of the preview screen 130 can be improved.

[0497] In addition, in the case where a plurality of people perform an imaging operation, it is preferable to display current position marks 139 a , 139 b , and 139 c including the positions of other people in the preview screen 130 so that each other's situation can be easily grasped.

[0498] The display control unit 30 may set the portion where imaging is not performed as follows. Figure 51A mark 136 is shown in FIG. A mark 137 is also displayed at a corresponding position in the navigation screen 120. In doing so, the user can efficiently find a portion where imaging is not performed and perform an imaging operation.

[0499] In the embodiment, the display control unit 30 performs display indicating an area in the preview image where the image obtaining process has been completed.

[0500] For example, as in Figure 26 As shown in , the display control unit 30 indicates the imaging position in the image obtaining process with a frustum 18 (quadrangular pyramid shape) or the like in the preview screen 130 so that the user can check the overall operation amount and progress.

[0501] Note that both the imaging position and the non-imaging position may be indicated by the frustum 18 etc. In this case, by making the display mode of the frustum 18 etc. different between the imaging position and the non-imaging position, it is easy for the user to check the portion where imaging is not performed or the portion where imaging is performed.

[0502] In addition, as in Figure 27 As shown in , the display control unit 30 adds green (stippled portion in the drawing) to a portion of the preview screen 130 where the image acquisition process has been performed, to clearly indicate that image data has been acquired for the portion.

[0503] Therefore, the user can perform the imaging operation while recognizing the progress status.

[0504] In the embodiment, the example has been described in which the display control unit 30 performs display indicating the quality of image data obtained in the image obtaining process or display indicating insufficiency of image data in the preview image.

[0505] For example, as in Figure 49 As shown in FIG, in the preview screen 130, the display control unit 30 clearly indicates the low-quality portion of the image data in the preview screen 130. Figure 27 As described, a portion where image data is insufficiently obtained is indicated by a color different from green (which indicates that imaging has been performed), ie, for example, red (indicated by shading in the drawing).

[0506] Therefore, even during the imaging operation, the user can recognize the portion on which re-imaging or additional imaging is to be performed, and this is suitable for improvement of the operation efficiency and the accuracy of the 3D model.

[0507] Note that, for a portion determined to be insufficiently imaged according to the preview process, the corresponding target indicator 1 may be emphasized in the navigation screen 120 .

[0508] In the embodiment, the example has been described in which the display control unit 30 performs display related to the moving speed of the user for the imaging operation.

[0509] For example, as in Figure 43 and Figure 44 As shown in , the display control unit 30 indicates that the movement is too fast or too slow. The user can adjust the movement speed while viewing the movement speed, and thus, high-quality image data can be obtained.

[0510] In the embodiment, an example has been described in which the display control unit 30 performs display for prompting the user to return the imaging position to the target pointer 1 according to the image obtaining process based on the quality evaluation result of the image captured in the image obtaining process.

[0511] For example, as in Figure 46 As shown in FIG, the display control unit 30 performs a display for prompting the user to return to the imaging inappropriate target indicator 1. This makes it easy for the user to notice that the imaging position is to be returned for re-imaging.

[0512] The program according to the embodiment is a program for causing a processor such as a CPU or a DSP or a device including the processor to execute Figure 28 、 Figure 34 The processing procedures shown in et al.

[0513] That is, the program according to the embodiment is a program that causes the information processing device 70 to perform the following processing: a processing for displaying a target indicator 1 (first indicator) indicating a recommended imaging position and a self-indicator 2 (second indicator) indicating a current imaging position in a superimposed manner on a monitor image 150 of an object to be subjected to 3D modeling; an image acquisition processing for obtaining image data to be used for 3D modeling based on a matching determination result between the target indicator 1 and the self-indicator 2 on the screen; and a processing for changing the display of the target indicator 1 based on the image acquisition processing.

[0514] By using such a program, the imaging device 51 having the user interface function in the embodiment can be realized by the information processing device 70 .

[0515] The program according to the embodiment described above can be pre-stored in an HDD as a storage medium built into a device such as a computer device, a ROM in a microcomputer including a CPU, or the like. In addition, such a program can be temporarily or permanently stored (recorded) in a removable storage medium such as a floppy disk, a compact disc read-only memory (CD-ROM), a magneto-optical (MO) disk, a digital versatile disc (DVD), a Blu-ray disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such a removable storage medium can be provided as so-called packaged software.

[0516] In addition, such a program can be installed from a removable storage medium into a personal computer or the like, or can be downloaded from a download site via a network such as a local area network (LAN) or the Internet.

[0517] In addition, such a program is suitable for providing the imaging device 51 according to the embodiment in various ways. For example, by downloading the program to a personal computer, a communication device, a mobile terminal device such as a smartphone or tablet, a mobile phone, a game device, a video device, a personal digital assistant (PDA), etc., these devices can be used as the imaging device 51 in the present disclosure.

[0518] Note that the effects described in this specification are merely examples and are not restrictive, and other effects may also be produced.

[0519] Note that the present technology can also have the following configurations. (1)

[0521] An imaging device, comprising:

[0522] a display control unit configured to display a first indicator indicating a recommended imaging position and a second indicator indicating a current imaging position in a superimposed manner on an imaging monitor image, and to perform processing for changing display of the first indicator in accordance with image acquisition processing for acquiring image data; and

[0523] An imaging control unit is configured to execute an image obtaining process according to a result of a matching determination between the first pointer and the second pointer on the screen. (2)

[0525] The imaging device according to (1), wherein

[0526] The display control unit changes a mode of changing the display of the first indicator according to the image obtaining process, depending on a quality evaluation result of an image captured in the image obtaining process. (3)

[0528] The imaging device according to (1) or (2), wherein

[0529] The first indicator includes a geometric shape. (4)

[0531] The imaging device according to any one of (1) to (3), wherein

[0532] The first indicator includes a hexagonal shape. (5)

[0534] The imaging device according to any one of (1) to (4), wherein

[0535] The first indicator indicates a direction facing the subject. (6)

[0537] The imaging device according to any one of (1) to (5), wherein

[0538] The display control unit performs processing for changing a pattern of the first indicator according to a positional relationship between the imaging position and the recommended imaging position. (7)

[0540] The imaging device according to any one of (1) to (6), wherein

[0541] The display control unit performs processing for changing a pattern of the first indicator according to a relationship between the facing direction of the recommended imaging position and the imaging direction. (8)

[0543] The imaging device according to any one of (1) to (7), wherein

[0544] The display control unit performs processing for causing the first pointer at the position where the image obtaining process was performed based on the matching determination with the second pointer to be displayed in a mode different from a mode before the image obtaining process was performed. (9)

[0546] The imaging device according to any one of (1) to (8), wherein

[0547] The display control unit performs processing for simultaneously displaying a plurality of first indicators corresponding to a plurality of recommended imaging positions on the imaging monitor image. (10)

[0549] The imaging device according to any one of (1) to (9), wherein

[0550] The display control unit performs processing for simultaneously displaying a plurality of first indicators corresponding to a plurality of recommended imaging positions on the imaging monitor image, and

[0551] The display density of the first indicator is made different according to the shape of each of the parts of the subject or the distance between the current imaging position and each part of the subject. (11)

[0553] The imaging device according to any one of (1) to (10), wherein

[0554] The display control unit performs display of a field of view indicating image data obtained in the image obtaining process on the imaging monitor image. (12)

[0556] The imaging device according to any one of (1) to (11), wherein

[0557] The display control unit performs display indicating an area on the subject for which the image obtaining process has been completed, on the imaging monitor image. (13)

[0559] The imaging device according to any one of (1) to (12), wherein

[0560] The display control unit displays information indicating a progress status of an operation related to 3D modeling on the imaging monitor image. (14)

[0562] The imaging device according to any one of (1) to (13), wherein

[0563] The display control unit performs processing for displaying a preview image of a 3D model generated based on the image data obtained in the image obtaining processing. (15)

[0565] The imaging device according to (14), wherein

[0566] The display control unit performs display indicating an area on the preview image where the image obtaining process has been completed. (16)

[0568] The imaging device according to (14) or (15), wherein

[0569] The display control unit performs display indicating the quality of the image data obtained in the image obtaining process or display indicating insufficiency of the image data in the preview image. (17)

[0571] The imaging device according to any one of (1) to (16), wherein

[0572] The display control unit performs display related to the moving speed of the user for the imaging operation. (18)

[0574] The imaging device according to any one of (1) to (17), wherein

[0575] The display control unit performs a function for causing the imaging position to return to display of the first indicator according to the image obtaining process, depending on a quality evaluation result of an image captured in the image obtaining process. (19)

[0577] An imaging method, performed by an imaging device, comprising:

[0578] a process for displaying a first indicator indicating a recommended imaging position and a second indicator indicating a current imaging position on an imaging monitor image in a superimposed manner;

[0579] an image obtaining process for obtaining image data according to a result of a matching determination between a first pointer and a second pointer on the screen; and

[0580] A process for changing the display of the first indicator according to the image obtaining process. (20)

[0582] A program that causes an information processing device to:

[0583] a process for displaying a first indicator indicating a recommended imaging position and a second indicator indicating a current imaging position on an imaging monitor image in a superimposed manner;

[0584] an image obtaining process for obtaining image data according to a result of a matching determination between a first pointer and a second pointer on the screen; and

[0585] A process for changing the display of the first indicator according to the image obtaining process.

[0586] Additionally, the present technology may also adopt the following configurations. (101)

[0588] An imaging device, comprising:

[0589] a display control unit configured to display a first indicator indicating a recommended imaging position and a second indicator indicating a current imaging position in a superimposed manner on an imaging monitor image of an object to be subjected to 3D modeling, and to perform processing for changing display of the first indicator in accordance with image acquisition processing for acquiring image data used for 3D modeling; and

[0590] An imaging control unit is configured to execute an image obtaining process according to a result of a matching determination between the first pointer and the second pointer on the screen. (102)

[0592] An imaging method, performed by an imaging device, comprising:

[0593] a process for displaying a first indicator indicating a recommended imaging position and a second indicator indicating a current imaging position in a superimposed manner on an imaging monitor image of an object to be subjected to 3D modeling;

[0594] an image obtaining process for obtaining image data to be used for 3D modeling according to a result of a matching determination between a first pointer and a second pointer on the screen; and

[0595] A process for changing the display of the first indicator according to the image obtaining process. (103)

[0597] A program that causes an information processing device to:

[0598] a process for displaying a first indicator indicating a recommended imaging position and a second indicator indicating a current imaging position in a superimposed manner on an imaging monitor image of an object to be subjected to 3D modeling;

[0599] an image obtaining process for obtaining image data to be used for 3D modeling according to a result of a matching determination between a first pointer and a second pointer on the screen; and

[0600] A process for changing the display of the first indicator according to the image obtaining process.

[0601] The elements in (2) to (18) can be added to the above (101), (102) and (103).

[0602] Reference Signs List

[0603] 1 target indicator

[0604] 2 self-indicator

[0605] 3 Navigation Box

[0606] 4-direction presentation section

[0607] 5A imaging success animation

[0608] 5B Imaging Inappropriate Animation

[0609] 6 field of view presentation frame

[0610] 9 Coloring Areas

[0611] 10Simple preview window

[0612] 11 Number of images

[0613] 12 Zoom out preview

[0614] 13 Operation presentation unit

[0615] 15 Imaging execution box

[0616] 20 imaging units

[0617] 24 Imaging Control Unit

[0618] 25 scoring processing units

[0619] 26 output units

[0620] 27 Superimposed image generation unit

[0621] 28 Display image generation unit

[0622] 29 Communication Unit

[0623] 30 Display control unit

[0624] 31 Depth Sensor

[0625] 32 imaging units

[0626] 33IMU

[0627] 34SLAM

[0628] 35TSDF update unit

[0629] 36 mesh generation units

[0630] 40 Server Devices

[0631] 41 Communication Unit

[0632] 42 Photogrammetry Processing Unit

[0633] 43 storage units

[0634] 51 Imaging device

[0635] 52 sensor units

[0636] 53 Interface device

[0637] 54,55 display

[0638] 70 Information processing device

[0639] 110ROI setting screen

[0640] 120 Navigation screen

[0641] 130 Preview screen

[0642] 131 3D models

[0643] 150 monitor screen

Claims

1. An imaging device comprising: a display control unit configured to display a first indicator indicating a recommended imaging position and a second indicator indicating a current imaging position in a superimposed manner on an imaging monitor image, and to perform processing for changing display of the first indicator in accordance with image acquisition processing for acquiring image data; as well as An imaging control unit configured to execute the image obtaining process according to a result of a matching determination between the first pointer and the second pointer on a screen.

2. The imaging device according to claim 1, wherein The display control unit changes a mode of changing display of the first indicator according to the image obtaining process, depending on a quality evaluation result of an image captured in the image obtaining process.

3. The imaging device according to claim 1, wherein The first indicator comprises a geometric figure.

4. The imaging device according to claim 1, wherein The first indicator comprises a hexagonal shape.

5. The imaging device according to claim 1, wherein The first indicator indicates a direction facing the subject.

6. The imaging device according to claim 1, wherein The display control unit performs processing for changing a pattern of the first indicator according to a positional relationship between an imaging position and the recommended imaging position.

7. The imaging device according to claim 1, wherein The display control unit performs processing for changing a mode of the first indicator according to a relationship between a facing direction of the recommended imaging position and an imaging direction.

8. The imaging device according to claim 1, wherein The display control unit performs processing for causing the first pointer at the position where the image obtaining process was performed based on matching determination with the second pointer to be displayed in a mode different from a mode before the image obtaining process was performed.

9. The imaging device according to claim 1, wherein The display control unit performs processing for simultaneously displaying a plurality of the first indicators corresponding to a plurality of the recommended imaging positions on the imaging monitor image.

10. The imaging device according to claim 1, wherein The display control unit performs processing for simultaneously displaying a plurality of the first indicators corresponding to a plurality of the recommended imaging positions on the imaging monitor image, and The display density of the first indicator is made different according to the shape of each of the parts of the object or the distance between the current imaging position and each part of the object.

11. The imaging device according to claim 1, wherein The display control unit performs display indicating a field of view of the image data obtained in the image obtaining process on the imaging monitor image.

12. The imaging device according to claim 1, wherein The display control unit performs display indicating a region on a subject for which the image obtaining process has been completed, on the imaging monitor image.

13. The imaging device according to claim 1, wherein The display control unit displays information indicating a progress status of an operation related to 3D modeling on the imaging monitor image.

14. The imaging device according to claim 1, wherein The display control unit performs processing for displaying a preview image of a 3D model generated based on the image data obtained in the image obtaining processing.

15. The imaging device according to claim 14, wherein The display control unit performs display indicating an area on the preview image where the image obtaining process has been completed.

16. The imaging device according to claim 14, wherein The display control unit performs display indicating quality of the image data obtained in the image obtaining process or display indicating insufficiency of the image data in the preview image.

17. The imaging device according to claim 1, wherein The display control unit performs display related to a moving speed of a user for an imaging operation.

18. The imaging device according to claim 1, wherein The display control unit performs a function for causing the imaging position to return to the display of the first indicator according to the image obtaining process, depending on a quality evaluation result of an image captured in the image obtaining process.

19. An imaging method, the imaging method being performed by the imaging device, the imaging method comprising: a process for displaying a first indicator indicating a recommended imaging position and a second indicator indicating a current imaging position on an imaging monitor image in a superimposed manner; an image acquisition process for acquiring image data according to a result of a matching determination between the first indicator and the second indicator on the screen; as well as A process for changing a display of the first indicator according to the image obtaining process.

20. A program causing an information processing device to execute: a process for displaying a first indicator indicating a recommended imaging position and a second indicator indicating a current imaging position on an imaging monitor image in a superimposed manner; an image acquisition process for acquiring image data according to a result of a matching determination between the first indicator and the second indicator on the screen; as well as A process for changing a display of the first indicator according to the image obtaining process.

Images