Information processing apparatus and method

Abstract

The present disclosure relates to an information processing device and method capable of suppressing a reduction in navigation accuracy of imaging for 3D modeling. In the present invention, first orientation information and first three-dimensional shape information are generated on the basis of a first image generated by first imaging in which a 3D object is imaged, and second orientation information is calibrated on the basis of the first image, the first orientation information, and a second image, and control a second image for generating the second three-dimensional shape information or guide information for generating the second image for generating the second three-dimensional shape information based on the second orientation information reflecting a result of the calibration and the first three-dimensional shape information, and control an output of the guide information. The present disclosure is applicable, for example, to an information processing device, an imaging device, an imaging communication device, an electronic device, an information processing method, a program, an information processing system, etc.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus and method, and more particularly, to an information processing apparatus and method capable of suppressing a decrease in the accuracy of navigation of imaging for 3D modeling. Background Art

[0002] As a known method for 3D modeling of a 3D object having a three-dimensional shape, there is a method called photogrammetry, which images a 3D object from multiple directions and generates 3D data based on multiple captured images (for example, see Patent Document 1). In addition, there is a method called real-time 3D modeling, which generates 3D data instantly (in real time) based on information such as captured images, orientation information, and depth. In addition, in recent years, methods generally referred to as "neural rendering" (such as neural radiance field (NeRF)) have been proposed, in which a neural field is constructed based on the captured image and the orientation of the captured image to generate an image or a three-dimensional model from any viewpoint.

[0003] Reference List

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2018-63693 Summary of the Invention

[0006] Problems to be solved by the present invention

[0007] Where navigation of imaging (imaging control, imaging guidance, or both) is performed to obtain captured images to be used in 3D modeling (as in such methods), there is the possibility that the lower the accuracy of the position and orientation of the imaging unit performing the imaging, the lower the accuracy of the navigation.

[0008] The present disclosure has been made in view of such circumstances, and an object of the present disclosure is to suppress a decrease in the accuracy of navigation of imaging for 3D modeling.

[0009] Solution to the problem

[0010] An information processing device according to one aspect of the present technology includes: a first 3D modeling processing unit that generates first orientation information indicating a position and orientation of a first imaging unit that performs first imaging and first three-dimensional shape information representing a three-dimensional shape of a 3D object based on a first captured image generated by first imaging that images a 3D object; a calibration unit that calibrates second orientation information indicating a position and orientation of a second imaging unit that performs second imaging based on the first captured image, the first orientation information, and a second captured image generated by second imaging that images the 3D object; and an imaging control unit that reflects a result of the calibration in the second orientation information and controls second imaging for generating second three-dimensional shape information representing the three-dimensional shape of the 3D object based on the second orientation information reflecting the result of the calibration and the first three-dimensional shape information.

[0011] An information processing method according to one aspect of the present technology includes: generating, based on a first captured image generated by first imaging that images a 3D object, first orientation information indicating the position and orientation of a first imaging unit that performs the first imaging and first three-dimensional shape information representing the three-dimensional shape of the 3D object; calibrating, based on the first captured image, the first orientation information, and a second captured image generated by second imaging that images the 3D object, second orientation information indicating the position and orientation of a second imaging unit that performs the second imaging; and reflecting the calibration result in the second orientation information, and controlling, based on the second orientation information reflecting the calibration result and the first three-dimensional shape information, second imaging for generating second three-dimensional shape information representing the three-dimensional shape of the 3D object.

[0012] An information processing device according to another aspect of the present technology includes: a first 3D modeling processing unit that generates first orientation information indicating the position and orientation of a first imaging unit that performs first imaging and first three-dimensional shape information representing the three-dimensional shape of the 3D object based on a first captured image generated by first imaging that images the 3D object; a calibration unit that calibrates second orientation information indicating the position and orientation of a second imaging unit that performs second imaging based on the first captured image, the first orientation information, and a second captured image generated by second imaging that images the 3D object; and a guidance information output control unit that reflects the calibration result in the second orientation information, generates guidance information for the second imaging based on the second orientation information reflecting the calibration result and the first three-dimensional shape information, and controls the output of the guidance information. The second imaging is used to generate second three-dimensional shape information representing the three-dimensional shape of the 3D object.

[0013] An information processing method in another aspect of the present technology includes: generating first orientation information indicating the position and orientation of a first imaging unit performing the first imaging and first three-dimensional shape information representing the three-dimensional shape of the 3D object based on a first captured image generated by first imaging that images the 3D object; calibrating second orientation information indicating the position and orientation of a second imaging unit performing the second imaging based on the first captured image, the first orientation information, and a second captured image generated by second imaging that images the 3D object; and reflecting the calibration result in the second orientation information, generating guidance information for the second imaging based on the second orientation information reflecting the calibration result and the first three-dimensional shape information, and controlling output of the guidance information, wherein the second imaging is used to generate second three-dimensional shape information representing the three-dimensional shape of the 3D object.

[0014] In an information processing device and method of one aspect of the present technology, based on a first captured image generated by first imaging that images a 3D object, first orientation information indicating the position and orientation of a first imaging unit that performs the first imaging and first three-dimensional shape information representing the three-dimensional shape of the 3D object are generated; based on the first captured image, the first orientation information, and a second captured image generated by second imaging that images the 3D object, second orientation information indicating the position and orientation of a second imaging unit that performs the second imaging is calibrated; the calibration result is reflected in the second orientation information, and based on the second orientation information reflecting the calibration result and the first three-dimensional shape information, second imaging for generating second three-dimensional shape information representing the three-dimensional shape of the 3D object is controlled.

[0015] In an information processing device and method on another aspect of the present technology, based on a first captured image generated by first imaging of a 3D object, first orientation information indicating the position and orientation of a first imaging unit performing the first imaging and first three-dimensional shape information representing the three-dimensional shape of the 3D object are generated; based on the first captured image, the first orientation information and a second captured image generated by second imaging of the 3D object, second orientation information indicating the position and orientation of the second imaging unit performing the second imaging is calibrated; the calibration result is reflected in the second orientation information, and guidance information for the second imaging is generated based on the second orientation information reflecting the calibration result and the first three-dimensional shape information, and the output of the guidance information is controlled, wherein the second imaging is used to generate second three-dimensional shape information representing the three-dimensional shape of the 3D object. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a diagram used to describe an overview of photogrammetry.

[0017] Figure 2 is a diagram for describing an overview of real-time 3D modeling.

[0018] Figure 3 A diagram comparing photogrammetry with real-time 3D modeling.

[0019] Figure 4 is a diagram of an example of a method for assisting imaging for 3D modeling.

[0020] Figure 5 is a diagram showing an example of how imaging control and imaging guidance are performed.

[0021] Figure 6 is a diagram showing an example of overlapping.

[0022] Figure 7 is a diagram showing an example of how imaging is performed according to the distance to the subject.

[0023] Figure 8 is a diagram showing an example of how parallel processing is performed.

[0024] Figure 9 is a diagram illustrating an example of scoring processing.

[0025] Figure 10 is a diagram illustrating an example of scoring processing.

[0026] Figure 11 is a diagram illustrating an example of scoring processing.

[0027] Figure 12 is a diagram illustrating an example of scoring processing.

[0028] Figure 13 is a diagram illustrating an example of scoring processing.

[0029] Figure 14 is a diagram showing an example of the output of the scoring result.

[0030] Figure 15 is a diagram showing an example of output of overlap information.

[0031] Figure 16 : is a figure which shows the display example of guidance information.

[0032] Figure 17 : is a figure which shows the display example of guidance information.

[0033] Figure 18 : is a figure which shows the display example of guidance information.

[0034] Figure 19 : is a figure which shows the display example of guidance information.

[0035] Figure 20 : is a figure which shows the display example of guidance information.

[0036] Figure 21 : is a figure which shows the display example of guidance information.

[0037] Figure 22 : is a figure which shows the display example of guidance information.

[0038] Figure 23 : is a figure which shows the display example of guidance information.

[0039] Figure 24 : is a figure which shows the display example of guidance information.

[0040] Figure 25 : is a figure which shows the display example of guidance information.

[0041] Figure 26 is a diagram showing an example of device calibration.

[0042] Figure 27 is a diagram illustrating an example of a calibration process.

[0043] Figure 28 is a diagram for describing the imaging timing calibration process.

[0044] Figure 29 is a diagram for describing the imaging timing calibration process.

[0045] Figure 30 is a diagram for describing the imaging timing calibration process.

[0046] Figure 31 This is a diagram for describing the installation boot process.

[0047] Figure 32 It is a diagram for describing the orientation information calibration process.

[0048] Figure 33 is a diagram illustrating an example of orientation information calibration processing.

[0049] Figure 34 It is a diagram for describing the orientation information calibration process.

[0050] Figure 35 3 is a diagram for describing the orientation information calibration process.

[0051] Figure 36 3 is a diagram for describing the orientation information calibration process.

[0052] Figure 37 is a diagram for describing the imaging timing calibration process.

[0053] Figure 38 is a diagram illustrating an example of a calibration process.

[0054] Figure 39is a block diagram showing a main configuration example of an imaging device.

[0055] Figure 40 is a flowchart for describing an example of the flow of a 3D modeling process.

[0056] Figure 41 is a flowchart for describing an example of the flow of a real-time 3D modeling process.

[0057] Figure 42 is a flowchart for describing an example of the flow of photogrammetry processing.

[0058] Figure 43 is a block diagram showing a main configuration example of an information processing system.

[0059] Figure 44 is a block diagram showing a main configuration example of an imaging communication device.

[0060] Figure 45 is a block diagram showing a main configuration example of an imaging device.

[0061] Figure 46 is a block diagram showing a main configuration example of a server.

[0062] Figure 47 is a flowchart for describing an example of the flow of a 3D modeling process.

[0063] Figure 48 This is an example of a flow for describing a 3D modeling process. Figure 47 Flowchart of the flowchart in .

[0064] Figure 49 is a block diagram showing a main configuration example of an imaging communication device.

[0065] Figure 50 is a block diagram showing a main configuration example of a server.

[0066] Figure 51 is a flowchart for describing an example of the flow of a 3D modeling process.

[0067] Figure 52 This is an example of a flow for describing a 3D modeling process. Figure 51 Flowchart of the flowchart in .

[0068] Figure 53 is a block diagram showing a main configuration example of an imaging communication device.

[0069] Figure 54 is a block diagram showing a main configuration example of a calibration unit.

[0070] Figure 55is a flowchart for describing an example of the flow of a 3D modeling process.

[0071] Figure 56 is a flowchart for describing an example of the flow of imaging timing calibration processing.

[0072] Figure 57 is a flowchart for describing an example of the flow of installation boot processing.

[0073] Figure 58 is a flowchart describing an example of the flow of orientation information calibration processing.

[0074] Figure 59 is a block diagram showing a main configuration example of a calibration unit.

[0075] Figure 60 is a flowchart for describing an example of the flow of a 3D modeling process.

[0076] Figure 61 is a flowchart for describing an example of the flow of orientation information calibration processing.

[0077] Figure 62 is a flowchart for describing an example of the flow of imaging timing calibration processing.

[0078] Figure 63 is a block diagram showing a main configuration example of a computer. DETAILED DESCRIPTION

[0079] Hereinafter, modes for carrying out the present disclosure (hereinafter, referred to as embodiments) will be described. Note that the description will be given in the following order.

[0080] 1.3D Modeling

[0081] 2. Imaging Control

[0082] 3. Imaging-guided output

[0083] 4. Combination

[0084] 5. Calibration Process

[0085] 6. First Embodiment (Imaging Device)

[0086] 7. Second Embodiment (Information Processing System)

[0087] 8. Third embodiment (information processing system)

[0088] 9. Appendix

[0089] 1.3D Modeling

[0090] <Photogrammetry>

[0091] As a known method for generating (reconstructing) a three-dimensional shape model of an object having a three-dimensional shape (also referred to herein as a 3D object), there is a method called photogrammetry, which images the 3D object from multiple directions and generates 3D data based on the multiple captured images. Note that generating a three-dimensional shape model of a 3D object is also referred to herein as 3D modeling.

[0092] Photogrammetry is a method of reconstructing a highly accurate three-dimensional model from multiple images captured from various viewpoints using the principle of triangulation. Note that in this article, the "accuracy" of 3D data (3D model) includes not only the reproducibility (accuracy, resolution, etc.) of the three-dimensional shape of the target 3D object, but also the reproducibility (accuracy, resolution, etc.) of the texture applied to the surface of the 3D model. For example, Figure 1 The camera devices shown in FIG. 1 , such as camera 11-1 to camera 11-5, image the 3D object 10 from multiple viewpoints to obtain multiple captured images. These captured images are then used to perform a process called structure from motion (SfM) and a process called multi-view stereo (MVS), and meshing and texturing are further performed as post-processing to generate 3D data 15.

[0093] For example, in SfM, corresponding points are searched between captured images. Epipolar constraints are used to derive the position and orientation of each camera. Based on the camera's position and orientation, triangulation is used to determine the position of each corresponding point in 3D space. This point in 3D space is also referred to herein as a 3D point. In other words, a 3D point is determined for each corresponding point. Bundle adjustment is then used to optimize the entire 3D point cloud.

[0094] For example, in MVS, denser corresponding points are searched using the three-dimensional point cloud derived as described above, and the obtained three-dimensional points are added.

[0095] As mentioned above, in photogrammetry, global optimization calculations, known as bundle adjustments, are performed to minimize errors. This can yield highly accurate results, but at the expense of a high computational load. Furthermore, photogrammetry is based on geometric calculations rather than physical measurements. Therefore, in principle, the higher the resolution of the image used, the more accurate the recovered model.

[0096] <Real-time 3D modeling>

[0097] As a 3D modeling method different from this photogrammetry, there is a method called real-time 3D modeling that generates 3D data instantly (in real time) based on information such as captured images, orientation information, and depth. For example, in this method, Figure 2As shown, the camera 21 images the 3D object 10 while moving around the 3D object 10 as shown by the dotted line 22. The camera 21 includes not only an image sensor but also a light detection and ranging (Lidar) scanner (direct time of flight (dToF) module), and obtains a captured image and detects the depth (distance to the subject).

[0098] In recent years, with the development of science and technology, the miniaturization and functional enhancement of dToF modules have been promoted, and they can accurately measure the depth of relatively long distances (for example, about 5 meters) both indoors and outdoors. This makes it easy to experience real-time modeling and capture at the consumer level.

[0099] Furthermore, the camera 21 also includes an inertial sensor, and detects the acceleration and angular velocity of the camera 21 (also referred to herein as inertial information about the camera 21 ).

[0100] In real-time 3D modeling, a process called simultaneous localization and mapping (SLAM) is performed to generate orientation information indicating the position and orientation of the camera 21. In addition, a truncated signed distance function (TSDF) is updated using the orientation information and depth, and 3D data 25 (mesh and texture) is generated through a process called marching cubes (MC).

[0101] For example, in SLAM, the position and orientation of the camera are estimated based on the captured image and inertial information (self-positioning). In updating the TSDF, the correspondence between the depth and the voxel is established, and the volume is detected. In MC, the calculation of the isosurface is performed using adjacent voxels. Using the real-time orientation information of SLAM, the volume of the voxel can be detected by superimposing multiple frames of depth (the distance reached by the light beam) (rather than via a point cloud). The voxel representation makes it possible to estimate viewpoints that are obscured and need to be captured (missing viewpoints). This makes it possible to detect perforated structures or protruding structures of 3D objects.

[0102] <Neural Rendering>

[0103] In addition, in recent years, methods collectively referred to as neural rendering (such as neural radiance field (NeRF) etc.) have been proposed, in which a neural field is constructed based on the captured image and the orientation of the captured image to generate an image or a three-dimensional model from an arbitrary viewpoint.

[0104] <Comparative Example>

[0105] Each of the above 3D modeling methods has different characteristics, and no one method is excellent in all aspects. Figure 3 Results comparing the properties of photogrammetry with real-time 3D modeling are shown. Figure 3As shown in Figure 2, when comparing various methods, photogrammetry uses SfM (including self-localization) and MVS, while real-time 3D modeling uses self-localization (SLAM) and TSDF. Furthermore, when comparing the data to be used, photogrammetry uses only image data, while real-time 3D modeling uses depth and orientation data in addition to image data. Furthermore, when comparing processing time, photogrammetry takes a long time, ranging from several minutes to tens of hours, while real-time 3D modeling allows for near-instant (real-time) processing, such as 30 fps (frames per second).

[0106] Furthermore, when comparing the required computing power, photogrammetry requires high-end central processing unit (CPU) and graphics processing unit (GPU)-level computing power, while real-time 3D modeling requires mobile application processor (AP)-level computing power. Furthermore, when comparing the resolution of the model to be generated, photogrammetry, while depending on factors such as the resolution, number, and capture method of the captured images, results in relatively high resolution, while real-time 3D modeling, while depending on factors such as depth and self-localization accuracy, results in relatively low resolution.

[0107] Furthermore, regarding the internal representation of the three-dimensional data to be generated, photogrammetry is based on point clouds, while real-time 3D modeling is based on voxels. Furthermore, photogrammetry has no subject size and resolution constraints, while real-time 3D modeling depends on the sensor. Furthermore, when comparing the absolute accuracy of the models, photogrammetry results in relatively high absolute accuracy due to optimization using bundle adjustment, while real-time 3D modeling, although dependent on factors such as sensor and self-positioning accuracy, results in relatively low absolute accuracy. Furthermore, when comparing sizes, for photogrammetry, the size is variable (unknown size), while for real-time 3D modeling, the size is uniquely determined (absolute size known).

[0108] For example, photogrammetry and real-time 3D modeling differ in their characteristics. Specifically, real-time 3D modeling can reduce the workload and processing volume of 3D modeling compared to photogrammetry or neural rendering. However, using photogrammetry or neural rendering can generate highly accurate 3D data compared to real-time 3D modeling.

[0109] <Convenience of 3D Modeling>

[0110] For example, to obtain more accurate 3D data, photogrammetry or neural rendering can be applied as described above. However, in this case, it is also desirable to reduce the workload and processing volume of 3D modeling. To reduce the workload and processing volume of 3D modeling, it is necessary to generate 3D data with the highest possible accuracy while minimizing the imaging frequency.

[0111] For example, if the captured images required for 3D modeling are unavailable, the accuracy of the 3D data may decrease. Conversely, attempting to obtain an excessive number of captured images to avoid a shortage may unnecessarily increase the imaging frequency, and the user's workload may also increase accordingly. Furthermore, in this case, the 3D modeling process may be performed using unnecessary captured images, which may unnecessarily increase the processing volume.

[0112] That is, in order to obtain more accurate 3D data with less workload and processing, it is necessary to image the 3D object in a more appropriate position and orientation. However, in each of the known 3D modeling methods, it is difficult for the photographer to determine the appropriate position and orientation for imaging.

[0113] For example, photogrammetry requires a significant amount of time for 3D modeling processing, making it difficult for the user to instantly check the 3D modeling results during imaging. Consequently, it is difficult for the user to determine the appropriate position and orientation for imaging during imaging. Consequently, for example, the number of images captured at the appropriate position and orientation may be insufficient, and the accuracy of the 3D data obtained through photogrammetry may be reduced. Furthermore, if imaging is performed excessively and arbitrarily at any position and orientation in order to avoid insufficient images captured at the appropriate position and orientation, not only is there a risk of increasing the user's workload, but the number of captured images may be unnecessarily increased, and the load on the 3D modeling processing (processing volume, processing time, etc.) may be unnecessarily increased.

[0114] <2. Imaging Control>

[0115] <Imaging control based on scoring results>

[0116] Thus, 3D modeling is performed twice, and the results of the first 3D modeling are used to control the imaging for the second 3D modeling.

[0117] For example, it is assumed that a second imaging is performed to image a 3D object having a three-dimensional shape, and a second 3D modeling process is performed to generate second 3D data (second three-dimensional shape information) representing the three-dimensional shape of the 3D object using a second captured image obtained by the second imaging. Figure 4 At this time, the second imaging for the second 3D modeling process is controlled so that the second imaging is performed at a more appropriate position and orientation ( Figure 4 In order to implement this control, the imaging control process 103 for the second 3D modeling is executed. Figure 4 The first 3D data generation process 101 and the scoring process 102 in .

[0118] The first 3D data generation process 101 generates first 3D data (first 3D shape information) representing the three-dimensional shape of a 3D object. Specifically, the first 3D data generation process 101 performs a first imaging process of the 3D object and a first 3D modeling process of generating first 3D data using a first captured image obtained through the first imaging process.

[0119] Scoring 102 evaluates (scores) the accuracy of second 3D data generated using the second captured image generated by the second imaging performed so far. This scoring is performed based on the first 3D data generated by the first 3D modeling process. In imaging control 103 for second 3D modeling, the second imaging is controlled based on the scoring result.

[0120] That is, based on the first 3D data generated based on the first captured image obtained by the first imaging, the accuracy of the second 3D data that can be generated based on the second captured image obtained by the second imaging up to this moment is evaluated (scoring is performed). By doing so, the scoring result can be generated more easily. In addition, the second imaging is controlled based on the scoring result. By doing so, the second imaging can be controlled so that the second imaging is performed at a more appropriate position and orientation. That is, the second 3D modeling process can be performed using the second captured image captured at a more appropriate position and orientation. Therefore, more accurate 3D data can be generated while suppressing the increase in the load (workload and processing volume) of 3D modeling. That is, 3D modeling can be performed more easily.

[0121] Note that, unless otherwise specified, the term "captured image" herein refers to any image captured by an image sensor, etc. For example, the following images are typically obtained using an imaging device, etc. For example, a still image is captured by an image sensor, etc., when a shutter button, etc., is operated, and this still image is stored as an imaging result in a storage medium, etc. Alternatively, a moving image is captured by an image sensor, etc., starting when a shutter button, etc. is operated, and the moving image is stored as an imaging result in a storage medium, etc. Furthermore, an image captured by an image sensor, etc. before the shutter button, etc. is operated (sometimes also referred to as an acquired image), is not stored as an imaging result in a storage medium, but is instead displayed on a monitor, etc. As used herein, "captured image" includes all of these images. In other words, a captured image can be either a still image or a moving image. Furthermore, a captured image may or may not be stored as an imaging result in a storage medium, etc. Furthermore, a captured image may be captured before, at, or after the shutter button, etc. is operated. Furthermore, the captured image may be data captured by an image sensor or the like (so-called RAW data). Furthermore, the captured image may be an image that has undergone color separation processing or color conversion processing. Furthermore, the captured image may be an image that has undergone signal processing (e.g., defect correction, noise reduction, automatic white balance (AWB), or gamma correction). Furthermore, other image processing may be performed.

[0122] <First Imaging>

[0123] Herein, the imaging unit (image sensor) that performs the first imaging is also referred to as the first imaging unit. In addition, the imaging unit (image sensor) that performs the second imaging is also referred to as the second imaging unit.

[0124] As described above, the first imaging is performed in the first 3D data generation process 101. That is, the first captured image is generated by the first imaging unit. At this time, the distance (depth) from the first imaging unit to the subject (3D object) appearing in the first captured image can be detected by the depth sensor. The depth detection method using the depth sensor may be any method. In addition, the depth sensor may be a sensor integrated with the first imaging unit, or a sensor that is different from the first imaging unit and is installed at a different position from the first imaging unit. Note that in the following description, unless otherwise specified, it is assumed that the depth has been properly calibrated for the first captured image. In addition, when performing the first imaging, inertial information (angular velocity and acceleration) about the first imaging unit may be detected by the inertial information sensor. The method for detecting inertial information using the inertial information sensor may be any method. In addition, the inertial information sensor may be a sensor integrated with the first imaging unit, or a sensor that is different from the first imaging unit and is installed at a different position from the first imaging unit.

[0125] The generated first captured image is used in the first 3D data generation process. In addition, in the case where depth and inertial information are generated, they are also used in the first 3D data generation process.

[0126] Note that the number of first imaging units (image sensors), depth sensors, and inertial information sensors can each be any number, whether singular or plural. That is, the number of first imaging units, depth sensors, and inertial information sensors can all be the same, or two of them can be the same, or they can all be different.

[0127] <First 3D Modeling Process>

[0128] As described above, the first 3D modeling process is performed in the first 3D data generation process 101. In the first 3D modeling process, first 3D data (first three-dimensional shape information) representing the three-dimensional shape of the 3D object is generated based on a first captured image generated by first imaging of the 3D object.

[0129] Compared to the second 3D data (second three-dimensional shape information) generated by the second 3D data generation process 104 , the first 3D data may have a smaller amount of information and may be less accurate.

[0130] By doing so, it is possible to suppress an increase in the load of the imaging control process 103 and the scoring process 102 for the second 3D modeling. In other words, by further simplifying the first 3D data (reducing the amount of information and accuracy of the first 3D data), it is possible to suppress an increase in the load of scoring and imaging control using the first 3D data. Furthermore, in general, an increase in the load of generating the first 3D data (the first 3D modeling process) can also be suppressed. In other words, the second imaging can be controlled with a lower load.

[0131] Furthermore, the first 3D modeling process may be any method. For example, in the first 3D modeling process, orientation information corresponding to the viewing angle of the first captured image may be derived, and first 3D data may be generated based on the orientation information, the first captured image, and the depth of the subject (3D object) in the first captured image. For example, the first 3D data may be generated by updating the TSDF and performing MC based on this information.

[0132] Note that the orientation information is information indicating the position and orientation of the first imaging unit in three-dimensional space. The method for deriving the orientation information can be any method. For example, the orientation information can be derived based on inertial information about the first imaging unit (acceleration and angular velocity of the first imaging unit). For example, SLAM can be applied.

[0133] That is, the above-mentioned real-time 3D modeling can be applied as a first 3D modeling processing method. By doing so, the first 3D modeling process can be performed instantly (in real time) and the first 3D data can be obtained instantly (in real time). Therefore, the imaging control process for the second 3D modeling can be performed instantly (in real time). That is, 3D modeling can be performed more easily. Note that the orientation information about the first imaging unit and the first 3D data can be generated using a neural network that takes the first captured image, the inertial information about the first imaging unit, and the depth as input.

[0134] Furthermore, the first 3D data may be any data as long as it represents the three-dimensional shape of the 3D object. Specifically, the first 3D data may be a point cloud, or may include a mesh representing the three-dimensional shape of the 3D object through vertices and connections, and a texture applied to the surface of the mesh. This first 3D data is provided to the scoring process 102.

[0135] <Rating Processing>

[0136] In the scoring process 102, as described above, the accuracy of the second 3D data generated using the second captured image generated by the second imaging performed so far is evaluated. The scoring is performed based on the first 3D data generated by the first 3D modeling process and the position and orientation of the second imaging performed so far. That is, the first 3D data is regarded as a 3D object to be modeled in the second 3D modeling process, and a score is calculated for each local part of the first 3D data. For example, in the case where the first 3D data includes a mesh representing the three-dimensional shape of the 3D object by vertices and connections and a texture applied to the surface of the mesh, a scoring result is generated for each polygon of the mesh. That is, the part of the first 3D data from which the more accurate second 3D data is obtained is evaluated higher (is set to a higher score).

[0137] For example, suppose Figure 5 The first 3D data 120 is shown by Figure 4 1. It is assumed that second imaging of the 3D object corresponding to the first 3D data 120 is performed at the positions and orientations of the cameras 121-1 to 121-3. In this case, in the scoring process 102, the upper side of the first 3D data 120 in the figure is evaluated with a relatively high score, and the lower side (gray portion) of the first 3D data 120 in the figure is evaluated with a relatively low score. An example of the scoring method will be described later.

[0138] Note that in Figure 5 In the example, for ease of description, only two scoring types, high score and low score, are described as scoring results, but the number of scoring types (the number of clusters) can be any number. For example, the scores can be classified into three levels (e.g., low score, medium score, high score), can be classified into 10 levels (e.g., 0 to 9 points), can be classified into 100 levels (e.g., 0 to 99 points), or can be classified into other levels.

[0139] The scoring result generated by the scoring process 102 is supplied to the imaging control process 103 for second 3D modeling.

[0140] <Imaging Control Processing for Second 3D Modeling>

[0141] In the imaging control process 103 for the second 3D modeling, the second imaging is controlled based on the position and orientation of the second imaging unit and the scoring result obtained by the scoring process 102. For example, control is performed so that the second imaging is performed at a position and orientation that leads to a better scoring result.

[0142] For example, suppose Figure 5The scoring result shown is obtained by the scoring process 102. The scoring result clearly shows that the lower side (eg, the gray portion) of the 3D object in the figure corresponding to the first 3D data 120 is insufficiently imaged.

[0143] Therefore, in the imaging control process 103 for the second 3D modeling, the second imaging is controlled so that the gray portion that is not adequately imaged is imaged from the lower side of the 3D object in the figure. For example, the position and orientation of the camera 121-4 are determined to be more suitable as the position and orientation for performing the second imaging, and the second imaging is controlled so that imaging is performed at the position and orientation of the camera 121-4.

[0144] By doing so, a second captured image captured at a more appropriate position and orientation can be generated. In other words, the second 3D modeling process can be performed using the second captured image captured at a more appropriate position and orientation. Therefore, more accurate 3D data can be generated while suppressing an increase in the 3D modeling load (workload and processing volume). In other words, 3D modeling can be performed more easily.

[0145] The method for obtaining the position and orientation at which the second imaging is to be performed may be any method. For example, in the imaging control process 103 for the second 3D modeling, the position and orientation (range) that enables the score of the portion (gray portion) where the second imaging is insufficient to be increased can be determined based on the scoring result. In addition, the current orientation information about the (position and orientation) of the second imaging unit can be provided to the scoring process 102 as imaging viewpoint information, and the scoring result can be obtained when the second captured image obtained at the current position and orientation is temporarily added. In the case where the score is higher than the score before the second captured image is added by more than a predetermined threshold, the current position and orientation can be determined as the position and orientation at which the second imaging is to be performed.

[0146] Note that if the positional and orientational relationship between the first imaging unit and the second imaging unit is known, the orientation information about the first imaging unit, rather than the orientation information about the second imaging unit, can be provided to the scoring process 102 as the imaging viewpoint information. In this case, in the scoring process 102, the orientation information about the second imaging unit can be used to derive the orientation information about the first imaging unit, and the orientation information about the second imaging unit can be used to generate a scoring result. Furthermore, a neural network that uses the orientation information about the first imaging unit as an input parameter can be used to generate a scoring result.

[0147] Furthermore, in the imaging control process 103 for the second 3D modeling, it is possible to determine whether the position and orientation are the position and orientation for performing the second imaging based on the overlap ratio of the imaging ranges of the second imaging performed so far. The overlap ratio indicates the degree (ratio) of the area (overlapping area) where the imaging ranges overlap. In other words, it is possible to determine whether the position and orientation for the second imaging are more appropriate based on the degree of overlap between the imaging range of the second imaging to be performed and the area captured in the second captured image obtained so far.

[0148] For example, when a method (such as photogrammetry) for performing 3D modeling based on corresponding points between a plurality of second captured images is applied as the second 3D modeling process, in order to obtain corresponding points, the imaging ranges of the plurality of second captured images need to at least partially overlap (exist an overlapping area). Therefore, with respect to the second captured images obtained to date, the position and orientation at which a second captured image with an overlapping ratio that makes the second 3D modeling process easier (allowing for a more accurate 3D modeling process) can be obtained can be determined as a more appropriate position and orientation for performing the second imaging.

[0149] Note that the overlap ratio that makes the second 3D modeling process easier (allows more accurate 3D modeling process) also depends on the three-dimensional shape of the 3D object, etc. For example, in the case of imaging using a so-called drone, the subject can be regarded as Figure 6 1 . For example, the imaging range when plane 130 is imaged from camera 131-1 is indicated by double-headed arrow 132-1. For example, the imaging range when plane 130 is imaged from camera 131-2 is indicated by double-headed arrow 132-2. Therefore, the overlapping area between these captured images is the range indicated by double-headed arrow 133. In this case, the captured images are overlapped in a simple manner, so that as long as an overlap ratio greater than or equal to a predetermined ratio can be obtained, more accurate 3D modeling processing can be performed.

[0150] However, in the case of the second imaging, the subject is a 3D object (first 3D data 135), and the subject is completely imaged so that the images overlap in a stereoscopic manner, as shown in FIG. Figure 6 1 and 136-2 in the right example. Therefore, the degree of overlap required for sufficiently accurate 3D modeling depends on the three-dimensional shape of the 3D object, etc. Therefore, when determining the position and orientation for performing the second imaging, while taking into account the overlap of the second captured images obtained so far, it is desirable to also take into account the three-dimensional shape of the 3D object, etc. (first 3D data) (so that the position and orientation for performing the second imaging can be more accurately determined).

[0151] Furthermore, when the position and orientation for performing the second imaging are determined, the distance from the imaging position to the subject (3D object) can be controlled. In other words, not only can the angle and the distance from which the part of the 3D object is imaged be controlled, but also the distance from which the part is imaged can be controlled.

[0152] As in Figure 7 In the example shown on the left, when imaging is performed at a position far from the 3D object 141 (the position indicated by the black triangle in the figure) as shown by the dotted line 142, the 3D object 141 can be completely imaged at a low imaging frequency. However, there may be cases where portions of the 3D object 141 having a complex three-dimensional shape (e.g., the shaded portion 141A) cannot be imaged. As a result, the accuracy of the second 3D modeling process (the accuracy of the second 3D data) may be reduced.

[0153] On the other hand, in Figure 7 In the example shown on the right, when imaging is performed at a position close to the 3D object 141 (the position indicated by the black triangle in the figure) as shown by the dotted line 143, Figure 7 Compared to the example on the left, the imaging frequency required to image the entire 3D object 141 increases. Figure 7 Compared to the example on the left, a portion of the 3D object 141 having a complex three-dimensional shape (eg, the shadow portion 141A, etc.) can be imaged. Figure 7 Compared with the example on the left, it is possible to more reliably image the entire 3D object 141. Therefore, it is possible to suppress a decrease in the accuracy of the second 3D modeling process (accuracy of the second 3D data).

[0154] That is, the appropriate distance from the 3D object as the position for the second imaging depends on the three-dimensional shape of the 3D object. Therefore, in the imaging control process 103 for the second 3D modeling, the distance between the position for the second imaging and the 3D object (subject) can be controlled based on (the complexity of) the three-dimensional shape of the 3D object. By doing so, as described above, it is possible to suppress an unnecessary increase in the frequency of the second imaging while suppressing a decrease in the accuracy of the second 3D modeling process (the accuracy of the second 3D data). In other words, control can be performed so that the second imaging is performed at a more appropriate position and orientation.

[0155] Note that any method can be used to derive the complexity of the 3D object's three-dimensional shape. For example, the complexity can be derived based on the first 3D data. However, in this case, for example, the first 3D data can be processed as a two-dimensional image, and the complexity of the 3D object's three-dimensional shape can be derived based on a pattern or the like. This can suppress an increase in the processing load associated with deriving the complexity of the 3D object's three-dimensional shape.

[0156] In addition, a detection frame can be set, and the complexity of the three-dimensional shape of the 3D object within the detection frame can be derived. The detection frame can have any shape or any size. For example, how many polygons in the first 3D data within the detection frame face the imaging surface of the second imaging is obtained, the degree of change in the normal direction of each polygon within the detection frame is quantified, and the complexity of the three-dimensional shape of the 3D object within the detection frame can be derived based on the degree of change. Generally speaking, the greater the change, the more complex the shape, and when facing the same direction, the shape can be regarded as close to a planar shape. In addition, the average value of the normal direction of each polygon within the detection frame can be used as a representative value of the degree of alignment with the imaging surface, and the complexity of the three-dimensional shape of the 3D object can be derived based on the representative value.

[0157] Furthermore, in the marching cubes method, in the case where there are many vertex arrangements that easily form a plane within the detection box, it can be determined that the complexity of the three-dimensional shape of the 3D object is low.

[0158] That is, as long as the complexity of the three-dimensional shape of the 3D object is a quantitative value used as a standard for estimating the direction, frequency, and distance necessary for imaging according to the outline of the subject in a certain area, the complexity of the three-dimensional shape of the 3D object can be any parameter (based on the value of any parameter). In addition, the method for controlling the distance between the second imaging and the 3D object based on the complexity of the three-dimensional shape of the 3D object can be any method. For example, when the three-dimensional shape of the 3D object is more complex, control can be performed so that the second imaging is performed at a position close to the 3D object. For example, when the three-dimensional shape of the 3D object is relatively simple, control can be performed so that the second imaging is performed at a position far from the 3D object.

[0159] In the imaging control process 103 for the second 3D modeling, as described above, the position and orientation (more appropriately, the position and orientation) at which the second imaging is to be performed is obtained, and control information (imaging control information) is generated. Based on this information, control is performed so that the second imaging is performed at this position and orientation. The imaging control information is then provided to the second 3D data generation process 104. For example, if a user or the like moves the second imaging unit and the position and orientation of the second imaging unit matches the obtained "position and orientation at which the second imaging is to be performed," imaging control information instructing the second imaging to be performed may be generated and provided to the second 3D data generation process 104 (i.e., the second imaging is to be performed at the "position and orientation at which the second imaging is to be performed").

[0160] <Second Imaging>

[0161] In the second 3D data generation process 104, the second imaging unit performs second imaging according to the control of the imaging control process 103 for the second 3D modeling to generate a second captured image. For example, the second imaging unit may perform the second imaging based on the imaging control information generated in the imaging control process 103 for the second 3D modeling. For example, the second imaging unit may perform the second imaging when the imaging control information indicates imaging (at the timing when imaging is indicated). In addition, the control unit that controls the position and orientation of the second imaging unit may move the second imaging unit to a position specified by the imaging control information and set the second imaging unit to an orientation specified by the imaging control information, and the second imaging unit may perform the second imaging at this position and orientation.

[0162] The number of the second imaging units can be any number, whether singular or plural. In addition, the first imaging unit and the second imaging unit can be a common imaging unit (the same imaging unit) or different imaging units installed at different positions.

[0163] The specifications of the second imaging unit (e.g., the number of pixels) may be the same as or different from those of the first imaging unit. For example, the second captured image may have higher image quality than the first captured image. Furthermore, the second captured image may have higher resolution than the first captured image. Furthermore, the second captured image may have higher dynamic range than the first captured image.

[0164] <Second 3D Modeling Process>

[0165] Furthermore, any method may be used for the second 3D modeling process executed in the second 3D data generating process 104. For example, the method for the second 3D modeling process may be the same as or different from the method for the first 3D modeling process.

[0166] For example, the above-mentioned photogrammetry can be applied as the second 3D modeling process. That is, in the second 3D modeling process, SfM and MVS can be applied, and a point cloud can be generated based on multiple second captured images. In addition, as post-processing, meshing and texturing can be performed on the point cloud to generate second 3D data. That is, the second 3D data can be any data as long as the second 3D data represents the three-dimensional shape of the 3D object; specifically, the first 3D data can be a point cloud, or can include a mesh representing the three-dimensional shape of the 3D object through vertices and connections and a texture applied to the surface of the mesh. In addition, the above-mentioned neural rendering can be applied as the second 3D modeling process.

[0167] For example, in addition to the second captured image, the second 3D data may be generated using orientation information about the second imaging unit that performs the second imaging (orientation information corresponding to the viewing angle of the second captured image obtained so far). This orientation information is information indicating the position and orientation of the second imaging unit in three-dimensional space.

[0168] Furthermore, if the positional and orientational relationship between the first imaging unit and the second imaging unit performing the first imaging is known, the second 3D data can be generated using orientation information about the first imaging unit (position and orientation in three-dimensional space). In other words, the second 3D data can be generated using the orientation information derived in the first 3D modeling process. For example, the orientation information about the first imaging unit can be used to derive the orientation information about the second imaging unit, and the second 3D data can be generated using the orientation information about the second imaging unit. Furthermore, the second 3D data can be generated using a neural network that takes the orientation information about the first imaging unit and the second captured image as input.

[0169] Furthermore, the second 3D data may be encoded using any encoding method.

[0170] <Manual Imaging>

[0171] In addition, if Figure 4 As shown, in the second 3D data generation process 104, the second imaging can be performed without relying on the imaging control information (for example, manually). In this article, this imaging method is also referred to as manual imaging. In the case of performing manual imaging, imaging timing information indicating the imaging timing is generated in the second 3D data generation process 104 (the second imaging), and the imaging timing information is provided to the imaging control process 103 for the second 3D modeling. Then, in the imaging control process 103 for the second 3D modeling, the orientation information about the second imaging unit at the imaging timing is obtained based on the imaging timing information, and the orientation information about the second imaging unit at the imaging timing is provided to the scoring process 102 as the imaging viewpoint information. Then, in the scoring process 102, the score is calculated based on the imaging viewpoint information. As described above, the second captured image obtained by manual imaging (the orientation information about the second imaging unit corresponding to the viewing angle) can be reflected in the scoring process 102 (the scoring result obtained).

[0172] <Camera Information>

[0173] In addition, if Figure 4As shown, in the second 3D data generation process 104 (second imaging), camera information about the second imaging unit can be generated, and the camera information can be provided to the scoring process 102. Then, in the scoring process 102, scoring can be performed based on the camera information, and a scoring result can be generated. The camera information can include any information. For example, the camera information can include internal parameters of the imaging unit. The camera information can also include external parameters of the imaging unit. The camera information can also include a captured image. The camera information can also include viewing angle information (focal length information) about the second captured image. The camera information can also include distortion correction information. The camera information can also include shading correction information. The camera information can also include breathing correction information. The camera information can also include focus position information. The camera information can also include image plane phase difference information. That is, this information can be used for scoring (evaluating the accuracy of the second three-dimensional shape information that can be generated).

[0174] <Real-time processing>

[0175] Notice, Figure 4 The first 3D data generation process 101 (first imaging and first 3D modeling process), the scoring process 102, and the imaging control process 103 for second 3D modeling in the 3D modeling process can be executed in parallel.

[0176] For example, in the first 3D data generation process 101, first 3D data of a portion of a 3D object (the 3D object is a subject) that has been subjected to the first imaging can be sequentially generated. For example, based on the captured image, depth information, etc., 3D data can be generated instantly (in real time) by applying real-time 3D modeling as the first 3D modeling process. That is, in this case, while the first imaging is being performed (while the first captured image is being obtained), the first 3D modeling can be performed to generate the first 3D data. For example, as shown in FIG. Figure 2 As described above, while the camera moves around the 3D object, each portion of the 3D object as the subject is imaged. However, before obtaining a captured image of the entire 3D object, 3D modeling can be performed based on the obtained captured image and depth. In other words, 3D data of the imaged portions can be sequentially generated.

[0177] Furthermore, in the scoring process 102, scoring of the first 3D data corresponding to a portion of the 3D object can be performed (evaluating the accuracy of the second three-dimensional shape information that can be generated using the second captured image generated by the second imaging performed so far). That is, each time first 3D data corresponding to a portion of the 3D object is generated by the first 3D modeling process (before first 3D data for the entire 3D object is generated), scoring can be sequentially performed on the portion of the 3D object from which the first 3D data has been generated (evaluating the accuracy of the second 3D data that can be generated). By doing so, the scoring process 102 can be started before the first 3D data generation process 101 ends (before first 3D data for the entire 3D object is generated). That is, the first 3D data generation process 101 and the scoring process 102 can be performed in parallel.

[0178] Furthermore, in the imaging control process 103 for the second 3D modeling, each time a scoring result is obtained by the scoring process 102 (before a scoring result for the entire 3D object is obtained), the second imaging can be controlled based on the obtained scoring result (the scoring result of the first 3D data corresponding to a portion of the 3D object). By doing so, the imaging control process 103 for the second 3D modeling can be started before the scoring process 102 is completed (before a scoring result for the entire 3D object is obtained). In other words, the scoring process 102 and the imaging control process 103 for the second 3D modeling can be executed in parallel.

[0179] By combining the above-described methods, the first 3D data generation process 101 , the scoring process 102 , and the imaging control process 103 for the second 3D modeling can be performed in parallel.

[0180] For example, in Figure 8 , it is assumed that the time axis extends from left to right in the figure, as indicated by the arrow. By performing the first imaging and the first 3D modeling process in parallel in the first 3D data generation process 101, the first 3D data of the portion that has undergone the first imaging, such as the first 3D data 151-1, the first 3D data 151-2, and the first 3D data 151-3, can be sequentially generated. In addition, by performing the first 3D data generation process (the first 3D modeling process) and the scoring process 102 in parallel, the scoring results of the portion of the first 3D data that has been sequentially generated therefrom, such as the scoring result 152-1, the scoring result 152-2, and the scoring result 152-3, can be obtained. In addition, by performing the scoring process 102 and the imaging control process 103 for the second 3D modeling in parallel, the second imaging can be controlled at each timing based on the scoring results (scoring result 152-1, scoring result 152-2, scoring result 152-3) obtained so far.

[0181] That is, by executing the first 3D data generation process 101, the scoring process 102, and the imaging control process 103 for the second 3D modeling in parallel, the second imaging can be controlled while executing the first imaging. That is, the first imaging and the second imaging can be executed in parallel (in real time).

[0182] <Rating>

[0183] This scoring method will be described. Examples of conditions for successful operation of photogrammetry include: ensuring that SfM is successfully operated, ensuring that MVS is successfully operated, and ensuring that texturing (texture mapping) is successfully operated. Examples of conditions for successful operation of SfM include: ensuring that a baseline can be ensured, ensuring that feature points can be matched, etc. In addition, examples of conditions for successful operation of MVS include: ensuring that a baseline can be ensured. Examples of conditions for successful operation of texturing include: ensuring that high-definition textures can be obtained from captured images, and ensuring that the surface to which the texture is applied is imaged from the front as directly as possible. The baseline indicates the distance between the imaging viewpoint positions (the position of the camera during imaging).

[0184] <Polygon Rating>

[0185] Examples of conditions under which a polygonal surface can be recovered by SfM or MVS include: minimum visibility conditions (whether the polygon is visible from the imaging position), favorable conditions for accuracy (conditions under which accuracy is improved), and favorable conditions for matching (corresponding point detection) (conditions under which matching becomes easier).

[0186] Examples of minimum visibility conditions include: ensuring that the centroid of the target polygon falls within the field of view (within the imaging perspective) when viewed from the viewpoint (imaging position); ensuring that the dot product of the target polygon's normal and the line of sight (the vector from the line of sight towards the centroid of the target polygon) is at least positive; that no other polygons block the line of sight; and that there are two or more (visible) lines of sight to the target polygon.

[0187] For example, in Figure 9 In this case, there exists a line of sight 162 from a certain viewpoint toward the centroid of target polygon 160, such that target polygon 160 is within the field of view. Furthermore, the dot product of normal 161 of target polygon 160 and line of sight 162 is positive. Furthermore, line of sight 162 reaches target polygon 160 without being blocked by other polygons and is a line of sight that allows target polygon 160 to be seen. Therefore, line of sight 162 satisfies the minimum visibility condition.

[0188] On the other hand, the line of sight 163 is not a “line of sight that can see the target polygon 160 ” because the line of sight 163 is blocked by the polygon 164 .

[0189] In addition, examples of favorable conditions for accuracy include: ensuring that the baseline is long enough, ensuring that the ratio of the length of the baseline to the distance to the subject (length of the baseline / distance to the subject) is large enough, and ensuring that there are a sufficient number of visible viewpoints and the variation in the angles formed between them is large.

[0190] For example, in Figure 10 In this case, examples of conditions for improving accuracy include: ensuring that the baseline 173 between the viewpoint 171 and the viewpoint 172 that can see the target polygon 170 is long enough (the viewpoint 171 and the viewpoint 172 are sufficiently separated); ensuring that the ratio of the length of the baseline 173 to the distance 174 to the subject is large enough (the value of "baseline length / distance to the subject" is large enough), etc.

[0191] In addition, Figure 11 In the case of the left example, the viewpoints from which the target polygon 180 can be seen include two points: viewpoint 181 and viewpoint 182. Figure 11 In the example on the right, the viewpoints that can see target polygon 180 include six points, viewpoints 181 through 186. This means that the example on the right has more visible viewpoints than the example on the left, resulting in a greater variation in the angles formed between viewpoints. This large number of visible points makes triangulation more robust than using multiple pieces of information, thereby improving accuracy. Therefore, the example on the right better meets the conditions for improved accuracy than the example on the left.

[0192] Examples of favorable conditions for matching include: ensuring that the angle formed between the normal of the target polygon and the line of sight extending from the viewpoint toward the centroid of the target polygon is sufficiently small; ensuring that the ratio of the distances from paired viewpoints to the subject is sufficiently small; and ensuring that there is a texture that can be matched.

[0193] exist Figure 12 In the case of the left example, the angle formed between the normal line 191 of the target polygon 190 and the viewpoint 192 is smaller than the angle formed between the normal line 191 and the viewpoint 193. Therefore, the viewpoint 192 enables more accurate detection of feature points on the surface of the target polygon 190 than the viewpoint 193. Figure 12 In the case of the example on the right side of FIG, the distance from viewpoint 194 to the subject (target polygon 190) is significantly longer than the distance from viewpoint 195 to the subject (target polygon 190). That is, the ratio of the distances from viewpoint 194 to the subject is large compared to the distances from viewpoint 195 to the subject. In this case, even if the baseline is long, the appearance of the feature points on the surface of the target polygon 190 will be significantly different between different viewpoints, so that the difficulty of matching may increase. In other words, the smaller the ratio of the distances from the viewpoint to the subject, for example Figure 12 The closer the viewpoint 192 and the viewpoint 193 in the left example are, the easier it is to match them.

[0194] <Texture Rating>

[0195] Examples of conditions for determining whether a polygonal surface has a sufficient number of viewpoints for textured include a minimum condition (whether the polygonal surface can be seen) and a favorable condition for textured (a condition for achieving clearer texture).

[0196] An example of a minimum condition includes ensuring that a viewpoint exists that satisfies the minimum visibility condition described above.

[0197] In addition, examples of favorable conditions for texturing include: ensuring that the angle formed between the normal of the target polygon and the line of sight extending from the viewpoint toward the centroid of the target polygon is small; and ensuring that sufficient resolution can be obtained when the distance from the viewpoint to the subject is less than or equal to a certain limit.

[0198] Note that each of the above conditions is an example. Any condition can be applied to the rating. Furthermore, the content can be of any type. For example, the above conditions can be omitted, or additional conditions can be added.

[0199] <Rating of Second Captured Image>

[0200] The second captured image obtained by the second imaging process can be scored. For example, the second captured image can be scored based on camera device information. For example, the second captured image can be evaluated to determine whether it is focused on the desired location. Furthermore, the camera can be evaluated to determine whether the image is shaken. Furthermore, the exposure can be evaluated to determine whether feature points are easily acquired.

[0201] <Example of Rating Calculation>

[0202] For example, Figure 13 As shown, the distance between the target captured image 201 and the target polygon 202 is represented as d. In addition, the ideal distance to the subject is represented as d opt In addition, c d In this case, the score s d It can be obtained as shown in the following equation (1).

[0203] [Mathematical formula 1]

[0204]

[0205] In addition, the center of the target polygon 202 is represented by c p From the target captured image 201 to the center c p The line of sight is represented by v p In addition, the normal of the target polygon 202 is represented by n p Then, the sight v p With normal np The angle between is denoted as α. The angle α formed in this case can be obtained as shown in the following equation (2). Then, the score s based on the angle α is α It can be obtained as the following equation (3). Note that c α Indicates the predetermined coefficient.

[0206] [Mathematical formula 2]

[0207]

[0208]

[0209] [Mathematical formula 3]

[0210]

[0211] The optical axis of the camera (the normal vector of the capture target image starting from the center of the capture target image 201) is represented as v c In addition, the optical axis v c With sight v p The angle formed between is denoted as β. The angle β formed in this case can be obtained as shown in the following equation (4). Then, the score s based on the angle β is β It can be obtained as the following equation (5). Note that c β Indicates the predetermined coefficient.

[0212] [Formula 4]

[0213]

[0214] [Formula 5]

[0215]

[0216] Total ratings total The scores derived as described above can be used d 、s α and s β This is obtained as shown in the following equation (6).

[0217] [Formula 6]

[0218] S total =S d ·S α ·S β ...(6)

[0220] Then, the weighted sum of the total scores of the first two viewpoints among all viewpoints derived as described above may be used as the final score.

[0221] Note that this calculation method is an example, and the calculation method applied to the scoring process 102 may be any method, and is not limited to this example.

[0222] <Example of Configuration for Execution Processing>

[0223] above Figure 4 Each process in can be executed by any device. For example, in an information processing device, the first 3D modeling process of the first 3D data generation process 101, the scoring process 102, and the imaging control process 103 for the second 3D modeling can be executed.

[0224] That is, the information processing device may include: a first 3D modeling processing unit that generates first three-dimensional shape information representing the three-dimensional shape of a 3D object based on a first captured image generated by first imaging of the 3D object; a scoring processing unit that uses the first three-dimensional shape information to evaluate the accuracy of second three-dimensional shape information that can be generated using a second captured image generated by second imaging performed so far, and generates a scoring result; and an imaging control unit that controls the second imaging of the 3D object based on the scoring result. In this section, the information processing device is also referred to as the first information processing device.

[0225] In addition, the information processing method performed by the first information processing device may include: generating first three-dimensional shape information representing the three-dimensional shape of a 3D object based on a first captured image generated by first imaging of the 3D object; using the first three-dimensional shape information to evaluate the accuracy of second three-dimensional shape information that can be generated using a second captured image generated by second imaging performed so far, and generating a scoring result; and controlling the second imaging of the 3D object based on the scoring result.

[0226] By doing so, it is possible to image the 3D object in a more appropriate position and orientation (performing second imaging), and to perform a second 3D modeling process using the obtained second captured image. Consequently, it is possible to generate more accurate 3D data while suppressing an increase in the 3D modeling load (workload or processing volume). In other words, 3D modeling can be performed more easily.

[0227] In addition, the first 3D modeling processing unit may include: an orientation information generation unit, which generates orientation information indicating the position and orientation of the first imaging unit based on the first captured image and the acceleration and angular velocity of the first imaging unit; and a three-dimensional shape generation unit, which generates first three-dimensional shape information about the 3D object based on the orientation information and the depth of the 3D object.

[0228] Furthermore, the first information processing device may further perform the first imaging of the first 3D data generation process 101. For example, the first information processing device may further include a first imaging unit. Furthermore, the first information processing device including the first imaging unit may include a depth detection unit for detecting depth, an inertial measurement unit for detecting acceleration and angular velocity of the first imaging unit, or both.

[0229] In addition, the first information processing apparatus may further perform the second imaging of the second 3D data generation process 104. For example, the first information processing apparatus may further include a second imaging unit.

[0230] Note that the second captured image generated by the second imaging process may be encoded. For example, a first information processing device including a second imaging unit may include an encoding unit that encodes the second captured image generated by the second imaging unit. The encoded second captured image may be provided to another information processing device via communication or may be stored in a storage medium.

[0231] In addition, in the first information processing device, a second 3D modeling process of the above-mentioned second 3D data generation process 104 can be further performed. For example, the first information processing device including the second imaging unit can also include a second 3D modeling processing unit, which generates second three-dimensional shape information based on the second captured image generated by the second imaging unit. For example, the second 3D modeling processing unit may include: a corresponding point position deriving unit, which derives the three-dimensional position of each corresponding point between the multiple second captured images; and a three-dimensional point adding unit, which adds three-dimensional points based on the three-dimensional positions of the corresponding points. In the second 3D modeling process, meshing and texturing can be further performed as post-processing. For example, the second three-dimensional shape information may include a mesh representing the three-dimensional shape of the 3D object through vertices and connections, and a texture applied to the surface of the mesh.

[0232] Note that the second 3D data generated by the second 3D modeling process may be encoded. For example, the first information processing device including the second imaging unit and the second 3D modeling processing unit may further include an encoding unit that encodes the second three-dimensional shape information generated by the second 3D modeling processing unit. The encoded second three-dimensional shape information (second 3D data) may be provided to another information processing device via communication or may be stored in a storage medium.

[0233] Note that the second imaging of the second 3D data generation process 104 described above may be performed in a second information processing device that is different from the first information processing device. For example, the first information processing device may include a communication unit that communicates with the second information processing device (imaging device) that includes the second imaging unit, the imaging control unit may generate imaging control information based on which the second imaging is controlled, and the communication unit may provide the imaging control information to the second information processing device.

[0234] Furthermore, in this case, the first information processing device may acquire a second captured image generated by the second information processing device. For example, the first information processing device including a communication unit may acquire a second captured image provided by the second information processing device. This second captured image may be encoded. For example, the first information processing device including a communication unit may include an encoding unit that encodes the second captured image acquired by the communication unit. The encoded second captured image may be provided to another information processing device via communication or may be stored in a storage medium.

[0235] Furthermore, the second captured image provided by the second information processing device may be encoded. That is, the communication unit may acquire encoded data of the second captured image. The encoded data may then be provided to another information processing device via communication or stored in a storage medium. Furthermore, the first information processing device may decode the encoded data acquired by the communication unit to generate (restore) the second captured image. For example, the first information processing device including the communication unit may include a decoding unit that decodes the encoded data of the second captured image acquired by the communication unit.

[0236] Even in the case where the second imaging is performed in the second information processing device as described above, the second 3D modeling process of the second 3D data generation process 104 described above may be further performed in the first information processing device. For example, the first information processing device including the communication unit may further include a second 3D modeling processing unit that generates second three-dimensional shape information based on the second captured image acquired by the communication unit. For example, the second 3D modeling processing unit may include: a corresponding point position deriving unit that derives the three-dimensional position of each corresponding point between the plurality of second captured images; and a three-dimensional point adding unit that adds three-dimensional points based on the three-dimensional positions of the corresponding points. In the second 3D modeling process, meshing and texturing may be further performed as post-processing. For example, the second three-dimensional shape information may include a mesh that represents the three-dimensional shape of the 3D object through vertices and connections, and a texture applied to the surface of the mesh.

[0237] Note that the second 3D data generated by the second 3D modeling process can be provided to another information processing device via communication, or can be stored in a storage medium. Furthermore, the second 3D data can be encoded. For example, a first information processing device including a communication unit and a second 3D modeling processing unit can further include an encoding unit that encodes the second three-dimensional shape information generated by the second 3D modeling processing unit. The encoded data of the generated second three-dimensional shape information (second 3D data) can then be provided to another information processing device via communication, or can be stored in a storage medium.

[0238] Incidentally, as described above, the second imaging can be performed by manual imaging. In this case, the second captured image obtained by manual imaging can be used in the second 3D modeling process. In the scoring process 102, as described above, the accuracy of the second three-dimensional shape information that can be generated using the second captured image obtained so far is evaluated. At this time, the second captured image can include the second captured image obtained by manual imaging. That is, the orientation information about the manual imaging can be reflected in the scoring process 102. For example, the scoring processing unit of the first information processing device can generate a scoring result based on the position and orientation of the second information processing device corresponding to the second imaging timing, without relying on the imaging control information, and the second imaging timing is indicated by the imaging timing information indicating the second imaging timing. For example, the imaging control unit can obtain orientation information about the second imaging unit at the imaging timing based on the imaging timing information, and the scoring processing unit can calculate the score based on the orientation information. By doing so, the orientation information about the manual imaging is reflected in the scoring result.

[0239] Note that in this case, the second imaging (manual imaging) can be performed by either the first information processing device or the second information processing device. If the first information processing device includes a second imaging unit, for example, when performing manual imaging, the second imaging unit can generate imaging timing information indicating the timing and provide the imaging timing information to the imaging control unit. Furthermore, if the first information processing device includes a communication unit, for example, the communication unit can acquire the imaging timing information provided by the second information processing device and provide the imaging timing information to the imaging control unit.

[0240] By doing so, control can be performed so that the second imaging is performed at a more appropriate position and orientation based on the imaging timing information.

[0241] Incidentally, in the first information processing device, as described above, the camera information about the second imaging unit can be reflected in the scoring process 102. For example, the scoring processing unit of the first information processing device can generate a scoring result based on the camera information. Note that in this case, the second imaging can be performed in the first information processing device or in the second information processing device. For example, in the case where the first information processing device includes the second imaging unit, the second imaging unit can generate camera information and provide the camera information to the scoring processing unit. In addition, in the case where the first information processing device includes a communication unit, for example, the communication unit can obtain the camera information provided from the second information processing device and provide the camera information to the scoring processing unit.

[0242] By doing so, control can be performed so that the second imaging is performed at a more appropriate position and orientation based on the camera information.

[0243] Incidentally, the second information processing device may perform the second imaging of the second 3D data generation process 104 described above. For example, the second information processing device may include a second imaging unit and a communication unit for communicating with the first information processing device. The communication unit may acquire imaging control information provided by the first information processing device, and the second imaging unit may image the 3D object based on the imaging control information to generate the second captured image. The imaging control information is information for controlling the second imaging, and is generated based on the scoring result obtained based on the first 3D data.

[0244] Furthermore, in the information processing method performed by the second information processing apparatus, imaging control information provided from the first information processing apparatus may be acquired, second imaging may be performed based on the imaging control information, and a second captured image for generating second 3D data may be generated.

[0245] By doing so, it is possible to image the 3D object in a more appropriate position and orientation (performing second imaging), and to perform a second 3D modeling process using the obtained second captured image. Consequently, it is possible to generate more accurate 3D data while suppressing an increase in the 3D modeling load (workload or processing volume). In other words, 3D modeling can be performed more easily.

[0246] The generated second captured image can be provided to the first information processing device. For example, the communication unit can provide the second captured image generated by the second imaging unit to the first information processing device. The second captured image is a captured image used to generate three-dimensional shape information representing the three-dimensional shape of the 3D object. In addition, the second captured image can be encoded. For example, the second information processing device may include an encoding unit that encodes the second captured image generated by the second imaging unit. Then, the communication unit can provide the encoded data of the second captured image generated by the encoding unit to the first information processing device. Note that the second captured image (or the encoded data of the second captured image) can be provided to an information processing device other than the first information processing device. For example, the communication unit can provide the second captured image (or the encoded data of the second captured image) to another information processing device. In addition, the second captured image (or the encoded data of the second captured image) can be stored in a storage medium. For example, the second information processing device may include a storage unit that stores the encoded data of the second captured image generated by the encoding unit.

[0247] In addition, the second information processing device can perform the above-mentioned second 3D modeling process. That is, in the second information processing device, a second 3D modeling process can be performed using the second captured image generated by the second imaging to generate second 3D data. For example, the second information processing device may further include a second 3D modeling processing unit, which generates second three-dimensional shape information (second 3D data) representing the three-dimensional shape of the 3D object based on the second captured image generated by the second imaging unit. For example, the second 3D modeling processing unit may include: a corresponding point position deriving unit, which derives the three-dimensional position of each corresponding point between multiple second captured images; and a three-dimensional point adding unit, which adds three-dimensional points based on the three-dimensional positions of the corresponding points. In the second 3D modeling process, meshing and texturing can be further performed as post-processing. For example, the second three-dimensional shape information may include a mesh representing the three-dimensional shape of the 3D object through vertices and connections and a texture applied to the surface of the mesh.

[0248] Note that the second 3D data generated by the second 3D modeling process can be provided to another information processing device via communication, or can be stored in a storage medium. Furthermore, the second 3D data can be encoded. For example, the second information processing device including the second 3D modeling processing unit can further include an encoding unit that encodes the second three-dimensional shape information generated by the second 3D modeling processing unit. The encoded data of the generated second three-dimensional shape information (second 3D data) can then be provided to another information processing device via communication, or can be stored in a storage medium.

[0249] Incidentally, as described above, the second imaging can be performed using manual imaging. In this case, the second captured image obtained through manual imaging can be used in the second 3D modeling process. In the scoring process 102, as described above, the accuracy of the second three-dimensional shape information that can be generated using the second captured image obtained so far is evaluated. In this case, the second captured image can include the second captured image obtained through manual imaging. In other words, the orientation information regarding the manual imaging can be reflected in the scoring process 102.

[0250] In this case, imaging timing information indicating the timing of manual imaging may be generated in the second information processing device and provided to the first information processing device. For example, the second imaging unit of the second information processing device may generate imaging timing information indicating the timing when manual imaging is performed, and the communication unit may provide the imaging timing information to the first information processing device.

[0251] By doing so, the 3D object can be imaged (second imaging is performed) at a more appropriate position and orientation based on the imaging timing information.

[0252] Incidentally, as described above, the camera information regarding the second imaging unit can be reflected in the scoring process 102. For example, the second imaging unit of the second information processing device can generate the camera information, and the communication unit can provide the camera information to the first information processing device. In this case, the communication unit can obtain imaging control information generated based on the camera information, and the second imaging unit can perform the second imaging based on the imaging control information. In addition, in the information processing method performed by the second information processing device, the camera information regarding the second imaging unit can be generated, and the camera information can be provided to the first information processing device. In addition, the imaging control information generated based on the camera information can be obtained, and the second imaging can be performed based on the imaging control information.

[0253] By doing so, it is possible to image the 3D object in a more appropriate position and orientation based on the camera information (perform second imaging).

[0254] <3. Imaging-Guided Output>

[0255] <Imaging-Guided Output Processing for Second 3D Modeling>

[0256] Furthermore, guidance information for assisting imaging for the second 3D modeling may be output instead of controlling imaging for the second 3D modeling. Figure 4In the embodiment, after executing the first 3D data generation process 101 and the scoring process 102, the imaging guidance output process 105 for the second 3D modeling can also be executed. In this case, the first 3D data generation process 101 and the scoring process 102 are executed similarly to the case described above in <2. Imaging Control>. However, the scoring process 102 provides the scoring results to the imaging guidance output process 105 for the second 3D modeling.

[0257] In the imaging guidance output process 105 for second 3D modeling, guidance information for second imaging is generated based on the scoring result obtained by the scoring process 102 , and output of the guidance information is controlled and output by an output device.

[0258] The user or the like manually performs the second imaging based on such guidance information. That is, in this case, the second imaging is manual imaging (imaging that does not rely on imaging control information). By performing the second imaging in this manner, a second captured image captured at a more appropriate position and orientation can be generated. Then, the second 3D data generation process 104 (second imaging and second 3D modeling process) is performed using the second captured image to generate the desired second 3D data. In other words, the second 3D modeling process can be performed using the second captured image captured at a more appropriate position and orientation. Therefore, more accurate 3D data can be generated while suppressing the increase in the load (workload or processing volume) of 3D modeling. That is, 3D modeling can be performed more easily.

[0259] <Generation of guidance information>

[0260] To generate this guidance information, in the imaging guidance output process 105 for the second 3D modeling, the position and orientation at which the second imaging is to be performed (a more appropriate position and orientation as the position and orientation at which the second imaging is to be performed) are obtained based on the scoring results. The method for obtaining such a position and orientation at which the second imaging is to be performed may be any method. For example, the method may be similar to the method of the imaging control process 103 for the second 3D modeling described above. For example, based on the scoring results, the position and orientation (range) that enables the score of the portion (gray portion) where the second imaging is insufficient can be determined.

[0261] In addition, in the imaging guidance output process 105 for the second 3D modeling, it is possible to determine whether the current position and orientation are the position and orientation at which the second imaging is to be performed based on a change in the scoring result based on the current orientation information about the second imaging unit (the position and orientation). For example, as a result of reflecting in the scoring the second captured image (the orientation information about the second captured image) obtained when the second imaging unit performs the second imaging at the current position and orientation, if the score is higher than the score before the second captured image is added by a predetermined threshold or more, the current position and orientation can be determined as the position and orientation at which the second imaging is to be performed. That is, in this case, in the scoring process 102, a scoring result is derived and compared between a case where the second imaging performed by the second imaging unit at the current position and orientation is included in the "second imaging performed so far" and a case where the second imaging is not included in the "second imaging performed so far". Therefore, in this case, in the scoring process 102, scoring is performed based on the current orientation information (imaging viewpoint information) about the second imaging unit.

[0262] This imaging viewpoint information can be provided by the imaging guidance output process 105 for the second 3D modeling. As described above, in this case, the imaging guidance output process 105 for the second 3D modeling is executed, and the second imaging is performed manually. Therefore, similar to the case described above in <2. Imaging Control>, imaging timing information indicating the imaging timing can be generated in the second 3D data generation process 104 (second imaging), and this imaging timing information can be provided to the imaging guidance output process 105 for the second 3D modeling. Then, in the imaging guidance output process 105 for the second 3D modeling, based on the imaging timing information, orientation information about the second imaging unit at the imaging timing can be obtained, and the orientation information about the second imaging unit at the imaging timing can be provided to the scoring process 102 as imaging viewpoint information.

[0263] Note that if the positional and orientational relationship between the first imaging unit and the second imaging unit is known, orientation information about the first imaging unit may be provided to the scoring process 102 as imaging viewpoint information instead of orientation information about the second imaging unit.

[0264] Furthermore, in the imaging guide output process 105 for the second 3D modeling, whether the position and orientation are the position and orientation at which the second imaging is to be performed may be determined based on the overlap rate with the imaging range of the second imaging performed so far.

[0265] Note that how much overlap makes the second 3D modeling process easier (allows for more accurate 3D modeling processing) depends on the three-dimensional shape of the 3D object, etc. Therefore, when obtaining the position and orientation at which the second imaging is to be performed while taking into account the overlap ratio with respect to the second captured images obtained thus far, it is desirable to also take into account the three-dimensional shape of the 3D object (first 3D data) (so that the position and orientation at which the second imaging is to be performed can be obtained more accurately).

[0266] In addition, as mentioned above Figure 7 As described above, when determining the position and orientation for performing the second imaging, the distance from the subject (3D object) to the imaging position can be controlled. In this case, the distance can be controlled based on the three-dimensional shape (complexity) of the 3D object. By doing so, it is possible to suppress a decrease in the accuracy of the second 3D modeling process (the accuracy of the second 3D data) while also suppressing an unnecessary increase in the frequency of the second imaging. In other words, control can be performed so that the second imaging is performed at a more appropriate position and orientation.

[0267] Then, in the imaging guidance output process 105 for second 3D modeling, guidance information is generated based on the position and orientation at which the second imaging is to be performed obtained as described above. The guidance information may be any type of information and may include, for example, image information or audio information.

[0268] Furthermore, output of the guide information is performed so that the content of the guide information is presented to, for example, the user who performs the second imaging, etc. The output device may be any device and may include, for example, a monitor that displays image information or a speaker that outputs audio information.

[0269] <Boot Information>

[0270] Next, the content of the guidance information will be described. The content of the guidance information can be of any type. For example, the guidance information can include information indicating to the user a more appropriate position and orientation for the second imaging.

[0271] For example, suppose Figure 5 The first 3D data 120 is shown by Figure 4 1. Next, assume that second imaging is performed on the 3D object corresponding to the first 3D data 120 using the positions and orientations of the cameras 121-1 to 121-3. In this case, in the scoring process 102, the upper side of the first 3D data 120 in the figure is evaluated with a relatively high score, while the lower side (gray portion) of the first 3D data 120 in the figure is evaluated with a relatively low score. This scoring result clearly indicates that the lower side (e.g., the gray portion) of the 3D object in the figure corresponding to the first 3D data 120 is insufficiently imaged.

[0272] Therefore, in the imaging guidance output process 105 for the second 3D modeling, guidance information for guiding the second imaging is generated and output so that a captured image of the under-imaged gray portion can be obtained. Specifically, this guidance information guides the second imaging so that the 3D object is imaged from the lower side in the image. For example, the position and orientation of the camera 121-4 are determined to be more suitable for performing the second imaging, and this determination is notified to the user, etc.

[0273] By doing so, the user can perform a second imaging operation based on the guidance information, thereby imaging the 3D object in a more appropriate position and orientation. In other words, 3D modeling can be performed using the captured image (second 3D data generation process 104). Consequently, more accurate 3D data can be generated while suppressing an increase in the 3D modeling load (workload or processing volume). In other words, 3D modeling can be performed more easily.

[0274] <Display of rating results>

[0275] Note that the guidance information may include information indicating the scoring result. That is, in the imaging guidance output process 105 for the second 3D modeling, guidance information including information indicating the scoring result may be generated, and an image indicating the scoring result may be displayed on the monitor as the guidance information. In addition, the guidance information may include information indicating the scoring result within the current viewing angle of the second imaging unit. That is, in the imaging guidance output process 105 for the second 3D modeling, guidance information including information indicating the scoring result within the viewing angle of the second imaging unit may be generated based on the current position and orientation of the second imaging unit, and an image indicating the scoring result may be displayed on the monitor as the guidance information.

[0276] For example, Figure 14 As shown, it is assumed that the second imaging unit is located at the position of the camera device 211 and is oriented to image a portion of the scored first 3D data 210 surrounded by the dotted frame 212. In this case, in the imaging guidance output process 105 for the second 3D modeling, an image indicating the scoring results within the current viewing angle (imaging range) of the second imaging unit (in other words, an image indicating the portion of the first 3D data 210 surrounded by the dotted frame 212) can be displayed on the monitor as guidance information, such as image 213. By doing so, the scoring results can be displayed in a state based on the current position and orientation of the second imaging unit. Therefore, the user can more easily determine a position and orientation suitable for the second imaging.

[0277] In addition, the guidance information (an image indicating the scoring result within the current viewing angle of the second imaging unit) may be displayed superimposed on the captured image generated by the second imaging unit. Figure 14The image 213 shown (an image indicating the scoring result within the current viewing angle of the second imaging unit) can be superimposed on the captured image generated by the second imaging unit with the current viewing angle for display. By doing so, the captured image and the guidance information (the image indicating the scoring result) with the same viewing angle can be superimposed to be displayed on the monitor. Based on such a display, the user can more easily make the 3D object in the real space correspond to the scoring result. Therefore, the user can more easily determine the position and orientation suitable for the second imaging. In addition, a bird's-eye view image indicating the scoring result of the entire 3D object can be displayed. By displaying such a bird's-eye view image, the user can more easily determine which part of the entire 3D object corresponds to the part included in the captured image of the 3D object currently displayed.

[0278] <Display of overlapping areas>

[0279] In addition, the guidance information may include information indicating an overlapping area where the respective imaging ranges of the plurality of second captured images overlap. For example, in the imaging guidance output process 105 for the second 3D modeling, guidance information including information indicating an overlapping area where the respective imaging ranges of the plurality of second images overlap may be generated, and an image indicating the overlapping area may be displayed as the guidance information. For example, when the second imaging unit is in Figure 15 In the case of the position and orientation of the camera 221-1 on the left, its imaging range is assumed to be the imaging range 222-1. In addition, in the case of the second imaging unit being at the position and orientation of the camera 221-2, its imaging range is assumed to be the imaging range 222-2. In this case, the imaging range 222-1 and the imaging range 222-2 overlap with each other. When there is such an area where the respective imaging ranges of a plurality of second captured images overlap, corresponding points between the two images can be detected. That is, when there is an appropriate overlapping area between the plurality of second captured images, accurate second 3D data can be generated in the second 3D modeling process (suppressing a reduction in the accuracy of the second 3D data).

[0280] Therefore, it is desirable to generate a second captured image (perform second imaging) such that an appropriate overlapping region is formed between the plurality of second captured images. As described above, since an image indicating such an overlapping region is displayed on the monitor as guidance information, a user operating the second imaging unit can determine the position and orientation of the second imaging based on the guidance information while taking the overlapping region into consideration. In other words, the user can more easily perform second imaging at a position and orientation that forms an appropriate overlapping region between the plurality of second captured images. In other words, the user can more easily perform second imaging at an appropriate position and orientation.

[0281] Note that the image indicating the overlapping area can indicate the overlapping area in any manner. For example, the overlapping area can be indicated by color, density, pattern, design, letters, symbols, numbers, etc. For example, the overlapping area can be highlighted compared to other areas (subjectively more obvious than other areas).

[0282] In addition, the overlapping area may be an overlapping area between the current viewing angle of the second imaging unit and the imaging range of the second captured image obtained so far. That is, an image indicating the overlapping area between the second captured image obtained so far and the second captured image to be generated may be displayed as guidance information. For example, in the imaging guidance output processing 105 for the second 3D modeling, guidance information including information indicating the overlapping area between the viewing angle of the second imaging unit and the imaging range of the second captured image obtained so far may be generated based on the current position and orientation of the second imaging unit, and an image indicating the overlapping area may be displayed on the monitor as guidance information. For example, in Figure 15 In the embodiment, it is assumed that the second imaging unit is at the position of the camera 221-2 and is oriented to image the imaging range 222-2. In this case, an image 223 indicating the overlapping area 224 within the imaging range 222-2 can be generated and displayed as guidance information.

[0283] By doing so, the overlapping area can be displayed based on the current position and orientation of the second imaging unit. Therefore, based on the guidance information, a user operating the second imaging unit can more easily determine how the imaging range of the second captured image obtained so far overlaps with the imaging range of the second captured image obtained by performing the second imaging at the current position and orientation. In other words, the user can more easily perform the second imaging so that the imaging range of the second captured image obtained so far appropriately overlaps. In other words, the user can more easily perform the second imaging at the appropriate position and orientation.

[0284] In addition, guidance information (an image indicating an overlapping area where the respective imaging ranges of the second captured images overlap, or an image indicating an overlapping area where the current viewing angle of the second imaging unit overlaps with the imaging range of the second captured image obtained so far) may be superimposed on the captured image generated by the second imaging unit for display. For example, Figure 15 The image 223 shown in (an image indicating an overlapping area where the current viewing angle of the second imaging unit overlaps with the imaging range of the second captured image obtained so far) can be displayed superimposed on the captured image generated by the second imaging unit at the current viewing angle.

[0285] By doing so, the captured image can be superimposed with guidance information having the same viewing angle (an image indicating the overlapping area where the current viewing angle of the second imaging unit overlaps with the imaging range of the second captured image obtained so far) for display on the monitor. This display allows the user to more easily align 3D objects in real space with the overlapping area. Consequently, the user can more easily determine a position and orientation suitable for the second imaging.

[0286] Note that an image of the overlap ratio indicating the proportion of the overlapping area within the viewing angle may also be displayed. The overlap ratio may be represented, for example, by a numerical value, or may be represented by, for example, color, density, or a pattern. Such a display enables the user to more intuitively determine how much overlap has occurred.

[0287] <Display of Imaging Auxiliary Image>

[0288] Furthermore, an imaging auxiliary image for assisting the second imaging may be included in the guidance information. For example, in the imaging guidance output process 105 for the second 3D modeling, guidance information including the imaging auxiliary image for assisting the second imaging may be generated, and the imaging auxiliary image may be displayed as the guidance information. The content of the imaging auxiliary image may be of any type.

[0289] For example, the imaging auxiliary image may include a recommended imaging position and orientation guide indicating a recommended imaging position and orientation as the recommended position and orientation for the second imaging. For example, in the imaging guide output process 105 for the second 3D modeling, a recommended imaging position and orientation as the recommended position and orientation for the second imaging may be derived based on the scoring result, and the recommended imaging position and orientation guide indicating the recommended imaging position and orientation may be displayed as the guidance information (imaging auxiliary image).

[0290] For example, in a case where the current position and orientation of the second imaging unit are the same as the recommended imaging position and orientation, an image indicating this state can be displayed as a recommended imaging position and orientation guide. That is, for example, in a case where the user etc. moves the second imaging unit so that the current position and orientation match the recommended imaging position and orientation, the user etc. can be notified of this state. This notification method can be any method. For example, when the current position and orientation of the second imaging unit matches the recommended imaging position and orientation, a completely different image, such as a white image, can be displayed. In addition, instead of such an image, the current position and orientation of the second imaging unit can be indicated as a recommended imaging position and orientation using letters, patterns, symbols, etc. The user etc. who operates the second imaging unit can easily determine that the current position and orientation of the second imaging unit matches the recommended imaging position and orientation based on such a display (recommended imaging position and orientation guide). Therefore, the user etc. can more easily perform the second imaging in an appropriate position and orientation.

[0291] Furthermore, an image indicating the recommended imaging position and orientation relative to the second imaging unit can be displayed as a recommended imaging position and orientation guide. Specifically, the direction and distance of the recommended imaging position and orientation relative to the current position and orientation of the second imaging unit can be indicated using, for example, letters, patterns, or symbols. This display allows a user, for example, operating the second imaging unit, to more easily move the second imaging unit toward the recommended imaging position and orientation, even if the current position and orientation of the second imaging unit does not match the recommended imaging position and orientation. Consequently, the user, for example, can more easily perform secondary imaging in the appropriate position and orientation.

[0292] Note that the recommended imaging position and orientation guide can be displayed superimposed on the captured image generated by the secondary imaging unit. This display makes it easier for the user to align the 3D object in real space with the recommended imaging position and orientation guide. This makes it easier for the user to determine a position and orientation suitable for secondary imaging.

[0293] <Imaging Guidance Based on Subject Complexity>

[0294] That is, the appropriate distance from the 3D object as the position of the second imaging depends on the three-dimensional shape of the 3D object. Therefore, the distance from the 3D object (subject) can be included in the recommended imaging position and orientation of the second imaging obtained in the imaging guidance output process 105 for the second 3D modeling. Then, when the recommended imaging position and orientation of the second imaging is obtained in the imaging guidance output process 105 for the second 3D modeling, the distance from the 3D object can be obtained based on the complexity of the three-dimensional shape of the 3D object.

[0295] The method for deriving the complexity of the three-dimensional shape of a 3D object may be any method, and may be, for example, the method described above in <2. Imaging Control>. Furthermore, the method for deriving the distance to the 3D object (recommended imaging position and orientation) based on the complexity of the three-dimensional shape of the 3D object may be any method. For example, when the three-dimensional shape of the 3D object is relatively complex, a position closer to the 3D object may be set as the recommended imaging position and orientation. Furthermore, when the three-dimensional shape of the 3D object is relatively simple, a position farther away from the 3D object may be set as the recommended imaging position and orientation.

[0296] In addition, a detection frame may be displayed in the guidance information displayed on the monitor, such as Figure 16 As shown. Figure 16In FIG. 1 , display image 230 is guidance information displayed on a monitor by imaging guidance output processing 105 for second 3D modeling. Display image 230 shows scored first 3D data 231 and a detection frame 232. Displaying detection frame 232 as described above allows the user to easily move the second imaging unit closer to or away from a portion of interest in a 3D object (subject) based on the complexity of the 3D object's three-dimensional shape. Of course, displaying the detection frame is not always necessary.

[0297] For example, a captured image generated by the second imaging unit can be displayed on a monitor, and a detection frame and the first 3D data corresponding to the 3D object (subject) can be superimposed on the captured image as guidance information, and the portion of the first 3D data (3D object) to be imaged can be indicated. Then, to ensure that the second imaging unit is in a position and orientation suitable for performing the second imaging, the user can move the second imaging unit so that the portion of the first 3D data to be imaged in the display is aligned with the detection frame.

[0298] For example, Figures 17 to 19 As shown, by displaying a display image 240 on a monitor, a detection frame 241 and a portion 242 of the 3D object to be imaged, which is derived based on the first 3D data, can be displayed in the display image 240. Then, in order to ensure that the second imaging unit is in a position and orientation more suitable for performing the second imaging, the user can move the second imaging unit so that the portion 242 to be imaged in the display image 240 is closer to the detection frame 241 (ideally aligned with the detection frame 241).

[0299] For example, in Figure 17 In the case of the left side in FIG, the portion to be imaged 242 is displayed smaller than the detection frame 241. In this case, in order to make the display of the portion to be imaged 242 as Figure 17 As shown on the right side of the image 240, the user moves the second imaging unit closer to the 3D object so that the portion 242 to be imaged appears larger. When the second imaging unit is moved as described above, the second imaging unit is in a position and orientation more suitable for performing the second imaging.

[0300] In addition, Figure 18 In the case of the left example, the imaging direction deviates from the direction of the normal to the portion 242 to be imaged (the portion 242 to be imaged and the detection frame 241 (imaging surface) are not directly facing each other). In this case, the user adjusts the direction of the second imaging unit (i.e., the imaging direction) and the like so that the portion 242 to be imaged faces (is more directly facing) the detection frame 241, as shown in FIG. Figure 18 When the second imaging unit is moved as described above, the second imaging unit is in a position and orientation that is more suitable for performing the second imaging.

[0301] In addition, Figure 19 In the case of the left example, the height of the portion 242 to be imaged is different from the detection frame 241. In this case, the user adjusts the distance between the second imaging unit and the 3D object, etc. so that the height of the portion 242 to be imaged is as shown in FIG. Figure 19 The right side of is shown to match (or approximate) the height of the detection frame 241. When the second imaging unit is moved as described above, the second imaging unit is in a position and orientation that is more suitable for performing the second imaging.

[0302] In addition, if Figure 20 In the example of , an arrow indicating the recommended moving direction of the second imaging unit (movement direction toward the recommended imaging position and orientation) may be displayed as a guide display. Figure 20 In the case of the left side of FIG, a display image 250 for displaying a guide display on the monitor is displayed, and an arrow 251 is displayed as a guide display in the display image 250. The arrow 251 is an arrow pointing to the far side (forward) in the figure, and guides the second imaging unit to move forward (toward the 3D object (subject)). In addition, Figure 20 In the example on the right side of the diagram, arrows 252 are displayed as guides in the display image 250 displayed on the monitor. Arrows 252 are arrows pointing toward the near side (backward) in the diagram and guide the second imaging unit to move backward (away from the 3D object (subject)). When the user moves the second imaging unit according to these arrows, the second imaging unit can be moved closer to the recommended imaging position and orientation.

[0303] In addition, if Figure 21 In the example of , an indicator indicating the positional relationship in the depth direction between the current position of the second imaging unit and the recommended imaging position and orientation may be displayed. Figure 21 In the case of the left side of the display image 260 for displaying the guide display, the display image 260 is displayed on the monitor, and the indicator 261 is displayed as the guide display in the display image 260. The indicator 261 indicates the positional relationship between the current position of the second imaging unit and the recommended imaging position and orientation in the depth direction. Figure 21 In the case of the left example, the indicator 261 indicates that the position of the recommended imaging position and orientation is deviated from the current position of the second imaging unit (in front of the current position of the second imaging unit), and guides the second imaging unit to move forward (toward the 3D object (subject)). Figure 21In the example on the right, indicator 261 indicates that the current position of the second imaging unit is approximately aligned with (approximately aligned with) the position of the recommended imaging position and orientation. In other words, in this case, indicator 261 indicates that there is little need to move the second imaging unit. When the user moves the second imaging unit according to indicator 261, the second imaging unit can be brought closer to the recommended imaging position and orientation.

[0304] Note that the indicator 261 can be of any design and is not limited to Figure 21 For example, you can use Figure 22 In the case of this example, the display changes as shown on the upper side in the figure in such a manner that it depends on the positional relationship in the depth direction between the current position of the second imaging unit and the recommended imaging position and orientation.

[0305] In addition, if Figure 23 In the example in FIG, the distance and alignment degree (orientation relationship) between the portion to be imaged of the first 3D data (3D object) and the second imaging unit can be displayed as guidance information. Figure 23 In this case, a display image 270 for displaying a guide display on the monitor is displayed, and the scored first 3D data 271 is displayed in the display image 270. In addition, a line (or a line corresponding thereto) 272 connecting the optical axis of the second imaging unit (the center of the pixel area of the second imaging unit) and the center of the portion to be imaged of the first 3D data (3D object) 271 is displayed as a guide display in the display image 270. In addition, an arrow 273 indicating the orientation of the subject surface in the center area of the portion to be imaged of the first 3D data (3D object) 271 is displayed as a guide display in the display image 270.

[0306] In the displayed image 270, line 272 and arrow 273 indicate the positional relationship between the current position of the second imaging unit and the recommended imaging position and orientation, as well as the distance and alignment degree (orientation relationship) between the portion to be imaged of the first 3D data (3D object) and the second imaging unit.

[0307] For example, Figure 24 As shown on the upper left side, when the directions of line 272 and arrow 273 are different from each other, it indicates that the surface (direction of the normal) of the part to be imaged of the first 3D data (3D object) deviates (is not facing) from the imaging surface (orientation of the second imaging unit) by a difference (angle).

[0308] On the other hand, Figure 24As shown in the upper center, with the directions of line 272 and arrow 273 aligned with each other, it indicates that the surface (direction of the normal) of the part to be imaged of the first 3D data (3D object) is facing the imaging surface (orientation of the second imaging unit).

[0309] In addition, if Figure 24 As shown in the upper right portion of FIG, if line 272 and arrow 273 are separated, this indicates that the distance between the portion of the first 3D data (3D object) to be imaged and the second imaging unit is longer than the distance suitable for second imaging. In other words, in this case, guidance is provided to move the second imaging unit toward the first 3D data (3D object).

[0310] In addition, if Figure 24 As shown in the lower left portion of FIG, if line 272 is shorter than arrow 273, this indicates that the distance between the portion of the first 3D data (3D object) to be imaged and the second imaging unit is shorter than a distance suitable for second imaging. In other words, in this case, guidance is provided to move the second imaging unit away from the first 3D data (3D object).

[0311] In addition, if Figure 24 As shown in the center of the lower portion of FIG, if a circle 274 is displayed at the connection between line 272 and arrow 273, it indicates that the distance between the portion of the first 3D data (3D object) to be imaged and the second imaging unit is approximately the distance suitable for second imaging. In other words, in this case, guidance is provided without moving the second imaging unit in the depth direction.

[0312] In addition, if Figure 24 As shown in the lower right portion of FIG, if a circle 274 is displayed at the connection between line 272 and arrow 273, and the directions of line 272 and arrow 273 are aligned, it indicates that the distance between the portion to be imaged of the first 3D data (3D object) and the second imaging unit is approximately the distance suitable for second imaging, and the surface (the direction of the normal line) of the portion to be imaged of the first 3D data (3D object) is facing the imaging surface (the orientation of the second imaging unit). In other words, in this case, guidance is provided indicating that the current position and orientation of the second imaging unit is aligned with or approximately the recommended imaging position and orientation.

[0313] The user can more easily bring the second imaging unit closer to the recommended imaging position and orientation by moving the second imaging unit according to such guide information.

[0314] Note that since the orientation information about the second imaging unit (first imaging unit) is obtained by SLAM, etc., the distance between the second imaging unit and the subject can be easily obtained. Therefore, the above display example can be updated in real time (instantly).

[0315] <Real-time processing>

[0316] Notice, Figure 4 The first 3D data generation process 101 (first imaging and first 3D modeling process), the scoring process 102, and the imaging guidance output process 105 for the second 3D modeling can be executed in parallel. As described above in <2. Imaging Control>, 3D data of the first imaged portion of the 3D object can be sequentially generated through the first 3D modeling process. Furthermore, the first 3D data generation process 101 and the scoring process 102 can be executed in parallel.

[0317] Furthermore, in the imaging guidance output process 105 for the second 3D modeling, each time a scoring result is obtained by the scoring process 102 (before a scoring result for the entire 3D object is obtained), guidance information for the second imaging can be generated and output based on the obtained scoring result (the scoring result of the first 3D data corresponding to a portion of the 3D object). By doing so, the imaging guidance output process 105 for the second 3D modeling can be started before the scoring process 102 ends (before a scoring result for the entire 3D object is obtained). In other words, the scoring process 102 and the imaging guidance output process 105 for the second 3D modeling can be executed in parallel.

[0318] By combining the above-described methods, the first 3D data generation process 101 , the scoring process 102 , and the imaging guidance output process 105 for the second 3D modeling can be performed in parallel.

[0319] For example, Figure 25 As shown, it is assumed that a display image 280 is displayed on a monitor, and a captured image captured by the second imaging unit is displayed in the display image 280. In the captured image, a 3D object 281 appears as a subject. As described above, the first 3D data generation process 101, the scoring process 102, and the imaging guidance output process 105 for the second 3D modeling are performed in parallel, so that the guidance information can be displayed in the display image 280 before the first 3D data and scoring of the entire 3D object 281 are completed. Figure 25 In the displayed image 280, shaded display 282 indicates the portion of 3D object 281 for which the first 3D data has been generated. Furthermore, gray display 283 indicates the portion for which the second captured image is insufficient as a result of the scoring. By executing the first 3D data generation process 101, the scoring process 102, and the imaging guidance output process 105 for the second 3D modeling in parallel, the imaging guidance can be displayed while the first imaging is being performed. Therefore, the user can perform the second imaging in parallel (instantaneously) with the first imaging.

[0320] <Camera Information>

[0321] Note that, also when the imaging guide output process 105 for the second 3D modeling is executed, similarly to the case described above in <2. Imaging Control>, camera information regarding the second imaging unit can be generated in the second 3D data generation process 104 (second imaging), and this camera information can be provided to the scoring process 102. Then, in the scoring process 102, scoring can be performed based on the camera information to generate a scoring result. Similar to the case described above in <2. Imaging Control>, the camera information can include any information.

[0322] <Example of Configuration for Execution Processing>

[0323] above Figure 4 Each process in can be executed by any device. For example, in an information processing device, the first 3D modeling process of the first 3D data generation process 101, the scoring process 102, and the imaging guidance output process 105 for the second 3D modeling can be executed.

[0324] That is, the information processing device may include: a first 3D modeling processing unit that generates first three-dimensional shape information representing the three-dimensional shape of a 3D object based on a first captured image generated by first imaging of the 3D object; a scoring processing unit that uses the first three-dimensional shape information to evaluate the accuracy of second three-dimensional shape information that can be generated using a second captured image generated by second imaging performed so far, and generates a scoring result; and a guidance information output control unit that generates guidance information for the second imaging of the 3D object based on the scoring result and controls the output of the guidance information. In this section, the information processing device is also referred to as the first information processing device.

[0325] In addition, the information processing method performed by the first information processing device may include: generating first three-dimensional shape information representing the three-dimensional shape of a 3D object based on a first captured image generated by first imaging of the 3D object; using the first three-dimensional shape information to evaluate the accuracy of second three-dimensional shape information that can be generated using a second captured image generated by second imaging performed so far, and generating a scoring result; and generating guidance information for the second imaging of the 3D object based on the scoring result, and controlling the output of the guidance information.

[0326] By doing so, the user can perform a second imaging process based on the guidance information, allowing them to image the 3D object in a more appropriate position and orientation. That is, 3D modeling (second 3D modeling process) can be performed using the captured image. Consequently, more accurate 3D data can be generated while suppressing an increase in the 3D modeling load. In other words, 3D modeling can be performed more easily.

[0327] Note that the guidance information output control unit may generate an image indicating the scoring result as guidance information and display the image. Furthermore, the guidance information output control unit may generate an image indicating the scoring result within the field of view of the second imaging unit based on the position and orientation of the second imaging unit and display the image. Furthermore, the guidance information output control unit may superimpose the captured image generated by the second imaging unit on the image indicating the scoring result within the field of view of the second imaging unit and display the image. Furthermore, the guidance information output control unit may also display a bird's-eye view image indicating the scoring result for the entire 3D object.

[0328] In addition, the guidance information output control unit can generate an image indicating an overlapping area where the imaging ranges of multiple second captured images overlap as guidance information, and display the image. Furthermore, based on the position and orientation of the second imaging unit, the guidance information output control unit can generate an image indicating an overlapping area where the current viewing angle of the second imaging unit overlaps with the imaging ranges of the second captured images obtained so far, and display the image. Furthermore, the guidance information output control unit can superimpose the captured images generated by the second imaging unit on the image for display. Furthermore, the guidance information output control unit can also display an image indicating an overlap ratio, which indicates the proportion of the overlapping area within the current viewing angle of the second imaging unit.

[0329] In addition, the guidance information output control unit can generate an imaging auxiliary image for assisting the second imaging as guidance information and display the imaging auxiliary image. In addition, the guidance information output control unit can derive a recommended imaging position and orientation as the recommended position and orientation for the second imaging based on the scoring result, and display a recommended imaging position and orientation guide indicating the recommended imaging position and orientation as guidance information. In addition, when the position and orientation of the second imaging unit are the same as the recommended imaging position and orientation, the guidance information output control unit can display an image indicating that the current position and orientation of the second imaging unit is the recommended imaging position and orientation as the recommended imaging position and orientation guide. In addition, the guidance information output control unit can display an image indicating the relative position and orientation of the recommended imaging position and orientation relative to the second imaging unit as the recommended imaging position and orientation guide. In addition, the guidance information output control unit can superimpose the captured image generated by the imaging unit performing the second imaging on the recommended imaging position and orientation guide for display.

[0330] Incidentally, in the first information processing device described above, the first three-dimensional shape information has less information content and lower accuracy than the second three-dimensional shape information. Furthermore, the first 3D modeling processing unit of the first information processing device may include: an orientation information generation unit that generates orientation information indicating the position and orientation of the first imaging unit based on the first captured image and the acceleration and angular velocity of the first imaging unit; and a three-dimensional shape generation unit that generates the first three-dimensional shape information based on the orientation information and the depth of the 3D object. Note that in this case, the first three-dimensional shape information may include a mesh representing the three-dimensional shape of the 3D object through vertices and connections, and a texture applied to the surface of the mesh.

[0331] Furthermore, in the first information processing device described above, the scoring processing unit may generate a scoring result for each local portion of the first three-dimensional shape information based on the first three-dimensional shape information and the position and orientation of the second imaging performed so far. Furthermore, the first three-dimensional shape information may include a mesh indicating the three-dimensional shape of the 3D object through vertices and connections, and a texture applied to a surface of the mesh, and the scoring processing unit may generate a scoring result for each polygon of the mesh.

[0332] Furthermore, the first information processing device may also execute the second imaging of the second 3D data generation process 104. The configuration of the first information processing device in this case is similar to that described above in <2. Imaging Control>. Furthermore, the first information processing device may also execute the second 3D modeling process of the second 3D data generation process 104. The configuration of the first information processing device in this case is also similar to that described above in <2. Imaging Control>.

[0333] Note that, as described above, the second imaging is performed by manual imaging. Therefore, the scoring processing unit of the first information processing device can generate a scoring result based on the position and orientation of the second information processing device corresponding to the second imaging timing indicated by the imaging timing information indicating the second imaging timing. For example, the guidance information output control unit can obtain orientation information about the second imaging unit at the imaging timing based on the imaging timing information, and the scoring processing unit can calculate the score based on the orientation information. By doing so, the orientation information about the manual imaging is reflected in the scoring result. The configuration of the first information processing device in this case is also similar to the case described above in <2. Imaging control>. However, the imaging timing information generated by the second imaging unit or the imaging timing information acquired by the communication unit is provided to the guidance information output control unit. By doing so, control can be performed so that the second imaging is performed in a more appropriate position and orientation based on the imaging timing information.

[0334] Furthermore, in the first information processing device, as described above, the camera information regarding the second imaging unit can be reflected in the scoring process 102. For example, the scoring processing unit of the first information processing device can generate a scoring result based on the camera information. The configuration of the first information processing device in this case is also similar to the case described above in <2. Imaging Control>. By doing so, control can be performed so that the second imaging is performed at a more appropriate position and orientation based on the camera information.

[0335] Incidentally, while the first information processing device is executing the imaging guidance output process 105 for the second 3D modeling, the second information processing device can also execute the second imaging of the second 3D data generation process 104 described above. The configuration of the second information processing device in this case is also similar to that described above in <2. Imaging Control>. The second information processing device can then further execute the second 3D modeling process described above. The configuration of the second information processing device in this case is also similar to that described above in <2. Imaging Control>.

[0336] Furthermore, imaging timing information indicating the timing of manual imaging may be generated in the second information processing apparatus and provided to the first information processing apparatus. The configuration of the second information processing apparatus in this case is also similar to that described above in <2. Imaging Control>.

[0337] Furthermore, camera information about the second imaging unit may be reflected in the scoring process 102. The configuration of the second information processing device in this case is also similar to the case described above in <2. Imaging control>.

[0338] <4. Combination>

[0339] <Combination of Imaging Control and Guidance Information Output>

[0340] Note that in Figure 4 In the embodiment, both the imaging control process 103 for the second 3D modeling and the imaging guide output process 105 for the second 3D modeling can be performed. By performing both the imaging control and the guide information output, the user can more easily perform the second imaging in an appropriate position and orientation.

[0341] For example, the first information processing device described in <2. Imaging Control> above may further include a guidance information output control unit that generates guidance information for second imaging of the 3D object based on the scoring result. In this case, the guidance information output control unit performs processing similar to that described in <3. Imaging Guidance Output> above.

[0342] Furthermore, the first information processing device described in <3. Imaging Guidance Output> may further include an imaging control unit that controls a second imaging operation for imaging the 3D object based on the scoring result. In this case, the imaging control unit performs processing similar to that described in <2. Imaging Control> above.

[0343] <5. Calibration Process>

[0344] <Decrease in Navigation Accuracy Due to Decrease in Accuracy of Orientation Information>

[0345] In the case of navigating (imaging control, imaging guidance, or both) imaging for obtaining captured images to be used in 3D modeling such as real-time 3D modeling or photogrammetry, it is possible that the lower the accuracy of the position and orientation of the imaging unit performing the imaging, the lower the accuracy of the navigation.

[0346] For example, as described above in <2. Imaging Control>, <3. Imaging Guidance Output>, etc., when performing 3D modeling twice and performing navigation (imaging control, imaging guidance, or both) for imaging for the second 3D modeling (second imaging) using the results of the first 3D modeling, there is a possibility that unless the position and orientation of the imaging unit performing the imaging for each 3D modeling are obtained with sufficient accuracy, it may be difficult to perform navigation with sufficiently high accuracy. The reduction in the accuracy of the navigation for the second imaging makes it difficult to obtain a captured image (second captured image) at an appropriate position and orientation in the second imaging, and the accuracy of the 3D data obtained by the second 3D modeling may decrease. In other words, when the second imaging is performed based on low-accuracy navigation, the number of second captured images captured at an appropriate position and orientation (also referred to as appropriate second captured images) may become insufficient. Therefore, in order to obtain a sufficient number of appropriate second captured images, it is necessary to increase the frequency of the second imaging. As a result, not only does the user's workload increase, but the number of captured images also increases unnecessarily, and the load of the 3D modeling processing (processing volume, processing time, etc.) may also increase unnecessarily.

[0347] For example, in the case where the first orientation information indicating the position and orientation of the first imaging unit is obtained in the first 3D modeling of the real-time 3D modeling as described above in <2. Imaging Control>, <3. Imaging Guidance Output>, etc., the above-mentioned first orientation information can be used for navigation of the second imaging. That is, by applying the first orientation information as information indicating the position and orientation of the second imaging unit that performs the second imaging, the second imaging can be navigated. However, for example, in the case where the first imaging unit that performs the first imaging for the first 3D modeling and the second imaging unit that performs the second imaging are not close enough to each other (in the case where the first imaging unit and the second imaging unit are separated by at least a certain distance), when the first orientation information is applied as the position and orientation of the second imaging unit, the accuracy of the position and orientation of the second imaging unit may be reduced, and the accuracy of the navigation of the second imaging may be reduced.

[0348] For example, suppose you use Figure 26 The smartphone 301 and the interchangeable lens camera (ILC) 302 shown perform such 3D modeling and navigation. The smartphone 301 is an imaging communication device having an imaging function, a communication function, and an information processing function, and includes a depth sensor 311, an imaging unit 312, and an inertial measurement unit (IMU) (not shown). In addition, the ILC 302 is an imaging device having an interchangeable optical system (e.g., a lens). The smartphone 301 and the ILC 302 are communicatively connected to each other and can exchange information through communication. In addition, the smartphone 301 can be mounted at a predetermined position on the ILC 302. In other words, the smartphone 301 can be detached from the ILC 302. In addition, it is assumed that when the smartphone 301 is mounted on the ILC, the position and orientation of the smartphone 301 relative to the ILC 302 are variable. That is, it is assumed that the position and orientation of the smartphone 301 relative to the ILC 302 can be adjusted. Furthermore, the ILC 302 (or the smartphone 301 ) is equipped with a component for stabilizing the position and orientation of the smartphone 301 relative to the ILC 302 , thereby preventing the position and orientation of the smartphone 301 relative to the ILC 302 from changing.

[0349] Furthermore, it is assumed that, with the smartphone 301 mounted on the ILC 302, the smartphone 301 performs processing such as the first imaging. Furthermore, the ILC 302 performs the second imaging, etc., based on navigation of the second imaging. Note that navigation of the second imaging using the first 3D modeling or the 3D model obtained by the first 3D modeling can be performed by any device. For example, these processes can be performed by the smartphone 301, by the ILC 302, or by another device.

[0350] In this case, the first imaging unit (imaging unit 312) and the second imaging unit (imaging unit of ILC 302) may be separated (not close enough). In addition, the orientations of the first imaging unit and the second imaging unit may be different from each other. Therefore, when the first orientation information indicating the position and orientation of the first imaging unit obtained in the first 3D modeling is applied as is as the position and orientation of the second imaging unit, the position and orientation of the second imaging unit may be misaligned. Therefore, when the second imaging is navigated using the position and orientation of the second imaging unit (i.e., the first orientation information), the position and orientation of the navigation may be misaligned, and the accuracy of the navigation may be reduced.

[0351] <Application of Calibration Processing>

[0352] So, for example, Figure 4 As shown, a calibration process 106 is performed to calibrate the second orientation information indicating the position and orientation of the second imaging unit, etc. In the calibration process 106, the second orientation information is calibrated based on the first captured image, the first orientation information, and the second captured image. In <5. Calibration Process>, the first captured image indicates a captured image generated by the first imaging unit that performs the first imaging. In addition, the second captured image indicates a captured image generated by the second imaging unit that performs the second imaging. The first imaging is imaging for generating a captured image used in the first 3D modeling, and the second imaging is imaging for generating a captured image used in the second 3D modeling. The first orientation information indicates the position and orientation of the first imaging unit. For example, the first orientation information can be generated in the first 3D modeling.

[0353] The calibration process 106 generates calibration information as a calibration result and provides the configuration information to the imaging control process 103 for second 3D modeling and the imaging guidance output process 105 for second 3D modeling. In the imaging control process 103 for second 3D modeling and the imaging guidance output process 105 for second 3D modeling, imaging control and imaging guidance are performed using the calibration information.

[0354] By doing so, it is possible to suppress a decrease in the accuracy of the position and orientation of the second imaging unit. Therefore, it is possible to suppress a decrease in the accuracy of the navigation of the second imaging (ie, the navigation of the imaging for 3D modeling).

[0355] For example, if the position and orientation relationship between the first imaging unit and the second imaging unit is known, the second orientation information can be calibrated based on the relationship. Figure 26In this case, the second orientation information (the position and orientation of the imaging unit of the ILC 302) can be calibrated based on the position and orientation relationship between the depth sensor 311 and the imaging unit 312 (the first imaging unit), and the position and orientation relationship between the imaging unit 312 and the imaging unit of the ILC 302 (the second imaging unit). The calibrated second orientation information can then be used for navigation of the second imaging.

[0356] Note that there may be situations where the position and orientation relationship between the first imaging unit and the second imaging unit is unknown. Figure 26 In the case of the above, when the smartphone 301 is mounted on the ILC 302, there may be a case where the position and orientation of (the imaging unit 312 of) the smartphone 301 relative to (the imaging unit of) the ILC 302 are freely determined (the position and orientation cannot be determined). In addition, there may be a case where the position and orientation of (the imaging unit 312 of) the smartphone 301 relative to (the imaging unit of) the ILC 302 changes after the smartphone 301 is mounted on the ILC 302. It goes without saying that there may be a case where the position and orientation relationship between the first imaging unit and the second imaging unit cannot be determined (that is, the relationship is unknown) due to other reasons.

[0357] In addition, you can Figure 26 The above-mentioned 3D modeling and navigation are performed in the smart phone 321 shown. The smart phone 321 includes a depth sensor 331, an imaging unit 332, an imaging unit 333 and an inertial measurement unit (IMU) (not shown). The imaging unit 332 is a first imaging unit that performs the first imaging. The imaging unit 333 is a second imaging unit that performs the second imaging. As described above, the above-mentioned 3D modeling and navigation can also be performed in a single information processing device (imaging communication device). Even with such a configuration, there may be a situation where the position and orientation relationship between the first imaging unit and the second imaging unit is unknown. For example, there may be a situation where the relationship changes due to the installation of movable parts, etc. In addition, there may be a situation where the relationship changes due to aging, etc.

[0358] As in these examples, when the positional and orientation relationship between the first and second imaging units is unknown, it is difficult to ensure the accuracy of the second imaging unit's position and orientation when the first orientation information is used as the second imaging unit's position and orientation. In other words, the accuracy of the second imaging unit's position and orientation may be reduced. Consequently, the accuracy of the second imaging navigation may be reduced. Furthermore, when the positional and orientation relationship between the first and second imaging units is unknown, it is difficult to calibrate the second orientation information based on the relationship, as is done when the relationship is known.

[0359] Therefore, for example, the second orientation information can be calibrated by performing the following calibration process 106 .

[0360] <Example of Calibration Processing>

[0361] In the calibration process 106, for example, Figure 27 As shown, imaging timing calibration processing 341, installation guidance processing 342, and orientation information calibration processing 343 can be performed. For example, imaging timing calibration processing 341, installation guidance processing 342, and orientation information calibration processing 343 can be performed in this order. Even if the positional and orientation relationship between the first imaging unit and the second imaging unit is unknown, the second orientation information can be calibrated by performing each of the above-described processes. Note that installation guidance processing 342 can be omitted. Furthermore, imaging timing calibration processing 341 or orientation information calibration processing 343 can be omitted.

[0362] <Imaging Timing Calibration Processing>

[0363] The imaging timing calibration process 341 is a process for calibrating the imaging timing of the second imaging unit. Generally speaking, the timing (imaging timing) at which the imaging unit actually images the subject has a predetermined delay (time lag) relative to the timing (instruction timing) at which imaging is instructed. For example, assuming that Figure 26 The second imaging is instructed in the illustrated smartphone 301, and the second imaging based on the instruction is performed in the ILC 302. In particular, in such a case, the communication latency (delay time) between the smartphone 301 and the ILC 302 is increased, and the time lag from the instruction timing to the imaging timing may increase.

[0364] When the time lag between the instruction timing and the imaging timing increases, even if an imaging instruction is issued at a timing when the second imaging unit is in the appropriate position, the second imaging unit may move from the appropriate position before the second imaging is performed. Therefore, the accuracy of the navigation of the second imaging may be reduced.

[0365] Therefore, in the imaging timing calibration process 341, the imaging timing of the second imaging unit is calibrated, and the calibration result (imaging timing calibration information) is provided to the imaging control process for second 3D modeling 103. In the imaging control process for second 3D modeling 103, when controlling the second imaging, the calibration result is reflected in the imaging timing, and the second imaging is instructed at an instruction timing that makes the imaging timing reflecting the calibration result appropriate.

[0366] By doing so, the second imaging can be controlled (instructed) so that the second imaging is performed at an appropriate timing (at a desired position), and thus a decrease in the accuracy of navigation of the second imaging can be suppressed.

[0367] Any method may be used to calibrate the imaging timing in the imaging timing calibration process 341. For example, a measurement image that changes over time (e.g., frame by frame) may be displayed on a display unit, the measurement image displayed on the display unit may be captured by a second imaging unit, and the imaging timing may be calibrated based on the obtained second captured image.

[0368] For example, when a command is issued to display a measurement image on the display unit, as shown in FIG. Figure 28 As shown in the "Display" section of the timing diagram, measurement of image display begins after a predetermined time (display latency) has elapsed from the command issuance timing. Assuming the display latency (delay time) is known, the image then changes over time (e.g., frame by frame).

[0369] The imaging instruction is issued simultaneously with the display instruction (at the same instruction timing). The second imaging unit captures the measurement image displayed on the display unit based on the instruction to generate a second captured image. Figure 28 As shown in the "Imaging" part of the timing diagram of , the imaging timing is delayed by a predetermined time from the command issuance timing due to communication delay, shutter lag, etc. Figure 28 In the example of FIG, the time period indicated by the double-headed arrow 351 is a time lag from the instruction timing to the imaging timing. This time lag can be obtained based on what type of image the measurement image included in the obtained second captured image is.

[0370] exist Figure 26 In the case of an example using a smartphone 301 and an ILC 302, as Figure 29 As shown, the imaging unit (second imaging unit) of the ILC 302 is positioned to face the display unit of the smartphone 301 so that the imaging unit of the ILC 302 can capture a measurement image displayed on the display unit of the smartphone 301 .

[0371] At this time, a prompt may be output to guide the measurement image to be imaged in the correct position. For example, an image providing such a prompt may be displayed on the display unit. For example, Figure 30 As shown, a guide image 372 indicating an area where a measurement image is displayed is displayed on the display unit 371 of the smartphone 301. Figure 29As in the example of , the measurement image displayed on the display unit of the smartphone 301 can be captured by pointing the imaging unit of the ILC 302 toward the display unit. Therefore, as in the example of ILC 302-1, the smartphone 301 is displayed on the display unit 373 of the ILC 302. As in the example of ILC 302-2, the user adjusts the positions of the smartphone 301 and the ILC 302 so that the guide image 372 is displayed on the entire display unit 373. As described above, the ILC 302 can capture the entire measurement image of a larger size. Therefore, it is easier to identify which type of image the measurement image included in the captured image is. That is, the time lag from the instruction timing to the imaging timing can be obtained more easily and accurately.

[0372] That is to say, if Figure 27 As shown, in the imaging timing calibration process 341, an instruction to display the measurement image and an instruction to perform the second imaging are issued at the same time. In addition, a second captured image obtained by the second imaging based on the instruction (obtained by capturing the measurement image) is provided. Then, a time lag is obtained based on the second captured image, and the imaging timing of the second imaging is calibrated. Then, timing calibration information is generated as a calibration result, and the timing calibration information is provided to the imaging control process 103 for the second 3D modeling. In the imaging control process 103 for the second 3D modeling, the issuance timing of the imaging instruction command for the second imaging is controlled so that the second imaging is performed at the desired timing (at an appropriate position) based on the timing calibration information generated as described above.

[0373] <Example of installation boot processing>

[0374] The installation guidance processing 342 is a processing for providing guidance when, for example, the smartphone 301 is attached to (mounted to) the ILC 302. For example, guidance is output that prompts the second imaging unit (device (ILC 302) equipped with the second imaging unit) to be installed on the first imaging unit (device (smartphone 301) equipped with the first imaging unit) in the correct relative position and orientation. Guidance can be provided by image or by audio. For example, in the installation guidance processing 342, it can be instructed to display a guidance (image). In addition, first orientation information (orientation information about the first imaging unit) can be provided, and an orientation image indicating the orientation of the first imaging unit can be superimposed on the guidance based on the orientation information.

[0375] Figure 31 An example of the above situation is shown. Note that Figure 31 , each of the smartphones 301 - 1 to 301 - 4 indicates the smartphone 301 , and indicates that images displayed on the display unit 371 are different from each other.

[0376] First, as in smartphone 301-1, a first captured image obtained by the first imaging unit is displayed on display unit 371. Next, as in smartphone 301-2, a target orientation 381 indicating the target position and orientation of the field of view of the second imaging unit (the imaging unit of ILC 302) is superimposed on the first captured image for display on display unit 371. Next, as in smartphone 301-3, a current orientation 382 indicating the current position and orientation of the field of view of the second imaging unit (the imaging unit of ILC 302) is further superimposed based on the second orientation information for display on display unit 371. Note that the second orientation information is essentially uncalibrated at this point. For example, the first orientation information can be used as the second orientation information. It goes without saying that second orientation information calibrated by any method can be used.

[0377] As in smartphone 301 - 4 , the user adjusts the relative position and orientation of ILC 302 with respect to smartphone 301 so that current orientation 382 is aligned with target orientation 381 .

[0378] By executing such an installation guidance process 342, the second imaging unit (ILC 302) can be installed on the first imaging unit (smartphone 301) in, for example, a correct relative position and orientation. Therefore, the positional and orientational relationship between the first and second imaging units can be calibrated with a certain degree of accuracy.

[0379] That is to say, if Figure 27 As shown, in the installation guidance process 342, an instruction is issued to display guidance prompting the installation of the second imaging unit (device equipped with the second imaging unit) on the first imaging unit (device equipped with the first imaging unit) in the correct relative position and orientation. In response to this instruction, a second captured image is provided. In other words, an image (orientation image) indicating the position and orientation of the second imaging unit is provided. In the installation guidance process 342, a guidance display instruction is issued so that the information obtained from the orientation image is reflected in the guidance.

[0380] Note that the guidance may be a guide for prompting the first imaging unit (device equipped with the first imaging unit (smartphone 301)) to be mounted on the second imaging unit (device equipped with the second imaging unit (ILC302)) in a correct relative position and orientation. Figure 31 , the second captured image, the target orientation 381, and the current orientation 382 may be displayed on the display unit 371 of the smartphone 301. In this case, the target orientation 381 indicates the target position and orientation of the field of view of the first imaging unit. In addition, the current orientation 382 indicates the current position and orientation of the field of view of the first imaging unit.

[0381] <Example of Orientation Information Calibration Processing>

[0382] However, the accuracy of calibrating the position and orientation relationship between the first and second imaging units based on the guidance in the installation guidance process 342 cannot be said to be necessarily sufficient. Therefore, even if installation is performed according to this guidance, it cannot be said that the second orientation information is always calibrated with sufficient accuracy. Therefore, the orientation information calibration process 343 is executed to calibrate the second orientation information.

[0383] By calibrating the second orientation information in orientation information calibration processing 343, it is possible to suppress a decrease in the accuracy of the position and orientation of the second imaging unit, and to suppress a decrease in the accuracy of navigation performed by the second imaging. Consequently, captured images can be obtained at a more appropriate position and orientation, and a decrease in the accuracy of 3D data obtained by the second 3D modeling can be suppressed. Furthermore, since the second imaging can be performed at a more appropriate position and orientation, an increase in the number of captured images used in the second 3D modeling can be suppressed. Consequently, an increase in the user's workload can be suppressed. Furthermore, an increase in the load (processing volume, processing time, etc.) of the 3D modeling process can be suppressed.

[0384] For example, Figure 32 As shown, orientation information calibration processing 343 can be performed by imaging a 3D object 392 positioned within a common field of view (e.g., the range indicated by a dotted double-headed arrow 391) by the first imaging unit and the second imaging unit. In this common field of view, the perspective of the first imaging unit (the imaging unit 312 of the smartphone 301) and the perspective of the second imaging unit (the imaging unit of the ILC 302) are aligned with each other. For example, the second orientation information can be calibrated based on the first captured image and the second captured image captured at the same time and the first orientation information obtained at that time.

[0385] At this time, in the orientation information calibration process 343, the same subject can be imaged by the first imaging unit and the second imaging unit, and a first captured image and a second captured image can be generated. In addition, the feature points of the obtained first captured image and the second captured image can be detected. In addition, corresponding points can be obtained between the first captured image and the second captured image based on the feature points. Then, the relative positional relationship between the first imaging unit and the second imaging unit can be obtained based on the positional relationship between the images. However, in this case, the size (dimensions) of the 3D object serving as the subject is unknown. Therefore, the distance to the subject (3D object) can be obtained based on the first orientation information, and the size of the 3D object can be determined based on the distance. Then, the second orientation information can be calibrated based on the corresponding points and size obtained so that the reprojection error becomes sufficiently small.

[0386] For example, Figure 33As shown, feature point verification processing 401, common field of view verification processing 402, imaging control processing 403, and orientation information calibration processing 404 may be performed. For example, feature point verification processing 401, common field of view verification processing 402, imaging control processing 403, and orientation information calibration processing 404 may be performed in this order.

[0387] In the feature point verification process 401, the first imaging unit and the second imaging unit are instructed to perform imaging, and a first captured image and a second captured image can be provided. Then, predetermined feature points can be detected from these captured images. The feature point is a point with a predetermined feature. The predetermined feature can be any feature. Note that how many feature points can be extracted depends on the captured images. If the number of extracted feature points is insufficient, it may be difficult to obtain the relative positional relationship between the first imaging unit and the second imaging unit. Therefore, it can be verified whether a sufficient number of feature points are detected from the first captured image and the second captured image. In the event that a sufficient number of feature points cannot be detected, a prompt can be provided to perform the first imaging and the second imaging again with a changed position and orientation (i.e., the subject).

[0388] In the common field of view verification process 402, the common field of view (the range of overlapping perspectives) between the first imaging unit and the second imaging unit can be verified. For example, the feature points detected as described above can be used to detect corresponding points between the first captured image and the second captured image. Note that in the absence of a common field of view between the first imaging unit and the second imaging unit, it is difficult to obtain the positional relationship between the first captured image and the second captured image. Therefore, it is possible to verify whether the first imaging unit and the second imaging unit have a common field of view. In the absence of a common field of view, the position or orientation can be changed (i.e., the subject can be changed) to prompt the first imaging and the second imaging to be performed again.

[0389] In the imaging control process 403 , the first imaging unit and the second imaging unit may be instructed to perform imaging.

[0390] In the orientation information calibration process 404, a first captured image and a second captured image can be generated and provided based on the instruction. In addition, first orientation information can be provided. Then, the distance to the subject (3D object) can be obtained based on the first orientation information, and the size of the 3D object can be determined based on the distance. Then, based on the obtained corresponding points and sizes, the second orientation information can be calibrated so that the reprojection error becomes sufficiently small. At this time, the second orientation information can be calibrated using the internal parameters of the second imaging unit so that the reprojection error becomes sufficiently small. Note that the internal parameters can be preset values preset for the second imaging unit or the optical system unit used in the second imaging unit. In other words, the internal parameters can be known. In addition, the internal parameters can be provided by the second imaging unit. As a result of such calibration of the second orientation information, orientation calibration information can be generated. For example, the orientation calibration information can be provided to the imaging control process 103 for the second 3D modeling. In addition, for example, the orientation calibration information can be provided to the imaging guidance output process 105 for the second 3D modeling.

[0391] Note that in Figure 32 In the case where the second orientation information is calibrated by imaging the 3D object as in the example in , the size of the 3D object may be determined using the depth of the 3D object (information indicating the distance to the 3D object) instead of the first orientation information.

[0392] In addition, in Figure 32 In the case where the second orientation information is calibrated by imaging a 3D object, as in the example in FIG, the calibration process 106 may be performed before the above-described navigation. Furthermore, the calibration process 106 may be performed in parallel with the above-described navigation (e.g., performed in the background while navigation is being performed). Furthermore, the navigation may be temporarily interrupted, and the calibration process 106 may be performed. As described above, the calibration process 106 may be performed at more different timings by imaging the 3D object and calibrating the second orientation information.

[0393] Note that the orientation information calibration process 343 may be performed based on a plurality of first captured images and second captured images obtained by performing the first imaging and second imaging at a plurality of different positions. In this case, in the orientation information calibration process 404, guidance for prompting the movement of the first imaging unit and the second imaging unit may be output. For example, Figure 34 As shown in the smartphone 301 - 11 in FIG, a message such as “With the camera fixed to the smartphone, move slowly left, right, up, and down” may be displayed on the display unit 371 .

[0394] In addition, in the case where the calibration of the second orientation information succeeds or fails, a notification of the result may be performed. For example, in the case where the calibration of the second orientation information succeeds, a notification indicating successful calibration may be output (displayed). Figure 34 As shown in the smart phone 301-12 in FIG, a message such as “calibration successful” may be displayed on the display unit 371. In addition, in the case where the calibration of the second orientation information fails, a prompt for re-imaging guidance may be output (displayed). For example, Figure 34 As shown in the smartphone 301-13 in FIG, a message prompting re-imaging such as “Try again in a textured place” or a message prompting confirmation of fixation such as “Fix the camera device firmly to the smartphone” may be displayed on the display unit 371.

[0395] Note that, for example, Figure 35 As shown, the orientation information calibration process 343 can be performed by imaging a calibration pattern 422 having a predetermined design positioned within a common field of view (e.g., the range indicated by the dotted double-headed arrow 391) with the first imaging unit and the second imaging unit, in which the viewing angle of the first imaging unit (the imaging unit 312 of the smart phone 301) overlaps with the viewing angle of the second imaging unit (the imaging unit of the ILC 302). The calibration pattern 422 is an image having a design for detecting feature points (corresponding points). The calibration pattern 422 can be any pattern. For example, it can be used Figure 35 Alternatively, another pattern may be used. In the case of such a calibration pattern 422, the size of the pattern is known. Therefore, the second orientation information can be calibrated without using the first orientation information. However, since calibration pattern 422 is used, it is difficult to perform orientation information calibration processing 343 while the above-mentioned navigation is being performed (in parallel and without interruption). In other words, in this case, orientation information calibration processing 343 is performed as preprocessing before the above-mentioned navigation or by interrupting the above-mentioned navigation.

[0396] That is to say, if Figure 27 As shown, in the orientation information calibration process 343, for example, an instruction is issued to display guidance prompting the movement of the first imaging unit and the second imaging unit. Furthermore, in the orientation information calibration process 343, an instruction is issued to execute the first imaging and an instruction to execute the second imaging. In response to these instructions, the first captured image, the second captured image, the first orientation information, and information such as internal parameters of the second imaging unit are provided as needed. In the orientation information calibration process 343, the second orientation information is calibrated based on this information. Then, orientation calibration information is generated as a calibration result and provided to the imaging control process 103 for the second 3D modeling.

[0397] <Example 2 of Calibration Processing>

[0398] In the calibration process 106, any process may be performed and is not limited to Figure 27 For example, in the orientation information calibration process, such as Figure 36 As shown by arrow 501 in , the user can slowly move the smartphone 301 and the ILC 302 (both firmly fixed to each other), and calibrate the second orientation information based on the trajectory. In this case, for example, visual SLAM is performed on the first imaging unit and the second imaging unit independently of each other, and the trajectory of the first imaging unit (arrow 502) and the trajectory of the second imaging unit (arrow 503) are obtained. The trajectories are then superimposed using a technique such as iterative closest point (ICP). The value of the orientation transformation required for the superimposed trajectory is applied as the difference between the positions of the first imaging unit and the second imaging unit, and the second orientation information is calibrated. Such a method can be applied as an orientation information calibration process.

[0399] Furthermore, in the imaging timing calibration process, for example, the imaging timing of the second imaging unit can be calibrated by applying the time difference between the first imaging unit performing the same physical movement and the second imaging unit being located at the same physical position as a time delay based on the result of superimposing the above-mentioned trajectory. Figure 37 As shown, the difference between the start time of the imaging part 521 performing the first imaging (exposure, etc.) and the start time of the imaging part 522 performing the second imaging (exposure, etc.) can be used as a time delay. Such a method can be applied as imaging timing calibration processing.

[0400] In this case, in the calibration process 106, for example, Figure 38 As shown, the installation guide process 342, the orientation information calibration process 541 and the imaging timing calibration process 542 can be performed. In the orientation information calibration process 541, as described above with reference to Figure 36 As described above, the second orientation information is calibrated based on the trajectory of the first imaging unit and the second imaging unit. Then, orientation calibration information is generated as a calibration result, and the orientation calibration information is provided to the imaging control process 103 for the second 3D modeling. In the imaging timing calibration process 542, as described above with reference to Figure 37 As described above, the time difference between the first imaging unit and the second imaging unit performing the same physical movement when they are at the same physical position is applied as a time delay, and the imaging timing of the second imaging unit is calibrated. Then, timing calibration information is generated as a calibration result, and the timing calibration information is provided to the imaging control process 103 for the second 3D modeling. For example, Figure 38As shown, the installation guidance process 342, the orientation information calibration process 541, and the imaging timing calibration process 542 can be performed in this order. Even if the positional and orientation relationship between the first imaging unit and the second imaging unit is unknown, the second orientation information can be calibrated by performing each of the above processes. Note that the installation guidance process 342 can be omitted. In addition, the orientation information calibration process 541 or the imaging timing calibration process 542 can also be omitted.

[0401] <Application Examples>

[0402] Note that by changing the reading mode of the depth sensor (depth detection unit), the depth sensor can be used as the first imaging unit to calibrate the second orientation information. Figure 26 In the case shown on the upper side of , when the depth sensor 311 is able to obtain a captured image, the depth sensor 311 can be used as the first imaging unit to calibrate the second orientation information.

[0403] In addition, Figure 26 In the embodiment, the example in which the processing such as the first imaging and the second imaging described above is performed by the smartphone 301 and the ILC 302 and the example in which the processing is performed by the smartphone 321 has been described; however, the configuration of the device that performs these processes may be any configuration. For example, the processing such as the first imaging and the second imaging may be performed by an ILC to which an auxiliary unit including a combination of a depth sensor and an image sensor is attached. That is, instead of Figure 26 In the example of the smartphone 301, the accessory unit can be attached to the ILC 302. Even with such a configuration, the second orientation information can be calibrated. For example, the calibration process can be performed similarly to the above example.

[0404] <Example of Configuration for Execution Processing>

[0405] Described above Figure 4 (as well as Figure 27 、 Figure 33 、 Figure 38 Each process in the above-mentioned first 3D data generation process 101, the first 3D modeling process, the calibration process 106, and the imaging control process 103 for the second 3D modeling can be executed by any device. For example, in an information processing device, the first 3D modeling process of the above-mentioned first 3D data generation process 101, the calibration process 106, and the imaging control process 103 for the second 3D modeling can be executed.

[0406] That is, the information processing device may include: a first 3D modeling processing unit that generates first orientation information indicating the position and orientation of a first imaging unit that performs first imaging and first three-dimensional shape information representing the three-dimensional shape of the 3D object based on a first captured image generated by first imaging that images the 3D object; a calibration unit that calibrates second orientation information indicating the position and orientation of a second imaging unit that performs second imaging based on the first captured image, the first orientation information, and a second captured image generated by second imaging that images the 3D object; and an imaging control unit that reflects the calibration result in the second orientation information and controls second imaging for generating second three-dimensional shape information representing the three-dimensional shape of the 3D object based on the second orientation information reflecting the calibration result and the first three-dimensional shape information. In this section, the information processing device is also referred to as the first information processing device.

[0407] In addition, the information processing method performed by the first information processing device may include: generating first orientation information indicating the position and orientation of a first imaging unit performing the first imaging and first three-dimensional shape information representing the three-dimensional shape of the 3D object based on a first captured image generated by first imaging of the 3D object; calibrating second orientation information indicating the position and orientation of a second imaging unit performing the second imaging based on the first captured image, the first orientation information and a second captured image generated by second imaging of the 3D object; and reflecting the calibration result in the second orientation information, and controlling second imaging for generating second three-dimensional shape information representing the three-dimensional shape of the 3D object based on the second orientation information and the first three-dimensional shape information reflecting the calibration result.

[0408] This configuration can suppress a decrease in the accuracy of the position and orientation of the second imaging unit, and can also suppress a decrease in navigation accuracy. Thus, captured images can be obtained at a more appropriate position and orientation, and a decrease in the accuracy of 3D data obtained through the second 3D modeling can be suppressed. Furthermore, since the second imaging can be performed at a more appropriate position and orientation, an increase in the number of captured images used in the second 3D modeling can be suppressed. Consequently, an increase in the user's workload can be suppressed. Furthermore, an increase in the load (processing volume, processing time, etc.) of the 3D modeling processing can be suppressed.

[0409] Note that in the first information processing apparatus, the calibration unit may calibrate the second orientation information based on the first captured image and the second captured image captured at the same timing and the first orientation information obtained at the timing.

[0410] In addition, the calibration unit can detect feature points of the first captured image and the second captured image, obtain corresponding points between the feature points of the first captured image and the feature points of the second captured image, determine the size of the 3D object based on the first orientation information, and calibrate the second orientation information based on the corresponding points and the size so that the reprojection error becomes sufficiently small.

[0411] In addition, the calibration unit may also calibrate the second orientation information using the internal parameters of the second imaging unit so that the re-projection error becomes sufficiently small. Note that the internal parameters may be preset values preset for the second imaging unit or the optical system unit used in the second imaging unit.

[0412] Furthermore, the calibration unit may output guidance prompting movement of the first imaging unit and the second imaging unit, and calibrate the second orientation information based on a plurality of first and second captured images obtained by performing the first and second imaging at a plurality of different positions according to the guidance.

[0413] Furthermore, in a case where calibration of the second orientation information is successful, the calibration unit may output a notification indicating successful calibration, and in a case where calibration of the second orientation information fails, the calibration unit may output guidance prompting re-imaging.

[0414] Furthermore, the calibration unit may output guidance prompting installation of the second imaging unit on the first imaging unit in a correct relative position and orientation.

[0415] Furthermore, the calibration unit may also calibrate the imaging timing of the second imaging, and when controlling the second imaging, the imaging control unit may instruct the second imaging at an instruction timing that ensures that the imaging timing reflecting the calibration result is appropriate.

[0416] Furthermore, the calibration unit may calibrate the imaging timing based on a second captured image obtained by capturing the measurement image that changes with time.

[0417] Furthermore, the calibration unit may output guidance prompting to capture the measurement image at the correct position.

[0418] The first information processing device may further include a scoring processing unit that uses the first three-dimensional shape information to evaluate the accuracy of second three-dimensional shape information that can be generated using the second captured image generated by the second imaging performed so far, and generates a scoring result. In this case, the imaging control unit may control the second imaging to generate the second three-dimensional shape information based on the second orientation information reflecting the calibration result, the first three-dimensional shape information, and the scoring result.

[0419] Furthermore, the first 3D modeling processing unit may include an orientation information generating unit that generates first orientation information based on the acceleration and angular velocity of the first imaging unit, and a three-dimensional shape generating unit that generates first three-dimensional shape information based on the first orientation information and the depth of the 3D object. Note that the first three-dimensional shape information may include a mesh indicating the three-dimensional shape of the 3D object through vertices and connections, and a texture applied to a surface of the mesh.

[0420] The first information processing device may further include a depth detection unit for detecting depth, a first imaging unit, and an inertial measurement unit for detecting acceleration and angular velocity. Furthermore, the first information processing device may further include a second imaging unit. Furthermore, the first information processing device may further include an associating unit for associating second orientation information reflecting the calibration result with the second captured image.

[0421] Furthermore, for example, in the first information processing apparatus, the first 3D modeling process of the above-described first 3D data generation process 101 , the calibration process 106 , and the imaging guide output process 105 for the second 3D modeling may be executed.

[0422] That is, the first information processing device may include: a first 3D modeling processing unit, which generates first orientation information indicating the position and orientation of the first imaging unit performing the first imaging and first three-dimensional shape information representing the three-dimensional shape of the 3D object based on a first captured image generated by first imaging of the 3D object; a calibration unit, which calibrates second orientation information indicating the position and orientation of the second imaging unit performing the second imaging based on the first captured image, the first orientation information and a second captured image generated by second imaging of the 3D object; and a guidance information output control unit, which reflects the calibration result in the second orientation information, generates guidance information for the second imaging based on the second orientation information reflecting the calibration result and the first three-dimensional shape information, and controls the output of the guidance information, wherein the second imaging is used to generate second three-dimensional shape information representing the three-dimensional shape of the 3D object.

[0423] In addition, the information processing method performed by the first information processing device may include: generating first orientation information indicating the position and orientation of a first imaging unit performing the first imaging and first three-dimensional shape information representing the three-dimensional shape of the 3D object based on a first captured image generated by first imaging of the 3D object; calibrating second orientation information indicating the position and orientation of a second imaging unit performing the second imaging based on the first captured image, the first orientation information and a second captured image generated by second imaging of the 3D object; and reflecting the calibration result in the second orientation information, generating guidance information for the second imaging based on the second orientation information reflecting the calibration result and the first three-dimensional shape information, and controlling the output of the guidance information, the second imaging being used to generate second three-dimensional shape information representing the three-dimensional shape of the 3D object.

[0424] This configuration can suppress a decrease in the accuracy of the position and orientation of the second imaging unit, and can also suppress a decrease in navigation accuracy. Thus, captured images can be obtained at a more appropriate position and orientation, and a decrease in the accuracy of 3D data obtained through the second 3D modeling can be suppressed. Furthermore, since the second imaging can be performed at a more appropriate position and orientation, an increase in the number of captured images used in the second 3D modeling can be suppressed. Consequently, an increase in the user's workload can be suppressed. Furthermore, an increase in the load (processing volume, processing time, etc.) of the 3D modeling processing can be suppressed.

[0425] Note that in the first information processing apparatus, the calibration unit may calibrate the second orientation information based on the first captured image and the second captured image captured at the same timing and the first orientation information obtained at the timing.

[0426] In addition, the calibration unit can detect feature points of the first captured image and the second captured image, obtain corresponding points between the feature points of the first captured image and the feature points of the second captured image, determine the size of the 3D object based on the first orientation information, and calibrate the second orientation information based on the corresponding points and the size so that the reprojection error becomes sufficiently small.

[0427] In addition, the calibration unit may also calibrate the second orientation information using the internal parameters of the second imaging unit so that the re-projection error becomes sufficiently small. Note that the internal parameters may be preset values preset for the second imaging unit or the optical system unit used in the second imaging unit.

[0428] Furthermore, the calibration unit may output guidance prompting movement of the first imaging unit and the second imaging unit, and calibrate the second orientation information based on a plurality of first and second captured images obtained by performing first and second imaging at a plurality of different positions according to the guidance.

[0429] Furthermore, in a case where calibration of the second orientation information is successful, the calibration unit may output a notification indicating successful calibration, and in a case where calibration of the second orientation information fails, the calibration unit may output guidance prompting re-imaging.

[0430] Furthermore, the calibration unit may output guidance prompting installation of the second imaging unit on the first imaging unit in a correct relative position and orientation.

[0431] The first information processing device may further include a scoring processing unit that uses the first three-dimensional shape information to evaluate the accuracy of second three-dimensional shape information that can be generated using the second captured image generated by the second imaging performed so far, and generates a scoring result. In this case, the guidance information output control unit may generate guidance information based on the second orientation information reflecting the calibration result, the first three-dimensional shape information, and the scoring result, and control the output of the guidance information.

[0432] Furthermore, the first 3D modeling processing unit may include an orientation information generating unit that generates first orientation information based on the acceleration and angular velocity of the first imaging unit, and a three-dimensional shape generating unit that generates first three-dimensional shape information based on the first orientation information and the depth of the 3D object. Note that the first three-dimensional shape information may include a mesh indicating the three-dimensional shape of the 3D object through vertices and connections, and a texture applied to a surface of the mesh.

[0433] The first information processing device may further include a depth detection unit for detecting depth, a first imaging unit, and an inertial measurement unit for detecting acceleration and angular velocity. Furthermore, the first information processing device may further include a second imaging unit. Furthermore, the first information processing device may further include an associating unit for associating second orientation information reflecting the calibration result with the second captured image.

[0434] <6. First embodiment>

[0435] <Imaging Device>

[0436] Figure 39 is a block diagram illustrating an example of the configuration of an imaging device as one aspect of an information processing device to which the present technology is applied. Figure 39 The imaging device 1300 shown in FIG is a device that images a 3D object and performs 3D modeling using the captured image. Note that Figure 39 shows the main configuration including processing units, data flow, etc., and Figure 39 The processing units and data flows shown in are not necessarily complete. That is, the imaging device 1300 may include Figure 39 In addition, there may be devices or processing units not shown as blocks. Figure 39 The data flow or process is shown as arrows or the like.

[0437] like Figure 39 As shown in FIG, the imaging device 1300 includes a first 3D data generation unit 1301, a scoring processing unit 1302, an imaging control unit 1303, a second 3D data generation unit 1304, an encoding unit 1305, a storage unit 1306, a communication unit 1307, an imaging guidance output control unit 1308, and an output unit 1309. Furthermore, the first 3D data generation unit 1301 includes a depth sensor 1311, an imaging unit 1312, an inertial measurement unit (IMU) 1313, and a real-time 3D modeling processing unit 1314. Furthermore, the real-time 3D modeling processing unit 1314 includes a simultaneous localization and mapping (SLAM) unit 1321, a truncated signed distance function (TSDF) updating unit 1322, and a mesh generation unit 1323. Furthermore, the second 3D data generation unit 1304 includes an operation unit 1331, an imaging unit 1332, an image processing unit 1333, and a photogrammetry processing unit 1334. Furthermore, the photogrammetry processing unit 1334 includes structure from motion (SfM) 1341 and multi-view stereo (MVS) 1342 .

[0438] The first 3D data generating unit 1301 performs processing related to the generation of the first 3D data. For example, the first 3D data generating unit 1301 performs Figure 4 The first 3D data generation process 101 in the image processing unit 101 is as follows. The depth sensor 1311 includes a Lidar sensor (dToF module) and the like, detects the depth of the subject, and provides the depth to the TSDF update unit 1322. The imaging unit 1312 includes an image sensor and images the subject to generate a captured image. The imaging unit 1312 performs Figure 4 The first imaging (that is, imaging for first 3D modeling (real-time 3D modeling)) of the first 3D data generation process 101 is performed. The imaging unit 1312 provides the generated captured image to the SLAM 1321. The IMU 1313 detects inertial information about (the acceleration and angular velocity of) the imaging device and provides the inertial information to the SLAM 1321.

[0439] The real-time 3D modeling processing unit 1314 performs processing related to real-time 3D modeling. For example, the real-time 3D modeling processing unit 1314 performs Figure 4 The first 3D modeling process (real-time 3D modeling) of the first 3D data generation process 101 in the real-time 3D modeling processing unit 1314 is to generate first three-dimensional shape information representing the three-dimensional shape of the 3D object based on the first captured image generated by the first imaging of the 3D object.

[0440] The SLAM 1321 performs self-positioning based on the provided first captured image and inertial information, and generates orientation information indicating the position and orientation of the imaging device 1300. The SLAM 1321 provides the generated orientation information to the TSDF updating unit 1322, the imaging control unit 1303, and the imaging guidance output control unit 1308. The TSDF updating unit 1322 updates the TSDF based on the orientation information and depth, and provides the updated TSDF to the mesh generation unit 1323. The mesh generation unit 1323 uses the updated TSDF to generate a mesh (or texture). The mesh generation unit 1323 provides the mesh and texture as first 3D data (first three-dimensional shape information) to the scoring processing unit 1302.

[0441] The scoring processing unit 1302 performs processing related to scoring. For example, the scoring processing unit 1302 performs scoring based on the supplied first 3D data and the imaging viewpoint information (information indicating the position and orientation to perform the second imaging) supplied from the imaging control unit 1303. Figure 4 Scoring processing 102 in the above example is performed. That is, scoring processing unit 1302 uses the first three-dimensional shape information to evaluate the accuracy of second three-dimensional shape information that can be generated using the second captured image generated by the second imaging performed so far, and generates a scoring result. For example, scoring processing unit 1302 may generate a scoring result for each local portion of the first three-dimensional shape information based on the first three-dimensional shape information and the position and orientation of the second imaging performed so far. For example, scoring processing unit 1302 may generate a scoring result for each polygon of the mesh. Scoring processing unit 1302 provides the scoring result to imaging control unit 1303 and imaging guidance output control unit 1308.

[0442] Note that the scoring processing unit 1302 may acquire camera information about the imaging unit 1332 and generate a scoring result based on the camera information. Furthermore, the scoring processing unit 1302 may generate a scoring result based on the position and orientation of the imaging device 1300 corresponding to the timing of the second imaging that does not depend on the imaging control information of the imaging unit 1332.

[0443] The imaging control unit 1303 performs processing related to the control of the second imaging. For example, the imaging control unit 1303 performs Figure 4The imaging control processing 103 for the second 3D modeling in the image processing unit 1303 is performed. That is, the imaging control unit 1303 generates imaging control information based on which the second imaging is controlled based on the provided scoring result and orientation information, and provides the imaging control information to the imaging unit 1332. The imaging control information is, for example, control information for causing the imaging unit 1332 to perform the second imaging (generate a second captured image). That is, the imaging control unit 1303 obtains a position and orientation suitable for the second imaging based on the scoring result, and causes the imaging unit 1332 to perform the second imaging at the position and orientation. In addition, the imaging control unit 1303 provides the imaging viewpoint information indicating the position and orientation of the second imaging to be performed to the scoring processing unit 1302.

[0444] In addition, the imaging control unit 1303 can obtain imaging timing information indicating the timing of the second imaging that is not dependent on the imaging control information of the imaging unit 1332, and provide the orientation information about the imaging device 1300 corresponding to the imaging timing as imaging viewpoint information to the scoring processing unit 1302.

[0445] The second 3D data generating unit 1304 performs processing related to the generation of the second 3D data. For example, the second 3D data generating unit 1304 performs Figure 4 The operation unit 1331 receives an instruction to the imaging unit 1332 from a user or the like, and supplies the instruction to the imaging unit 1332 .

[0446] The imaging unit 1332 includes an image sensor and images a subject to generate a captured image. Figure 4 The imaging unit 1332 supplies the generated captured image to the image processing unit 1333 .

[0447] For example, the imaging unit 1332 may perform the second imaging according to the control of the imaging control unit 1303 (based on the imaging control information provided by the imaging control unit 1303) and generate a second captured image. Furthermore, the imaging unit 1332 may perform the second imaging according to the instruction provided by the operation unit 1331 to generate a second captured image. Furthermore, the imaging unit 1332 may provide camera information (internal parameters, external parameters, and viewing angle information of the imaging unit 1332, etc.) to the scoring processing unit 1302. Furthermore, the imaging unit 1332 may provide imaging timing information indicating the timing of the second imaging that is not dependent on the imaging control information to the imaging control unit 1303 and the imaging guidance output control unit 1308.

[0448] Image processing unit 1333 performs predetermined image processing on the captured image (second captured image) generated by imaging unit 1332. The content of this image processing is arbitrary. Image processing unit 1333 provides the captured image to SfM 1341. In addition, image processing unit 1333 may provide the captured image to encoding unit 1305 and imaging guidance output control unit 1308.

[0449] The photogrammetry processing unit 1334 performs photogrammetry-related processing on the second captured image. For example, the photogrammetry processing unit 1334 performs Figure 4 That is, the photogrammetry processing unit 1334 generates second three-dimensional shape information based on the second captured image generated by the imaging unit 1332.

[0450] For example, the SfM 1341 searches for corresponding points between the second captured images, derives the position and orientation of the camera using epipolar constraints, determines the position of each corresponding point in three-dimensional space based on the position and orientation of the camera using triangulation, optimizes all determined three-dimensional point clouds using bundle adjustment, and provides the optimized three-dimensional point clouds to the MVS 1342. For example, the MVS 1342 further performs a dense corresponding point search using the three-dimensional point clouds, adds the three-dimensional points, and further performs meshing and texturing as post-processing to generate second 3D data. The MVS 1342 provides the generated second 3D data to the encoding unit 1305.

[0451] The encoding unit 1305 encodes the supplied second 3D data and supplies the encoded data to the storage unit 1306 and the communication unit 1307. In addition, the encoding unit 1305 may encode the supplied second captured image and supply the encoded data to the storage unit 1306 and the communication unit 1307.

[0452] The storage unit 1306 stores the supplied coded data. The communication unit 1307 transmits the supplied coded data to another information processing device (eg, a server, etc.).

[0453] The imaging guidance output control unit 1308 performs processing related to guidance for the second imaging. For example, the imaging guidance output control unit 1308 performs Figure 41305 for the second 3D modeling. That is, the imaging guidance output control unit 1308 generates guidance information for the second imaging and controls the output of the guidance information. For example, the imaging guidance output control unit 1308 generates the above-mentioned guidance information based on the provided scoring result and the orientation information about the imaging device 1300. In addition, the imaging guidance output control unit 1308 can generate the guidance information based on the provided imaging timing information. The imaging guidance output control unit 1308 provides the generated guidance information to the output unit 1309 so that the output unit 1309 outputs the guidance information as, for example, an image, audio, etc. In addition, the imaging guidance output control unit 1308 can superimpose the provided captured image on the guidance information (image) for display.

[0454] The output unit 1309 outputs guidance information as an image, audio, or the like according to control of the imaging guidance output control unit 1308 .

[0455] Such a configuration enables the imaging device 1300 to image the 3D object in a more appropriate position and orientation and perform 3D modeling (second 3D modeling process) using the captured image. Therefore, the imaging device 1300 can generate more accurate 3D data while suppressing the increase in the load of 3D modeling. In addition, the imaging device 1300 can output guidance information so that the user can perform the second imaging in a more appropriate position and orientation. In other words, the imaging device 1300 can perform 3D modeling (second 3D modeling process) using the captured image. Therefore, the imaging device 1300 can generate more accurate 3D data while suppressing the increase in the load of 3D modeling. In other words, the user can perform 3D modeling more easily.

[0456] <3D Modeling Process>

[0457] Will refer to Figure 40 An example of the flow of the 3D modeling process performed by the imaging device 1300 is described with reference to the flowchart in FIG.

[0458] When the 3D modeling process starts, in step S301 , the depth sensor 1311 , the imaging unit 1312 , and the IMU 1313 acquire depth, captured images, and inertial information.

[0459] In step S302 , the real-time 3D modeling processing unit 1314 performs real-time 3D modeling processing to generate first 3D data.

[0460] In step S303 , the scoring processing unit 1302 performs scoring on the first 3D data based on the second imaging performed so far.

[0461] In step S304 , the imaging guide output control unit 1308 generates imaging guide (guidance information) for the second imaging based on the scoring result, orientation information, etc. The output unit 1309 outputs the imaging guide (guidance information).

[0462] In step S305 , the imaging control unit 1303 controls imaging for photogrammetry (second imaging) based on the scoring result, orientation information, and the like.

[0463] In step S306 , the imaging unit 1332 performs imaging according to the control (performs the second imaging).

[0464] In step S307, the imaging control unit 1303 and the imaging guidance output control unit 1308 acquire camera information from the imaging unit 1332. Furthermore, the scoring processing unit 1302 acquires imaging timing information from the imaging unit 1332.

[0465] In step S308, the imaging control unit 1303 determines whether to terminate the photogrammetric imaging (second imaging). If it is determined that the photogrammetric imaging is not to be terminated, the process returns to step S303. If it is determined that the photogrammetric imaging is to be terminated in step S308, the process proceeds to step S309.

[0466] In step S309 , the photogrammetry processing unit 1334 performs photogrammetry processing to generate second 3D data.

[0467] In step S310 , the encoding unit 1305 encodes the second 3D data.

[0468] In step S311, the storage unit 1306 stores the encoded data. In addition, the communication unit 1307 transmits the encoded data to another device (for example, a server, etc.).

[0469] Completion of step S311 ends the 3D modeling process.

[0470] <Process of real-time 3D modeling>

[0471] Will refer to Figure 41 The flowchart in Figure 40 An example of the flow of the real-time 3D modeling process performed in step S302 in FIG.

[0472] When the real-time 3D modeling process is started, in step S331 , the SLAM 1321 derives orientation information indicating the three-dimensional orientation of the imaging device 1300 based on the captured image and the inertial information.

[0473] In step S332 , the TSDF updating unit 1322 updates the TSDF based on the captured image, orientation information, and depth.

[0474] In step S333 , the mesh generation unit 1323 generates first 3D data based on the updated TSDF.

[0475] Completion of step S333 ends the real-time 3D modeling process, and the process returns to Figure 40 .

[0476] <Photogrammetry Processing Flow>

[0477] Will refer to Figure 42 The flowchart in Figure 40 An example of the flow of the photogrammetry processing performed in step S309 in FIG.

[0478] When the photogrammetry process is started, in step S351 , the SfM 1341 detects corresponding points between captured images.

[0479] In step S352 , the SfM 1341 finds the three-dimensional orientation of the camera using epipolar constraints.

[0480] In step S353 , the SfM 1341 uses triangulation to obtain three-dimensional points.

[0481] In step S354 , the SfM 1341 performs overall optimization using bundle adjustment.

[0482] In step S355 , the MVS 1342 obtains three-dimensional points through dense corresponding point search and generates second 3D data.

[0483] Completion of step S355 ends the photogrammetry process, and the process returns to Figure 40 .

[0484] By performing each of the processes described above, the imaging device 1300 can image the 3D object in a more appropriate position and orientation, and can perform 3D modeling (second 3D modeling process) using the captured image. Therefore, the imaging device 1300 can generate more accurate 3D data while suppressing the increase in the load of 3D modeling. In addition, the imaging device 1300 can output guidance information so that the user can perform the second imaging in a more appropriate position and orientation. That is, the imaging device 1300 can perform 3D modeling (second 3D modeling process) using the captured image. Therefore, the imaging device 1300 can generate more accurate 3D data while suppressing the increase in the load of 3D modeling. That is, the user can perform 3D modeling more easily.

[0485] <7. Second embodiment>

[0486] <Information Processing System 1>

[0487] The present technology is not limited to the above examples and can be applied to any configuration. For example, the present technology can be applied to an information processing system that performs 3D modeling.

[0488] For example, in an information processing system including an information processing device and an imaging device, the information processing device may include: a first 3D modeling processing unit that generates first three-dimensional shape information representing the three-dimensional shape of a 3D object based on a first captured image generated by first imaging of the 3D object; a scoring processing unit that uses the first three-dimensional shape information to evaluate the accuracy of second three-dimensional shape information that can be generated using a second captured image generated by second imaging performed so far, and generates a scoring result; an imaging control unit that generates imaging control information based on which the second imaging of the 3D object is controlled based on the position and orientation of the imaging device and the scoring result; and a first communication unit that provides the imaging control information to the imaging device. Furthermore, the imaging device may include: a second communication unit that acquires the imaging control information provided from the information processing device; and an imaging unit that images the 3D object based on the imaging control information to generate the second captured image.

[0489] Figure 43 is a diagram illustrating a configuration example of one aspect of an information processing system to which the present technology is applied. Figure 43 The information processing system 1400 shown in FIG is a system for imaging a 3D object and performing 3D modeling using the captured image. Figure 43 As shown in FIG, an information processing system 1400 includes an imaging communication device 1401, an imaging device 1402, and a server 1403. The imaging communication device 1401 and the server 1403 are communicatively connected via a network 1404. The network 1404 is a communication path including any communication medium such as the Internet, a local area network (LAN), or a wireless LAN.

[0490] Imaging communication device 1401 is an information processing device that has both communication and imaging functions and is capable of communicating with any device, such as a smartphone, via network 1404. Imaging device 1402 is an information processing device that has an imaging function, such as a digital camera. Imaging device 1402 can communicate only with imaging communication device 1401. Imaging communication device 1401 and imaging device 1402 are fixedly connected to each other and are each used by a user as a terminal device 1410. Server 1403 acquires a second captured image generated in terminal device 1410 (imaging device 1402), performs second 3D modeling (photogrammetry processing) using the second captured image to generate second 3D data, and stores (manages) the second 3D data.

[0491] Figure 44 1401 is a block diagram showing a main configuration example of the imaging communication device 1401. Note that Figure 44 shows the main configuration including processing units, data flow, etc., and Figure 44 The processing units and data flows shown in are not necessarily complete. That is, the imaging communication device 1401 may include Figure 44 In addition, there may be devices or processing units not shown as blocks. Figure 44 The data flow or process is shown as arrows or the like.

[0492] like Figure 44 As shown in FIG, the imaging communication device 1401 includes a communication unit 1421, rather than the imaging device 1300 ( Figure 39 ) is a second 3D data generating unit 1304 of the components of the imaging device 1300. That is, the other components are similar to those of the imaging device 1300.

[0493] The communication unit 1421 is communicatively connected to the imaging device 1402 and communicates with the imaging device 1402 to exchange information. For example, the communication unit 1421 can provide the imaging control information provided by the imaging control unit 1303 to the imaging device 1402. In addition, the communication unit 1421 can obtain a second captured image generated by the imaging device 1402 and provide the second captured image to the encoding unit 1305 and the imaging guidance output control unit 1308. In addition, the communication unit 1421 can obtain camera information provided by the imaging device 1402 and provide the camera information to the scoring processing unit 1302. The camera information may include internal parameters, external parameters, and viewing angle information of the imaging device 1402 (or the imaging unit 1332 thereof). In addition, the communication unit 1421 can obtain imaging timing information provided by the imaging device 1402 and provide the imaging timing information to the imaging control unit 1303 and the imaging guidance output control unit 1308. This imaging timing information indicates the timing of imaging performed by (the imaging unit 1332 of) the imaging device 1402 without depending on the imaging control information.

[0494] Note that the communication unit 1307 is communicatively connected to the server 1403 via the network 1404 and communicates with the server 1403 to exchange information. For example, the encoding unit 1305 encodes the second captured image provided from the communication unit 1421 and provides the encoded data to the storage unit 1306 and the communication unit 1307. The storage unit 1306 stores the encoded data of the second captured image. The communication unit 1307 provides the encoded data of the second captured image to the server 1403 via the network 1404.

[0495] Figure 45 1402 is a block diagram showing a main configuration example of the imaging device 1402. Note that Figure 45 shows the main configuration including processing units, data flow, etc., and Figure 45 The processing units and data flows shown in FIG are not necessarily exhaustive. That is, the imaging device 1402 may include Figure 45 In addition, there may be devices or processing units not shown as blocks. Figure 45 The data flow or process is shown as arrows or the like.

[0496] like Figure 45 As shown in FIG, the imaging device 1402 includes an operation unit 1331, an imaging unit 1332, an image processing unit 1333, a communication unit 1431, an encoding unit 1432, and a storage unit 1433. The operation unit 1331, the imaging unit 1332, and the image processing unit 1333 perform the same Figure 39 The case of the imaging device 1300 is processed similarly.

[0497] Communication unit 1431 is communicatively connected to imaging communication device 1401 and communicates with imaging communication device 1401 to exchange information. For example, communication unit 1431 can acquire imaging control information provided by imaging communication device 1401 and provide the imaging control information to imaging unit 1332. Furthermore, communication unit 1431 can acquire camera information provided by imaging unit 1332 and provide the camera information to imaging communication device 1401. The camera information may include internal parameters, external parameters, and viewing angle information of imaging unit 1332. Furthermore, communication unit 1431 can acquire imaging timing information provided by imaging unit 1332 and provide the imaging timing information to imaging communication device 1401. The imaging timing information indicates the timing of imaging performed by imaging unit 1332, independent of the imaging control information. Furthermore, communication unit 1431 can acquire a second captured image provided by image processing unit 1333 and provide the second captured image to imaging communication device 1401.

[0498] The encoding unit 1432 encodes the second captured image supplied from the image processing unit 1333, and supplies the encoded data to the storage unit 1433. The storage unit 1433 stores the encoded data.

[0499] Figure 46 1403 is a block diagram showing a main configuration example of the server 1403. Note that Figure 46 shows the main configuration including processing units, data flow, etc., and Figure 46 The processing units and data flows shown in are not necessarily complete. That is, the server 1403 may include Figure 46 In addition, there may be devices or processing units not shown as blocks. Figure 46 The data flow or process is shown as arrows or the like.

[0500] like Figure 46 As shown in FIG, the server 1403 includes a communication unit 1441, a decoding unit 1442, a photogrammetry processing unit 1334, an encoding unit 1444, and a storage unit 1445. The photogrammetry processing unit 1334 has Figure 39 The imaging device 1300 is configured similarly to the configuration in FIG. 1 and performs similar processing.

[0501] The communication unit 1441 is communicatively connected to the imaging communication device 1401 via the network 1404, and communicates with other devices such as the imaging communication device 1401 to exchange information. For example, the communication unit 1441 acquires the encoded data of the second captured image provided from the imaging communication device 1401, and provides the encoded data to the decoding unit 1442. In addition, the communication unit 1441 can provide the encoded data of the second 3D data provided from the encoding unit 1444 to another device (for example, the imaging communication device 1401) via the network 1404.

[0502] The decoding unit 1442 decodes the encoded data of the second captured image provided by the communication unit 1441 to generate (restore) the second captured image. The decoding unit 1442 provides the second captured image to the photogrammetry processing unit 1334 (SfM1341). The photogrammetry processing unit 1334 performs second 3D modeling (photogrammetry processing) using the second captured image to generate second 3D data. The photogrammetry processing unit 1334 (MVS1342) provides the generated second 3D data to the encoding unit 1444.

[0503] The encoding unit 1444 encodes the supplied second 3D data and supplies the encoded data to the storage unit 1445. Also, the encoding unit 1444 may supply the encoded data of the second 3D data to the communication unit 1441. The storage unit 1445 stores the supplied encoded data of the second 3D data.

[0504] Since each device has such a configuration, the information processing system 1400 can image the 3D object in a more appropriate position and orientation and perform 3D modeling (second 3D modeling process) using the captured image. Therefore, the information processing system 1400 can generate more accurate 3D data while suppressing the increase in the load of 3D modeling. In addition, the information processing system 1400 can output guidance information so that the user can perform the second imaging in a more appropriate position and orientation. That is, the information processing system 1400 can perform 3D modeling (second 3D modeling process) using the captured image. Therefore, the information processing system 1400 can generate more accurate 3D data while suppressing the increase in the load of 3D modeling. That is, the user can perform 3D modeling more easily.

[0505] <3D Modeling Process>

[0506] Will refer to Figure 47 and Figure 48 An example of the flow of 3D modeling processing performed by the information processing system 1400 is described with reference to the flowchart in FIG.

[0507] When starting the 3D modeling process, Figure 47 In step S401 of FIG. 1 , the depth sensor 1311 , the imaging unit 1312 , and the IMU 1313 of the imaging communication device 1401 acquire depth, capture images, and inertial information.

[0508] In step S402, the real-time 3D modeling processing unit 1314 of the imaging communication device 1401 performs real-time 3D modeling processing to generate first 3D data. This real-time 3D modeling processing is similar to Figure 41 The examples in are executed similarly.

[0509] In step S403 , the scoring processing unit 1302 of the imaging communication device 1401 performs scoring on the first 3D data based on the second imaging performed so far.

[0510] In step S404 , the imaging guide output control unit 1308 of the imaging communication device 1401 generates imaging guide (guide information) for the second imaging based on the scoring result, orientation information, etc. The output unit 1309 outputs the imaging guide (guide information).

[0511] In step S405, the imaging control unit 1303 of the imaging communication device 1401 generates imaging control information for controlling the imaging for photogrammetry (second imaging) based on the scoring result, orientation information, etc. The communication unit 1421 provides the imaging control information to the imaging device 1402. In step S411, the communication unit 1431 of the imaging device 1402 acquires the imaging control information.

[0512] In step S412, the imaging unit 1332 of the imaging device 1402 performs imaging (performs second imaging) according to the control to generate a second captured image. The image processing unit 1333 performs predetermined image processing on the second captured image.

[0513] In step S413, the communication unit 1431 of the imaging communication device 1402 provides the second captured image to the imaging communication device 1401. In step S406, the communication unit 1421 of the imaging communication device 1401 acquires the second captured image.

[0514] Furthermore, in step S414, the communication unit 1431 of the imaging device 1402 provides the imaging device information and imaging timing information about the imaging unit 1332 to the imaging communication device 1401. In step S407, the communication unit 1421 of the imaging communication device 1401 acquires the imaging device information and imaging timing information.

[0515] exist Figure 48 In step S441 in FIG. 1 , the encoding unit 1432 of the imaging device 1402 encodes the second captured image. The storage unit 1433 stores the encoded data of the second captured image.

[0516] In step S431, encoding unit 1305 of imaging communication device 1401 encodes the second captured image. Communication unit 1307 provides the encoded data of the second captured image to server 1403. In step S451, communication unit 1441 of server 1403 obtains the encoded data of the second captured image. Decoding unit 1442 decodes the encoded data to generate (restore) the second captured image.

[0517] In step S452, the photogrammetric processing unit 1334 of the server 1403 performs photogrammetric processing to generate second 3D data. Figure 42 The examples in are executed similarly.

[0518] In step S453 , the encoding unit 1444 of the server 1403 encodes the second 3D data.

[0519] In step S454, the storage unit 1445 of the server 1403 stores the encoded data. Furthermore, the communication unit 1441 transmits the encoded data to another device (for example, the imaging communication device 1401 or the like).

[0520] Furthermore, in step S432, the imaging control unit 1303 of the imaging communication device 1401 determines whether to terminate the imaging for photogrammetry (second imaging). In the case where it is determined that the imaging for photogrammetry is not to be terminated, the process returns to Figure 47 In step S403. Figure 48 In a case where it is determined in step S432 that the imaging for photogrammetry is terminated, the 3D modeling process is ended.

[0521] By performing each of the processes described above, the information processing system 1400 can image the 3D object in a more appropriate position and orientation and perform 3D modeling using the captured image (second 3D modeling process). Therefore, the information processing system 1400 can generate more accurate 3D data while suppressing the increase in the load of 3D modeling. In addition, the information processing system 1400 can output guidance information so that the user can perform the second imaging in a more appropriate position and orientation. That is, the information processing system 1400 can perform 3D modeling using the captured image (second 3D modeling process). Therefore, the information processing system 1400 can generate more accurate 3D data while suppressing the increase in the load of 3D modeling. That is, the user can perform 3D modeling more easily.

[0522] <Information Processing System 2>

[0523] Note that in the information processing system 1400 , the scoring process may be performed by the server 1403 .

[0524] Figure 49 1401 in this case. Figure 49 shows the main configuration including processing units, data flow, etc., and Figure 49 The processing units and data flows shown in are not necessarily complete. That is, the imaging communication device 1401 may include Figure 49 In addition, there may be devices or processing units not shown as blocks. Figure 49 The data flow or process is shown as arrows or the like.

[0525] like Figure 49 As shown in FIG, in the imaging communication device 1401 in this case, from Figure 44 The configuration in omits the scoring processing unit 1302. In this case, the communication unit 1307 provides the imaging viewpoint information provided from the imaging control unit 1303 to the server 1403.

[0526] In this case, the real-time 3D modeling processing unit 1314 (mesh generation unit 1323) provides the generated first 3D data to the encoding unit 1305. The encoding unit 1305 encodes the first 3D data and provides the encoded data to the communication unit 1307. The communication unit 1307 provides the encoded data of the first 3D data provided by the encoding unit 1305 to the server 1403.

[0527] Furthermore, the communication unit 1307 acquires the scoring result by (the scoring processing unit 1302 of) the server 1403 , and supplies the scoring result to the imaging control unit 1303 and the imaging guidance output control unit 1308 .

[0528] In addition, with Figure 45 Similar to the case in , the communication unit 1307 provides the encoded data of the second captured image provided from the encoding unit 1305 to the server 1403.

[0529] Furthermore, in this case, the communication unit 1421 acquires the camera information (regarding the imaging unit 1332) provided from the imaging device 1402, and provides the camera information to the encoding unit 1305. The encoding unit 1305 encodes the camera information and provides the encoded data to the communication unit 1307. The communication unit 1307 provides the encoded data of the camera information to the server 1403.

[0530] Figure 50 1403 is a block diagram showing a main configuration example of the server 1403. Note that Figure 50 shows the main configuration including processing units, data flow, etc., and Figure 50 The processing units and data flows shown in are not necessarily complete. That is, the server 1403 may include Figure 50 In addition, there may be devices or processing units not shown as blocks. Figure 50 The data flow or process is shown as arrows or the like.

[0531] like Figure 50 In this case, in addition to Figure 46 In addition to the components in FIG. 1 , server 1403 further includes scoring processing unit 1302. In this case, communication unit 1441 acquires the encoded data of the first 3D data provided from imaging communication device 1401 and provides the encoded data to decoding unit 1442. Decoding unit 1442 decodes the encoded data to generate (restore) the first 3D data. Decoding unit 1442 provides the first 3D data to scoring processing unit 1302.

[0532] Furthermore, the communication unit 1441 acquires the imaging viewpoint information supplied from the imaging communication device 1401, and supplies the imaging viewpoint information to the decoding unit 1442. The decoding unit 1442 supplies the imaging control information to the scoring processing unit 1302.

[0533] Furthermore, the communication unit 1441 acquires the encoded data of the camera information supplied from the imaging communication device 1401 and supplies the encoded data to the decoding unit 1442. The decoding unit 1442 decodes the encoded data to generate (restore) the camera information. The decoding unit 1442 supplies the camera information to the scoring processing unit 1302.

[0534] In addition, with Figure 46Similar to the case in FIG, the communication unit 1441 acquires the encoded data of the second captured image provided from the imaging communication device 1401 and provides the encoded data to the decoding unit 1442. The decoding unit 1442 decodes the encoded data to generate (restore) the second captured image. The decoding unit 1442 provides the second captured image to the photogrammetry processing unit 1334.

[0535] In this case as well, the scoring processing unit 1302 performs the following operations based on the supplied first 3D data and imaging viewpoint information: Figure 4 Scoring process 102 in the image sensor is performed to obtain a scoring result. In addition, scoring process 102 can be performed based on the camera information. Scoring process 102 is provided by encoding unit 1444. Encoding unit 1444 provides the scoring result to communication unit 1441. Communication unit 1441 provides the scoring result to imaging communication device 1401.

[0536] Other processing and Figure 46 The processing in is similar.

[0537] Since each device has such a configuration, in this case, the information processing system 1400 can also image the 3D object in a more appropriate position and orientation and perform 3D modeling using the captured image (second 3D modeling process). Therefore, the information processing system 1400 can generate more accurate 3D data while suppressing the increase in the load of 3D modeling. In addition, the information processing system 1400 can output guidance information so that the user can perform the second imaging in a more appropriate position and orientation. That is, the information processing system 1400 can perform 3D modeling using the captured image (second 3D modeling process). Therefore, the information processing system 1400 can generate more accurate 3D data while suppressing the increase in the load of 3D modeling. That is, the user can perform 3D modeling more easily.

[0538] <3D Modeling Process>

[0539] Will refer to Figure 51 and Figure 52 An example of the flow of 3D modeling processing performed by the information processing system 1400 is described with reference to the flowchart in FIG.

[0540] When starting the 3D modeling process, Figure 51 In step S501 of FIG. 1 , the depth sensor 1311 , the imaging unit 1312 , and the IMU 1313 of the imaging communication device 1401 acquire depth, capture images, and inertial information.

[0541] In step S502, the real-time 3D modeling processing unit 1314 of the imaging communication device 1401 performs real-time 3D modeling processing to generate first 3D data. This real-time 3D modeling processing is similar to Figure 41 The examples in are executed similarly.

[0542] In step S503, the communication unit 1307 of the imaging communication apparatus 1401 provides the generated first 3D data to the server 1403. In step S521, the communication unit 1441 of the server 1403 acquires the first 3D data.

[0543] In step S522 , the scoring processing unit 1302 of the server 1403 performs scoring on the first 3D data based on the second imaging performed so far.

[0544] In step S523, the communication unit 1441 of the server 1403 provides the scoring result to the imaging communication apparatus 1401. In step S504, the communication unit 1307 of the imaging communication apparatus 1401 acquires the scoring result.

[0545] In step S505 , the imaging guide output control unit 1308 of the imaging communication device 1401 generates imaging guide (guide information) for the second imaging based on the scoring result, orientation information, etc. The output unit 1309 outputs the imaging guide (guide information).

[0546] In step S506, the imaging control unit 1303 of the imaging communication device 1401 generates imaging control information for controlling the imaging for photogrammetry (second imaging) based on the scoring result, orientation information, etc. The communication unit 1421 provides the imaging control information to the imaging device 1402. In step S511, the communication unit 1431 of the imaging device 1402 acquires the imaging control information.

[0547] Furthermore, in step S507, the communication unit 1307 of the imaging communication device 1401 provides the imaging viewpoint information to the server 1403. In step S524, the communication unit 1441 of the server 1403 acquires the imaging viewpoint information.

[0548] exist Figure 52 In step S541 in FIG. 5 , the imaging unit 1332 of the imaging device 1402 performs imaging (performs second imaging) according to the imaging control information to generate a second captured image. The image processing unit 1333 performs predetermined image processing on the second captured image.

[0549] In step S542, the communication unit 1431 of the imaging communication device 1402 provides the second captured image to the imaging communication device 1401. In step S531, the communication unit 1421 of the imaging communication device 1401 acquires the second captured image.

[0550] Furthermore, in step S543, the communication unit 1431 of the imaging device 1402 provides the camera information and imaging timing information about the imaging unit 1332 to the imaging communication device 1401. In step S532, the communication unit 1421 of the imaging communication device 1401 acquires the camera information and imaging timing information.

[0551] In step S544, the encoding unit 1432 of the imaging device 1402 encodes the second captured image. The storage unit 1433 stores the encoded data of the second captured image.

[0552] In step S533, encoding unit 1305 of imaging communication device 1401 encodes the second captured image. Communication unit 1307 provides the encoded data of the second captured image to server 1403. In step S551, communication unit 1441 of server 1403 obtains the encoded data of the second captured image. Decoding unit 1442 decodes the encoded data to generate (restore) the second captured image.

[0553] In step S552, the photogrammetric processing unit 1334 of the server 1403 performs photogrammetric processing to generate second 3D data. Figure 42 The examples in are executed similarly.

[0554] In step S553 , the encoding unit 1444 of the server 1403 encodes the second 3D data.

[0555] In step S554, the storage unit 1445 of the server 1403 stores the encoded data. Furthermore, the communication unit 1441 transmits the encoded data to another device (for example, the imaging communication device 1401 or the like).

[0556] Furthermore, in step S534, the imaging control unit 1303 of the imaging communication device 1401 determines whether to terminate the imaging for photogrammetry (second imaging). In the case where it is determined that the imaging for photogrammetry is not to be terminated, the process returns to Figure 51 In step S522. Figure 52 In a case where it is determined in step S534 that the imaging for photogrammetry is terminated, the 3D modeling process is ended.

[0557] By performing each of the processes described above, the information processing system 1400 can image the 3D object in a more appropriate position and orientation and perform 3D modeling using the captured image (second 3D modeling process). Therefore, the information processing system 1400 can generate more accurate 3D data while suppressing the increase in the load of 3D modeling. In addition, the information processing system 1400 can output guidance information so that the user can perform the second imaging in a more appropriate position and orientation. That is, the information processing system 1400 can perform 3D modeling using the captured image (second 3D modeling process). Therefore, the information processing system 1400 can generate more accurate 3D data while suppressing the increase in the load of 3D modeling. That is, the user can perform 3D modeling more easily.

[0558] <8. Third embodiment>

[0559] <Applying Calibration Processing to Information Processing System>

[0560] The present technology described above in <5. Calibration Process> can be applied to any information processing device. For example, the present technology can be applied to any system and device described in the first embodiment and the second embodiment.

[0561] For example, the present technology described above in <5. Calibration Process> can be applied to Figure 43 Hereinafter, such a case will be described.

[0562] <Imaging Communication Device>

[0563] Figure 53 1401 is a block diagram showing a main configuration example of the imaging communication device 1401. Note that Figure 53 shows the main configuration including processing units, data flow, etc., and Figure 53 The processing units and data flows shown in are not necessarily complete. That is, the imaging communication device 1401 may include Figure 53 In addition, there may be devices or processing units not shown as blocks. Figure 53 The data flow or process is shown as arrows or the like.

[0564] like Figure 53 In this case, in addition to the reference Figure 44 In addition to the components (processing units) described above, the imaging communication device 1401 also includes a calibration unit 1601. That is, the other components are the same as those described above. Figure 44 Similar to those components in .

[0565] Calibration unit 1601 performs Figure 4 That is, the calibration unit 1601 can perform the calibration process 106. Figure 27Each process and Figure 33 . For example, the calibration unit 1601 acquires a first captured image generated by the imaging unit 1312. In addition, the calibration unit 1601 acquires first orientation information derived by the real-time 3D modeling processing unit 1314. In addition, the calibration unit 1601 acquires a second captured image generated by the imaging unit 1332 of the imaging device 1402 via the communication unit 1421. The calibration unit 1601 calibrates the second orientation information indicating the position and orientation of the imaging unit 1332 (second imaging unit) based on this information, and generates configuration information as a calibration result. The calibration unit 1601 provides the configuration information to the imaging control unit 1303, the imaging guidance output control unit 1308, or both.

[0566] The imaging control unit 1303 controls the second imaging using the calibration information supplied from the calibration unit 1601. Similarly, the imaging guide output control unit 1308 generates guide information for the second imaging using the calibration information supplied from the calibration unit 1601, and controls output of the guide information.

[0567] <Calibration Unit>

[0568] Figure 54 16 is a block diagram showing a main configuration example of the calibration unit 1601. Figure 54 As shown in , the calibration unit 1601 includes an imaging timing calibration unit 1611 , an installation guide processing unit 1612 , and an orientation information calibration unit 1613 .

[0569] Imaging timing calibration unit 1611 performs Figure 27 341 in the imaging timing calibration process. For example, the imaging timing calibration unit 1611 provides an image display instruction for instructing the display of the measurement image to the output unit 1309. Furthermore, the imaging timing calibration unit 1611 provides an imaging instruction to the imaging unit 1312 and the imaging unit 1332 of the imaging device 1402. Furthermore, in response to the instruction, the imaging timing calibration unit 1611 obtains the first captured image provided by the imaging unit 1312 and the second captured image provided by the imaging unit 1332 of the imaging device 1402. Based on this information, the imaging timing calibration unit 1611 calibrates the imaging timing of the second imaging unit and generates timing calibration information as a calibration result. The imaging timing calibration unit 1611 provides the generated timing calibration information to the imaging control unit 1303.

[0570] Installation boot processing unit 1612 executes Figure 27Installation guidance processing 342 in the process. For example, the installation guidance processing unit 1612 instructs the output unit 1309 to display a guide (image) prompting the user to install the second imaging unit (ILC 302) on the first imaging unit (smartphone 301) in the correct relative position and orientation. In addition, the installation guidance processing unit 1612 obtains a captured image obtained by imaging the guide displayed on the output unit 1309.

[0571] The orientation information calibration unit 1613 performs Figure 27 That is, the orientation information calibration unit 1613 can perform the orientation information calibration process 343. Figure 33 Each process in the process. For example, the orientation information calibration unit 1613 instructs display guidance or instructs imaging. Furthermore, the orientation information calibration unit 1613 obtains the first captured image, the second captured image, orientation information, internal parameters, and the like. The orientation information calibration unit 1613 uses this information to calibrate the orientation information and generates orientation calibration information as a calibration result. The orientation information calibration unit 1613 provides the generated orientation calibration information to the imaging control unit 1303, the imaging guidance output control unit 1308, or both.

[0572] Through such a configuration, the imaging communication device 1401 can control (instruct) the second imaging so that the second imaging is performed at the appropriate timing (desired position). In addition, the imaging communication device 1401 can install the second imaging unit on the first imaging unit in the correct relative position and orientation. In addition, the imaging communication device 1401 can suppress the reduction in the accuracy of the position and orientation of the second imaging unit. Therefore, the imaging communication device 1401 can suppress the reduction in the accuracy of the navigation of the second imaging. Therefore, the imaging communication device 1401 can obtain captured images in a more appropriate position and orientation, and can suppress the reduction in the accuracy of the 3D data obtained by the second 3D modeling. In addition, since the second imaging can be performed in a more appropriate position and orientation, the imaging communication device 1401 can suppress the increase in the number of captured images used in the second 3D modeling. Therefore, the imaging communication device 1401 can suppress the increase in the user's workload. In addition, the imaging communication device 1401 can suppress the increase in the load (processing volume, processing time, etc.) of the 3D modeling processing.

[0573] <3D Modeling Process>

[0574] Next, refer to Figure 55 An example of the flow of the 3D modeling process performed by the imaging communication device 1401 in this case is described with reference to the flowchart in FIG.

[0575] When the 3D modeling process is started, in step S601 , the imaging timing calibration unit 1611 of the imaging communication device 1401 performs imaging timing calibration processing.

[0576] In step S602 , the installation boot processing unit 1612 executes installation boot processing.

[0577] In step S603 , the orientation information calibration unit 1613 performs orientation information calibration processing.

[0578] Steps S604 to S606 are respectively Figure 40 Steps S301 to S303 in are similarly executed.

[0579] In step S607, the imaging guide output control unit 1308 reflects the orientation calibration information in the second orientation information. In addition, the imaging guide output control unit 1308 generates imaging guidance based on the second orientation information reflecting the orientation calibration information, the scoring result, etc., and provides the imaging guidance to the output unit 1309 for output.

[0580] In step S608, the imaging control unit 1303 reflects the orientation calibration information in the second orientation information. In addition, the imaging control unit 1303 controls the second imaging (imaging for photogrammetry) at the timing reflecting the timing calibration information based on the second orientation information reflecting the orientation calibration information, the scoring result, etc.

[0581] Completion of step S608 ends the 3D modeling process.

[0582] <Flow of Imaging Timing Calibration Processing>

[0583] Next, refer to Figure 56 The flowchart in Figure 55 An example of the flow of the imaging timing calibration process performed in step S601 in FIG.

[0584] When the imaging timing calibration process is started, in step S621 , the imaging timing calibration unit 1611 displays a guide image and directs the second imaging unit toward the display unit.

[0585] In step S622 , the imaging timing calibration unit 1611 sets the imaging parameters of the second imaging unit.

[0586] In step S623 , the imaging timing calibration unit 1611 displays the measurement image and instructs the second imaging.

[0587] In step S624 , the imaging timing alignment unit 1611 acquires a second captured image.

[0588] In step S625, the imaging timing calibration unit 1611 obtains a time lag based on the second captured image, calibrates the second orientation information based on the time lag, and generates timing calibration information as a calibration result. In step S626, the generated timing calibration information is provided to the imaging control unit 1303. This enables the imaging control unit 1303 to control the second imaging using the timing calibration information.

[0589] Completion of step S625 ends the imaging timing calibration process, and the process returns to Figure 55 .

[0590] <Installation Boot Processing Flow>

[0591] Next, refer to Figure 57 The flowchart in Figure 55 An example of the flow of the installation boot process performed in step S602 in .

[0592] When the installation guidance process starts, in step S641, the installation guidance processing unit 1612 displays the second captured image. In step S642, the installation guidance processing unit 1612 displays the target viewing angle position. In step S643, the installation guidance processing unit 1612 displays the viewing angle of the first imaging unit based on the first orientation information.

[0593] In step S644, the installation guidance processing unit 1612 determines whether the angle of view of the first imaging unit is aligned with the target angle of view position. If it is determined that the angle of view of the first imaging unit is not aligned with the target angle of view position, the process returns to step S643 and the subsequent processing is repeated.

[0594] Furthermore, in the case where it is determined in step S644 that the viewing angle of the first imaging unit is aligned with the target viewing angle position due to the user moving the first imaging unit (the imaging communication device 1401 equipped with the first imaging unit (imaging unit 1312)), the installation guidance processing is ended, and the processing returns to step S644. Figure 55 .

[0595] <Flow of Orientation Information Calibration Processing>

[0596] Next, refer to Figure 58 The flowchart in Figure 55 An example of the flow of the orientation information calibration processing performed in step S603 in .

[0597] When the orientation information calibration process is started, in step S661 , the orientation information calibration unit 1613 instructs the first imaging and the second imaging.

[0598] In step S662, the orientation information calibration unit 1613 detects feature points in the first and second captured images. In step S663, the orientation information calibration unit 1613 determines whether a sufficient number of feature points have been detected. If the number of feature points is insufficient, the process proceeds to step S664. In step S664, the orientation information calibration unit 1613 displays an image indicating movement.

[0599] Upon completion of step S664, the process returns to step S663, and the subsequent processes are repeated. In addition, in the event that it is determined in step S663 that a sufficient number of feature points have been detected, the process proceeds to step S665.

[0600] In step S665 , the orientation information calibration unit 1613 compares the detected feature points to detect corresponding points between the first captured image and the second captured image, and verifies a common field of view between the first captured image and the second captured image.

[0601] In step S666, the orientation information calibration unit 1613 determines whether there is a sufficient common field of view. In the case where it is determined that there is not a sufficient common field of view, the process proceeds to step S667.

[0602] In step S667, the orientation information calibration unit 1613 displays an image prompting installation orientation adjustment. Upon completion of step S667, the process returns to step S663 and the subsequent processes are repeated. In addition, if it is determined in step S666 that there is a sufficient common field of view, the process proceeds to step S668.

[0603] In step S668 , the orientation information calibration unit 1613 instructs the first imaging and the second imaging to simultaneously image the object.

[0604] In step S669 , the orientation information calibration unit 1613 generates orientation calibration information for obtaining the position and orientation of the second imaging unit using the first captured image, the second captured image, the orientation information, and the internal parameters.

[0605] In step S670, the orientation information calibration unit 1613 determines whether the imaging is sufficient. If it is determined that the imaging is insufficient, the process returns to step S668 and the subsequent processes are repeated. In addition, if it is determined that the imaging is sufficient in step S670, the process proceeds to step S671.

[0606] In step S671, the orientation information calibration unit 1613 provides the orientation calibration information to the imaging control unit 1303, the imaging guidance output control unit 1308, or both. This enables the imaging control unit 1303, the imaging guidance output control unit 1308, or both to control the second imaging using the orientation calibration information.

[0607] Completion of step S671 ends the orientation information calibration process, and the process returns to Figure 55 .

[0608] As described above, by performing each process, the imaging communication device 1401 can control (instruct) the second imaging so that the second imaging is performed at the appropriate timing (desired position). In addition, the imaging communication device 1401 can install the second imaging unit on the first imaging unit in the correct relative position and orientation. In addition, the imaging communication device 1401 can suppress the reduction in the accuracy of the position and orientation of the second imaging unit. Therefore, the imaging communication device 1401 can suppress the reduction in the accuracy of the navigation of the second imaging. Therefore, the imaging communication device 1401 can obtain captured images at a more appropriate position and orientation, and can suppress the reduction in the accuracy of the 3D data obtained by the second 3D modeling. In addition, since the second imaging can be performed at a more appropriate position and orientation, the imaging communication device 1401 can suppress the increase in the number of captured images used in the second 3D modeling. Therefore, the imaging communication device 1401 can suppress the increase in the user's workload. In addition, the imaging communication device 1401 can suppress the increase in the load (processing volume, processing time, etc.) of the 3D modeling process.

[0609] <Example 2 of Calibration Processing>

[0610] <Calibration Unit>

[0611] A case where the above-described example is applied in <Example 2 of calibration processing> will be described. Figure 59 1601. In this case, the calibration unit 1601 includes an installation guide processing unit 1612 and a calibration processing unit 1710. The installation guide processing unit 1612 has a Figure 54 The configuration of the case in FIG is similar to that in FIG, and similar processing is performed. The calibration processing unit 1710 includes a SLAM 1711 and a trajectory superposition unit 1712.

[0612] The SLAM 1711 acquires the second captured image, performs SLAM on the second captured image, performs self-positioning, and generates orientation information (second orientation information) indicating the position and orientation of the second imaging unit (i.e., the imaging unit 1332). The SLAM 1711 provides the generated orientation information to the trajectory superimposition unit 1712.

[0613] Trajectory superimposition unit 1712 obtains orientation information (second orientation information) provided by SLAM 1711. Furthermore, trajectory superimposition unit 1712 obtains orientation information (first orientation information) provided by real-time 3D modeling processing unit 1314 (SLAM 1321). Trajectory superimposition unit 1712 derives the trajectory of the first imaging unit (imaging unit 1312) based on the orientation information (first orientation information) provided by real-time 3D modeling processing unit 1314 (SLAM 1321). Furthermore, trajectory superimposition unit 1712 derives the trajectory of the second imaging unit (imaging unit 1332) based on the orientation information (second orientation information) provided by SLAM 1711. Trajectory superimposition unit 1712 superimposes the trajectories using, for example, a technique such as ICP and calibrates the second orientation information using the difference. Trajectory superimposition unit 1712 generates orientation calibration information indicating the calibration result and provides the orientation calibration information to imaging control unit 1303, imaging guidance output control unit 1308, or both.

[0614] Furthermore, based on the result of the superimposed trajectory, trajectory superimposition unit 1712 applies the time difference between the first imaging unit (imaging unit 1312) and the second imaging unit (imaging unit 1332) when they are located at the same physical location as a time delay, and calibrates the imaging timing of the second imaging unit. Trajectory superimposition unit 1712 generates timing calibration information indicating the calibration result and provides the timing calibration information to imaging control unit 1303, imaging guidance output control unit 1308, or both.

[0615] Through such a configuration, the imaging communication device 1401 can install the second imaging unit on the first imaging unit in the correct relative position and orientation. In addition, the imaging communication device 1401 can suppress the reduction in the accuracy of the position and orientation of the second imaging unit. In addition, the imaging communication device 1401 can control (instruct) the second imaging so that the second imaging is performed at the appropriate timing (desired position). Therefore, the imaging communication device 1401 can suppress the reduction in the accuracy of the navigation of the second imaging. Therefore, the imaging communication device 1401 can obtain the captured image in a more appropriate position and orientation, and can suppress the reduction in the accuracy of the 3D data obtained by the second 3D modeling. In addition, since the second imaging can be performed in a more appropriate position and orientation, the imaging communication device 1401 can suppress the increase in the number of captured images used in the second 3D modeling. Therefore, the imaging communication device 1401 can suppress the increase in the user's workload. In addition, the imaging communication device 1401 can suppress the increase in the load (processing volume, processing time, etc.) of the 3D modeling processing.

[0616] <3D Modeling Process>

[0617] Next, refer to Figure 60An example of the flow of the 3D modeling process performed by the imaging communication device 1401 in this case is described with reference to the flowchart in FIG.

[0618] When the 3D modeling process is started, the installation guidance processing unit 1612 performs the installation guidance process. Figure 57 The processing is performed in a similar manner to the case described in the flowchart.

[0619] In step S702, the calibration processing unit 1710 performs orientation information calibration processing. In step S703, the calibration processing unit 1710 performs imaging timing calibration processing.

[0620] Steps S704 to S708 are respectively Figure 55 Steps S604 to S608 in are similarly executed.

[0621] Completion of step S708 ends the 3D modeling process.

[0622] <Flow of Orientation Information Calibration Processing>

[0623] Next, refer to Figure 61 The flowchart in Figure 60 An example of the flow of the orientation information calibration processing performed in step S702 in .

[0624] At the start of the orientation information calibration process, the first and second captured images are captured as moving images. Then, in step S731, the calibration processing unit 1710 detects feature points for each frame of the first and second captured images and determines whether a sufficient number of feature points are detected. If it is determined that the number of feature points is insufficient, the process proceeds to step S732. In step S732, the calibration processing unit 1710 issues instructions such as changing the viewpoint to ensure that any imaging has a sufficient field of view and that a sufficient number of feature points can be detected. For example, the calibration processing unit 1710 displays an image that indicates movement.

[0625] Upon completion of step S732, the process returns to step S731. That is, steps S731 and S732 are repeated until a sufficient number of feature points are determined to exist in step S731. If a sufficient number of feature points are determined to have been detected in step S731, the process proceeds to step S733.

[0626] In step S733, the calibration processing unit 1710 issues an instruction to slowly move the imaging communication device 1401 (eg, smartphone) and the imaging device 1402 (eg, ILC). For example, the calibration processing unit 1710 displays an image that indicates such movement (change of viewpoint).

[0627] In step S734, the calibration processing unit 1710 performs visual SLAM on the first captured image and the second captured image independently to derive the trajectory of the first imaging unit and the trajectory of the second imaging unit. For example, the SLAM 1321 performs visual SLAM on each frame of the first captured image to derive first orientation information. The trajectory overlay unit 1712 derives the trajectory of the first imaging unit based on the first orientation information. In addition, the SLAM 1711 performs visual SLAM on each frame of the second captured image to derive second orientation information. The trajectory overlay unit 1712 derives the trajectory of the second imaging unit based on the second orientation information.

[0628] In step S735, the trajectory superimposition unit 1712 superimposes the trajectories, derives the transformation parameters required for superimposition, and generates orientation calibration information. For example, the trajectory superimposition unit 1712 uses a technique such as ICP to superimpose the trajectory of the first imaging unit and the trajectory of the second imaging unit, and uses the difference to calibrate the second orientation information. The trajectory superimposition unit 1712 generates orientation calibration information indicating the calibration result.

[0629] Completion of step S735 ends the orientation information calibration process, and the process returns to Figure 60 .

[0630] <Flow of Imaging Timing Calibration Processing>

[0631] Next, refer to Figure 62 The flowchart in Figure 60 An example of the flow of the imaging timing calibration process performed in step S703 in FIG.

[0632] When the imaging timing calibration process starts, in step S761, the calibration processing unit 1710 issues an instruction to slowly move the imaging communication device 1401 (e.g., smartphone) and the imaging device 1402 (e.g., ILC). For example, the calibration processing unit 1710 displays an image that indicates such movement (change of viewpoint).

[0633] In step S762, the calibration processing unit 1710 causes the first imaging unit (imaging unit 1312) to perform first imaging (capturing a moving image). Furthermore, the calibration processing unit 1710 causes the second imaging unit (imaging unit 1332) to perform second imaging (capturing a moving image).

[0634] In step S763, the calibration processing unit 1710 performs visual SLAM on the first captured image and the second captured image independently to derive the trajectory of the first imaging unit and the trajectory of the second imaging unit. For example, the SLAM 1321 performs visual SLAM on each frame of the first captured image to derive first orientation information. The trajectory overlay unit 1712 derives the trajectory of the first imaging unit based on the first orientation information. In addition, the SLAM 1711 performs visual SLAM on each frame of the second captured image to derive second orientation information. The trajectory overlay unit 1712 derives the trajectory of the second imaging unit based on the second orientation information.

[0635] In step S764, the trajectory superimposition unit 1712 superimposes the trajectories, obtains the time difference between the internal stamps at the same trajectory point (i.e., the time difference when the first imaging unit (imaging unit 1312) and the second imaging unit (imaging unit 1332) are located at the same physical location), applies the average of the time differences as the time delay, and calibrates the imaging timing of the second imaging unit. The trajectory superimposition unit 1712 generates timing calibration information indicating the calibration result.

[0636] Completion of step S764 ends the imaging timing calibration process, and the process returns to Figure 60 .

[0637] By performing each of the processes described above, the imaging communication device 1401 can install the second imaging unit on the first imaging unit in the correct relative position and orientation. In addition, the imaging communication device 1401 can suppress the reduction in the accuracy of the position and orientation of the second imaging unit. In addition, the imaging communication device 1401 can control (instruct) the second imaging so that the second imaging is performed at the appropriate timing (desired position). Therefore, the imaging communication device 1401 can suppress the reduction in the accuracy of the navigation of the second imaging. Therefore, the imaging communication device 1401 can obtain captured images in a more appropriate position and orientation, and can suppress the reduction in the accuracy of the 3D data obtained by the second 3D modeling. In addition, since the second imaging can be performed in a more appropriate position and orientation, the imaging communication device 1401 can suppress the increase in the number of captured images used in the second 3D modeling. Therefore, the imaging communication device 1401 can suppress the increase in the user's workload. In addition, the imaging communication device 1401 can suppress the increase in the load (processing volume, processing time, etc.) of the 3D modeling process.

[0638] <Other application examples of this technology>

[0639] Note that, in the above description, the case where the present technology described above in <5. Calibration Process> is applied to Figure 43The present invention is a case of the information processing system 1400 (imaging communication device 1401 ) in FIG. 4 ; however, the present technology can also be applied to devices other than the imaging communication device 1401 .

[0640] For example, this technology can be applied to Figure 39 In this case, the imaging device 1300 is Figure 53 The situation of the imaging communication device 1401 in is similar, Figure 39 The imaging device 1300 in the embodiment may include a calibration unit 1601. In addition, in this case, the imaging device 1300 may perform Figure 55 3D modeling processing in .

[0641] <9. Appendix>

[0642] <Computer>

[0643] The above series of processes can be performed by hardware or software. In the case of performing the series of processes by software, the program forming the software is installed in a computer. Here, examples of computers include, for example, computers built into dedicated hardware, general-purpose personal computers that can perform various functions by installing various programs, etc.

[0644] Figure 63 : is a block diagram showing a configuration example of hardware of a computer that executes the above-described series of processes by a program.

[0645] exist Figure 63 In a computer 1900 shown in FIG, a central processing unit (CPU) 1901 , a read-only memory (ROM) 1902 , and a random access memory (RAM) 1903 are interconnected via a bus 1904 .

[0646] The bus 1904 is also connected to an input / output interface 1910 . An input unit 1911 , an output unit 1912 , a storage unit 1913 , a communication unit 1914 , and a drive 1915 are connected to the input / output interface 1910 .

[0647] Input unit 1911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, and input terminals. Output unit 1912 includes, for example, a display, a speaker, and output terminals. Storage unit 1913 includes, for example, a hard disk, a RAM disk, and nonvolatile memory. Communication unit 1914 includes, for example, a network interface. Drive 1915 drives removable recording medium 1921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0648] In the computer configured as described above, for example, the CPU 1901 loads a program stored in the storage unit 1913 into the RAM 1903 via the input / output interface 1910 and the bus 1904 and executes the program. Thus, the above-described series of processes is performed. The RAM 1903 can appropriately store data and the like required for the CPU 1901 to execute various types of processes.

[0649] For example, the program executed by the computer can be recorded in the removable recording medium 1921 as a package medium or the like and used. In this case, the program can be read from the removable recording medium 1921 attached to the drive 1915 and installed in the storage unit 1913 via the input / output interface 1910.

[0650] In addition, the program can be provided via any wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. In this case, the program can be received by the communication unit 1914 and installed in the storage unit 1913 via the input / output interface 1910.

[0651] Furthermore, the program may be pre-installed in the ROM 1902 , the storage unit 1913 , or both.

[0652] <Target application of this technology>

[0653] The present technology can be applied to any configuration. For example, the present technology can be applied to various electronic devices.

[0654] In addition, for example, the present technology can also be implemented as a partial configuration of a device, such as a processor as a system large-scale integration (LSI) etc. (e.g., a video processor), a module using multiple processors etc. (e.g., a video module), a unit using multiple modules etc. (e.g., a video unit), or a group obtained by further adding other functions to the unit (e.g., a video group).

[0655] Furthermore, for example, the present technology can also be applied to a network system comprising multiple devices. For example, the present technology can be implemented as cloud computing in which multiple devices collaboratively share and process data via a network. For example, the present technology can be implemented in a cloud service that provides image (moving image) related services to any terminal such as a computer, audio-visual (AV) device, portable information processing terminal, or Internet of Things (IoT) device.

[0656] Note that in this article, a system refers to a group of multiple components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, multiple devices stored in different housings and connected via a network and a single device in which multiple modules are stored in one housing are both systems.

[0657] <Other>

[0658] Note that in this article, the term "association" means, for example, that when processing one data, other data is allowed to be used (linked). That is, data associated with each other can be collected as one data or can become separate data. For example, information associated with certain data can be transmitted on a transmission path different from the transmission path of the data. In addition, for example, information associated with certain data can be recorded in a recording medium different from the recording medium of the data (or another recording area of the same recording medium). Note that this "association" may not be all data, but a part of the data. For example, mobile 3D data and information corresponding to the mobile 3D data can be associated with each other in any unit such as multiple frames, one frame or a part within a frame.

[0659] Note that in this document, terms such as "combine", "multiplex", "add", "merge", "include", "store", "put in", "introduce" and "insert" mean, for example, combining multiple objects into one object, such as combining encoded data and metadata into one data, and mean a method of the above-mentioned "association".

[0660] Furthermore, the embodiment of the present technology is not limited to the above-described embodiment, and various modifications are possible without departing from the scope of the present technology.

[0661] For example, a configuration described as one device (or processing unit) may be divided and configured as a plurality of devices (or processing units). Conversely, a configuration described as a plurality of devices (or processing units) may be collectively configured as one device (or processing unit). Furthermore, it goes without saying that a configuration other than the above configuration may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, a portion of the configuration of a certain device (or processing unit) may be included in the configuration of another device (or another processing unit).

[0662] In addition, for example, the above program can be executed in any device. In this case, the device only needs to have necessary functions (functional blocks, etc.) and obtain necessary information.

[0663] Furthermore, for example, each step in a flowchart may be performed by a single device, or may be shared and performed by multiple devices. Furthermore, when multiple processes are included in a single step, the multiple processes may be performed by a single device, or may be shared and performed by multiple devices. In other words, the multiple processes included in a single step may also be performed as a single process. Conversely, a process described as a single process may also be performed collectively as a single step.

[0664] Furthermore, for example, in a program executed by a computer, the processing of the steps describing the program may be performed in a time series in the order described herein, or may be performed individually or in parallel at necessary timings, such as when a call is made. That is, the processing of the individual steps may be performed in an order different from the order described above, as long as there is no contradiction. Furthermore, the processing of the steps describing the program may be performed in parallel with the processing of another program, or may be performed in combination with the processing of another program.

[0665] Furthermore, for example, multiple technologies related to the present technology may be independently implemented as a single entity, as long as no contradiction exists. It goes without saying that any multiple technologies may be implemented in combination. For example, part or all of the technology described in any embodiment may be implemented in combination with part or all of the technology described in another embodiment. Furthermore, part or all of any of the above-described technologies may be implemented together with another technology not described above.

[0666] Note that the present technology can also have the following configurations.

[0667] (1) An information processing device comprising:

[0668] a first 3D modeling processing unit that generates, based on a first captured image generated by first imaging that images a 3D object, first orientation information indicating a position and orientation of a first imaging unit that performs the first imaging and first three-dimensional shape information representing a three-dimensional shape of the 3D object;

[0669] a calibration unit that calibrates second orientation information indicating a position and orientation of a second imaging unit that performs the second imaging based on the first captured image, the first orientation information, and a second captured image generated by second imaging that images the 3D object; and

[0670] An imaging control unit is configured to reflect a calibration result in the second orientation information and control the second imaging for generating second three-dimensional shape information representing the three-dimensional shape of the 3D object based on the second orientation information reflecting the calibration result and the first three-dimensional shape information.

[0671] (2) The information processing device according to (1), wherein

[0672] The calibration unit calibrates the second orientation information based on the first captured image and the second captured image both captured at the same timing and the first orientation information obtained at the timing.

[0673] (3) The information processing device according to (2), wherein

[0674] The calibration unit

[0675] detecting feature points of the first captured image and the second captured image,

[0676] obtaining corresponding points between feature points of the first captured image and feature points of the second captured image,

[0677] determining a size of the 3D object based on the first orientation information, and

[0678] The second orientation information is calibrated based on the corresponding points and the size so that a reprojection error becomes sufficiently small.

[0679] (4) The information processing device according to (3), wherein

[0680] The calibration unit further calibrates the second orientation information using internal parameters of the second imaging unit so that the re-projection error becomes sufficiently small.

[0681] (5) The information processing device according to (4), wherein

[0682] The internal parameters include preset values preset for the second imaging unit or an optical system unit used in the second imaging unit.

[0683] (6) The information processing device according to (2), wherein

[0684] The calibration unit

[0685] outputting guidance prompting movement of the first imaging unit and the second imaging unit, and

[0686] The second orientation information is calibrated based on a plurality of the first captured images and the second captured images obtained by performing the first imaging and the second imaging at a plurality of different positions according to the guidance.

[0687] (7) The information processing device according to (2), wherein

[0688] In a case where calibration of the second orientation information is successful, the calibration unit outputs a notification indicating successful calibration, and

[0689] If calibration of the second orientation information fails, the calibration unit outputs guidance prompting re-imaging.

[0690] (8) The information processing device according to (1), wherein

[0691] The calibration unit outputs guidance prompting installation of the second imaging unit in a correct relative position and orientation with respect to the first imaging unit.

[0692] (9) The information processing device according to (1), wherein

[0693] The calibration unit further calibrates the imaging timing of the second imaging, and

[0694] When controlling the second imaging, the imaging control unit instructs the second imaging at an instruction timing that ensures that the imaging timing reflecting the calibration result is appropriate.

[0695] (10) The information processing device according to (9), wherein

[0696] The calibration unit calibrates the imaging timing based on the second captured image obtained by capturing a measurement image that changes with time.

[0697] (11) The information processing device according to (10), wherein

[0698] The calibration unit outputs a guide prompting to capture the measurement image at a correct position.

[0699] (12) The information processing device according to (1), further comprising:

[0700] a scoring processing unit that uses the first three-dimensional shape information to evaluate the accuracy of the second three-dimensional shape information that can be generated using the second captured image generated by the second imaging performed so far, and generates a scoring result, wherein

[0701] The imaging control unit controls the second imaging for generating the second three-dimensional shape information based on the second orientation information reflecting the calibration result, the first three-dimensional shape information, and the scoring result.

[0702] (13) The information processing device according to (1), wherein

[0703] The first 3D modeling processing unit includes:

[0704] an orientation information generating unit that generates the first orientation information based on the acceleration and angular velocity of the first imaging unit; and

[0705] A three-dimensional shape generating unit generates the first three-dimensional shape information based on the first orientation information and the depth of the 3D object.

[0706] (14) The information processing device according to (13), wherein

[0707] The first three-dimensional shape information includes a mesh representing the three-dimensional shape of the 3D object through vertices and connections and a texture applied to a surface of the mesh.

[0708] (15) The information processing device according to (13), further comprising:

[0709] a depth detection unit, configured to detect the depth;

[0710] the first imaging unit; and

[0711] An inertial measurement unit detects the acceleration and the angular velocity.

[0712] (16) The information processing device according to (1), further comprising:

[0713] The second imaging unit.

[0714] (17) The information processing device according to (1), further comprising:

[0715] An associating unit that associates the second orientation information reflecting the calibration result with the second captured image.

[0716] (18) An information processing method comprising:

[0717] generating, based on a first captured image generated by first imaging that images a 3D object, first orientation information indicating a position and orientation of a first imaging unit that performs the first imaging and first three-dimensional shape information representing a three-dimensional shape of the 3D object;

[0718] calibrating second orientation information indicating a position and orientation of a second imaging unit that performs the second imaging based on the first captured image, the first orientation information, and a second captured image generated by second imaging that images the 3D object; and

[0719] The calibration result is reflected in the second orientation information, and the second imaging for generating second three-dimensional shape information representing the three-dimensional shape of the 3D object is controlled based on the second orientation information reflecting the calibration result and the first three-dimensional shape information.

[0720] (21) An information processing device comprising:

[0721] a first 3D modeling processing unit that generates, based on a first captured image generated by first imaging that images a 3D object, first orientation information indicating a position and orientation of a first imaging unit that performs the first imaging and first three-dimensional shape information representing a three-dimensional shape of the 3D object;

[0722] a calibration unit that calibrates second orientation information indicating a position and orientation of a second imaging unit that performs the second imaging based on the first captured image, the first orientation information, and a second captured image generated by second imaging that images the 3D object; and

[0723] a guidance information output control unit that reflects a calibration result in the second orientation information, generates guidance information for a second imaging based on the second orientation information reflecting the calibration result and the first three-dimensional shape information, and controls output of the guidance information, wherein the second imaging is used to generate second three-dimensional shape information representing the three-dimensional shape of the 3D object.

[0724] (22) The information processing device according to (21), wherein

[0725] The calibration unit calibrates the second orientation information based on the first captured image and the second captured image both captured at the same timing and the first orientation information obtained at the timing.

[0726] (23) The information processing device according to (22), wherein

[0727] The calibration unit detects feature points of the first captured image and the second captured image,

[0728] obtaining corresponding points between feature points of the first captured image and feature points of the second captured image,

[0729] determining a size of the 3D object based on the first orientation information, and

[0730] The second orientation information is calibrated based on the corresponding points and the size so that a reprojection error becomes sufficiently small.

[0731] (24) The information processing device according to (23), wherein

[0732] The calibration unit further calibrates the second orientation information using internal parameters of the second imaging unit so that the re-projection error becomes sufficiently small.

[0733] (25) The information processing device according to (24), wherein

[0734] The internal parameters include preset values preset for the second imaging unit or an optical system unit used in the second imaging unit.

[0735] (26) The information processing device according to (22), wherein

[0736] The calibration unit outputs guidance prompting movement of the first imaging unit and the second imaging unit, and calibrates the second orientation information based on a plurality of first captured images and second captured images obtained by performing the first imaging and the second imaging at a plurality of different positions according to the guidance.

[0737] (27) The information processing device according to (22), wherein

[0738] In a case where calibration of the second orientation information is successful, the calibration unit outputs a notification indicating successful calibration, and

[0739] If calibration of the second orientation information fails, the calibration unit outputs guidance prompting re-imaging.

[0740] (28) The information processing device according to (21), wherein

[0741] The calibration unit outputs guidance prompting installation of the second imaging unit in a correct relative position and orientation with respect to the first imaging unit.

[0742] (29) The information processing device according to (21), further comprising:

[0743] a scoring processing unit that uses the first three-dimensional shape information to evaluate the accuracy of the second three-dimensional shape information that can be generated using the second captured image generated by the second imaging performed so far, and generates a scoring result, wherein

[0744] The guide information output control unit generates the guide information based on the second orientation information reflecting the calibration result, the first three-dimensional shape information, and the scoring result, and outputs the guide information.

[0745] (30) The information processing device according to (21), wherein

[0746] The first 3D modeling processing unit includes:

[0747] an orientation information generating unit that generates the first orientation information based on the acceleration and angular velocity of the first imaging unit; and

[0748] A three-dimensional shape generating unit generates the first three-dimensional shape information based on the first orientation information and the depth of the 3D object.

[0749] (31) The information processing device according to (30), wherein

[0750] The first three-dimensional shape information includes a mesh representing the three-dimensional shape of the 3D object through vertices and connections and a texture applied to a surface of the mesh.

[0751] (32) The information processing device according to (30), further comprising:

[0752] a depth detection unit, configured to detect the depth;

[0753] the first imaging unit; and

[0754] An inertial measurement unit detects the acceleration and the angular velocity.

[0755] (33) The information processing device according to (21), further comprising:

[0756] The second imaging unit.

[0757] (34) The information processing device according to (21), further comprising:

[0758] An associating unit that associates the second orientation information reflecting the calibration result with the second captured image.

[0759] (35) An information processing method comprising:

[0760] generating, based on a first captured image generated by first imaging that images a 3D object, first orientation information indicating a position and orientation of a first imaging unit that performs the first imaging and first three-dimensional shape information representing a three-dimensional shape of the 3D object;

[0761] calibrating second orientation information indicating a position and orientation of a second imaging unit that performs the second imaging based on the first captured image, the first orientation information, and a second captured image generated by second imaging that images the 3D object; and

[0762] The calibration result is reflected in the second orientation information, and guidance information for the second imaging is generated based on the second orientation information reflecting the calibration result and the first three-dimensional shape information, and the output of the guidance information is controlled, and the second imaging is used to generate second three-dimensional shape information representing the three-dimensional shape of the 3D object.

[0763] Reference Signs List

[0764] 101 First 3D Data Generation Process

[0765] 102 Rating Processing

[0766] 103 Imaging Control Processing for Second 3D Modeling

[0767] 104 Second 3D data generation process

[0768] 105 Imaging-guided output processing for second 3D modeling

[0769] 106 Calibration Process

[0770] 1300 Imaging Device

[0771] 1301 First 3D data generating unit

[0772] 1302 Scoring Processing Unit

[0773] 1303 Imaging Control Unit

[0774] 1304 Second 3D data gene...

Claims

1. An information processing device, comprising: a first 3D modeling processing unit that generates, based on a first captured image generated by first imaging that images a 3D object, first orientation information indicating a position and orientation of a first imaging unit that performs the first imaging and first three-dimensional shape information representing a three-dimensional shape of the 3D object; a calibration unit that calibrates second orientation information indicating a position and orientation of a second imaging unit that performs the second imaging based on the first captured image, the first orientation information, and a second captured image generated by second imaging that images the 3D object; as well as An imaging control unit is configured to reflect a calibration result in the second orientation information and control the second imaging for generating second three-dimensional shape information representing the three-dimensional shape of the 3D object based on the second orientation information reflecting the calibration result and the first three-dimensional shape information.

2. The information processing device according to claim 1, wherein The calibration unit calibrates the second orientation information based on the first captured image and the second captured image both captured at the same timing and the first orientation information obtained at the timing.

3. The information processing device according to claim 2, wherein: The calibration unit detecting feature points of the first captured image and the second captured image, obtaining corresponding points between feature points of the first captured image and feature points of the second captured image, determining a size of the 3D object based on the first orientation information, and The second orientation information is calibrated based on the corresponding points and the size so that a reprojection error becomes sufficiently small.

4. The information processing device according to claim 3, wherein: The calibration unit further calibrates the second orientation information using internal parameters of the second imaging unit so that the re-projection error becomes sufficiently small.

5. The information processing apparatus according to claim 4, wherein: The internal parameters include preset values preset for the second imaging unit or an optical system unit used in the second imaging unit. The information processing apparatus according to claim 2 , wherein: The calibration unit outputting guidance prompting movement of the first imaging unit and the second imaging unit, and The second orientation information is calibrated based on a plurality of the first captured images and the second captured images obtained by performing the first imaging and the second imaging at a plurality of different positions according to the guidance.

7. The information processing apparatus according to claim 2, wherein: In a case where calibration of the second orientation information is successful, the calibration unit outputs a notification indicating successful calibration, and If calibration of the second orientation information fails, the calibration unit outputs guidance prompting re-imaging.

8. The information processing apparatus according to claim 1, wherein: The calibration unit outputs guidance prompting installation of the second imaging unit in a correct relative position and orientation with respect to the first imaging unit.

9. The information processing apparatus according to claim 1, wherein: The calibration unit further calibrates the imaging timing of the second imaging, and When controlling the second imaging, the imaging control unit instructs the second imaging at an instruction timing that ensures that the imaging timing reflecting the calibration result is appropriate.

10. The information processing apparatus according to claim 9, wherein: The calibration unit calibrates the imaging timing based on the second captured image obtained by capturing a measurement image that changes with time. The information processing apparatus according to claim 10 , wherein: The calibration unit outputs a guide prompting to capture the measurement image at a correct position.

12. The information processing apparatus according to claim 1, further comprising: a scoring processing unit that uses the first three-dimensional shape information to evaluate the accuracy of the second three-dimensional shape information that can be generated using the second captured image generated by the second imaging performed so far, and generates a scoring result, wherein The imaging control unit controls the second imaging for generating the second three-dimensional shape information based on the second orientation information reflecting the calibration result, the first three-dimensional shape information, and the scoring result.

13. The information processing apparatus according to claim 1, wherein: The first 3D modeling processing unit includes: an orientation information generating unit that generates the first orientation information based on the acceleration and angular velocity of the first imaging unit; and A three-dimensional shape generating unit generates the first three-dimensional shape information based on the first orientation information and the depth of the 3D object. The information processing apparatus according to claim 13 , wherein: The first three-dimensional shape information includes a mesh representing the three-dimensional shape of the 3D object through vertices and connections and a texture applied to a surface of the mesh.

15. The information processing apparatus according to claim 13, further comprising: a depth detection unit, configured to detect the depth; the first imaging unit; as well as An inertial measurement unit detects the acceleration and the angular velocity.

16. The information processing apparatus according to claim 1, further comprising: The second imaging unit.

17. The information processing apparatus according to claim 1, further comprising: An associating unit that associates the second orientation information reflecting the calibration result with the second captured image.

18. An information processing method, comprising: generating, based on a first captured image generated by first imaging that images a 3D object, first orientation information indicating a position and orientation of a first imaging unit that performs the first imaging and first three-dimensional shape information representing a three-dimensional shape of the 3D object; calibrating second orientation information indicating a position and orientation of a second imaging unit that performs the second imaging based on the first captured image, the first orientation information, and a second captured image generated by second imaging that images the 3D object; as well as The calibration result is reflected in the second orientation information, and the second imaging for generating second three-dimensional shape information representing the three-dimensional shape of the 3D object is controlled based on the second orientation information reflecting the calibration result and the first three-dimensional shape information.

19. An information processing device comprising: a first 3D modeling processing unit that generates, based on a first captured image generated by first imaging that images a 3D object, first orientation information indicating a position and orientation of a first imaging unit that performs the first imaging and first three-dimensional shape information representing a three-dimensional shape of the 3D object; a calibration unit that calibrates second orientation information indicating a position and orientation of a second imaging unit that performs the second imaging based on the first captured image, the first orientation information, and a second captured image generated by second imaging that images the 3D object; as well as a guidance information output control unit that reflects a calibration result in the second orientation information, generates guidance information for a second imaging based on the second orientation information reflecting the calibration result and the first three-dimensional shape information, and controls output of the guidance information, wherein the second imaging is used to generate second three-dimensional shape information representing the three-dimensional shape of the 3D object.

20. An information processing method, comprising: generating, based on a first captured image generated by first imaging that images a 3D object, first orientation information indicating a position and orientation of a first imaging unit that performs the first imaging and first three-dimensional shape information representing a three-dimensional shape of the 3D object; calibrating second orientation information indicating a position and orientation of a second imaging unit that performs the second imaging based on the first captured image, the first orientation information, and a second captured image generated by second imaging that images the 3D object; as well as The calibration result is reflected in the second orientation information, and guidance information for the second imaging is generated based on the second orientation information reflecting the calibration result and the first three-dimensional shape information, and the output of the guidance information is controlled, and the second imaging is used to generate second three-dimensional shape information representing the three-dimensional shape of the 3D object.

Images