Information processing device, information processing method, and program

Abstract

This enables optimal capture for 3D modeling. [Solution] The camera unit photographs the subject, the field of view calculation unit calculates the coordinate range that fits within the camera unit's field of view in the subject's three-dimensional coordinate system based on parallax information from a sensor unit that acquires distance information to the subject, and the display control unit visualizes the coordinate range. The technology according to this disclosure can be applied, for example, to a shooting assist system for performing photogrammetry.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program, and more particularly to an information processing apparatus, an information processing method, and a program that can preferably realize shooting for 3D modeling.

Background Art

[0002] Conventionally, as a method for 3D modeling of a subject (3D object) having a three-dimensional shape, a method called photogrammetry is known in which the subject is photographed from multiple directions and 3D data is generated based on a plurality of captured images obtained.

[0003] Patent Document 1 discloses a technique for controlling the shooting of a 3D object based on a scoring result that evaluates the accuracy of 3D data that can be generated using captured images obtained by shooting performed so far. According to this technique, higher-definition 3D data can be generated, and thus 3D modeling can be performed more easily.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] When performing 3D modeling as described above, in order to reduce the amount of work and processing, it is desirable to generate 3D data with as few shooting times as possible. For this purpose, it is required to shoot the subject in a more appropriate view.

[0006] The present disclosure has been made in view of such a situation, and enables the shooting for 3D modeling to be preferably realized. [Means for solving the problem]

[0007] The information processing device disclosed herein comprises a camera unit for photographing a subject and a sensor unit for acquiring distance information from the subject, a field of view calculation unit for calculating a coordinate range that falls within the field of view of the camera unit in the three-dimensional coordinates of the subject, and a display control unit for visualizing the coordinate range.

[0008] The information processing method disclosed herein includes calculating a coordinate range that falls within the field of view of the camera unit in the three-dimensional coordinates of the subject, based on parallax information between a camera unit that photographs the subject and a sensor unit that acquires distance information to the subject, and visualizing the coordinate range.

[0009] The program disclosed herein is a program that causes a computer to perform a process that includes calculating the coordinate range within the field of view of the camera unit in the three-dimensional coordinates of the subject, based on parallax information between a camera unit that photographs the subject and a sensor unit that acquires distance information to the subject, and visualizing the coordinate range.

[0010] In this disclosure, based on parallax information between a camera unit that photographs a subject and a sensor unit that acquires distance information to the subject, the coordinate range that falls within the field of view of the camera unit in the three-dimensional coordinate system of the subject is calculated, and the coordinate range is visualized. [Brief explanation of the drawing]

[0011] [Figure 1] This diagram shows the external configuration of the shooting assistance system. [Figure 2] This diagram illustrates the difference in field of view between a smartphone and the International Linear Collider (ILC). [Figure 3] This is a block diagram showing an example of the functional configuration of a shooting assistance system. [Figure 4] This diagram illustrates the visualization of the field of view in ILC (Internal Lithography). [Figure 5]This is a diagram for explaining the perspective of ILC considering depth. [Figure 6] This is a diagram for explaining the visualization of scores. [Figure 7] This is a diagram for explaining the position and orientation of ILC with respect to the subject. [Figure 8] This is a diagram showing an example of the display of discrimination information. [Figure 9] This is a diagram showing an example of the display of discrimination information. [Figure 10] This is a diagram showing an example of the display of discrimination information. [Figure 11] This is a diagram showing an example of photographed focus position information. [Figure 12] This is a diagram showing an example of photographed focus position information. [Figure 13] This is a diagram showing an example of photographed focus position information. [Figure 14] This is a flowchart for explaining the flow of camera field visualization processing. [Figure 15] This is a schematic diagram for explaining camera field visualization processing. [Figure 16] This is a block diagram showing another example of the functional configuration of a shooting assist system. [Figure 17] This is a block diagram showing yet another example of the functional configuration of a shooting assist system. [Figure 18] This is a block diagram showing yet another example of the functional configuration of a shooting assist system. [Figure 19] This is a block diagram showing an example of the configuration of a computer.

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments for implementing the present disclosure (hereinafter referred to as embodiments) will be described. The description will be made in the following order.

[0013] 1. Photogrammetric photography and its problems 2. Configuration and technical features of a shooting assist system 3. Camera field visualization processing 4. Variations 5. Example of a computer configuration

[0014] <1. Photogrammetry and its challenges> Traditionally, a method called photogrammetry has been known for 3D modeling of objects with three-dimensional shapes. This method involves photographing the object from multiple directions and generating 3D data based on the resulting images. Photogrammetry photography is the process of capturing a set of photographs to generate such 3D data (3D models).

[0015] In recent years, a photogrammetry imaging assist system 1 (hereinafter simply referred to as imaging assist system 1), as shown in Figure 1, has been proposed for performing photogrammetry imaging.

[0016] The shooting assist system 1 is configured by mechanically and electrically connecting a smartphone 10 and an ILC (Interchangeable lens Cameras) 20. In the shooting assist system 1, the camera and sensors of the smartphone 10 are used to estimate the user's posture, and an AR mark, which serves as a navigation function, is projected into the real world and displayed on the screen of the smartphone 10. The smartphone 10 also controls the automatic shooting of the ILC 20 with appropriate shooting conditions and timing. Meanwhile, the ILC 20 takes pictures under the control of the smartphone 10.

[0017] At this time, as shown in Figure 2, the screen of the smartphone 10 (display unit 11) and the screen of the ILC 20 (display unit 21) display images of the same subject TR that is the target of 3D modeling. The image displayed on the display unit 11 of the smartphone 10 also includes the image of the field of view R21 captured by the ILC 20 and displayed on its display unit 21.

[0018] However, because the field of view (FOV1, FOV2) of the cameras and lenses of the smartphone 10 and the ILC20 are different, and there is an offset in their relative position and orientation, there is a difference between the field of view of the smartphone 10 and the field of view of the ILC20. In other words, there is parallax between the image of field of view R21 displayed on the display unit 11 of the smartphone 10 and the image displayed on the display unit 21 of the ILC20.

[0019] In the shooting assist system 1, the user performs shooting with the ILC20 while focusing on the navigation function (shooting assist function) displayed on the display unit 11 of the smartphone 10, and as a result, performs photogrammetry shooting without being aware of the parallax mentioned above. As a result, it may not be possible to photograph the subject with the appropriate field of view, which may lead to an increase in the amount of work and processing required due to reshoots, and a decrease in the accuracy of 3D modeling.

[0020] In contrast, the shooting assist system applying the technology described herein corrects the parallax that occurs between the smartphone and the ILC, and visualizes the ILC's field of view, thereby enabling the subject to be photographed with a more appropriate field of view.

[0021] <2. Configuration and Technical Features of the Imaging Assist System> (System Configuration) Figure 3 is a block diagram showing an example of the functional configuration of a shooting assist system to which the technology described herein is applied.

[0022] As shown in Figure 3, the shooting assist system applying the technology described herein consists of a smartphone 10 and an ILC 20, as described with reference to Figure 1. In the shooting assist system of Figure 3, the smartphone 10 is equipped with a sensor unit that acquires distance information to the subject, and functions such as spatial recognition, posture calculation, camera control, shooting assist display, and parallax calibration are integrated into the application. The ILC 20 also transfers camera information and shooting completion signals to the smartphone 10.

[0023] The smartphone 10 includes an IMU (Inertial Measurement Unit) 101, a sensor unit 102, a camera unit 103, and a parallax calibration unit 104. Furthermore, the smartphone 10 implements the following functional blocks by executing a program stored in memory (not shown): a parallax information acquisition unit 111, a position and orientation calculation unit 112, a three-dimensional coordinate acquisition unit 113, a field of view calculation unit 114, a display control unit 115, an appropriate position and orientation determination unit 121, a focus position calculation unit 131, and a score calculation unit 141.

[0024] The ILC20 consists of a camera unit 211 with interchangeable lenses and an image sensor, which has a shooting function for capturing subjects to be 3D modeled, and an image recording unit 212 that records the captured images obtained by the camera unit 211. The captured images (also called through images, etc.) captured by the camera unit 211 are displayed in real time on the display unit 21.

[0025] The IMU 101 detects the three-dimensional inertial motion (translational and rotational motion in the three orthogonal axes) of the smartphone 10 and outputs the detection results to the position and orientation calculation unit 112.

[0026] The sensor unit 102 is configured as a distance sensor. The sensor unit 102 detects the distance to the subject and outputs the detection result as distance information to the position and attitude calculation unit 112. For example, the sensor unit 102 may be configured as a depth sensor or as a stereo camera. Alternatively, the sensor unit 102 may be configured as an image sensor having image plane phase difference pixels or as a dToF (direct Time of Flight) type SPAD (Single Photon Avalanche Diode) distance sensor.

[0027] The camera unit 103 consists of a fixed-focus lens and an image sensor, and, like the camera unit 211, has a shooting function for capturing subjects that are the target of 3D modeling. The captured image captured by the camera unit 103 is output to the position and orientation calculation unit 112.

[0028] The parallax calibration unit 104 performs parallax calibration to adjust the parallax between the smartphone 10 and the ILC 20 using any method, and outputs the calibration result to the parallax information acquisition unit 111.

[0029] The parallax information acquisition unit 111 acquires parallax information between the sensor unit 102 of the smartphone 10 and the camera unit 211 of the ILC20 based on the calibration results from the parallax calibration unit 104, known information such as the field of view information of the camera unit 103 and camera information transferred from the ILC20. The parallax information includes at least difference information representing the difference in position and orientation (direction) between the smartphone 10 (sensor unit 102) and the ILC20 (camera unit 211) in three-dimensional coordinate space, and field of view information that determines their respective fields of view (angle of view FOV1, FOV2). The difference information is acquired based on the external parameters of the sensor unit 102 and the camera unit 211, respectively. The field of view information of the sensor unit 102 and the camera unit 211, respectively, is acquired as internal parameters of the sensor unit 102 and the camera unit 211, respectively. In addition, the camera information from the ILC20 includes lens information, information indicating the focus position and shutter timing, as well as the field of view information of the camera unit 211.

[0030] The position and orientation calculation unit 112 calculates the position and orientation of the smartphone 10 and ILC20 in three-dimensional coordinate space based on various sensor data from the IMU 101, sensor unit 102, and camera unit 103, as well as parallax information acquired by the parallax information acquisition unit 111.

[0031] The three-dimensional coordinate acquisition unit 113 recognizes the three-dimensional coordinate space based on the position and orientation of the smartphone 10 and ILC20 calculated by the position and orientation calculation unit 112. Then, the three-dimensional coordinate acquisition unit 113 acquires the three-dimensional coordinates of the subject to be 3D modeled based on the distance information acquired by the sensor unit 102.

[0032] The field of view calculation unit 114 calculates the coordinate range that falls within the field of view of the ILC20 based on the parallax information between the smartphone 10 and the ILC20, using the three-dimensional coordinates of the subject acquired by the three-dimensional coordinate acquisition unit 113.

[0033] The display control unit 115 visualizes the coordinate range (field of view of the ILC20) calculated by the field of view calculation unit 114 in the image captured by the camera unit 103, which is displayed in real time on the display unit 11.

[0034] The appropriate position and orientation determination unit 121 determines whether the position and orientation of the ILC20 (camera unit 211) relative to the subject are appropriate and supplies the determination result to the display control unit 115. The display control unit 115 displays the determination information for the display mode corresponding to the determination result from the appropriate position and orientation determination unit 121 within the coordinate range (field of view of the ILC20) visualized on the display unit 11.

[0035] The focus position calculation unit 131 calculates the focus position of the ILC20 (camera unit 211) in three-dimensional coordinate space based on camera information from the ILC20, and supplies the calculation result to the display control unit 115. Based on the calculation result from the focus position calculation unit 131, the display control unit 115 displays focus position information indicating the focus position of the camera unit 211 within the coordinate range (field of view of the ILC20) visualized on the display unit 11.

[0036] The focus position recording unit 132 records the focus position of the camera unit 211 in three-dimensional coordinate space, which is calculated by the focus position calculation unit 131. The display control unit 115 displays the captured focus position information, which is recorded in the focus position recording unit 132, within the coordinate range (field of view of the ILC 20) visualized on the display unit 11, in response to user instructions.

[0037] The score calculation unit 141 uses parallax information to calculate a score for evaluating the accuracy of the three-dimensional model data (3D data) of the subject, which is generated based on the captured image taken by the camera unit 211.

[0038] Furthermore, the display control unit 115 can also retrieve captured images taken by the ILC20 from the captured image recording unit 212 as appropriate and display them on the display unit 11.

[0039] The shooting assist system configured as described above can achieve the following technical features.

[0040] (Visualization of the ILC's field of view) As shown in Figure 3, according to the shooting assist system described, even if there is a difference (discrepancy) between the shooting range P10 of the smartphone 10 and the shooting range P20 of the ILC20, the field of view R21 of the ILC20 is visualized on the display unit 11 of the smartphone 10 based on the parallax information of the smartphone 10 and the ILC20. Furthermore, the display unit 11 of the smartphone 10 also displays the focus position information FP1' corresponding to the focus position information FP1 of the ILC20 displayed on the display unit 21 of the ILC20.

[0041] This allows users to visualize the ILC's field of view while shooting with the ILC, enabling them to capture subjects with a more appropriate field of view.

[0042] Furthermore, conventionally, focus position information indicating the focal point of the ILC was only displayed on the ILC's display unit. This meant that users had to switch their gaze between checking the navigation function on their smartphone's display and checking the focal point on the ILC's display unit. In contrast, the shooting assist system applying the technology disclosed herein allows for the visualization of the ILC's field of view and focus position information on the smartphone's display unit, enabling navigation and confirmation of the ILC's focal point to be performed solely on the smartphone screen.

[0043] In the example shown in Figure 4, the field of view R21 of the ILC20 visualized on the display unit 11 of the smartphone 10 is shown with a dashed frame. However, the area outside the field of view R21 may be superimposed with a transparent mask image such as gray, or it may be hidden by black pixels. Also, in the example shown in Figure 4, the focus position information FP1' displayed on the display unit 11 of the smartphone 10 is shown with a rectangular frame at the four corners, similar to the focus position information FP1 displayed on the display unit 21 of the ILC20. However, it may be shown with a frame of a different shape (for example, circular) than the focus position information FP1.

[0044] (ILC field of view considering depth) Furthermore, in a shooting assist system to which the technology described herein is applied, the display control unit 115 projects focus position information indicating the focus position of the ILC20 within a coordinate range having depth information. That is, on the display unit 11 of the smartphone 10, the field of view R21 of the ILC20 and the focus position information FP1' are visualized taking into account the depth of the subject included in the shooting range of the ILC20 (the distance between the ILC20 and the subject) based on the parallax information of the smartphone 10 and the ILC20.

[0045] For example, as shown on the left side of Figure 5, the display unit 21 is showing an image captured when the ILC20's shooting range includes a three-dimensional subject TR. The ILC20's focus position information FP1 is superimposed on the captured image displayed on the display unit 21.

[0046] In this case, the shape of the field of view R21 of the ILC20 visualized on the display unit 11 of the smartphone 10 will not be rectangular, as shown on the right side of the figure, due to the difference in distance and orientation based on the parallax between the smartphone 10 and the ILC20. Furthermore, the focus position information FP1' corresponding to the focus position information FP1 of the ILC20 is projected onto the field of view R21 of the ILC20 visualized on the display unit 11 of the smartphone 10 in a display manner that corresponds to the shape and orientation of the subject TR at that position.

[0047] In this way, the ILC20's field of view R21 and focus position information FP1' are visualized while taking into account the depth of the subject, allowing the user to spatially understand the extent of the ILC20's shooting range and where the focus is, thus preventing blurry photos.

[0048] The depth granularity within the shooting range of the ILC20 may be at a level where each pixel of the ILC20 has depth information, or it may be at a level where the focus position information FP1' is projected onto the plane facing the subject.

[0049] (Visualization of scores) As described above, in a shooting assist system to which the technology of this disclosure is applied, the score calculation unit 141 uses parallax information to calculate a score for evaluating the accuracy of the 3D data of the subject. At this time, the display control unit 115 visualizes the score calculated by the score calculation unit 141 on the mesh and texture representing the 3D data displayed in the coordinate range visualized on the display unit 11, similar to the method disclosed in Patent Document 1.

[0050] For example, as shown in Figure 6, the score SC is made visible on the mesh and texture representing the 3D data of the subject within the field of view R21 region of the ILC20 visualized on the display unit 11 of the smartphone 10.

[0051] In this way, the difference in position and orientation between the smartphone 10 and the ILC20, as well as a score that takes into account the field of view of each device, are fed back to the user regarding the ILC20's field of view, allowing the user to correctly determine whether or not any shots have been missed.

[0052] (Discrimination information based on the ILC's position and orientation) As described above, in a shooting assist system to which the technology of this disclosure is applied, the display control unit 115 displays discrimination information for the display mode according to whether the position and orientation of the ILC20 relative to the subject is appropriate or not.

[0053] Here, with reference to Figure 7, the position and orientation of the ILC20 relative to the subject will be explained.

[0054] Figure 7 shows the ILC20 and a side cross-sectional view of the subject TR, which is the target of 3D modeling.

[0055] The position of the ILC20 relative to the subject TR is determined by the distance L1 between the ILC20 and the subject portion indicated by the focus position information FP1. In other words, whether the position of the ILC20 relative to the subject TR is appropriate is determined, for example, by whether the distance L1 is within a predetermined distance range.

[0056] The orientation of the ILC20 relative to the subject TR is determined by the normal direction D2 of the subject portion (the direction in which the surface of the subject portion faces directly), as indicated by the focus position information FP1. In other words, whether the orientation of the ILC20 relative to the subject TR is appropriate is evaluated, for example, by whether the angle of the normal direction D2 relative to the ILC20 is within a predetermined angular range.

[0057] Then, discrimination information CR, which indicates whether the position and orientation of ILC20 relative to the subject TR is appropriate, or in other words, whether the angle between the distance L1 and the normal direction D2 is appropriate, is displayed in the field of view R21 visualized on the display unit 11 of the smartphone 10. In particular, if the position or orientation of ILC20 relative to the subject TR is inappropriate, the display mode of the discrimination information CR is changed in the field of view R21 to indicate that the position or orientation of ILC20 is inappropriate.

[0058] For example, if the angle between the distance L1 and the normal direction D2 is appropriate, that is, if the position and orientation of the ILC20 relative to the subject TR is appropriate, then, as shown in Figure 8, the cross-shaped discrimination information CR is displayed in green, for example, based on the focus position information FP1' of the field of view R21 on the display unit 11 of the smartphone 10. The two intersecting line segments that make up the cross-shaped discrimination information CR are displayed with an inclination corresponding to the angle of the normal direction D2. That is, in the example in Figure 8, the angle of the normal direction D2 is approximately the same as the optical axis direction of the ILC20, and the two line segments that make up the cross-shaped discrimination information CR are displayed so as to be orthogonal.

[0059] Furthermore, if the distance L1 is appropriate but the angle of the normal direction D2 is not appropriate, that is, if the position of the ILC20 relative to the subject TR is appropriate but the orientation is not, then, as shown in Figure 9, the cross-shaped discrimination information CR is displayed in a warning color different from green (such as red or orange), based on the focus position information FP1' of the field of view R21 on the display unit 11 of the smartphone 10. In the example in Figure 9, the angle of the normal direction D2 is significantly off from the optical axis direction of the ILC20, and the two line segments constituting the cross-shaped discrimination information CR are displayed intersecting diagonally.

[0060] On the other hand, if the distance L1 is inappropriate, that is, if the position of ILC20 relative to the subject TR is inappropriate, regardless of whether the angle of the normal direction D2 (the orientation of ILC20 relative to the subject TR) is appropriate or not, as shown in Figure 10, only the focus position information FP1' is displayed in the field of view R21 of the display unit 11 of the smartphone 10, and the cross-shaped discrimination information CR is hidden.

[0061] In other words, the display / hide of the cross-shaped discrimination information CR is determined by whether the distance L1 (the position of ILC20 relative to the subject TR) is appropriate, and the angles and colors of the two line segments constituting the cross-shaped discrimination information CR are determined by whether the angle of the normal direction D2 (the orientation of ILC20 relative to the subject TR) is appropriate.

[0062] For 3D modeling, high-quality images are required to reproduce the realism of the subject, and this requires shooting from the appropriate distance and direction. However, it is not easy to intuitively remember or measure how to maintain the appropriate distance and angle from the subject while moving during shooting.

[0063] In response to this, the display unit 11 of the smartphone 10 displays discrimination information CR, which indicates whether the position and orientation of the ILC 20 relative to the subject TR are appropriate or not. This allows the user to intuitively understand whether the distance and orientation relative to the subject TR are appropriate. As a result, if the user determines that the distance and orientation relative to the subject TR are appropriate, they can proceed with shooting. If the user determines that the distance or orientation relative to the subject TR is inappropriate, they can adjust the position or orientation of the ILC 20 relative to the subject TR before shooting.

[0064] Furthermore, the shape of the discrimination information CR is not limited to the cross shape formed by the intersection of two line segments as described above; it can be any shape based on a circle, a regular hexagon, or other similar shapes. Also, the color of the discrimination information CR, which indicates whether the orientation of the ILC20 relative to the subject TR is appropriate, can be any color.

[0065] (Focus position information of captured images) As described above, in a shooting assist system to which the technology relating to this disclosure is applied, the display control unit 115 further displays captured focus position information indicating the captured focus position within the coordinate range (field of view R21) visualized on the display unit 11 of the smartphone 10.

[0066] For example, as shown in Figure 11, in addition to the focus position information FP1' indicating the current focus position, multiple point-shaped captured focus position information APFs indicating the focus position at the time the already recorded captured image was taken are displayed for the subject TR displayed in the field of view R21 on the display unit 11 of the smartphone 10.

[0067] Furthermore, as shown in Figure 12, even when the display unit 11 of the smartphone 10 displays a captured image PIC that has already been recorded, a rectangular APF (Photon Focus Position Information) indicating the focus position at the time the captured image PIC was taken may also be displayed.

[0068] Furthermore, as shown in Figure 13, the 3D data MDL generated based on the multiple acquired images may also display a rectangular APF (Augmented Focus Position Information) indicating the focus position at the time each of the multiple acquired images used in its generation was captured.

[0069] Normally, when taking images for 3D modeling, there is no way to check the focus position of already captured images. Therefore, users had to remember or mark the position, or individually check the recorded images afterward.

[0070] In response to this, the display unit 11 of the smartphone 10 displays the AFP (Automatic Focus Position Information) which indicates the focus position that has already been captured. This allows the user to check the parts of the subject that have already been photographed and determine whether there are any missed shots or shooting errors. Furthermore, even after the fact, the user can check the parts of the subject that have already been photographed in the recorded captured image PIC or the generated 3D data MDL and determine whether further shooting is necessary.

[0071] Furthermore, the shape of the captured focus position information (AFP) is not limited to the point-like or rectangular shapes described above; it can be any shape, such as a circle.

[0072] <3. Camera field of view visualization processing> Referring to the flowchart in Figure 14 and the schematic diagram in Figure 15, the camera field of view visualization process performed in a shooting assist system to which the technology of this disclosure is applied will be described. The process in Figure 14 is performed in the shooting assist system while the ILC20 is capturing images of the subject TR that is the target of 3D modeling.

[0073] In step S1, as shown in Figure 15, the parallax information acquisition unit 111 acquires the difference information DIFF6dof for the position and orientation of the smartphone 10 and the ILC20 as parallax information between the smartphone 10 and the ILC20. Furthermore, the parallax information acquisition unit 111 acquires field of view information that determines the field of view FOV1 of the smartphone 10 and the field of view FOV2 of the ILC20, which are known information.

[0074] In step S2, the position and orientation calculation unit 112 calculates the position (viewpoint position) and orientation (direction) of the smartphone 10 and ILC20 in three-dimensional coordinate space based on the parallax information acquired by the parallax information acquisition unit 111. Specifically, the position and orientation calculation unit 112 uses the viewpoint position of the smartphone 10 as the origin (0,0,0) and uses the difference information DIFF6dof to acquire the viewpoint position (xilc,yilc,zilc) and orientation (roll,pitch,yaw) of the ILC20 in three-dimensional coordinate space.

[0075] In step S3, the three-dimensional coordinate acquisition unit 113 acquires the three-dimensional coordinates (xn, yn, zn) (n=1 to 6 in the example in Figure 15) of the subject TR included in the field of view FOV1 of the smartphone 10. The three-dimensional coordinates of the subject TR are acquired pixel by pixel using any spatial recognition technology.

[0076] In step S4, the field of view calculation unit 114 calculates the coordinate range ([x) that fits within the field of view (angle of view FOV2) of the camera unit 211 (ILC20) based on the viewpoint position (xilc, yilc, zilc) and orientation (roll, pitch, yaw) of the ILC20. min ,x max ][y min ,y max ][z min ,z maxThe field of view calculation unit 114 then calculates the ]). The field of view calculation unit 114 then determines whether the three-dimensional coordinates (xn,yn,zn) of each pixel of the subject TR are included within the coordinate range. In the example in Figure 15, the three-dimensional coordinates (x3,y3,z3) to (x5,y5,z5) of the subject TR are included within the coordinate range.

[0077] In step S5, the display control unit 115 visualizes the coordinate range. For example, in the image of the field of view FOV1 displayed on the display unit 11 of the smartphone 10, the display control unit 115 displays the pixels within the coordinate range in their original colors, and displays the pixels in the area outside the coordinate range by blending a color such as gray with their original colors.

[0078] Furthermore, the display control unit 115 obtains the three-dimensional coordinates of the pixels of the subject TR corresponding to the focus position of the ILC20 within the coordinate range, and displays focus position information indicating the focus position of the ILC20 within the visualized coordinate range. At this time, the display control unit 115 can further display discrimination information CR, as explained with reference to Figures 8 and 9, by determining the normal direction of the subject TR corresponding to the focus position.

[0079] Through the above processing, when shooting with the ILC while paying attention to the navigation function displayed on the smartphone screen, the ILC's field of view becomes visible, making it possible to photograph the subject with a more appropriate field of view. In addition, by visualizing the ILC's field of view and focus position information on the smartphone's display, it becomes possible to use the navigation function and confirm the ILC's focus point solely on the smartphone screen.

[0080] <4. Variation> In the above, the shooting assist system to which the technology relating to this disclosure is applied is assumed to consist of a smartphone 10 and an ILC20. Specifically, in the shooting assist system shown in Figure 3, the camera unit 211 is mounted on the ILC20, which is a shooting device, the sensor unit 102 is mounted on the smartphone 10, which is a terminal device attached to the ILC20, and the display control unit 115 visualizes the coordinate range on the display unit 21, which is a display screen provided by the smartphone 10.

[0081] The imaging assist system applying the technology described herein may have other configurations, but this is not limited to this example.

[0082] (Sensor accessories and ILC) Figure 16 is a block diagram showing another example of a functional configuration of a shooting assist system to which the technology described herein is applied.

[0083] The shooting assist system shown in Figure 16 consists of a sensor accessory 310 and an ILC320. In the shooting assist system of Figure 16, the sensor accessory 310 outputs sensor data necessary for spatial recognition and attitude calculation to the ILC320. The ILC320 also integrates functions such as spatial recognition, attitude calculation, camera control, and shooting assist display into the application. Parallax calibration, camera information acquisition, and transmission of the shooting completion signal are also completed internally by the ILC320.

[0084] The sensor accessory 310 is configured as an accessory device to be attached to the ILC20 and is equipped with an IMU 101 and a sensor unit 102.

[0085] The ILC320 differs from the ILC20 in Figure 3 in that it further includes a parallax calibration unit 104 and implements the following functional blocks by executing a program stored in memory (not shown): a parallax information acquisition unit 111, a position and orientation calculation unit 112, a three-dimensional coordinate acquisition unit 113, a field of view calculation unit 114, a display control unit 115, an appropriate position and orientation determination unit 121, a focus position calculation unit 131, a focus position recording unit 132, and a score calculation unit 141. In other words, in the ILC320, the display control unit 115 visualizes the coordinate range that falls within the field of view of the camera unit 211 on the display screen (display unit 21) of the ILC320, which is an imaging device.

[0086] Even with a shooting assist system configured as described above, the visualization of the ILC's field of view makes it possible to photograph subjects with a more appropriate field of view.

[0087] (ILC unit only) Figure 17 is a block diagram showing yet another example of a functional configuration of a shooting assist system to which the technology described herein is applied.

[0088] The shooting assist system shown in Figure 17 consists solely of the ILC420. In the shooting assist system of Figure 17, the ILC420 is equipped with a sensor unit that acquires distance information to the subject, and functions such as spatial recognition, attitude calculation, camera control, and shooting assist display are integrated into the application. Parallax calibration, acquisition of camera information, and transmission of the shooting completion signal are also completed internally within the ILC420.

[0089] The ILC420 differs from the ILC20 in Figure 3 in that it further includes an IMU 101, a sensor unit 102, and a parallax calibration unit 104, and by executing a program stored in memory (not shown), it realizes the following functional blocks: a parallax information acquisition unit 111, a position and orientation calculation unit 112, a three-dimensional coordinate acquisition unit 113, a field of view calculation unit 114, a display control unit 115, an appropriate position and orientation determination unit 121, a focus position calculation unit 131, a focus position recording unit 132, and a score calculation unit 141. In other words, in the ILC420, the display control unit 115 visualizes the coordinate range that falls within the field of view of the camera unit 211 on the display screen (display unit 21) of the ILC420, which is an imaging device.

[0090] Even with a shooting assist system configured as described above, the visualization of the ILC's field of view makes it possible to photograph subjects with a more appropriate field of view.

[0091] (Smartphone only) Figure 18 is a block diagram showing yet another example of a functional configuration of a shooting assist system to which the technology described herein is applied.

[0092] The shooting assist system shown in Figure 18 consists solely of a smartphone 510. In the shooting assist system of Figure 18, the smartphone 510 is equipped with a sensor unit for acquiring distance information to the subject, a camera unit, etc., and functions such as spatial recognition, posture calculation, camera control, and shooting assist display are integrated into the application. Parallax calibration, acquisition of camera information, and transfer of shooting completion signals and focus position information are also completed internally by the smartphone 510.

[0093] Smartphone 510 differs from smartphone 10 in Figure 3 in that it further includes a captured image recording unit 212. Specifically, in smartphone 510, the display control unit 115 visualizes the coordinate range that falls within the field of view of the camera unit 103 on the display screen (display unit 11) of the smartphone 510, which is a shooting device.

[0094] Even with a shooting assistance system configured as described above, the visualization of the smartphone's field of view makes it possible to photograph subjects with a more appropriate field of view.

[0095] <5. Example of computer configuration> The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up the software are installed from a program storage medium onto a computer that is built into dedicated hardware, or a general-purpose personal computer.

[0096] Figure 19 is a block diagram showing an example of a computer hardware configuration in which the series of processes described above are executed by a program.

[0097] A smartphone or ILC to which the technology described herein can be applied is implemented by a computer 600 having the configuration shown in Figure 19.

[0098] In computer 600, the CPU (Central Processing Unit) 601, ROM (Read Only Memory) 602, and RAM (Random Access Memory) 603 are interconnected by a bus 604.

[0099] An input / output interface 605 is further connected to the bus 604. An input unit 606, an output unit 607, a storage unit 608, a communication unit 609, and a drive 610 are connected to the input / output interface 605.

[0100] The input unit 606 consists of a keyboard, mouse, microphone, etc. The output unit 607 consists of a display, speaker, etc. The storage unit 608 consists of a hard disk, non-volatile memory, etc. The communication unit 609 consists of a network interface, etc. The drive 610 drives removable media 611 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0101] In the computer 600 configured as described above, the CPU 601 loads, for example, a program stored in the memory unit 608 into the RAM 603 via the input / output interface 605 and the bus 604, and executes it, thereby performing the series of processes described above.

[0102] The program executed by computer 600 (CPU 601) can be provided by recording it on removable media 611, such as a packaged media. The program can also be provided via wired or wireless transmission media, such as a local area network, the internet, or digital satellite broadcasting.

[0103] In computer 600, programs can be installed in the storage unit 608 via the input / output interface 605 by inserting removable media 611 into drive 610. Alternatively, programs can be received by the communication unit 609 via a wired or wireless transmission medium and installed in the storage unit 608. Furthermore, programs can be pre-installed in ROM 602 or storage unit 608.

[0104] The program executed by the computer 600 may be a program that is processed chronologically in the order described herein, or it may be a program that is processed in parallel or at necessary times, such as when it is called.

[0105] In this specification, the step of describing a program to be recorded on a recording medium includes not only processes that are performed chronologically in the order described, but also processes that are not necessarily performed chronologically, but are executed in parallel or individually.

[0106] The embodiments of the technology relating to this disclosure are not limited to those described above, and various modifications are possible without departing from the gist of the technology relating to this disclosure.

[0107] For example, the technology disclosed herein can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0108] Furthermore, each step described in the flowchart above can be performed by a single device, or it can be divided and performed by multiple devices.

[0109] Furthermore, if a single step includes multiple processes, those processes can be executed by a single device or shared among multiple devices.

[0110] The effects described herein are merely illustrative and not limited to those described herein; other effects may also occur.

[0111] Furthermore, the technology relating to this disclosure can have the following configuration. (1) A camera unit that photographs a subject and a sensor unit that acquires distance information from the subject, and a field of view calculation unit that calculates the coordinate range that falls within the field of view of the camera unit in the three-dimensional coordinates of the subject, A display control unit that visualizes the aforementioned coordinate range and An information processing device equipped with the following features. (2) The display control unit displays focus position information indicating the focus position of the camera unit within the visualized coordinate range. (1) The information processing device described above. (3) The display control unit projects the focus position information within the coordinate range having depth information. (2) The information processing device described above. (4) The parallax information includes the difference in position and orientation between the camera unit and the sensor unit, as well as the field of view information of the camera unit and the sensor unit, respectively. An information processing device as described in any of (1) to (3). (5) The difference information is acquired based on the external parameters of the camera unit and the sensor unit, respectively. The field of view information is acquired as internal parameters of the camera unit and the sensor unit, respectively. (4) The information processing device described above. (6) The system further comprises a three-dimensional coordinate acquisition unit that acquires the three-dimensional coordinates of the subject based on the distance information acquired by the sensor unit. An information processing device as described in any of (1) to (5). (7) The display control unit displays information indicating whether the position and orientation of the camera unit relative to the subject are appropriate. An information processing device as described in any of (3) to (6). (8) If the position or orientation of the camera unit relative to the subject is inappropriate, the display control unit changes the display mode of the discrimination information to indicate that the position or orientation of the camera unit is inappropriate. (7) The signal processing device described above. (9) The display control unit hides the discrimination information if the position of the camera unit relative to the subject is inappropriate. (8) The information processing device described above. (10) The display control unit displays the discrimination information in a warning color if the camera unit's orientation relative to the subject is inappropriate. (8) The information processing device described above. (11) The system further includes a focus position recording unit that records the focus position already captured by the camera unit, The display control unit further displays captured focus position information indicating the captured focus position within the visualized coordinate range. An information processing device as described in any of (3) to (10). (12) The system further includes a score calculation unit that uses the parallax information to calculate a score for evaluating the accuracy of the three-dimensional model data of the subject generated based on the captured image taken by the camera unit. An information processing device as described in any of (4) to (11). (13) The aforementioned camera unit is mounted on the imaging device, The sensor unit is mounted on a terminal device attached to the imaging device, The display control unit visualizes the coordinate range on the display screen provided by the terminal device. An information processing device as described in any of (1) to (12). (14) The aforementioned display screen displays the captured images captured by the shooting function of the terminal device in real time. The display control unit displays the area outside the coordinate range in the captured image in a display manner different from that of the coordinate range. (13) The information processing device described above. (15) The aforementioned camera unit is mounted on the imaging device, The sensor unit is mounted on an accessory device attached to the imaging device, The display control unit visualizes the coordinate range on the display screen provided by the imaging device. An information processing device as described in any of (1) to (12). (16) The camera unit and the sensor unit are mounted on the imaging device. The display control unit visualizes the coordinate range on the display screen provided by the imaging device. An information processing device as described in any of (1) to (12). (17) The aforementioned sensor unit is composed of a depth sensor. An information processing device as described in any of (1) to (16). (18) The aforementioned sensor unit is composed of a stereo camera. An information processing device as described in any of (1) to (16). (19) Based on the parallax information between the camera unit that photographs the subject and the sensor unit that acquires distance information to the subject, the coordinate range that falls within the field of view of the camera unit in the three-dimensional coordinate system of the subject is calculated. To visualize the aforementioned coordinate range and Information processing methods including (20) Based on the parallax information between the camera unit that photographs the subject and the sensor unit that acquires distance information to the subject, the coordinate range that falls within the field of view of the camera unit in the three-dimensional coordinate system of the subject is calculated. To visualize the aforementioned coordinate range and A program that causes a computer to perform a process that includes [a specific action]. [Explanation of symbols]

[0112] 1 Shooting assist system, 10 Smartphone, 11 Display unit, 20 ILC, 21 Display unit, 101 IMU, 102 Sensor unit, 103 Camera unit, 104 Parallax calibration unit, 111 Parallax information acquisition unit, 112 Position and orientation calculation unit, 113 Three-dimensional coordinate acquisition unit, 114 Field of view calculation unit, 115 Display control unit, 121 Appropriate position and orientation determination unit, 131 Focus position calculation unit, 132 Focus position recording unit, 141 Score calculation unit, 211 Camera unit, 212 Captured image recording unit, 310 Sensor accessory, 420 ILC, 510 Smartphone

Claims

1. A camera unit that photographs a subject and a sensor unit that acquires distance information from the subject, and a field of view calculation unit that calculates the coordinate range that falls within the field of view of the camera unit in the three-dimensional coordinates of the subject, A display control unit that visualizes the aforementioned coordinate range and An information processing device equipped with the following features.

2. The display control unit displays focus position information indicating the focus position of the camera unit within the visualized coordinate range. The information processing apparatus according to claim 1.

3. The display control unit projects the focus position information within the coordinate range having depth information. The information processing apparatus according to claim 2.

4. The parallax information includes the difference in position and orientation between the camera unit and the sensor unit, as well as the field of view information of the camera unit and the sensor unit, respectively. The information processing apparatus according to claim 1.

5. The difference information is acquired based on the external parameters of the camera unit and the sensor unit, respectively. The field of view information is acquired as internal parameters of the camera unit and the sensor unit, respectively. The information processing apparatus according to claim 4.

6. The system further comprises a three-dimensional coordinate acquisition unit that acquires the three-dimensional coordinates of the subject based on the distance information acquired by the sensor unit. The information processing apparatus according to claim 1.

7. The display control unit displays information indicating whether the position and orientation of the camera unit relative to the subject are appropriate. The information processing apparatus according to claim 3.

8. If the position or orientation of the camera unit relative to the subject is inappropriate, the display control unit changes the display mode of the discrimination information to indicate that the position or orientation of the camera unit is inappropriate. The signal processing apparatus according to claim 7.

9. The display control unit hides the discrimination information if the position of the camera unit relative to the subject is inappropriate. The information processing apparatus according to claim 8.

10. The display control unit displays the discrimination information in a warning color if the camera unit's orientation relative to the subject is inappropriate. The information processing apparatus according to claim 8.

11. The system further includes a focus position recording unit that records the focus position already captured by the camera unit, The display control unit further displays captured focus position information indicating the captured focus position within the visualized coordinate range. The information processing apparatus according to claim 3.

12. The system further includes a score calculation unit that uses the parallax information to calculate a score for evaluating the accuracy of the three-dimensional model data of the subject generated based on the captured image taken by the camera unit. The information processing apparatus according to claim 4.

13. The aforementioned camera unit is mounted on the imaging device, The sensor unit is mounted on a terminal device attached to the imaging device, The display control unit visualizes the coordinate range on the display screen provided by the terminal device. The information processing apparatus according to claim 1.

14. The aforementioned display screen displays the captured images captured by the shooting function of the terminal device in real time. The display control unit displays the area outside the coordinate range in the captured image in a display manner different from that of the coordinate range. The information processing apparatus according to claim 13.

15. The aforementioned camera unit is mounted on the imaging device, The sensor unit is mounted on an accessory device attached to the imaging device, The display control unit visualizes the coordinate range on the display screen provided by the imaging device. The information processing apparatus according to claim 1.

16. The camera unit and the sensor unit are mounted on the imaging device. The display control unit visualizes the coordinate range on the display screen provided by the imaging device. The information processing apparatus according to claim 1.

17. The aforementioned sensor unit is composed of a depth sensor. The information processing apparatus according to claim 1.

18. The aforementioned sensor unit is composed of a stereo camera. The information processing apparatus according to claim 1.

19. Based on the parallax information between the camera unit that photographs the subject and the sensor unit that acquires distance information to the subject, the coordinate range that falls within the field of view of the camera unit in the three-dimensional coordinate system of the subject is calculated. To visualize the aforementioned coordinate range and Information processing methods including

20. Based on the parallax information between the camera unit that photographs the subject and the sensor unit that acquires distance information to the subject, the coordinate range that falls within the field of view of the camera unit in the three-dimensional coordinate system of the subject is calculated. To visualize the aforementioned coordinate range and A program that causes a computer to perform a process that includes [a specific action].

Images