Information processing apparatus, information processing method, and program

Abstract

An information processing apparatus including circuitry configured to acquire one or more input images, acquire information related to each acquired input image of the one or more input images based on position information indicating a shooting position of each acquired input image, and orientation information indicating a shooting orientation of each acquired input image, and output at least one output notification indicating the acquired information related to at least one acquired input image of the one or more input images.

Description

INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING METHOD, AND PROGRAMCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of Japanese Priority Patent Application JP 2023-182567 filed October 24, 2023, the entire contents of which are incorporated herein by reference.

[0002] The present technology relates to an information processing apparatus, an information processing method, and a program that can be applied to generation of a 3D model.

[0003] Patent Literature 1 has disclosed an unmanned aircraft to be used for generation of a 3D model. In this unmanned aircraft, a plurality of images of a scan target is acquired and a 3D model is updated in real time. Accordingly, the 3D model is quickly scanned in practice.

[0004] Japanese Patent Application Laid-open No. 2023-72064

[0005] As described above, it is desirable to provide a technology that allows generation of a high-quality 3D model.

[0006] In view of the above-mentioned circumstances, it is an objective of the present technology to provide an information processing apparatus, an information processing method, and a program that allow generation of a high-quality 3D model.

[0007] According to the present disclosure, there is provided an image processing apparatus that includes circuitry configured to acquire one or more input images, acquire information related to each acquired input image of the one or more input images based on position information indicating a shooting position of each acquired input image, and orientation information indicating a shooting orientation of each acquired input image, and output at least one output notification indicating the acquired information related to at least one acquired input image of the one or more input images.

[0008] Furthermore, according to the present disclosure, an image processing method includes acquiring one or more input images, acquiring information related to each acquired input image of the one or more input images based on position information indicating a shooting position of each acquired input image, and orientation information indicating a shooting orientation of each acquired input image, and outputting at least one output notification indicating the acquired information related to at least one acquired input image of the one or more input images.

[0009] In addition, according to the present disclosure, a non-transitory computer-readable medium is provided having embodied thereon a program, which when executed by a computer causes the computer to execute an image processing method, the method including acquiring one or more input images, acquiring information related to each acquired input image of the one or more input images based on position information indicating a shooting position of each acquired input image, and orientation information indicating a shooting orientation of each acquired input image, and outputting at least one output notification indicating the acquired information related to at least one acquired input image of the one or more input images.

[0010] Fig. 1A is a schematic view showing the outline of a photogrammetry system.Fig. 1B is a schematic view showing the outline of the photogrammetry system.Fig. 2A is a schematic view showing an example of an image-shooting screen on a display.Fig. 2B is a schematic view showing an example of the image-shooting screen on the display.Fig. 3 is a schematic diagram showing a configuration example of an image-shooting device.Fig. 4 is an example of a processing flowchart by the image-shooting device.Fig. 5A is a graph showing a relationship between a position error and a position score.Fig. 5B is a graph showing a relationship between an orientation error and an orientation position score.Fig. 6A is a schematic view showing an example of a subjective view.Fig. 6B is a schematic view showing an example of a wide view.Fig. 7A is a schematic view showing a state of shooting of a wide-view image.Fig. 7B is a schematic view showing a state of shooting of the automobile.Fig. 8A is a schematic view showing an example of images of gauges.Fig. 8B is a schematic view showing an example of the images of the gauges.Fig. 9 is a schematic view showing a state in which a moving object has been detected.Fig. 10A is a schematic view showing an example of wide-view display.Fig. 10B is a schematic view showing an example of the wide-view display.Fig. 11 is a schematic view showing an example of the wide-view display.Fig. 12A is a schematic view showing a variation of the image-shooting device.Fig. 12B is a schematic view showing a variation of the image-shooting device.Fig. 13 is a schematic diagram showing a configuration example of the image-shooting device.Fig. 14 is a schematic view showing a variation example of an image shooting method.Fig. 15 is a block diagram showing a hardware configuration example of a computer capable of realizing the image-shooting device.

[0011] Hereinafter, embodiments according to the present technology will be described with reference to the drawings.

[0012] Photogrammetry Figs. 1A and 1B are schematic views each showing the outline of a photogrammetry system 100. The present technology is a technology used in the field of photogrammetry. The photogrammetry is a term referring to a technology of measuring a target object on the basis of images. This technology allows generation of a 3D model of an object existing in a real space, for example.

[0013] Specifically, a user 1 takes a plurality of taken images of a target object and a 3D model of the object is generated on the basis of the plurality of taken images. In a case of generating a 3D model to be used for metaverse or the like, the user 1 could draw a 3D model by his or herself using dedicated software, but it requires advanced techniques, so such a method is often inefficient in view of time and effort and costs. On the other hand, in the photogrammetry, a relatively simple and convenient method of shooting an object existing in the real space allows generation of the 3D model.

[0014] In the photogrammetry system 100 according to an embodiment of the present technology, as shown in Fig. 1A, the user 1 takes a plurality of images of an automobile 2. In this example, the automobile 2 is illustrated as a target object of shooting, though a specific type of the target object is not limited. The target object may be any object that can be a shooting target in the photogrammetry, such as natural objects, e.g., trees, artificial objects, e.g., telegraph poles, humans, and other animals.

[0015] In an embodiment, the user 1 takes an image at a predetermined target position in a predetermined target orientation. The target position and the target orientation refer to a position at which the user 1 should take an image of the automobile 2 and a shooting orientation, respectively. The contents and generation method therefor will be described later in detail.

[0016] Fig. 1A schematically shows nine target positions 3 (3a to 3i) as white triangles. Each triangle is an isosceles triangle. The orientation from the apex opposite to the bottom side to the midpoint on the bottom side represents a target orientation at the target position. Fig. 1A shows the target orientation at the target position 3a as the broken-line arrow. In this manner, the target orientation is an orientation of the perpendicular bisector on the bottom side. The target orientation at the target position 3a is an orientation toward the automobile 2.

[0017] The user 1 holds an image-shooting device 4 in the target orientation at the target position 3a and takes a still image of the automobile 2. Then, at the target position 3b, the user 1 takes an image in the target orientation at the target position 3b. Similarly, the user 1 takes an image in each target orientation at each of the target positions 3c to 3i in the order indicated by the elliptical arrow in Fig. 1A. Since the target orientations are all orientations toward the automobile 2, the automobile 2 is shot in various directions at the respective target positions 3. For example, the front side of the automobile 2 is shot at the target positions 3b and 3c and the rear side is shot at the target positions 3f and 3g.

[0018] In this example, the number of target positions 3 is set to be nine for the sake of description, though the number of target positions 3, i.e., the number of shooting times is not limited. Depending on the type and size of the target object, demanded quality of the 3D model, and the like, the number of shooting times can be several hundreds to several thousands. Moreover, the specific order of image shooting is also not limited, and for example, shooting may be performed in the counter-clockwise order in Fig. 1A. Alternatively, the user 1 may take images in a desired order without specifying the order.

[0019] Virtual View Figs. 2A and 2B are schematic views each showing an example of an image-shooting screen on a display 7. The image-shooting device 4 includes the display 7. When the user 1 takes an image, an image as shown in each of Figs. 2A and 2B is displayed on the display 7. In this example, a smartphone with a camera function is used as the image-shooting device 4 and performs image shooting in a state in which a dedicated application program has been activated. In addition, a specific type of the image-shooting device 4 is not limited, and for example, a dedicated device with an image-shooting function and a display function may be used.

[0020] Fig. 1B shows a periphery of the target position 3c in Fig. 1A. Fig. 2A shows an image-shooting screen in a case where the user 1 holds the camera toward the same orientation as the target orientation at the target position 3c at a device position 8a in a dot-pattern triangular shape in Fig. 1B. The automobile 2 in an angle of view of the camera is displayed on the display 7. In this manner, as in a generally-used digital camera or the like, the target object in the current angle of view is displayed on the display 7 as it is. The automobile 2 is displayed with a smaller size in Fig. 2A because the device position 8a is located relatively far from the automobile 2.

[0021] Additionally, various graphical user interfaces (GUIs) are displayed on the display 7. In the example shown in each of Figs. 2A and 2B, a virtual view 9 (9z, 9x) and a view cone 10 (10c) are displayed as GUIs. The virtual view 9 is constituted by a virtual view 9z displayed at the upper left of the display 7 and a virtual view 9x displayed on a right-hand side of the virtual view 9z. The virtual view 9z is a window displaying positional relationships between the device position 8a and the device orientation and between the target position 3 and the target orientation when viewed from above in a vertical direction.

[0022] In the virtual view 9z, a state when a region 6a shown by the broken-line rectangle in Fig. 1B is viewed from above (front side on the drawing sheet) is displayed with a left side of the region 6a arranged downwards. Then, a target icon 11c is displayed at a position corresponding to the target position 3c. Moreover, a device icon 12a is displayed at a position corresponding to the device position 8a.

[0023] For example in a case where the device position 8a moves upwards in Fig. 1B from this state, the device icon 12a moves substantially leftwards in the virtual view 9z. Moreover, in a case where the device position 8a moves leftwards, the device icon 12a moves substantially downwards.

[0024] Moreover, the orientation of the target icon 11c is an orientation corresponding to the target orientation at the target position 3c. Specifically, the orientation of the target icon 11c is an orientation of the perpendicular bisector on the bottom side of the triangle as in the broken-line arrow shown at the target position 3a in Fig. 1A. In the same way, the orientation of the device icon 12a is an orientation corresponding to the device orientation at the device position 8a.

[0025] For example, when the device is oriented leftwards in Fig. 1B and the device orientation becomes leftwards from this state, the orientation of the device icon 12a becomes downwards. Moreover, when the device is oriented rightwards, the orientation of the device icon 12a becomes upwards.

[0026] In this example, the target orientation at the target position 3c and the device orientation at the device position 8a are the same. Therefore, the orientations of the target icon 11c and the device icon 12a are also the same in the virtual view 9z. However, when the device orientation is different from the target orientation, the orientation of the target icon 11c can also be different from the orientation of the device icon 12a.

[0027] The size of the region 6 displayed in the virtual view 9z is not limited. For example, a state when a wider region 6 is viewed from above in the vertical direction may be displayed. Moreover, the shape of the region 6 is also not limited, and a circular region 6 may be displayed in a circular window. Moreover, for example in a case of displaying the virtual view 9z with a right side of the rectangle in Fig. 1B oriented downwards, the direction of display can be various directions at 360 degrees, and the display direction is also not limited. Moreover, the "state when viewed in the vertical direction" also includes the state when viewed from below in the vertical direction. That is, the state when viewed from below in the vertical direction may be displayed in the virtual view 9z.

[0028] Moreover, depending on needs, the region 6 displayed in the virtual view 9z and the direction of display may be changed. For example in a case where the distance between the target icon 11c and the device icon 12a decreases, processing of decreasing the size (zooming) of the region 6 to be displayed may be performed so that the user can easily see the respective icons. Conversely, processing of increasing the size (up) of the region 6 to be displayed may be performed.

[0029] Alternatively, for example when the orientation of the device icon 12a changes, the direction of display may be changed. For example, the direction of display may rotate so that the device icon 12a is constantly oriented leftwards following a change in the orientation of the device icon 12a. As a matter of course, the region 6 to be displayed and the direction of display may be fixed.

[0030] Specific types of the target icon 11c and the device icon 12a are also not limited. For example, each icon may be circular and the orientation may be expressed as an arrow. Moreover, the color and pattern are also not limited, and the icons may be transparent. Moreover, in a case where a target object is present in the region 6, an icon representing the target object may be displayed. Alternatively, an arrow representing the order of image shooting may be displayed. In addition, a specific display mode in the virtual view 9z is not limited.

[0031] Fig. 2B shows an image-shooting screen in a case where the user 1 holds the camera to the same orientation as the target orientation at the target position 3c, at a device position 8b in Fig. 1B. In this example, the automobile 2 is displayed to be bigger because the device position 8b is relatively close to the automobile 2. Moreover, the distance between the target icon 11c and the device icon 12a is shorter than in Fig. 2A because the device position 8b is closer to the target position 3c. Thus, the region displayed in the virtual view 9z is changed from 6a to 6b so that those icons are displayed near the center of the virtual view 9z.

[0032] In the virtual view 9x in Fig. 2A, a state when the target position 3c and the device position 8a in Fig. 1B are viewed from a lateral side is displayed. Here, the lateral side may be any direction orthogonal to the vertical direction, and for example in a case where the upper side is a positive side on a Z-axis in an orthogonal coordinate system, an orientation when the target position 3c or the like is viewed from an orientation parallel to an XY-plane is the "lateral side." In this example, a state when the target position 3c or the like is viewed from the left side in Fig. 1B is displayed with a lower side in the vertical direction oriented downwards. That is, the upper and lower directions in the virtual view 9z are identical to the vertical direction in the real space.

[0033] In Figs. 2A and 2B, the device icon 12 is displayed at a lower position than the target icon 11c. It means that the device position 8 is located on the lower side than the target position 3c also in the real space. In a case where the user 1 moves the image-shooting device 4 upwards, for example, by lifting it up from this state, the position of the device icon 12 also moves upwards.

[0034] In this manner, in an embodiment, the states when viewed from two directions of the upper side and the lateral side are displayed in the virtual views 9a and 9b, respectively. In which direction a state displayed in each of the virtual views 9 is viewed is not limited. For example, a state when viewed from a direction (e.g., obliquely lower side) which is not a direction orthogonal to the vertical direction may be displayed. Moreover, a state when viewed from a single direction may be displayed in a single virtual view 9. Alternatively, states when viewed from three or more directions may be displayed in three or more virtual views 9. The target icon 11 and the device icon 12 displayed in the virtual view 9 correspond to an embodiment of guide information according to the present technology.

[0035] The view cone 10 is an icon in a square pyramid shape. As shown in each of Figs. 2A and 2B, the view cone 10 displayed, superimposed on the image-shooting screen in the real space. The view cone 10 is superimposed at a position corresponding to the target position 3 on the image-shooting screen. In each of Figs. 2A and 2B, the view cone 10c corresponding to the target position 3c is superimposed on the image-shooting screen. Moreover, the view cone 10c is superimposed in an orientation corresponding to the target orientation at the target position 3c. In this example, an orientation from the apex of the square pyramid to the center of the bottom surface is the corresponding orientation.

[0036] In Fig. 2A, the view cone 10c is displayed to be smaller because the target position 3c is relatively far from the device position 8a. Meanwhile, in Fig. 2B, the view cone 10 is displayed on a major part of the image-shooting screen because the device position 8b is closer to the target position 3c. In this manner, the view cone 10 is displayed to be bigger as the device position 8 approaches the target position 3, and the view cone 10 can also cover the entire image-shooting screen.

[0037] A specific type of the view cone 10 is not limited. For example, the view cone 10 may be another cone- or frustum-shaped icon and the orientation of its bottom surface may express the orientation corresponding to the target orientation. Alternatively, the view cone 10 may be an icon in a spherical shape or the like and an arrow may express the orientation. Moreover, in this example, the transparent view cone 10 is displayed, superimposed, which can improve the visibility, though not limited thereto. An opaque icon may be used. In addition, the color, pattern, and the like are also not limited.

[0038] The view cone 10 corresponds to an embodiment of an icon according to the present technology, which is displayed, superimposed at a position corresponding to the target position on an image-shooting screen when the taken image is taken, the icon being displayed in an orientation corresponding to the target orientation.

[0039] The user 1 adjusts a position where the user 1 stands, a height at which the user 1 holds the camera, an orientation in which the user 1 holds the camera, and the like while seeing the virtual views 9z and 9x and the view cone 10. Then, the user 1 takes an image of the automobile 2 by releasing the shutter toward the automobile 2. Such image shooting is performed at each target position 3, such that a plurality of images of the automobile 2 is taken.

[0040] It should be noted that in the display of the view cone 10 and the target icon 11, a plurality of icons corresponding to all the target positions 3a to 3i may be simultaneously displayed or only an icon corresponding to the target position 3 that the user should face next may be displayed and an icon corresponding to the next target position 3 may be newly displayed at the time when shooting at such a target position 3 is completed.

[0041] Image-Shooting Device Fig. 3 is a schematic diagram showing a configuration example of the image-shooting device 4. The image-shooting device 4 includes a camera 15, a controller 16, the display 7, and a loudspeaker 8. The camera 15 may be any type of camera, and for example in a case where the image-shooting device 4 is a smartphone, the camera 15 is arranged inside a casing of the smartphone so that its lens is exposed from the casing. Otherwise, in a case where the image-shooting device 4 is a device dedicated to this system, the camera 15 may be externally connected to the casing.

[0042] The camera 15 continues to constantly acquire a single image for each frame for example at a predetermined frame rate (e.g., 1 / 60 seconds per frame). The image is acquired irrespective of whether or not the user 1 has released the shutter in the frame (has sent a shooting trigger). That is, although an image in a frame for which the shutter has not been released is not stored as a photograph, the image continues to be constantly acquired as a consecutive still image because it is necessary to display the image on the display 7 as an image in the current angle of view.

[0043] The controller 16 includes hardware necessary for configurations of a computer, for example, processors such as a CPU, a GPU, and a DSP, memories such as a ROM and a RAM, and a storage device such as an HDD. An image processing method according to an embodiment of the present technology is executed for example by the CPU loading a program according to an embodiment of the present technology recorded on the ROM or the like in advance to the RAM and executing the program. For example, a programmable logic device (PLD) such as a field programmable gate array (FPGA) or another device such as an application specific integrated circuit (ASIC) may be used as the controller 16.

[0044] In an embodiment, by the CPU of the controller 16 executing the program (e.g., an application program) according to the present technology, a self-position estimation unit 18, a shooting error calculation unit 19, a shooting score calculation unit 20, a backlight detection unit 21, a blur detection unit 22, a moving object detection unit 23, a viewpoint information generation unit 24, a visualized information generation unit 25, the sound generation unit 26, and a viewpoint information recording unit 27 are realized as functional blocks. It should be noted that in order to realize each functional block, dedicated hardware such as an integrated circuit (IC) may be used as appropriate.

[0045] The self-position estimation unit 18 calculates a shooting position and a shooting orientation. The shooting position is information indicating a position of the image-shooting device 4, and is for example information indicating the position of the image-shooting device 4 as three values of (X, Y, Z) in a three-dimensional coordinate system defined in a space. Otherwise, another coordinate system such as a polar coordinate system may be used or the position may be represented, not relying on any coordinate system.

[0046] The shooting orientation is information indicating an orientation of the image-shooting device 4, and the orientation of the image-shooting device 4 is represented by three values of roll, pitch, and yaw, for example. It should be noted that the term "orientation" set forth herein has a meaning similar to an attitude of the image-shooting device 4, and it can be also said that the shooting orientation is a shooting attitude, for example. The shooting orientation may be information indicating the orientation of the image-shooting device 4 in another way.

[0047] The shooting position corresponds to an embodiment of a device position according to the present technology. The shooting orientation corresponds to an embodiment of a device orientation according to the present technology. Moreover, the shooting position and the shooting orientation correspond to an embodiment of device information according to the present technology.

[0048] The image-shooting device 4 includes a storage unit (not shown) and a target shooting viewpoint database (DB) 28 is configured in the storage unit. The target shooting viewpoint DB 28 is a database storing a target shooting position and a target shooting orientation. The target position and orientation are position and orientation in which the user 1 should take an image of the automobile 2. Such information is for example information of three-dimensional coordinate values or three values of roll, pitch, and yaw like the shooting position and orientation.

[0049] Shooting positions and orientations typically necessary for generation of the 3D model are generated as target positions and orientations. For example, the target positions and orientations are generated so that all surfaces of the target object can be shot without exception. Moreover, many target positions and orientations may be generated for generating a high-definition 3D model or a relatively few target positions or orientations may be generated in a case where even a slightly rough 3D model is acceptable.

[0050] The target position and orientation may be generated by any method such as an existing technique. For example, there is an approach in which viewpoint candidates are sampled in a shooting space, scores are set thereto, and a combinatorial optimization problem with constraints using shooting distance and time as constraints is solved for planning. Moreover, an end-to-end planning process using a deep neural network (DNN) has been found in recent years.

[0051] Moreover, the target positions and orientations may be generated by this system. For example, the user 1 may takes rough images of the target object (e.g., from three directions) in advance and the target positions and orientations may be generated on the basis of the rough taken images. The target position and the target orientation corresponds to an embodiment of plan information according to the present technology. It should be noted that the plan information may include information regarding the order of image shooting.

[0052] The shooting error calculation unit 19 calculates a position error and an orientation error. The position error is an error between the target position and the shooting position. Moreover, the orientation error is an error between the target orientation and the shooting orientation. In an embodiment, for example, the position error is calculated as information of three-dimensional coordinate values and the orientation error is calculated as information of three values of roll, pitch, and yaw.

[0053] The shooting score calculation unit 20 calculates a shooting score on the basis of the position error and the orientation error. The shooting score is typically an indicator indicating how accurately the user 1 has taken an image in terms of the shooting position and orientation with respect to the target position and orientation. A specific calculation method for the shooting score will be described later.

[0054] The backlight detection unit 21, the blur detection unit 22, and the moving object detection unit 23 respectively detect inclusion of the backlight, generation of the blur, and inclusion of the moving object in the image acquired by the camera 15 and generates a backlight detection result, a blur detection result, and a moving object detection result. Specific detection methods by those functional blocks will be described later.

[0055] The viewpoint information generation unit 24 generates current-viewpoint information. The current-viewpoint information is information obtained by integrating a target position, a target orientation, a shooting position, a shooting orientation, a position error, an orientation error, a shooting score, a backlight detection result, a blur detection result, and a moving object detection result at a certain point of time (in a frame).

[0056] The visualized information generation unit 25 generates an image to be displayed on the display 7. Specifically, the visualized information generation unit 25 generates an image obtained by superimposing the view cone 10 shown in each of Figs. 2A and 2B on the image acquired by the camera 15. Moreover, the visualized information generation unit 25 generates the virtual view 9. In addition, the visualized information generation unit 25 may generate various GUIs and images of gauges to be described later.

[0057] The sound generation unit 26 generates a sound to be output by the loudspeaker 8. In an embodiment, the sound generation unit 26 generates a warning sound to the user 1 under predetermined conditions. In addition, the sound generation unit 26 may generate any type of sound such as a sound effect emitted when the user 1 presses a button on the display 7.

[0058] In a case where the user 1 has input a shooting trigger, the viewpoint information recording unit 27 records an image at the time when the shooting trigger has been input as a taken image. Hereinafter, the image constantly acquired by the camera 15 will be simply referred to as an "image" and the image at the time when the shooting trigger has been input will be referred to as a "taken image" for distinguishing them.

[0059] Moreover, the viewpoint information recording unit 27 records the current-viewpoint information at the time when the shooting trigger has been input as viewpoint information at the time of shooting. That is, the viewpoint information at the time of shooting includes a target position, a target orientation, a shooting position at the time of shooting, a shooting orientation at the time of shooting, a position error at the time of shooting, an orientation error at the time of shooting, a shooting score at the time of shooting, a backlight detection result at the time of shooting, a blur detection result at the time of shooting, and a moving object detection result at the time of shooting.

[0060] For example, by the user 1 pressing a shutter button, a shooting trigger is input to the viewpoint information recording unit 27. The shutter button may be a physical button provided on a casing of the image-shooting device 4 or may be a virtual button displayed on the display 7. In addition, any operation may be used to input the shooting trigger. In an embodiment, a shooting viewpoint DB 29 is configured in the storage unit (not shown) and a taken image at each time when the shooting trigger has been input and viewpoint information at the time of shooting are recorded.

[0061] In addition, specific configurations of the functional blocks of the controller 16 are not limited, and any configurations may be employed as long as the present technology can be realized. The self-position estimation unit 18 corresponds to an embodiment of a first acquisition unit according to the present technology. The shooting error calculation unit 19 corresponds to an embodiment of a second acquisition unit according to the present technology. The shooting score calculation unit 20, the viewpoint information generation unit 24, the visualized information generation unit 25, the sound generation unit 26, and the viewpoint information recording unit 27 correspond to an embodiment of an output unit according to the present technology. The controller 16 corresponds to an embodiment of an information processing apparatus according to the present technology.

[0062] The display 7 is for example a display device using liquid-crystal, electro-luminescence (EL), or the like. A display control unit (not shown) controls the display on the display 7 of various images and GUIs generated by the visualized information generation unit 25. The loudspeaker 8 is provided for example inside the casing of the image-shooting device 4 and outputs various sounds generated by the sound generation unit 26.

[0063] In addition, the storage unit, an operation unit, a communication unit, and the like (not shown) may be provided. The storage unit, the operation unit, the communication unit, the controller 16, the display 7, and the loudspeaker 8 are connected to one another via a bus, for example. Alternatively, these may be connected to one another using a communication network, an original communication scheme which is not standards, or the like instead of the bus.

[0064] The storage unit stores a control program for controlling comprehensive operations of the image-shooting device 4, for example. A method of installing the control program into the image-shooting device 4 is not limited. For example, the control program may be installed via various recording media or may be installed via the Internet or the like. Moreover, for example, the storage unit stores the target shooting viewpoint DB 28 and the shooting viewpoint DB 29. The type and the like of a recording medium that functions as the storage unit are not limited, and any computer-readable recording medium may be used. For example, any recording medium for non-temporarily recording data may be used.

[0065] The operation unit is for example a keyboard, a pointing device, a touch panel, or another operation device other than the above-mentioned shutter button. In a case where the operation unit includes a touch panel, the touch panel can be integral with the display 7.

[0066] The communication unit is a communication module for communicating with other devices via a network such as a local area network (LAN) or a wide area network (WAN). A communication module for near-field communication such as Bluetooth (registered trademark) may be provided. Moreover, communication devices such as a modem and a router may be used. The communication unit performs communication for example between the image-shooting device 4 and an external device.

[0067] The acquisition of the target positions and orientations and the recording of the taken image and the viewpoint information at the time of shooting may be performed by the communication unit performing communication with a database on a cloud without the target shooting viewpoint DB 28 and the shooting viewpoint DB 29 configured in the image-shooting device 4. Alternatively, various types of processing may be performed on the cloud by configuring some or all of the functional blocks of the controller 16 on the cloud.

[0068] Processing Flow Fig. 4 is an example of a processing flowchart by the image-shooting device 4. In an embodiment, processing in Steps 101 to 119 shown in this flow are performed at a predetermined frame rate (e.g., 1 / 60 seconds per frame).

[0069] A self-position is estimated (Step 101). Specifically, the self-position estimation unit 18 estimates (calculates) position and orientation of the camera 15 as shooting position and orientation. The shooting position and orientation may be calculated by any method such as an existing technique. Moreover, the image taken by the camera 15 may be acquired by the self-position estimation unit 18, and shooting position and orientation may be calculated on the basis of the image. In such a manner, three values of (X, Y, Z) are calculated for example as a shooting position and three values of roll, pitch, and yaw are calculated as a shooting orientation.

[0070] Shooting errors are calculated (Step 102). To be specific, first of all, the shooting error calculation unit 19 acquires shooting position and orientation from the self-position estimation unit 18 and acquires target position and orientation from the target shooting viewpoint DB 28. Then, the shooting error calculation unit 19 calculates a position error and an orientation error on the basis of such information.

[0071] The position error is a difference between the shooting position and the target position. For example in a case where the shooting position is (X, Y, Z) = (10, 20, 30) and the target position is (20, 40, 50), a difference between them (target position - shooting position), i.e., (20-10, 40-20, 50-30) = (10, 20, 20), is calculated as a position error. That is, moving the camera 15 by an amount corresponding to the position error (by (10, 20, 20)) allows the camera 15 to be adjusted to the same position as the target position.

[0072] Similarly, the orientation error is a difference between the shooting orientation and the target orientation. For example, differences in the three values of the roll, pitch, and yaw are calculated as an orientation error.

[0073] It should be noted that the position error may be any information regarding the error between the target position and the shooting position. That is, the above-mentioned calculation method for the position error is merely exemplary and a specific calculation method for the position error is not limited. Moreover, the position error may be information other than the three-dimensional coordinate values. Similarly, the orientation error may be any information regarding the error between the target orientation and the shooting orientation, a specific calculation method therefor is also not limited, and information other than three values of roll, pitch, and yaw may be employed.

[0074] Score Calculation The shooting score calculation unit 20 calculates shooting scores (Step 103). To be specific, first of all, the shooting score calculation unit 20 calculates a position score. The position score is a score calculated on the basis of a difference between a certain target position and a shooting position and is an indicator indicating how accurately the user has made the camera 15 closer to the target position. The position score is a score calculated for each target position.

[0075] Fig. 5A is a graph showing a relationship between the position error and the position score. The shooting score calculation unit 20 acquires a position error from the shooting error calculation unit 19. Then, the shooting score calculation unit 20 calculates a position score on the basis of a relationship between the position error and the position score defined as in Fig. 5A, for example. In the graph, the horizontal axis denotes the position error and the vertical axis denotes the position score. It should be noted that the position error denoted by the horizontal axis is an absolute value of a difference between the target position and the shooting position. That is, the position error on the horizontal axis is a value constantly equal to or larger than 0.

[0076] As shown in Fig. 5A, in a case where the position error is 0.01 m or more and 0.1 m or less, the position score is calculated in accordance with a linear function equation. Accordingly, it is possible to calculate the position score in a simple and convenient manner. Moreover, in a case where the position error is 0.01 m or less, the position score is constantly calculated to be 100 points. For example, position scores with respect to position errors of 0.005 m and 0.009 m are both 100 points.

[0077] Even though the shooting positions are slightly deviated from the target positions depending on a plan algorithm, sufficient images can be often taken for generation of the 3D model. That is, on the premise that it is difficult to achieve complete positioning, slight position errors are acceptable in many cases. In such cases, by clamping the lower limit for position errors (treating errors of 0.01 m or less as if they were 0.01 m) as in this example, the position scores can be all 100 points as long as the errors are acceptable. Moreover, in this example, in a case where the position error is 0.1 m or more, the position score is constantly calculated to be 0 points. In this manner, the upper limit for position errors may be clamped.

[0078] Moreover, the shooting score calculation unit 20 calculates an orientation score. The orientation score is a score calculated on the basis of a difference between a target orientation at the certain target position and a shooting orientation, and is an indicator indicating how accurately the user has adjusted the orientation of the camera 15 to the target orientation. The orientation score is a score calculated for each target position.

[0079] In an embodiment, first of all, the shooting score calculation unit 20 acquires an orientation error from the shooting error calculation unit 19. Then, on the basis of the acquired orientation error as three values of roll, pitch, and yaw, the shooting score calculation unit 20 calculates an orientation error in a rotation vector (a unit vector and a rotation amount on a rotation axis). The orientation error in the rotation vector is calculated in an angle unit (degrees) in the degree measure.

[0080] Then, the shooting score calculation unit 20 calculates an orientation score on the basis of a relationship between the orientation error and the orientation score defined as in Fig. 5B, for example. Fig. 5B is a graph showing the relationship between the orientation error and the orientation score. In the graph, the horizontal axis denotes an absolute value of the orientation error and the vertical axis denotes the orientation score. As in Fig. 5A, the lower limit (3 deg) and the upper limit (30 deg) for orientation errors are clamped and a linear function graph is shown in the angle range between the lower and upper limits.

[0081] Specific calculation methods for the position score and the orientation score are not limited. For example, without clamping at the lower limit, the score may be calculated to be 100 points only in a case where the shooting position and orientation are completely identical to the target position and orientation. Similarly, clamping at the upper limit may be omitted. Moreover, the relationship between the error and the score is also not limited, and the score may be calculated in accordance with a non-linear relational expression such as a quadratic function. Alternatively, the score may be calculated on the basis of correspondence in a table, for example.

[0082] Next, the shooting score calculation unit 20 calculates a viewpoint score. Specifically, the shooting score calculation unit 20 calculates a mean value of the position score and the orientation score at the certain target position as a viewpoint score at such a position. Alternatively, in a case where either one of the position score and the orientation score is more important, weighting may be performed on the respective scores and a weighted arithmetic mean of them may be calculated as a viewpoint score. In addition, a specific calculation method for the viewpoint score is not limited.

[0083] Last of all, the shooting score calculation unit 20 calculates a total score. Specifically, the shooting score calculation unit 20 calculates a mean value of scores of viewpoints at all target positions as a total score. Alternatively, in a case where a degree of importance differs for each target position, weighting may be performed on the respective viewpoint scores and a weighted arithmetic mean of them may be calculated as a total score. In addition, a specific calculation method for the total score is not limited.

[0084] As described above, four scores of the position score, the orientation score, the viewpoint score, and the total score are calculated as shooting scores. Since the position score is calculated on the basis of the position error, it can be said to be a score calculated on the basis of a difference between the target position and the device position. Similarly, the orientation score can be said to be a score calculated on the basis of a difference between the target orientation and the device orientation. Furthermore, since the viewpoint score and the total score are calculated on the basis of the position score and the orientation score, they can also be said to be scores calculated on the basis of the difference between the target position and the device position and the difference between the target orientation and the device orientation.

[0085] Moreover, in a case where the position error is in the range of 0.01 m or more and 0.1 m or less with respect to the position score, the position score is calculated in accordance with the monotonically decreasing linear function equation, and therefore it can be said that the position score is calculated to become larger in value as the difference between the target position and the device position becomes smaller (hereinafter, such a relationship will be referred to as a "monotonically decreasing relationship").

[0086] The present technology is not limited thereto, and the position score may be calculated on the basis of the monotonically decreasing relationship such as a non-linear expression or a table. Moreover, although the position scores are all clamped to be 100 points in a case where the position error is in the range of 0.01 m or less, the relationship in which the position score value does not change depending on the position error in this manner can be included in the monotonically decreasing relationship, broadly considered as one monotonically decreasing. Similarly, a relationship in which the position error is in the range of 0.1 m or more can also be included in the monotonically decreasing relationship.

[0087] Moreover, it can also be said that the orientation score is also calculated to become larger in value as the difference between the target orientation and the device orientation becomes smaller. In addition, considering the relationship in which as the difference between the target position and the device position becomes smaller, the position score becomes larger, the viewpoint score becomes larger, and the total score becomes larger, it can be said that the viewpoint score and the total score are also calculated to become larger in value as the difference between the target position and the device position becomes smaller. Similarly, it can also be said that the viewpoint score and the total score are calculated to become larger in value as the difference between the target orientation and the device orientation becomes smaller.

[0088] As a matter of course, the score does not need to be calculated on the basis of the monotonically decreasing relationship. For example, a part of the range may include a monotonically increasing relationship. Moreover, although the relationship in which each score continuously changes depending on the position error or the like is established in this example, a relationship in which the score stepwisely changes (e.g., 80 points are provided in a case where the position error is larger than 0.01 m, 100 points are provided in a case where the position error is 0.01 m or less) may be used. Moreover, although each score is calculated to be 100 points for 100 points in this example, the score may be calculated in any form such as 10 points for 10 points or assessment using grades A to E. In addition, a specific calculation method for each score is not limited. The viewpoint score and the total score correspond to an embodiment of a score according to the present technology.

[0089] The backlight detection unit 21 detects backlight (Step 104). The backlight detection unit 21 acquires an image from the camera 15 and detects inclusion of backlight in the image. For example in a case where the camera 15 faces the sun or a light and strong light enters the lens of the camera 15, the light can be detected as backlight. The backlight detection may be performed by any method such as a known approach. The backlight detection unit 21 generates information regarding a detection result of the backlight as a backlight detection result. The backlight detection result may include any information regarding the backlight, e.g., a position of inclusion of the backlight in the image and strength of the backlight. Moreover, in a case where no backlight has been detected, the backlight detection result may include the information itself, "no backlight has been detected."

[0090] The blur detection unit 22 detects a blur (Step 105). The blur detection unit 22 acquires an image from the camera 15 and detects generation of a blur in the image. For example in a case where the camera 15 is out of focus or in a case where the camera 15 vibrates because the user 1 cannot stably hold the camera 15, a blur can be generated. The blur detection may be performed by any method. The blur detection unit 22 generates information regarding a detection result of the blur as a blur detection result. The blur detection result may include any information such as intensity of the blur.

[0091] The moving object detection unit 23 detects a moving object (Step 105). The moving object detection unit 23 acquires an image from the camera 15 and detects inclusion of a moving object in the image. For example in a case where a moving object that should not be included in the angle of view of the camera 15, such as a person, a bird, or an insect, enters the angle of view of the camera 15, it is detected as inclusion of the moving object. The moving object detection may be performed by any method. For example, one of known techniques is a technique of detecting a moving object by estimating a self-position of the device by simultaneous localization and mapping (SLAM) and tracking positions of the surrounding objects by object tracking. The moving object detection unit 23 generates information regarding a detection result of the moving object as a moving object detection result. The moving object detection result may include any information such as the type of moving object and a position of inclusion of the moving object in the image.

[0092] The inclusion of the backlight, generation of the blur, and inclusion of the moving object correspond to an embodiment of a predetermined problem according to the present technology. In addition, any problem other than those problems, which should not occur in shooting, may be detected.

[0093] The viewpoint information generation unit 24 generates current-viewpoint information (Step 107). Specifically, the viewpoint information generation unit 24 acquires target position and orientation from the target shooting viewpoint DB 28. Moreover, the viewpoint information generation unit 24 acquires shooting position and orientation from the self-position estimation unit 18. Moreover, the viewpoint information generation unit 24 acquires a position error, an orientation error, and a shooting score from the shooting score calculation unit 20. Moreover, the viewpoint information generation unit 24 acquires a backlight detection result, a blur detection result, and a moving object detection result from the backlight detection unit 21, the blur detection unit 22, and the moving object detection unit 23, respectively. Then, the viewpoint information generation unit 24 generates current-viewpoint information by integrating them.

[0094] Whether or not a shooting trigger has been input in this frame (frame on which processing has been performed) is determined (Step 108). A dedicated functional block for determination may be configured. In a case where the shooting trigger has been input (Yes in Step 108), the viewpoint information recording unit 27 stores viewpoint information at the time of shooting and a taken image in the shooting viewpoint DB 29. The viewpoint information recording unit 27 acquires viewpoint information from the viewpoint information generation unit 24 and saves the acquired viewpoint information in the shooting viewpoint DB 29 as the viewpoint information at the time of shooting. That is, the viewpoint information at the time of shooting is information obtained by integrating information such as the shooting position at the time of shooting and the shooting orientation at the time of shooting.

[0095] Moreover, the viewpoint information recording unit 27 acquires an image from the camera 15 and saves the acquired image in the shooting viewpoint DB 29 as a taken image. It should be noted that the viewpoint information at the time of shooting and the taken image may include information regarding the time when the shooting trigger has been input.

[0096] Subjective View and Wide View In a case where it is determined in Step 108 that no shooting trigger has been input (No in Step 108) or after the processing in Step 109 is performed, whether or not the current view is a subjective view or a wide view is determined (Step 110).

[0097] Figs. 6A and 6B are schematic views each showing an example of the subjective view and the wide view. In an embodiment, the user 1 is able to switch the view to the subjective view shown in Fig. 6A or the wide view shown in Fig. 6B. The view is switched in a case where a view switching button 32 is pressed. The view switching button 32 is a GUI that is generated by the visualized information generation unit 25 and displayed on the display 7.

[0098] The subjective view shown in Fig. 6A is a view in which an object or the like in the angle of view of the camera 15 is subjectively displayed as it is. Also in the example of Fig. 2B is also display in the subjective view. It should be noted that in Fig. 6A, the illustration of the view cone 10 has been omitted for the sake of simplicity.

[0099] The wide view shown in Fig. 6B is a view in which a bird's-eye view of the automobile 2 and the surrounding space is displayed. That is, for example a bird's-eye view of the automobile 2 or the like viewed from above in the vertical direction is displayed in the wide view. The present technology is not limited thereto, and for example, a bird's-eye view of the automobile 2 or the like viewed from obliquely above may be displayed.

[0100] Figs. 7A and 7B are schematic views showing states of shooting of the wide-view image and shooting of the automobile 2, respectively. In an embodiment, as shown in Fig. 7A, the user 1 has taken a wide-view image showing the entire automobile 2 as a bird's-eye view in advance. The taken wide-view image is stored in the storage unit, for example and is used for display of an image when the wide view is selected. On the other hand, in normal shooting (operation of approaching the target position and shooting the automobile 2), image shooting is performed near the automobile 2 as shown in Fig. 7B.

[0101] Hereinafter, the contents or the like of display in the subjective view will be described. In a case where it is determined that the current view is a subjective view in Step 110, the visualized information generation unit 25 generates visualized information (Step 111). First of all, the visualized information generation unit 25 acquires an image and current-viewpoint information from the viewpoint information generation unit 24.

[0102] The visualized information generation unit 25 superimposes the view cone 10 on the acquired image. Specifically, on the basis of the target position, the target orientation, the shooting position, and the shooting orientation (included in the acquired current-viewpoint information), the visualized information generation unit 25 performs processing of calculating a position on the image, at which the view cone 10 should be superimposed, and superimposing the view cone 10 at the position. The superimposition position calculation may be performed by any method such as an existing technique.

[0103] Next, the visualized information generation unit 25 generates a virtual view 9. Specifically, on the basis of the target position, the target orientation, the shooting position, and the shooting orientation, the visualized information generation unit 25 calculates positions in virtual views 9z and 9x, at which the target icon 11 and the device icon 12 should be displayed, and generates a virtual view 9. The icon position calculation may be performed by any method such as an existing technique.

[0104] Gauge Display In an embodiment, the visualized information generation unit 25 generates images of gauges according to the difference between the target position and the shooting position and the difference between the target orientation and the shooting orientation. Figs. 8A and 8B are schematic views each showing an example of the images of the gauges. In each of Figs. 8A and 8B, two gauges of a distance error gauge 34 and a rotation error gauge 35 are displayed on the lower side of the display 7.

[0105] The distance error gauge 34 is a gauge visually indicating how much large a difference (position error) between the target position and the shooting position is. In this example, the distance error gauge 34 is constituted by a frame part and a dot pattern gauge part. Then, as the gauge part becomes wider, it indicates that the difference between the target position and the shooting position becomes larger.

[0106] A relationship between a difference in distance between the target position and the shooting position and the width of the gauge part may be arbitrarily defined. For example, only in a case where the shooting position is completely identical to the target position, the width of the gauge part may be defined to become zero (disappear) or the width of the gauge part may become zero in a case where the distance difference drops below a predetermined threshold. Conversely, the threshold for the distance difference when the width of the gauge part reaches the upper limit (the gauge part covers the entire frame part) may also be arbitrarily defined.

[0107] Moreover, in this example, an arrow indicating a target line 36 is displayed above the distance error gauge 34. The target line 36 is a reference indicator of the position error for the user 1. Typically, the target line 36 indicates that the position error is acceptable when the gauge part of the distance error gauge 34 remains on the left side of the target line 36 and that the position error is inacceptable when the gauge part exceeds rightwards from the target line 36. Therefore, in this case, it is desirable that the user 1 reduce the position error so that the gauge part drops below the target line 36 and takes an image in state in which the gauge part has dropped below the target line 36.

[0108] The rotation error gauge 35 is a gauge visually indicating how much large a difference (orientation error) between the target orientation and the shooting orientation is. As in the distance error gauge 34, the rotation error gauge 35 is constituted by a frame part and a dot pattern gauge part and means that as the gauge part becomes wider, a difference between the target orientation and the shooting orientation becomes larger. In this example, the arrow indicating the target line 36 for the distance error gauge 34 displayed above the distance error gauge 34 also indicates the target line 36 for simultaneously the rotation error gauge 35. Therefore, the user 1 reduces the orientation error while seeing the arrow. It should be noted that the present technology is not limited thereto, and one arrow may be displayed for each of the distance error gauge 34 and the rotation error gauge 35 and the respective arrows may be at different positions in a horizontal direction.

[0109] In addition, specific design such as the color and pattern of the distance error gauge 34 and the rotation error gauge 35 is not limited. Moreover, the respective gauges may be images of gauges in any form according to a difference between the target position and the device position and a difference between the target orientation and the device orientation, such as pie chart gauges. Alternatively, one gauge obtained by integrating the distance error gauge 34 and the rotation error gauge 35 (e.g., representing a mean value of the respective gauges) may be displayed.

[0110] Although the target line is set to the position error and the orientation error for example to indicate that a sufficient 3D model can be generated if the user shoots the target object with these position and orientation errors, a specific setting reference of the positions is also not limited. The distance error gauge 34 and the rotation error gauge 35 correspond to an embodiment of an image of a gauge according to the present technology.

[0111] The visualized information generation unit 25 generates an image of the distance error gauge 34 on the basis of the acquired position error. Moreover, the visualized information generation unit 25 generates an image of the rotation error gauge 35 on the basis of the acquired orientation error. In addition, the visualized information generation unit 25 generates an image of an icon such as an arrow representing the target line 36.

[0112] Although in the example of Fig. 8A, the illustration of the view cone 10 has been omitted for the sake of simplicity, the target position is near the right front of the automobile 2 (as viewed from the side of the automobile 2) and the target orientation is an orientation toward the automobile 2. Thus, in this example, the shooting position is relatively far from the target position and the difference between the target orientation and the shooting orientation is relatively large. Thus, the values of the distance error gauge 34 and the rotation error gauge 35 are large and they are significantly beyond the target line 36.

[0113] On the other hand, in the example of Fig. 8B, the shooting position is relatively close to the target position and the difference between the target orientation and the shooting orientation is relatively small. Thus, the values of the distance error gauge 34 and the rotation error gauge 35 are small and the rotation error gauge 35 is slightly beyond the target line 36 while the distance error gauge 34 is below the target line 36.

[0114] Score Display Furthermore, the visualized information generation unit 25 generates an image for displaying the shooting score. In this example, the value of the shooting score is displayed as it is, e.g., "Score: x ("x" indicates any number)" at the upper right of the display 7 as shown in each of Figs. 8A and 8B. That is, in this case, a simple image of the letters, "Score: x," is generated as the image for displaying the shooting score. The present technology is not limited thereto, and the score may be displayed in another form such as an image of a gauge. Moreover, although the viewpoint score at a certain target position is displayed in this example, the position score, the orientation score, or the total score may be displayed.

[0115] In the example of Fig. 8A, the shooting position is far from the target position, and therefore the position score is calculated to be lower. Similarly, the difference between the target orientation and the shooting orientation is large, and therefore the orientation score is also calculated to be lower. Therefore, the viewpoint score is also lower, i.e., 30 points. On the other hand, in the example of Fig. 8B, the position score and the orientation score are calculated to be higher, and therefore the viewpoint score is also higher, i.e., 95 points.

[0116] Warning Information Furthermore, the visualized information generation unit 25 refers to the acquired backlight detection result, and outputs warning information in a case where backlight in the image has been detected. The warning information is information for warning the user 1 of predetermined contents. For example in a case where backlight has been detected, the visualized information generation unit 25 generates an image of the letters, e.g., "Backlight has been detected," as the warning information. Alternatively, an icon (e.g., picture of light) indicating that the backlight has been detected may be generated.

[0117] Moreover, the visualized information generation unit 25 refers to the acquired blur detection result, and outputs warning information, e.g., "Blur has been detected," in a case where a blur in the image has been detected.

[0118] Fig. 9 is a schematic view showing a state in which the moving object has been detected. In this example, a person 39 who is not the user 1 enters between the image-shooting device 4 and the automobile 2 and the person 39 has been detected as a moving object. In this example, the letters, "Inclusion of moving object," are displayed at the upper right of the display 7 and the frame part of the image displayed on the display 7 has changed in predetermined color (e.g., violet color). Such information may be generated and displayed as the warning information.

[0119] Otherwise, any event, e.g., an unexpectable problem in shooting, a problem that affects the photogrammetry, may be detected and information corresponding to the event may be output as the warning information. Moreover, the contents of the warning information may also be any contents such as contents of an event or properties (e.g., intensity of backlight). Furthermore, the form of the warning information is also not limited to the letters and image, and may be a sound (to be described later) or may be any other form.

[0120] Moreover, the visualized information generation unit 25 generates a view switching button or another GUI. Specific information generated by the visualized information generation unit 25 is not limited, and any information that is visually recognizable may be generated.

[0121] The display control unit (not shown) or the like controls the display of the image on the display 7 (Step 112). Specifically, The display control unit (not shown) or the like controls the display of the image in which the view cone 10 has been superimposed, the virtual view 9, the distance error gauge 34 and the rotation error gauge 35, the shooting score, the warning information, various GUIs, and the like, which have been generated by the visualized information generation unit 25.

[0122] Sound Generation The sound generation unit 26 generates a sound (Step 113). In an embodiment, the sound generation unit 26 generates a sound of the contents according to a difference between the target position and the device position and a difference between the target orientation and the device orientation. To be specific, first of all, the sound generation unit 26 acquires current-viewpoint information from the viewpoint information generation unit 24.

[0123] For example, the sound generation unit 26 generates a sound so that the sound is output when the distance error gauge 34 or the rotation error gauge 35 exceeds (becomes beyond) the target line 36. In this case, for example, the sound generation unit 26 stores values of the position error and the orientation error in the previous frame, and generates a sound, e.g., "BEEP" only in a case where the condition, "the error in the previous frame has not exceeded the target line 36, but the error in the current frame has exceeded the target line 36," is satisfied. Conversely, a sound, e.g., "DING-DONG" in a case where the error has dropped below the target line 36 may be generated. Alternatively, a sound may be generated to continuously increase in volume as the error becomes smaller.

[0124] Moreover, the sound generation unit 26 also generates a sound as warning information. For example in a case where a moving object has been detected, the sound generation unit 26 generates a voice, such as "Moving object has been detected." Alternatively, the sound generation unit 26 may generate a simple warning sound. In addition, specific contents of the sound are not limited, and a sound of any contents may be generated.

[0125] A sound control unit or the like (not shown) controls the output of the sound through the loudspeaker 8 (Step 114). Accordingly, the sound generated by the sound generation unit 26 is output through the loudspeaker 8.

[0126] Although the contents or the like of the display in the subjective view have been described above, the examples described so far relate to the contents of the display in a state in which the shutter has not been released (photo shooting has not been performed). That is, in the state in which the user just holds the image-shooting device 4 without releasing the shutter, for example, the virtual view 9 and the view cone 10 are displayed as a matter of course, and the gauges and the scores are also displayed. Moreover, the display or the like of the warning information is also performed. That is, also in the state in which the user just holds the image-shooting device 4 without releasing the shutter, warning information is displayed in real time when a moving object or the like enters the angle of view.

[0127] Next, the contents or the like of the wide-view display will be described. In a case where it is determined in Step 110 that the current view is a wide view, the visualized information generation unit 25 generates visualized information (Step 115). First of all, the visualized information generation unit 25 acquires viewpoint information at the time of shooting from the shooting viewpoint DB 29.

[0128] Figs. 10A and 10B are schematic views each showing an example of the wide-view display. In the example of Fig. 10A, an image in which a bird's-eye view of the automobile 2 is displayed on the display 7 and view cones 41 (41a to 41i) are also displayed at positions in the image, which correspond to respective target positions.

[0129] In an embodiment, the visualized information generation unit 25 outputs the view cones 41 of types according to the scores of the viewpoints, which are displayed at positions corresponding to the target positions. Then, in this example, the types according to the scores of the viewpoints are colors according to the scores of the viewpoints.

[0130] In this example, shooting has been performed in vicinity of all target positions at least once and a viewpoint score of shooting at a certain target position is expressed by a color of the view cone 41 at the target position. Specifically, the color of the view cone 41 is in black in a case where the viewpoint score exceeds 90 points, the color of the view cone 41 is in dark grey in a case where the viewpoint score is 70 points or more and 90 points or less, the color of the view cone 41 is in light grey in a case where the viewpoint score is below 70 points. That is, in this example, shooting with a score above 90 points has been performed at each of the target positions corresponding to the view cones 41a to 41g. Moreover, shooting with a score below 70 points has been performed with respect to the view cone 41h and shooting with a score of 70 points or more and 90 points or less has been performed with respect to the view cone 41i.

[0131] Although three types of view cones 41 are displayed in accordance with the viewpoint scores in this example, more types of view cones 41 may be displayed by dividing the scores more finely. Moreover, the view cone 41 associated with a case where shooting has not been performed yet may be displayed in white. Moreover, the types according to the scores of the viewpoints are not limited to the colors of the view cones 41, and for example, the scores may be expressed by patterns, sizes, shapes, or the like of the view cones 41. Alternatively, not only the viewpoint scores, but also the position scores and the orientation scores may be expressed. Moreover, the target icons 11 and the device icons 12 in the virtual views 9 may be colored or the view cone 10 colored may be displayed in the subjective view. The view cone 41 corresponds to an embodiment of an icon of a type according to the score according to the present technology.

[0132] In Fig. 10A, the user 1 has pressed one view cone 41e of the view cones 41a to 41i and the one view cone 41e is in a selected state. In this case, a rectangular frame is displayed around the view cone 41e to indicate that the view cone 41e has been selected. In addition, the distance error gauge 34 and the rotation error gauge 35 in the selected view cone 41e are displayed. The viewpoint score in the view cone 41e is relatively high, i.e., 90 points or more, and therefore the distance error gauge 34 and the rotation error gauge 35 are below the target line, indicating that the position error and the orientation error are small values.

[0133] In the example of Fig. 10B, the view cone 41h has been selected. The viewpoint score in the view cone 41h is relatively low, i.e., 70 points, and therefore the distance error gauge 34 and the rotation error gauge 35 have largely exceeded the target line, indicating that the position error and the orientation error are large values.

[0134] Furthermore, in this example, a retake button is displayed at the upper right of the display 7. The retake button is a button for retaking an image at a shooting position corresponding to a selected view cone 41. For example in a case where the viewpoint score is below the predetermined threshold, the visualized information generation unit 25 generates a retake button. Accordingly, the user 1 can smoothly retake an image at a target position with a low viewpoint score. As a matter of course, the retake button may be constantly generated irrespective of the value of the viewpoint score.

[0135] Fig. 11 is a schematic view showing an example of the wide-view display. In an embodiment, the visualized information generation unit 25 generates predetermined types of view cones 41 displayed at positions corresponding to the target positions as the warning information. In this example, the predetermined types are colors of the view cones 41.

[0136] Specifically, inclusion of the moving object has been generated in shooting in the view cone 41g and the view cone 41g is displayed in a predetermined color (although it is shown with a dot pattern, it can be actually violet). Here, the view cone 41g has been selected by the user 1.

[0137] The predetermined type is not limited to the color of the view cone 41, and for example, the predetermined problem may be expressed by the pattern, the size, shape, or the like of the view cone 41. Alternatively, the view cone 41 of the type that differs in accordance with the type of the predetermined problem may be displayed.

[0138] Moreover, as in the example of Fig. 10B, a retake button 42 may be displayed at the upper right. Additionally, a warning window 44 for informing of a problem in shooting may be displayed at the lower right or the like. The warning window 44 displays, for example, the taken image in which the problem has been generated and the contents of the problem. As a matter of course, even in a case where no problem has been generated, the warning window 44 may be displayed as a screen just for checking the taken image.

[0139] The visualized information generation unit 25 generates images of various GUIs such as the view cone 41, the virtual view 9, the distance error gauge 34, the rotation error gauge 35, the target line 36, the warning window 44, and the retake button.

[0140] The display control unit (not shown) or the like controls the display of the image on the display 7 (Step 116). Specifically, the display control unit (not shown) or the like controls the display of the wide-view image, the view cone 41 generated by the visualized information generation unit 25, and the like.

[0141] Here, whether or not the user 1 has pressed the retake button 42 is determined (Step 117). In a case where the retake button 42 has been pressed (Yes in Step 117), the user 1 is encouraged to retake an image for example by the view returned to the subjective view (Step 118).

[0142] After the processing in Step 114, in a case where it is determined in Step 117 that the retake button 42 has not been pressed (No in Step 117), or after the processing in Step 118, whether or not shooting has been completed at all viewpoints, i.e., all target positions is determined. For example in a case where viewpoint scores at all target positions are equal to or larger than a predetermined reference value, it is determined that shooting has been completed at all target positions. Alternatively, in a case where the user 1 has pressed a shooting completion button or the like, it may be determined that shooting has been completed at all target positions. In addition, the reference for determination in Step 119 is not limited thereto.

[0143] In a case where there is a target position at which shooting has not been performed (No in Step 119), the series of processing is performed from Step 101 again. In a case where shooting has been completed at all target positions (Yes in Step 119), the processing is terminated.

[0144] The specific processing order of the steps in this flowchart is not limited, and any order may be employed as long as the present technology can be realized. For example, a processing order in which the moving object detection processing is first performed in the processing of Step 104 to 106 may be employed.

[0145] Guide Information As described above, in an embodiment, the shooting score calculation unit 20, the viewpoint information generation unit 24, the visualized information generation unit 25, the sound generation unit 26, and the viewpoint information recording unit 27 outputs guide information for guiding the shooting position and the shooting orientation to the target position and the target orientation on the basis of the acquired target position, target orientation, shooting position, and shooting orientation, the guide information including error information regarding errors between the target position and the device position and between the target orientation and the device orientation. Moreover, the guide information is information for guiding the shooting position and the shooting orientation to the target position and the target orientation to reduce the errors between the target position and the device position and between the target orientation and the device orientation.

[0146] The target icon 11 and the device icon 12 in the virtual view 9 correspond to an embodiment of guide information according to the present technology. The user 1 adjusts the user's shooting position and orientation so that the position and orientation of the device icon 12 become identical to the position and orientation of the target icon 11 while seeing the target icon 11 and the device icon 12. Accordingly, the shooting position and orientation are guided to the target position and orientation to reduce the errors.

[0147] Moreover, the distance error gauge 34 and the rotation error gauge 35 also correspond to an embodiment of the guide information according to the present technology. The user 1 adjusts the user's shooting position and orientation to reduce the values of the errors indicated by the distance error gauge 34 and the rotation error gauge 35 as much as possible. Accordingly, the shooting position and orientation are guided to the target position and orientation to reduce the errors.

[0148] Moreover, the viewpoint score and the total score also correspond to an embodiment of the guide information according to the present technology. The user 1 adjusts the user's shooting position and orientation to increase the values of the viewpoint score and the total score as much as possible. Accordingly, the shooting position and orientation are guided to the target position and orientation to reduce the errors.

[0149] Moreover, the sound for informing of the error also corresponds to an embodiment of the guide information according to the present technology. For example in a case where the sound is generated to continuously increase in volume as the error becomes smaller, the user 1 adjusts the user's shooting position and orientation to increase the volume as much as possible while listening to the sound. Accordingly, the shooting position and orientation are guided to the target position and orientation to reduce the errors.

[0150] In this manner, in a case where the user 1 is motivated to move the shooting position and orientation closer to the target position and orientation by referring to certain information, such information is the guide information. In addition, the present technology is not limited to the above-mentioned contents, and guide information may be output in any form.

[0151] Such guide information includes the error information regarding the errors between the target position and the device position and between the target orientation and the device orientation. For example, the target icon 11 and the device icon 12 represent an error between the target position and the shooting position by a position difference between them and represent an error between the target orientation and the shooting orientation by an orientation difference between them, and therefore the target icon 11 and the device icon 12 include the error information.

[0152] Moreover, the distance error gauge 34 and the rotation error gauge 35 express the position error and the orientation error by the widths of the gauge parts as they are, and therefore the distance error gauge 34 and the rotation error gauge 35 also include the error information. Moreover, the viewpoint score and the total score are calculated on the basis of the position error and the orientation error, and therefore the viewpoint score and the total score also include the error information. Moreover, the sound generated by the sound generation unit 26 changes in volume in accordance with the error, and therefore the sound generated by the sound generation unit 26 also includes the error information.

[0153] It can be thus said that the information that changes in contents in accordance with the position error and the orientation error is the information including the error information. In addition, the present technology is not limited to the above-mentioned contents, and the guide information may be information including error information with any contents.

[0154] Although four types of the guide information of the target icon 11 and the device icon 12, the gauges, the scores, and the sound for informing of the error are all output in an embodiment, an embodiment in which only some of them are output is also included in the scope of the present technology. For example, an embodiment in which only the target icon 11 and the device icon 12 are output and none of the gauges, the scores, or the sound is not output may be employed. Moreover, any combination of the guide information, e.g., a combination of only the gauges and the scores, may be output.

[0155] Moreover, the superimposition of the view cone 10 and the output of the warning information are also arbitrary, and any mode, e.g., "one or more types of guide information only," "one or more types of guide information + view cone 10," "one or more types of guide information + warning information," "one or more types of guide information + view cone 10 + warning information," may be employed.

[0156] As described above, in the photogrammetry system 100 according to an embodiment, the plan information including the target shooting position and orientation and the device information including the device position and orientation are acquired and the guide information for guiding the device position and orientation to the target shooting position and orientation is output on the basis of the plan information and the device information. Accordingly, it is possible to accurately take an image to be used for generation of the 3D model. As a result, a high-quality 3D model can be generated.

[0157] In recent years, a photogrammetry technology has been increasingly used as one of realization methods for 3D reconstruction. However, advanced shooting knowledge is required and numerous images need to be taken for generating an accurate 3D model in the photogrammetry, so it takes costs and time. Although there is an automated shooting technology for the photogrammetry using an autonomous movable object in recent years, a navigation technology to assist a person to take an image is still developing.

[0158] The photogrammetry technology is still in its infancy. Although the photogrammetry technology has attracted attention in recent years, it is still a developing technology. In recent years, with the emergence of technologies such as deep learning, the photogrammetry technology has made significant progress, and an environment where the photogrammetry technology can be tested on smartphones and other devices has been created. However, it is believed that not much consideration has been done in terms of the shooting assistance.

[0159] In recent years, a viewpoint planning technology for the photogrammetry has begun to be used for navigation when a person takes an image. However, a problem arises as to how accurately a person can move the device closer to a shooting-planned viewpoint. The present technology is an idea to solve problems that arise when the viewpoint planning technology is used for human photographic navigation, and the device position and orientation are guided to the target position and orientation on the basis of the guide information. This makes it possible to shoot at a planned viewpoint (target position) more accurately than with existing techniques.

[0160] Moreover, in the present technology, the guide information is the information for guiding the device position and orientation to the target position and orientation to reduce the errors. Accordingly, the user 1 is motivated to reduce the errors, and therefore the user 1 can more accurately adjust the position and the orientation.

[0161] Moreover, in the present technology, as the guide information, the target icon 11 is displayed in the orientation corresponding to the target orientation at the position corresponding to the target position when viewed from the one or more predetermined directions and the device icon 12 is displayed in the orientation corresponding to the device orientation at the position corresponding to the device position. Accordingly, the user 1 can recognize the relationship between the target position and orientation and the device position and orientation when viewed from a predetermined direction, and therefore the user 1 can perform more accurate adjustment.

[0162] Moreover, in the present technology, the target icon 11 and the device icon 12 when viewed from a plurality of directions are displayed. Accordingly, for example, fine position adjustment and orientation adjustment can be performed not only in a single direction, but also in other directions, and the user 1 can perform more accurate adjustment.

[0163] Moreover, in the present technology, the target icons 11 and the device icons 12 when viewed in the vertical direction and the direction (horizontal direction) orthogonal to the vertical direction as a plurality of directions are displayed. Since a person can easily recognize the vertical direction and the horizontal direction, this enables the user 1 to perform much more accurate adjustment.

[0164] Moreover, in the present technology, the images of the distance error gauge 34 and the rotation error gauge 35 are output as the guide information. Accordingly, the user 1 can intuitively visually recognize the errors. Moreover, in the present technology, the sound is output as the guide information. Accordingly, the user 1 can intuitively auditorily recognize the errors.

[0165] Moreover, in the present technology, the score is output as the guide information. Accordingly, the user 1 can perform more accurate adjustment while seeing the score. Moreover, the user 1 can roughly estimate a final 3D model before the 3D model is generated by referring to the total score.

[0166] In a case where a setting algorithm for the target position and orientation has a problem, an inaccurate 3D model is sometimes generated even though the position adjustment and the orientation adjustment are accurately performed. That is, in a case where an inaccurate 3D model has been generated despite a high score, the user 1 can consider that his or her adjustment of the position and orientation has been successfully done, but there has been a problem with the settings of the target position and orientation, and can identify the problem.

[0167] Moreover, in the present technology, the score is calculated to become larger in value as the difference between the target position and the device position and the difference between the target orientation and the device orientation become smaller. Accordingly, the user 1 can more accurately recognize the errors.

[0168] Moreover, in the present technology, the view cones 41 of the types according to the scores, which are displayed at positions corresponding to the target positions, are output. Accordingly, the user 1 can easily grasp a score at each shooting position. Moreover, in the present technology, the view cones 41 in the colors according to the scores are output as the types according to the scores. Accordingly, the user 1 can more intuitively grasp a score at each shooting position.

[0169] Moreover, in the present technology, the predetermined problem in the taken image is detected and the warning information according to the contents of the predetermined problem is output. Accordingly, the user 1 is informed of the problem, and the user 1 can suitably determine whether or not to retake an image, for example. Moreover, in the present technology, the warning information according to the contents including at least one of the inclusion of the backlight, the generation of the blur, or the inclusion of the moving object is output as the predetermined problem. Accordingly, the user 1 can easily grasp those relatively important problems.

[0170] Moreover, in the present technology, the icons of the predetermined types displayed at positions corresponding to the target positions are output as the warning information. Accordingly, the user 1 can easily determine at which target position a problem has been generated in the taken image. Moreover, in the present technology, the icons in the predetermined colors are output as the predetermined types. Accordingly, the user 1 can more intuitively grasp a target position at which a problem has been generated.

[0171] Moreover, in the present technology, the view cone 10 displayed, superimposed in the orientation corresponding to the target orientation at the position corresponding to the target position on the image-shooting screen, is output. For example in a case where only the view cone 10 is displayed and the target icon 11 and the device icon 12 are not displayed, the display region of the view cone 10 becomes larger as the user 1 approaches the target position. Then, in a case where the display region of the view cone 10 is displayed in the entire display 7, it becomes difficult for the user 1 to perform the position adjustment and the orientation adjustment by referring to the view cone 10. On the other hand, in an embodiment, the guide information such as the target icon 11 and the device icon 12 are displayed in addition to the view cone 10, and therefore the user can use an adjustment method of, for example, performing rough adjustment while viewing the view cone 10, for example, at a position far from the target position, and performing fine adjustment while viewing the target icon 11 and the device icon 12 once the user approaches the target position.

[0172] Other Embodiments The present technology is not limited to the above-mentioned embodiments, and various other embodiments can be realized. Single-Lens Reflex Camera Figs. 12A and 12B are schematic views each showing a variation of the image-shooting device 4. In the examples of Figs. 2A and 2B or the like, the image-shooting device 4 is a smartphone, the camera 15 built in the smartphone has executed shooting, and the display 7 has performed the display. The present technology is not limited thereto, and for example, a combination of a smartphone 45 and a single-lens reflex camera 46 may be used as the image-shooting device 4.

[0173] As shown in Fig. 12A, the smartphone 45 and the single-lens reflex camera 46 are connected so that an orientation of a front side (side of a display 47) of the smartphone 45 is identical to an orientation of a rear side of the single-lens reflex camera 46 (side on which a lens 48 is not arranged). These are connected and fixed via a jig 49, for example. Fig. 12A schematically shows the jig 49 as a line. Similarly, Fig. 12B shows the rear side of the smartphone 45 and the front side of the single-lens reflex camera 46.

[0174] In the image-shooting device 4 in this example, image shooting is performed through the lens 48 of the single-lens reflex camera 46 while the display of the guide information or the like is performed by the display 47 of the smartphone 45. That is, the device that performs shooting is separate from the device that performs the display.

[0175] Fig. 13 is a schematic diagram showing a configuration example of the image-shooting device 4. Fig. 13 shows a configuration example of the image-shooting device 4 in an embodiment of Figs. 12A and 12B. The image-shooting device 4 includes a camera 51 such as the single-lens reflex camera 46 and the smartphone 45.

[0176] The camera 51 includes an image capturing unit 52 and a sending unit 53. The image capturing unit 52 is a mechanism for image capturing, which is constituted by the lens 48 and the like, and captures an image of the target object. The sending unit 53 acquires a captured image from the image capturing unit 52 and also receives the input of a shooting trigger from the user 1. Then, the sending unit 53 sends the image to the smartphone 45 via a communication unit or the like (not shown). Furthermore, in a case where the shooting trigger has been input, the sending unit 53 also sends the shooting trigger to the smartphone 45.

[0177] The smartphone 45 acquires the image and the shooting trigger via a communication unit or the like (not shown). The subsequent processing is similar to that in an embodiment shown in Fig. 3. It should be noted that in the self-position estimation, it is necessary to estimate the position and orientation of the apparatus itself that performs shooting. Therefore, in the self-position estimation unit 18, the shooting position and orientation (device position and orientation) of the camera 51, not the smartphone 45, are calculated. For example, first of all, the shooting position and orientation of the smartphone 45 are calculated, and the calculated values are corrected on the basis of predefined offsets (differences in position and orientation between the smartphone 45 and the camera 51). It should be noted that in a case where the self-position estimation is performed on the basis of the image taken by the camera 51, such correction processing may be unnecessary.

[0178] In an embodiment, a device specialized for shooting, such as the single-lens reflex camera 46, performs shooting, and therefore an image with higher quality can be acquired as compared to a case where the camera built in the smartphone 45 performs shooting. In addition, any other type of camera 51 may be used. Moreover, the device for display is also not limited to the smartphone 45, and any device may be employed.

[0179] Cooperative Shooting Fig. 14 is a schematic view showing a variation example of an image shooting method. In this example, both a person 57 and a drone 58 cooperatively shot a target object such as a rocky place 56. The drone 58 flies in accordance with an operation made by the person or automatically. A camera 59 is connected to the drone 58, and therefore the drone 58 is capable of shooting the rocky place 56 while flying.

[0180] First of all, the person 57 performs shooting at all shooting positions and viewpoint scores at the respective shooting positions are calculated. Next, the drone 58 performs shooting at all shooting positions and viewpoint scores at the respective shooting positions are calculated. Then, the viewpoint score of the person 57 and the viewpoint score of the drone 58 at each shooting position are compared with each other. The score comparison may be performed by, for example, a controller of an image-shooting device on the side of the person 57 or may be performed on the side of the drone 58. Alternatively, the viewpoint scores may be sent to a cloud from both the side of the person 57 and the side of the drone 58 and comparison processing may be performed on the cloud.

[0181] In a case where a viewpoint score of the person 57 at a certain shooting position is higher, the taken image on the side of the person 57 is employed as the taken image at such a shooting position. On the other hand, in a case where the score of the drone 58 is higher, the taken image on the side of the drone 58 is employed.

[0182] For example, it is often difficult for the person 57 to perform accurate position adjustment and orientation adjustment in a place where it is difficult to stably stand, such as the rocky place 56. On the other hand, the drone 58 is capable of flying, and therefore the drone 58 can relatively easily perform position adjustment and the like also in such a place. Therefore, the score of the drone 58 tends to be higher on the side of the person 57.

[0183] In addition, the score of the drone 58 tends to be higher also in a dangerous zone or an elevated place. On the other hand, a narrow path, a dark place, and the like are places where the drone 58 does not work well, and the score of the person 57 tends to be higher in such a place. In an embodiment, a taken image with a higher viewpoint score at each target position is employed, and therefore the person 57 and the drone 58 can compensate for the shooting positions at which they are not good. Furthermore, a high-quality image can be obtained.

[0184] It should be noted that although the person 57 and the drone 58 perform shooting at all target positions and then their viewpoint scores are compared with each other in this example, for example in a case where the viewpoint score of the person 57 will be obviously lower at a certain target position, the person 57 does not need to take an image and an image taken by the drone 58 may be employed as it is. Conversely, there may a target position where only the person 57 takes an image and the drone 58 takes no images. Accordingly, the person 57 or the drone 58 does not need to go to dangerous places, and it is possible to lower the risk of injury or malfunction.

[0185] Moreover, the present technology is not limited to the drone 58, and shooting may be performed by an autonomous walking robot or the like. Alternatively, three or more persons 57 or machines may perform shooting. In addition, a specific carrying out format in this example is not limited.

[0186] Angle of View The bottom surface of the view cone 10 in each of Figs. 2A and 2B or the like may correspond to the angle of view of the camera 15. That is, in a case where it is necessary to perform shooting in an up state (in a wide angle of view) at the target position, a view cone 10 with a larger rectangle as its bottom surface is superimposed. Conversely, in a case where it is necessary to perform shooting in a zoomed state (in a narrow angle of view), a view cone 10 with a smaller rectangle as its bottom surface is superimposed.

[0187] Similarly, the angle of view may be expressed by the bottom sides of the target icon 11 and the device icon 12. That is, in a case where it is necessary to perform shooting in a wide angle of view, those icons are triangles with a longer bottom side and a larger top angle. Moreover, in a case where it is necessary to perform shooting in a narrow angle of view, those icons are tapered triangles with a shorter bottom side and a smaller top angle.

[0188] The user 1 adjusts the degrees of close-up and zoom of the camera 15 so that the length of the bottom side of the device icon 12 becomes identical to the length of the bottom side of the target icon 11. Alternatively, warning information, e.g., "Too closed up" and "Too zoomed-up," may be output. With them, an image with a much higher quality can be taken.

[0189] Shooting of Moving Image In the example of each of Figs. 2A and 2B for example, the user 1 takes a still image of the target object by pressing the shutter button, for example. The present technology is not limited thereto, and for example, a moving image may continue to be constantly shot and a still image may be automatically extracted from the moving image in a frame where the position and orientation of the image-shooting device 4 match the target position and orientation. That is, it may be possible to perform a series of shooting by work, e.g., adjusting the position and orientation at each target position while taking the moving image without the need for pressing the shutter button or the like.

[0190] Some or all of the functions of the image-shooting device 4 shown in Fig. 3 may be mounted on another information processing apparatus. For example, the other information processing apparatus may be provided with the shooting error calculation unit 19 and the like shown in Fig. 3. That is, the image-shooting device 4 may be realized by a plurality of computers or may be realized by a single computer.

[0191] Fig. 15 is a block diagram showing a hardware configuration example of a computer 500 capable of realizing the image-shooting device 4. The computer 500 includes a CPU 501, a ROM 502, a RAM 503, an input / output interface 505, and a bus 504 that connects them to one another. A display unit 506, an input unit 507, a storage unit 508, a communication unit 509, a drive unit 510, and the like are connected to the input / output interface 505.

[0192] The display unit 506 is for example a display device using liquid crystals, EL, or the like. The input unit 507 is for example a keyboard, a pointing device, a touch panel, or another operation device. In a case where the input unit 507 includes a touch panel, the touch panel can be integral with the display unit 506. The storage unit 508 is a nonvolatile storage device. The storage unit 508 is for example an HDD, a flash memory, or another solid-state memory. The drive unit 510 is for example a device capable of driving a removable recording medium 511 such as an optical recording medium or a magnetic record tape. The communication unit 509 is a modem, a router, or another communication device for communicating with other devices, which is connectable to a LAN, a WAN, or the like. The communication unit 509 may perform wired communication or may perform wireless communication. The communication unit 509 is often used separately from the computer 500.

[0193] Cooperation of software stored in the storage unit 508, the ROM 502, or the like with hardware resources of the computer 500 achieves information processing of the computer 500 having the hardware configurations as described above. Specifically, loading a program that configures the software, which has been stored in the ROM 502 or the like, to the RAM 503 and executing the program achieves the information processing method according to an embodiment of the present technology.

[0194] The computer 500 installs the program via the recording medium 511, for example. Alternatively, the computer 500 may install the program via a global network or the like. Otherwise, any computer-readable non-transitory storage medium may be used.

[0195] It should be noted that in the present disclosure, the system means a set of a plurality of components (e.g., apparatuses, modules (parts)) and it does not matter whether or not all the components are housed in the same casing. Therefore, both of a plurality of apparatuses housed in separate casings and connected to one another via a network and a single apparatus having a plurality of modules housed in a single casing are the system.

[0196] Executing the information processing method according to an embodiment of the present technology by the computer system includes, for example, both of a case where a single computer executes self-position estimation, calculation of the position error and orientation error, score calculation, detection of the moving object and the like, output of the guide information and the warning information, generation of the visualized information, and the like, and a case where different computers execute the respective processes. Moreover, executing the respective processes by a predetermined computer includes causing another computer to execute some or all of those processes and acquiring the results. That is, the information processing method according to an embodiment of the present technology can also be applied to a cloud computing configuration in which a plurality of apparatuses shares and cooperatively processes a single function via a network.

[0197] The respective configurations such as the photogrammetry system 100, the image-shooting device 4, and various GUIs, the respective processing flows, and the like, which have been described with reference to the respective drawings, are merely embodiments, and can be arbitrarily modified without departing from the gist of the present technology. That is, any other configurations, algorithms, and the like for carrying out the present technology may be employed.

[0198] In the present disclosure, when the term "substantially" is used, it is the use for easy understanding of the description, and the use and non-use of the term "substantially" do not have any special meaning. That is, in the present disclosure, it is assumed that the concepts that define the shape, the size, the position relationship, the state, and the like such as "center," "middle," "uniform," "equal," "the same," "orthogonal," "parallel," "symmetric," "extending," "axial," "circular," and "square pyramid shape" are concepts including "substantially center," "substantially middle," "substantially uniform," "substantially equal," "substantially the same," "substantially orthogonal," "substantially parallel," "substantially symmetric," "substantially extending," "substantially axial," "substantially circular," and "substantially square pyramid shape," and the like. For example, states included in a predetermined range (e.g.,+ / -10% range) using "completely center," "completely middle," "completely uniform," "completely equal," "completely the same," "completely orthogonal," "completely parallel," "completely symmetric," "completely extending," "completely axial," "completely circular," "completely square pyramid shape," and the like as the basis are also included. Thus, also when the term "substantially" is not added, concepts expressed with so-called "substantially" can be included. On the contrary, a state express with "substantially" does not exclude a complete state.

[0199] In the present disclosure, the expressions with "than," e.g., "it is larger than A" and "it is smaller than A," are expressions comprehensively including both a concept including a case where it is equal to A and a concept not including a case where it is equal to A. For example, "it is larger than A" is not limited to a case not including it is equal to A, and also includes "equal to and larger than A." Moreover, "it is smaller than A" is not limited to" it is below A," and it also includes "it is equal to or smaller than A." When carrying out the present technology, specific settings and the like may be employed as appropriate from the concepts included in "it is larger than A" and "it is smaller than A" so that the above-mentioned effects can be provided.

[0200] At least two features of the features according to the present technology, which have been described above, may be combined. That is, the various features described in the respective embodiments may be arbitrarily combined across the respective embodiments. Moreover, the above-mentioned various effects are merely exemplary and not limitative, and other effects may be provided.

[0201] It should be noted that the present technology can also take the following configurations. (1) An information processing apparatus including: circuitry configured to acquire one or more input images, acquire information related to each acquired input image of the one or more input images based on position information indicating a shooting position of each acquired input image, and orientation information indicating a shooting orientation of each acquired input image, and output at least one output notification indicating the acquired information related to at least one acquired input image of the one or more input images. (2) The information processing apparatus according to (1), wherein the acquired information related to each acquired input image includes a shooting error of each acquired input image. (3) The information processing apparatus according to (1) or (2), wherein the shooting error is determined based on a position error of the position information and an orientation error of the orientation information of each acquired input image. (4) The information processing apparatus according to any of (1) to (3), wherein the position error is determined based on a difference between a predetermined target position and the shooting position of each acquired input image indicated by the position information. (5) The information processing apparatus according to any of (1) to (4), wherein the orientation error is determined based on a difference between a predetermined target orientation and the shooting orientation of each acquired input image indicated by the orientation information. (6) The information processing apparatus according to any of (1) to (5), wherein the at least one output notification includes at least one of an output sound or an output image generated for output according to a degree of the shooting error of the at least one acquired input image. (7) The information processing apparatus according to any of (1) to (6), wherein the at least one output notification including the at least one of the output sound or the output image is generated for output according to a degree of the shooting error of a current viewpoint of a device used to acquire the one or more input images. (8) The information processing apparatus according to any of (1) to (7), wherein the at least one output notification includes an output image indicating at least one of the position error or the orientation error of the at least one acquired input image. (9) The information processing apparatus according to any of (1) to (8), wherein the output image indicates the at least one of the position error or the orientation error of the at least one acquired input image in relation to one or more score thresholds. (10) The information processing apparatus according to any of (1) to (9), wherein the at least one output notification includes an output image indicating the shooting position and the shooting orientation of the at least one acquired input image in spatial relation to a predetermined target position and a predetermined target orientation set for each acquired input image. (11) The information processing apparatus according to any of (1) to (10), wherein the output image indicates the shooting position and the shooting orientation of the at least one acquired input image in spatial relation to the predetermined target position and the predetermined target orientation set for each acquired input image when viewed from a plurality of directions. (12) The information processing apparatus according to any of (1) to (11), wherein the output image indicates the shooting position and the shooting orientation of the at least one acquired input image in spatial relation to the predetermined target position and the predetermined target orientation set for each acquired input image with respect to a target object in the one or more input images. (13) The information processing apparatus according to any of (1) to (12), wherein the predetermined target position and the predetermined target orientation are indicated in the output image for each acquired input image as a view cone set with respect to the target object. (14) The information processing apparatus according to any of (1) to (13), wherein the view cone is superimposed on the at least one acquired input image. (15) The information processing apparatus according to any of (1) to (14), wherein the at least one output notification includes an output image indicating an overall shooting score determined based on the shooting error of the one or more input images. (16) The information processing apparatus according to any of (1) to (15), wherein the circuitry is further configured to acquire a plurality of input images, and wherein the at least one output notification includes an output image related to the acquired information for each acquired input image of the plurality of input images as a virtual view reflecting the position information and the orientation information of the plurality of input images. (17) The information processing apparatus according to any of (1) to (16), wherein the position information and the orientation information are generated using self-position estimation of a device used to acquire the one or more input images. (18) The information processing apparatus according to any of (1) to (17), wherein the circuitry is further configured to detect at least one of a backlight, a blur, or a moving object in the at least one acquired input image. (19) An information processing method including: acquiring one or more input images; acquiring information related to each acquired input image of the one or more input images based on position information indicating a shooting position of each acquired input image, and orientation information indicating a shooting orientation of each acquired input image; and outputting at least one output notification indicating the acquired information related to at least one acquired input image of the one or more input images. (20) A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer causes the computer to execute an information processing method, the method including: acquiring one or more input images; acquiring information related to each acquired input image of the one or more input images based on position information indicating a shooting position of each acquired input image, and orientation information indicating a shooting orientation of each acquired input image; and outputting at least one output notification indicating the acquired information related to at least one acquired input image of the one or more input images. (21) An information processing apparatus, including: a first acquisition unit that acquires plan information for taking a plurality of taken images with respect to a target object, the plurality of taken images being to be used for generation of a 3D model of the target object in a real space, the plan information including a target shooting position and a target shooting orientation; a second acquisition unit that acquires device information of an image-shooting device that shoots the target object, the device information including a device position and a device orientation; and an output unit that outputs, on the basis of the acquired plan information and the acquired device information, guide information for guiding the device position and the device orientation to the target position and the target orientation, the guide information including error information regarding errors between the target position and the device position and between the target orientation and the device orientation. (22) The information processing apparatus according to (21), in which the guide information is information for guiding the device position and the device orientation to the target position and the target orientation so that the errors are reduced. (23) The information processing apparatus according to (21) or (22), in which the guide information is icons that are displayed in orientations corresponding to the target orientation and the device orientation at positions corresponding to the target position and the device position when viewed from the one or more predetermined directions. (24) The information processing apparatus according to (23), in which the one or more predetermined directions is a plurality of directions. (25) The information processing apparatus according to (24), in which the plurality of directions is a vertical direction and a direction orthogonal to the vertical direction. (26) The information processing apparatus according to any one of (21) to (25), in which the guide information is an image of a gauge according to a difference between the target position and the device position and a difference between the target orientation and the device orientation. (27) The information processing apparatus according to any one of (21) to (26), in which the guide information is a sound of contents according to a difference between the target position and the device position and a difference between the target orientation and the device orientation. (28) The information processing apparatus according to any one of (21) to (27), in which the guide information is a score calculated on the basis of a difference between the target position and the device position and a difference between the target orientation and the device orientation. (29) The information processing apparatus according to (28), in which the score is calculated to become larger in value as the difference between the target position and the device position and the difference between the target orientation and the device orientation become smaller. (30) The information processing apparatus according to (28) or (29), in which the output unit outputs an icon of a type according to the score, the icon being displayed at a position corresponding to the target position. (31) The information processing apparatus according to (30), in which the type according to the score is a color according to the score. (32) The information processing apparatus according to any one of (21) to (31), further including a detection unit that detects a predetermined problem in the taken image, in which the output unit outputs, in a case where the detection unit has detected the predetermined problem, warning information according to contents of the predetermined problem. (33) The information processing apparatus according to (32), in which the predetermined problem includes at least one of inclusion of backlight, generation of a blur, or inclusion of a moving object. (34) The information processing apparatus according to (32) or (33), in which the warning information is a predetermined type of icon that is displayed at a position corresponding to the target position. (35) The information processing apparatus according to (34), in which the predetermined type is a predetermined color. (36) The information processing apparatus according to any one of (21) to (35), in which the output unit outputs an icon that is displayed, superimposed at a position corresponding to the target position on an image-shooting screen when the taken image is taken, the icon being displayed in an orientation corresponding to the target orientation. (37) An information processing method, including: acquiring plan information for taking a plurality of taken images with respect to a target object, the plurality of taken images being to be used for generation of a 3D model of the target object in a real space, the plan information including a target shooting position and a target shooting orientation; acquiring device information of an image-shooting device that shoots the target object, the device information including a device position and a device orientation; and outputting, on the basis of the acquired plan information and the acquired device information, guide information for guiding the device position and the device orientation to the target position and the target orientation, the guide information including error information regarding errors between the target position and the device position and between the target orientation and the device orientation. (38) A program that causes a computer system to execute the steps of: acquiring plan information for taking a plurality of taken images with respect to a target object, the plurality of taken images being to be used for generation of a 3D model of the target object in a real space, the plan information including a target shooting position and a target shooting orientation; acquiring device information of an image-shooting device that shoots the target object, the device information including a device position and a device orientation; and outputting, on the basis of the acquired plan information and the acquired device information, guide information for guiding the device position and the device orientation to the target position and the target orientation, the guide information including error information regarding errors between the target position and the device position and between the target orientation and the device orientation.

[0202] It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and alterations may occur depending on design requirements and other factors insofar as they are within the scope of the appended claims or the equivalents thereof.

[0203] 2 automobile 3 target position 4 image-shooting device 8 device position 9 virtual view 10, 41 view cone 11 target icon 12 device icon 15 camera 18 self-position estimation unit 19 shooting error calculation unit 20 shooting score calculation unit 21 backlight detection unit 22 blur detection unit 23 moving object detection unit 24 viewpoint information generation unit 25 visualized information generation unit 26 sound generation unit 27 viewpoint information recording unit 34 distance error gauge 35 rotation error gauge 100 photogrammetry system

Claims

1. An image processing apparatus comprising: circuitry configured to acquire one or more input images, acquire information related to each acquired input image of the one or more input images based on position information indicating a shooting position of each acquired input image, and orientation information indicating a shooting orientation of each acquired input image, and output at least one output notification indicating the acquired information related to at least one acquired input image of the one or more input images.

2. The image processing apparatus according to claim 1, wherein the acquired information related to each acquired input image includes a shooting error of each acquired input image.

3. The image processing apparatus according to claim 2, wherein the shooting error is determined based on a position error of the position information and an orientation error of the orientation information of each acquired input image.

4. The image processing apparatus according to claim 3, wherein the position error is determined based on a difference between a predetermined target position and the shooting position of each acquired input image indicated by the position information.

5. The image processing apparatus according to claim 3, wherein the orientation error is determined based on a difference between a predetermined target orientation and the shooting orientation of each acquired input image indicated by the orientation information.

6. The image processing apparatus according to claim 2, wherein the at least one output notification includes at least one of an output sound or an output image generated for output according to a degree of the shooting error of the at least one acquired input image.

7. The image processing apparatus according to claim 6, wherein the at least one output notification including the at least one of the output sound or the output image is generated for output according to a degree of the shooting error of a current viewpoint of a device used to acquire the one or more input images.

8. The image processing apparatus according to claim 3, wherein the at least one output notification includes an output image indicating at least one of the position error or the orientation error of the at least one acquired input image.

9. The image processing apparatus according to claim 8, wherein the output image indicates the at least one of the position error or the orientation error of the at least one acquired input image in relation to one or more score thresholds.

10. The image processing apparatus according to claim 1, wherein the at least one output notification includes an output image indicating the shooting position and the shooting orientation of the at least one acquired input image in spatial relation to a predetermined target position and a predetermined target orientation set for each acquired input image.

11. The image processing apparatus according to claim 10, wherein the output image indicates the shooting position and the shooting orientation of the at least one acquired input image in spatial relation to the predetermined target position and the predetermined target orientation set for each acquired input image when viewed from a plurality of directions.

12. The image processing apparatus according to claim 10, wherein the output image indicates the shooting position and the shooting orientation of the at least one acquired input image in spatial relation to the predetermined target position and the predetermined target orientation set for each acquired input image with respect to a target object in the one or more input images.

13. The image processing apparatus according to claim 12, wherein the predetermined target position and the predetermined target orientation are indicated in the output image for each acquired input image as a view cone set with respect to the target object.

14. The image processing apparatus according to claim 13, wherein the view cone is superimposed on the at least one acquired input image.

15. The image processing apparatus according to claim 2, wherein the at least one output notification includes an output image indicating an overall shooting score determined based on the shooting error of the one or more input images.

16. The image processing apparatus according to claim 1, wherein the circuitry is further configured to acquire a plurality of input images, and wherein the at least one output notification includes an output image related to the acquired information for each acquired input image of the plurality of input images as a virtual view reflecting the position information and the orientation information of the plurality of input images.

17. The image processing apparatus according to claim 1, wherein the position information and the orientation information are generated using self-position estimation of a device used to acquire the one or more input images.

18. The image processing apparatus according to claim 1, wherein the circuitry is further configured to detect at least one of a backlight, a blur, or a moving object in the at least one acquired input image.

19. An image processing method comprising: acquiring one or more input images; acquiring information related to each acquired input image of the one or more input images based on position information indicating a shooting position of each acquired input image, and orientation information indicating a shooting orientation of each acquired input image; and outputting at least one output notification indicating the acquired information related to at least one acquired input image of the one or more input images.

20. A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer causes the computer to execute an image processing method, the method comprising: acquiring one or more input images; acquiring information related to each acquired input image of the one or more input images based on position information indicating a shooting position of each acquired input image, and orientation information indicating a shooting orientation of each acquired input image; and outputting at least one output notification indicating the acquired information related to at least one acquired input image of the one or more input images.

Images