Information processing device, information processing method, information processing program, and recording medium

Abstract

Provided are an information processing device, an information processing method, and an information processing program which are capable of acquiring information for imaging a desired subject in a desired composition. The information processing device comprises a processor, wherein the processor: acquires information about an imaging device and information about a first subject and a second subject which are to be imaged; and performs control for estimating an imaging position where the first subject and the second subject can be imaged, on the basis of display states of a first image showing the first subject and a second image showing the second object in a display device and / or another display device.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. and recording media In particular, an information processing device, an information processing method, and an information processing program for processing information for capturing an image of a desired subject with a desired composition and recording media Regarding. [Background technology]

[0002] Patent Document 1 describes an imaging device that displays an icon resembling a subject on a display unit, sets a composition on the screen using the icon, calculates the distance at which the subject can be captured with the set composition, and presents the calculated distance to the user. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-017912 Summary of the Invention

[0004] One embodiment of the technique of the present disclosure provides an information processing device, an information processing method, and an information processing program capable of acquiring information for capturing an image of a desired subject with a desired composition. and recording media to provide. [Means for solving the problem]

[0005] (1) An information processing device that includes a processor, which acquires information about an imaging device and information about a first subject and a second subject to be imaged, and performs control to estimate an imaging position at which the first subject and the second subject can be imaged based on the display state of a first image representing the first subject and a second image representing the second subject on a display device and / or another display device.

[0006] (2) The information processing device of (1), wherein the processor controls displaying information about the estimated imaging position on the display device and / or another display device.

[0007] (3) An information processing device according to (1) or (2), wherein the processor receives input of the composition of the first image and the second image on the display device and / or another display device, and performs control to estimate the imaging position based on the received input composition.

[0008] (4) The information processing device of (3), wherein the composition is the position and size of the first image and the position and size of the second image on the display device and / or another display device.

[0009] (5) The information processing device according to any one of (1) to (4), wherein the information on the first subject and the second subject is information on the positions and sizes of the first subject and the second subject.

[0010] (6) The information processing device of (5), wherein the position information of the first subject and / or the position information of the second subject is information on the altitude and direction of the first subject and / or information on the altitude and direction of the second subject.

[0011] (7) An information processing device according to (5), wherein the processor receives input of information identifying a first subject and a second subject, and controls the acquisition of information on the position and size of the identified first subject and information on the position and size of the identified second subject from a database.

[0012] (8) The information processing device of (5), wherein the processor controls acquisition of information obtained by measuring the position and size of the first subject and the position and size of the second subject.

[0013] (9) An information processing device according to any one of (1) to (8), in which the processor acquires information on the current location, generates guidance information leading to the imaging position based on the acquired information on the current location, and controls the display of the generated guidance information on the display device and / or another display device.

[0014] (10) An information processing device according to any one of (1) to (9), wherein the processor calculates the time periods during which backlight and / or insufficient light will occur when imaging the first subject and the second subject at the imaging position, and controls the display device and / or another display device to display the calculated time periods or time periods excluding the calculated time periods.

[0015] (11) The information processing device of (3) or (4), wherein the processor, when unable to estimate the imaging position, performs control to correct the composition so that the imaging position can be estimated.

[0016] (12) An information processing device according to (3) or (4), wherein the processor controls the display device and / or another display device to display a frame having the same aspect ratio as an image captured by the imaging device.

[0017] (13) An information processing device according to (12), wherein, when the size of the first image and the second image in the composition for which input has been received is inappropriate and the imaging position cannot be estimated, the processor controls to change the size of the first image and the second image within the frame and / or the relative positions of the first and second images to correct the composition so that the imaging position can be estimated.

[0018] (14) An information processing device according to (12) or (13), wherein, when the imaging position cannot be estimated because the relative positions of the first and second images in the input composition are inappropriate, the processor controls the size of the first and second images within the frame and / or the relative positions of the first and second images to correct the composition so that the imaging position can be estimated.

[0019] (15) An information processing device according to any one of (12) to (14), wherein, when the distance between the first image and the second image in the composition for which input has been received is inappropriate and the imaging position cannot be estimated, the processor changes the distance between the first image and the second image within the frame to correct the composition so that the imaging position can be estimated.

[0020] (16) An information processing device according to any one of (12) to (15), wherein the processor detects obstacles when capturing an image of the first subject and the second subject at the estimated capturing position, and when an obstacle is detected, performs control to correct the composition so that the obstacle can be avoided and the first subject and the second subject can be captured.

[0021] (17) An information processing device according to any one of (1) to (16), wherein the processor determines whether the estimated imaging position is an accessible location and performs control to issue a warning if the location is an inaccessible location.

[0022] (18) Any one of the information processing devices (1) to (17), in which the imaging position includes altitude information.

[0023] (19) The information processing device according to any one of (1) to (18), wherein the processor determines whether the estimated imaging position is a reachable position and performs control to issue a warning if the position is an unreachable position.

[0024] (20) The information processing device of (12), wherein the processor further performs control to accept input of the orientation of the first image and the second image within the frame.

[0025] (21) An information processing method including the steps of acquiring information about an imaging device and information about a first subject and a second subject to be imaged, and estimating an imaging position at which the first subject and the second subject can be imaged based on the display state of a first image representing the first subject and a second image representing the second subject on a display device and / or another display device.

[0026] (22) An information processing program that causes a computer to realize a function of acquiring information about an imaging device and information about a first subject and a second subject to be imaged, and a function of estimating an imaging position at which the first subject and the second subject can be imaged based on the display state of a first image representing the first subject and a second image representing the second subject on a display device and / or another display device. [Brief explanation of the drawings]

[0027] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of an imaging position estimation system. [Figure 2] FIG. 1 is a block diagram showing an example of the hardware configuration of a user terminal; [Figure 3] Block diagram of the main functions realized by the user terminal [Figure 4] FIG. 10 is a diagram showing an example of a subject selection screen; [Figure 5] FIG. 10 is a diagram showing an example of a map displayed in a map display area. [Figure 6] FIG. 10 is a diagram showing an example of a screen display when the selection of the first subject is accepted. [Figure 7] FIG. 10 is a diagram showing an example of a screen display when the selection of a second subject is accepted. [Figure 8] Conceptual diagram of how to display icons in the composition setting area [Figure 9] A diagram showing an example of the display after dragging is completed [Figure 10] Conceptual diagram of composition setting operations [Figure 11] Conceptual diagram of composition setting operations [Figure 12] Conceptual diagram of composition setting operations [Figure 13] Conceptual diagram of composition setting operations [Figure 14] Conceptual diagram of composition setting operations [Figure 15] Conceptual diagram of composition setting operations [Figure 16] Conceptual diagram of imaging position estimation process [Figure 17] Flowchart showing the procedure for processing to estimate an imaging position [Figure 18] FIG. 10 is a diagram showing an example of a display of the result of the imaging position estimation process. [Figure 19] FIG. 10 is a diagram showing an example of a display of the result of the imaging position estimation process. [Figure 20] A block diagram showing an example of the hardware configuration of an information providing server. [Figure 21] Block diagram of the main functions provided by the information server [Figure 22] Flowchart showing processing operations of a user terminal [Figure 23]FIG. 10 is a diagram showing an example of a display of the results of estimation processing including the imaging direction. [Figure 24] FIG. 10 is a diagram showing an example of a screen for performing a subject selection operation. [Figure 25] FIG. 10 is a diagram showing an example of a subject selection result. [Figure 26] A diagram showing an example of a set composition. [Figure 27] A diagram showing an example of a set composition. [Figure 28] Conceptual diagram of the method for estimating the imaging position [Figure 29] FIG. 10 is a diagram showing an example of displaying the results of the estimation process. [Figure 30] Block diagram of the main functions realized by the user terminal [Figure 31] Block diagram of the main functions realized by the user terminal [Figure 32] An example of size correction [Figure 33] FIG. 10 is a diagram showing an example of correcting the front-to-back positional relationship; [Figure 34] FIG. 10 is a diagram showing an example of correcting the distance between subjects; [Figure 35] A diagram showing an example of correction when an obstacle is present [Figure 36] Block diagram of the main functions realized by the user terminal [Figure 37] FIG. 10 is a diagram showing an example of a screen for accepting corrections to the imaging position and imaging direction; [Figure 38] Block diagram of the main functions realized by the user terminal [Figure 39] Conceptual diagram of spatial measurement DETAILED DESCRIPTION OF THE INVENTION

[0028] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0029] [First embodiment] The present invention will be described by taking as an example a case where it is applied to a system (image capture position estimation system) that estimates and presents a position (image capture position) where a desired subject can be captured with a desired composition. The "image capture position" is a geographical position.

[0030] Note that the subjects to be imaged are limited to those whose location can be identified at least at the time of image capture. Therefore, even if an object is moving, it can be included in the subjects to be imaged as long as its location can be identified at the time of image capture. "Location" refers to a geographical location.

[0031] In the imaging position estimation system described below, an example will be described in which two subjects are selected and images are captured with a desired composition.

[0032] [System Configuration] FIG. 1 is a diagram showing a schematic configuration of an imaging position estimation system.

[0033] As shown in the figure, the imaging position estimation system 1 of this embodiment includes a user terminal 10 and an information providing server 100. The user terminal 10 and the information providing server 100 are connected via a network 2 so as to be able to communicate with each other.

[0034] [User device] The user terminal 10 is configured as a mobile terminal, particularly a mobile terminal with a camera. When the user terminal 10 is an information terminal with a camera, an image is captured by the camera provided in the information terminal. In this embodiment, a case where the user terminal 10 is configured as a smartphone with a camera will be described.

[0035] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a user terminal.

[0036] As shown in the figure, the user terminal 10 includes a CPU (Central Processing Unit) that controls the overall operation. )11, ROM (Read Only Memory) 12 storing basic input / output programs, etc., RAM (Random Access Memory) 13 used as a work area for CPU 11, EEPROM (Electrically Erasable and Programmable ROM) 14 storing various programs executed by CPU 11 and various data, etc., display 15, touch panel 16 detecting touch operations on the display screen, GPS receiving unit 17 receiving GPS (Global Positioning Systems) signals, camera 18 electronically capturing images, audio input unit 19 inputting audio, audio output unit 20 outputting audio, communication unit 21 communicating wirelessly with the nearest base station, etc., short-range wireless communication unit 22 communicating with external devices via short-range wireless, and sensor unit 23, etc.

[0037] The sensor unit 23 includes various sensors such as a geomagnetic sensor, a gyrocompass, an acceleration sensor, and a range sensor. The range sensor is a two-dimensional scanning optical distance sensor that measures the distance to a detected object while scanning light. The range sensor includes a laser scanner, a laser range finder, and a laser scanner. d These include LRF (Light Detection and Ranging) and LIDAR (Light Imaging Detection and Ranging or Laser Imaging Detection and Ranging). By having a range sensor, it is possible to scan space in three dimensions.

[0038] Camera 18 is a so-called digital camera and includes a lens, an image sensor, and the like. In this embodiment, camera 18 has a zoom function. Camera 18 is an example of an imaging device. CPU 11 is an example of a processor. Display 15 is an example of a display device.

[0039] FIG. 3 is a block diagram of the main functions realized by the user terminal.

[0040] As shown in the figure, the user terminal 10 has functions such as a camera information acquisition unit 10A, a subject selection unit 10B, a subject information acquisition unit 10C, a composition setting unit 10D, an imaging position estimation unit 10E, and a result output unit 10F. The functions of each unit are realized by a CPU 11 executing a predetermined program.

[0041] The camera information acquisition unit 10A performs processing to acquire camera information (camera information) required for estimating the imaging position. The camera information includes information on the size of the image sensor, information on the settable focal length of the lens, etc. As described above, in this embodiment, imaging is performed by the camera 18 provided in the user terminal 10. Therefore, in this embodiment, camera information of the camera 18 provided in the user terminal 10 is acquired. This information is stored in the ROM 12 and / or the EEPROM 14. The camera information acquisition unit 10A acquires the camera information from the ROM 12 and / or the EEPROM 14. The acquired camera information is added to the imaging position estimation unit 10E.

[0042] The subject selection unit 10B performs a process of accepting the selection of a subject to be imaged. In this embodiment, the selection of the subject is accepted using a map.

[0043] FIG. 4 is a diagram showing an example of a subject selection screen.

[0044] As shown in the figure, subject selection unit 10B displays a map M for subject selection on display 15. Map M is displayed in a predetermined map display area 15A set within the screen of display 15.

[0045] The map M displayed in the map display area 15A can be enlarged by pinching out and reduced by pinching in. Furthermore, the map M displayed in the map display area 15A can be moved in the direction of the operation by swiping.

[0046] The subject selection unit 10B acquires data of the map M from the information providing server 100 and displays it on the display 15.

[0047] FIG. 5 is a diagram showing an example of a map displayed in the map display area.

[0048] As shown in the figure, subject candidates Ob that can be selected within the displayed area are displayed on the map M. The subject candidates Ob are displayed as a combination of dots and names. The dots are displayed corresponding to the location of the subject candidate Ob on the map M. Note that since the names are also displayed, subject candidates Ob with unique names are selected.

[0049] The user selects a subject to be imaged by touching a subject candidate Ob desired to be imaged for a predetermined period of time or longer (selecting by so-called long press). Subject selection unit 10B accepts the selection of a subject based on an operation input to touch panel 16.

[0050] When the selection process is confirmed, an icon of the selected subject is displayed in a predetermined information display area 15B set within the screen of the display 15. In this embodiment, the information display area 15B is set within the same screen as the map display area 15A.

[0051] As described above, two subjects are selected in the imaging position estimation system 1 of this embodiment, and therefore two icons are displayed in the information display area 15B.

[0052] FIG. 6 is a diagram showing an example of a screen display when the selection of the first subject is accepted.

[0053] As shown in the figure, when the subject selection unit 10B accepts the selection of the first subject (first subject), it displays an icon Ic1 of the accepted subject in a first icon display area 15b1 set in the information display area 15B. In this embodiment, the name of the selected subject is displayed together with the icon Ic1. The icon Ic1 of the first subject is an example of a first image.

[0054] FIG. 7 is a diagram showing an example of a screen display when the selection of the second subject is accepted.

[0055] As shown in the figure, when the subject selection unit 10B accepts the selection of a second subject (second subject), it displays an icon Ic2 of the accepted subject in a second icon display area 15b2 set in the information display area 15B. In this embodiment, the name of the selected subject is displayed together with the icon Ic2. The icon Ic2 of the second subject is an example of a second image.

[0056] An icon is composed of an image representing a subject. In this embodiment, the icon is composed of an image that illustrates the subject. As will be described later, the icon is used to set the composition. Therefore, it is preferable that the icon is composed of an image that reproduces the subject as much as possible. In addition to an image that illustrates the subject, an image in which the subject portion is cut out from an image of the subject can be used as the icon.

[0057] The subject selection unit 10B acquires icon data of the selected subject from the information providing server 100 and displays it on the display 15.

[0058] The subject information acquisition unit 10C performs a process of acquiring information (subject information) about the subject selected by the subject selection unit 10B.

[0059] The subject information includes information necessary for estimating the imaging position. The information necessary for estimating the imaging position includes information on the position and size of the subject. The position is a geographical position. The geographical position is specified by, for example, latitude and longitude. When specifying the geographical position in more detail, altitude is included. The size information includes at least information on the height of the subject. The subject information acquisition unit 10C acquires the subject information from the information providing server 100. The acquired subject information is added to the imaging position estimation unit 10E.

[0060] The composition setting unit 10D performs processing to accept the input of a composition. The composition to be input is a composition for capturing an image of the subjects (first subject and second subject) selected by the subject selection unit 10B. The composition is input using the subject icons Ic1 and Ic2. The composition is also input in a predetermined composition setting area 15C set on the screen of the display 15. As shown in FIG. 4, in this embodiment, the composition setting area 15C is set on the same screen as the map display area 15A and the information display area 15B. A frame F having the same aspect ratio as the image captured by the camera 18 is displayed in the composition setting area 15C, and the composition setting operation is performed within the frame F. In other words, an operation to determine the position and size of the subject is performed within the frame F.

[0061] FIG. 8 is a conceptual diagram of an operation for displaying an icon in the composition setting area.

[0062] As shown in the figure, the icon of the subject to be captured is displayed in the composition setting area 15C by dragging the icon displayed in the information display area 15B to the composition setting area 15C. Fig. 8 shows an example in which the icon Ic1 of the first subject is dragged to the composition setting area 15C. The composition setting unit 10D controls the display on the display 15 based on an operation input to the touch panel 16.

[0063] FIG. 9 is a diagram showing an example of what appears on the display after the drag is completed.

[0064] As shown in the figure, when all of the icons Ic1 and Ic2 displayed in the information display area 15B are dragged to the composition setting area 15C, the display of the information display area 15B switches and a predetermined button (Check & Search button) Bt1 is displayed. The function of this Check & Search button Bt1 will be described later.

[0065] 10 to 15 are conceptual diagrams of the composition setting operation.

[0066] The composition setting operation is performed by moving and / or enlarging / reducing the subject icons Ic1 and Ic2 within the frame F displayed in the composition setting area 15C. The user moves and / or enlarges / reduces the subject icons Ic1 and Ic2 within the frame F to adjust the position and size of the icons Ic1 and Ic2 to achieve the desired composition.

[0067] Fig. 10 shows an example of an initial state. Fig. 11 shows an example of a movement operation. Fig. 12 shows an example of a zoom operation. Fig. 13 and Fig. 14 show an example of a posterior-posterior relationship setting operation.

[0068] As shown in FIG. 10, in the initial state, the icons Ic1 and Ic2 of the respective subjects are displayed at the dragged positions.

[0069] As shown in Fig. 11, the movement operation is performed by touching the icon of the subject to be moved and sliding it in the desired direction (a so-called swipe operation). Fig. 11 shows an example of moving the icon Ic1 of the first subject.

[0070] As shown in FIG. 12, the zooming operation is performed by pinching in and out. Specifically, by pinching in on the icon of a subject that you want to zoom out, the icon of that subject is reduced. Also, by pinching out on the icon of a subject that you want to zoom in, the icon of that subject is enlarged. FIG. 12 shows an example of reducing the icon Ic2 of a second subject. In this case, a pinch in is performed on the icon Ic2 of the second subject.

[0071] The front-to-back relationship setting operation is performed by touching the icon of the subject to be operated for a certain period of time or more (so-called long press). When the icon of the subject to be operated for a certain period of time or more, a front-to-back relationship setting menu Me is displayed on the screen, as shown in FIG. 13. The front-to-back relationship setting menu Me includes a button for moving the icon of the subject to be operated to the forefront and a button for moving it to the backmost. The user touches the buttons displayed in this front-to-back relationship setting menu Me to set the front-to-back relationship of the subject icon. FIG. 13 shows an example in which the icon Ic1 of the first subject is set as the operation target. In this case, the icon Ic1 of the first subject is touched for a certain period of time or more. FIG. 14 shows an example in which moving to the backmost is selected for the icon Ic1 of the first subject. In this case, as shown in the same figure, the icon Ic1 of the first subject is placed below the icon Ic2 of the second subject.

[0072] 15 shows an example of a composition set by a series of operations. Information about the set composition is added to the imaging position estimation unit 10E.

[0073] The imaging position estimation unit 10E performs processing to estimate a position (imaging position) where the first and second subjects can be imaged with a set composition, based on the subject information and camera information. The imaging position here includes a position where an image can be captured with approximately the same composition as the set composition. In other words, the imaging position estimation unit 10E estimates a position where an image can be captured with approximately the same composition. As described above, the composition is set on the display using the subject icons Ic1 and Ic2. Therefore, the imaging position is estimated based on the display state of the icons Ic1 and Ic2 on the display 15. More specifically, the imaging position is estimated based on the display state of the icons Ic1 and Ic2 within the frame F displayed on the display 15.

[0074] As described above, the subject information includes information on the position and size of the subject to be imaged. The camera information includes information on the size of the image sensor, information on the focal length of the lens that can be set, etc. The imaging position estimation unit 10E uses this information to estimate the imaging position.

[0075] FIG. 16 is a conceptual diagram of the imaging position estimation process.

[0076] In the figure, point P1 is the position of a first subject selected as an imaging target, and point P2 is the position of a second subject selected as an imaging target.

[0077] Consider the case where an image of a subject with a known size is captured. In this case, if the size of the subject in the captured image is known, the distance to the subject can be calculated from information about the camera used to capture the image (image sensor size, lens focal length, etc.).

[0078] In the imaging position estimation system 1 of this embodiment, the size and position of the subject to be imaged are known, and the size of the image sensor and the focal length of the lens that can be set are also known. Therefore, if the size of the subject in the set composition is known, the distance for imaging the subject at that size can be calculated. Here, the size of the subject in the set composition refers to the size of the subject on the image sensor.

[0079] 16, a dashed-dotted circle C11 centered at point P1 indicates a position where the first subject can be imaged at a set size when the lens focal length is set to a first focal length f1 [mm]. A solid-line circle C12 centered at point P1 indicates a position where the first subject can be imaged at a set size when the lens focal length is set to a second focal length f2 [mm] (f2>f1). A dashed-line circle C13 centered at point P1 indicates a position where the first subject can be imaged at a set size when the lens focal length is set to a third focal length f3 [mm] (f3>f2).

[0080] 16, a dashed-dotted circle C21 centered at point P2 indicates a position where the second subject can be imaged at the set size when the lens focal length is set to the first focal length f1 [mm]. A solid-line circle C22 centered at point P2 indicates a position where the second subject can be imaged at the set size when the lens focal length is set to the second focal length f2 [mm]. A dashed-line circle C23 centered at point P2 indicates a position where the second subject can be imaged at the set size when the lens focal length is set to the third focal length f3 [mm].

[0081] Therefore, in FIG. 16, the intersection of the same kind of circles is the position where the first subject and the second subject can be imaged at the set size.

[0082] Here, the intersections of the same kind of circles occur at two locations. When the first subject and the second subject are photographed from the positions of the respective intersections, the left-right positional relationship of the first subject and the second subject is swapped in the photographed image. Therefore, the intersection that will result in the same left-right positional relationship as the set composition is the intersection that should be selected. In other words, it is the position where the first subject and the second subject can be photographed with the same left-right positional relationship as the set composition. For example, in the example of FIG. 16, the solid-line circle C 12 and circle C22 have two intersection points X1 and X2. As shown in FIG. 15, the set composition has a first subject on the left side of the image and a second subject on the right side. When the first and second subjects are imaged from the position of intersection point X1, an image is captured in which the first subject is positioned on the left side of the image and the second subject is positioned on the right side. On the other hand, when the first and second subjects are imaged from the position of intersection point X2, an image is captured in which the first subject is positioned on the right side of the image and the second subject is positioned on the left side. Therefore, in this case, intersection point X1 is the intersection point that allows for an image to be captured with the correct left-right positional relationship.

[0083] In the example shown in Figure 16, there is no intersection between the dashed-dotted circles C11 and C21. This means that the first and second subjects cannot be captured in the set composition at the first focal length f1 [mm]. On the other hand, there are intersections between the solid-line circles C12 and C22 and the dashed-line circles C13 and C23. Therefore, when the focal length is set to the second focal length f2 [mm] or the third focal length f3 [mm], the first and second subjects can be captured in the set size.

[0084] However, the distance between the first and second objects captured when the focal length is set to the second focal length f2 [mm] differs from the distance between the first and second objects captured when the focal length is set to the third focal length f3 [mm]. Therefore, by further determining the focal length at which the first and second objects can be captured at the set distance, it is possible to determine the position at which the first and second objects can be captured with the set composition.

[0085] The imaging position estimating unit 10E changes the focal length in stages to search for a position where the first subject and the second subject can be imaged with the set composition.

[0086] FIG. 17 is a flowchart showing the procedure of the imaging position estimation process.

[0087] First, the size and positional relationship between the first and second subjects in the set composition is calculated (step S1). Specifically, the size and positional relationship between the first and second subjects on the image sensor are calculated.

[0088] Next, calculation parameters for calculating the imaging position are set (step S2). As for the focal length, the value at the wide end is set as the initial value.

[0089] Next, candidate imaging positions are calculated based on the set calculation parameters (step S3). As described above, candidate imaging positions are calculated as positions where the first and second subjects can be imaged at the set composition size. Two such positions are calculated as intersections of circles. Of the two intersections, the intersection where the first and second subjects can be imaged in the same left-right positional relationship as the set composition is selected as the candidate imaging position.

[0090] Next, it is determined whether an image having the same composition as the set composition can be captured when the first and second subjects are captured at the selected imaging position (step S4). That is, it is determined whether the first and second subjects can be captured with the same left-right positional relationship (spacing) as the set composition.

[0091] If it is determined that the set composition and image can be captured, the selected position is estimated as the image capturing position (step S5), and the process ends.

[0092] On the other hand, if it is determined that an image with the same composition as the set cannot be captured, it is determined whether the focal length can be changed (step S6). If the focal length can be changed, the focal length is changed (step S7). The focal length is changed to the telephoto side by a preset amount. Thereafter, based on the changed focal length, candidate imaging positions are calculated again (step S3).

[0093] If the focal length cannot be changed, it is determined that the imaging position cannot be estimated (step S9), and the process ends. In this embodiment, the case where the focal length cannot be changed refers to the case where the telephoto end has been reached.

[0094] In this way, the imaging position estimation unit 10E searches for the imaging position by switching the focal length in stages. Note that if the focal length cannot be changed, it is determined that the imaging position cannot be estimated, and therefore the processing performed by the imaging position estimation unit 10E includes processing for determining whether or not imaging can be performed with the set composition.

[0095] The imaging position estimation unit 10E performs an imaging position estimation process in response to a user's instruction to start the process. The user touches the Check & Search button Bt1 (see FIG. 9) displayed in the information display area 15B to instruct the process to start.

[0096] The processing result of the imaging position estimation unit 10E is sent to the result output unit 10F. The information of the processing result includes information on the estimated imaging position as well as information on the focal length. If it is determined that the imaging position cannot be estimated, information indicating that imaging is not possible is sent to the result output unit 10F as the processing result.

[0097] The result output unit 10F performs a process of outputting the processing result by the imaging position estimation unit 10E. This process is performed by displaying the processing result on the display 15 in a predetermined format.

[0098] 18 and 19 are diagrams showing an example of a display of the results of the imaging position estimation process. Fig. 18 shows an example of a display when the imaging position can be estimated. On the other hand, Fig. 19 shows an example of a display when the imaging position cannot be estimated.

[0099] 18, when the imaging position can be estimated, information about the estimated imaging position is displayed on the display 15. In this embodiment, a mark Mx indicating the imaging position is displayed on a map displayed in the map display area 15A. Information about the latitude and longitude of the estimated imaging position is also displayed in the information display area 15B. Information about the focal length when capturing an image is also displayed in the information display area 15B. By viewing this display, the user can understand the position and imaging conditions (focal length) at which the image composition set by the user can be captured.

[0100] On the other hand, if the imaging position cannot be estimated, that is, if the set composition cannot be captured, a predetermined error message is displayed in the information display area 15B as shown in FIG. to In this embodiment, the words "Not Found" are displayed as an error message in the information display area 15B.

[0101] When an error message is displayed, the user is required to reset the composition. When the user touches the composition setting area 15C to reset the composition, the error message disappears. When the error message disappears, the Check & Search button Bt1 is displayed again in the information display area 15B, as shown in FIG. 9. This allows the user to instruct the start of the imaging position estimation process again for the reset composition.

[0102] [Information server] The information providing server 100 provides various types of information to user terminals. The information providing server 100 is configured as a general server computer.

[0103] FIG. 20 is a block diagram showing an example of the hardware configuration of the information providing server.

[0104] As shown in the figure, the information providing server 100 comprises a CPU 101 that controls the overall operation, a ROM 102 that stores programs used to drive the CPU 101, such as an IPL (Initial Program Loader), a RAM 103 that is used as a work area for the CPU 101, an HDD (Hard Disk Drive) 104 that stores various programs and various data for the information providing server 100, a display 105, a communication unit 106 for data communication using a network 2, a keyboard 107, a mouse 108, an optical drive 110, etc.

[0105] FIG. 21 is a block diagram of the main functions provided by the information providing server.

[0106] As shown in the figure, the information providing server 100 has functions such as a request receiving unit 100A, a request processing unit 100B, and a request processing result output unit 100C. 101 This is realized by executing a predetermined program.

[0107] The request receiving unit 100A performs processing to receive requests for providing information from the user terminal 10. That is, it receives requests for providing map data, subject information, and the like.

[0108] The request processing unit 100B processes the requests received by the request receiving unit 100A. For example, in response to a request for map data, the request processing unit 100B performs a process to acquire the corresponding map data. Also, in response to a request for subject information, the request processing unit 100B performs a process to acquire the corresponding subject information.

[0109] Map data, object information, etc. are stored in an information database (DB) 100D. Therefore, the request processing unit 100B acquires the map data, object information, etc. from the information database 100D. The information database 100D is stored in the HDD 104, for example.

[0110] The request processing result output unit 100C acquires the processing result for the request accepted by the request accepting unit 100A from the request processing unit 100B, and outputs (transmits) the processing result to the user terminal 10.

[0111] [Operation] 22 is a flowchart showing the processing operation of the user terminal, which illustrates the flow of a series of processes from selecting a subject to outputting the estimation result of the imaging position.

[0112] First, a process of acquiring camera information is performed (step S11). As described above, in this embodiment, the camera information is stored in the ROM 12 and / or the EEPROM 14. Therefore, the camera information is acquired from the ROM 13 and / or the EEPROM 14.

[0113] Next, a process for accepting the selection of a subject to be imaged is performed (step S12). As shown in FIG. 4, the selection of the subject is performed using a map. The user selects the subject to be imaged on the map displayed on the display 15. More specifically, the user selects the subject to be imaged from the subject candidate Ob displayed on the map. The selection is performed by touching the subject candidate Ob to be imaged for a predetermined period of time or more. Once the selection is confirmed, an icon of the selected subject is displayed in the information display area 15B on the screen of the display 15, as shown in FIGS. 6 and 7. In this embodiment, two subjects are selected.

[0114] Next, a process of acquiring subject information of the selected subject is performed (step S13). The subject information is acquired from the information providing server 100.

[0115] Next, a process for accepting input of a composition is performed (step S14). The composition is input using an icon of the subject. As shown in FIG. 8, the user drags icons Ic1 and Ic2 displayed in the information display area 15B to the composition setting area 15C, thereby displaying icons used for setting the composition in the composition setting area 15C. As shown in FIGS. 11 to 14, the user sets the composition by moving and / or enlarging or reducing the icons Ic1 and Ic2 displayed in the composition setting area 15C.

[0116] After the composition setting is completed, the image capturing position estimation process is performed in response to a command from the user to start the process (step S15). The command to start the process is given by touching the Check & Search button Bt1 (see FIG. 9) displayed in the information display area 15B.

[0117] After the estimation process is completed, a process for outputting the results is performed (step S16). The results are displayed on the display 15. At this time, the display content changes depending on whether the imaging position could be estimated or not. As shown in FIG. 18, if the imaging position could be estimated, information on the estimated imaging position is displayed on the display 15. Specifically, a mark Mx indicating the imaging position is displayed on the map displayed in the map display area 15A. Furthermore, information on the latitude and longitude of the estimated imaging position and information on the focal length when capturing an image are displayed in the information display area 15B. By looking at this display, the user can understand the position where the composition set by the user can be captured and the imaging conditions (focal length). On the other hand, if the imaging position could not be estimated, a predetermined error message is displayed in the information display area 15B as shown in FIG. 19. to The user re-sets the composition in accordance with this display.

[0118] As described above, according to the imaging position estimation system of this embodiment, it is possible to obtain an imaging position (imaging position) by simply selecting a subject to be imaged and inputting a composition, thereby easily capturing an image with a desired composition.

[0119] [Variations] (1) Specifying the subject In the above embodiment, the imaging target is specified (selected) using a map, but the method of specifying the imaging target is not limited to this. For example, the imaging target may be specified by inputting a name. In this case, the information database 100D of the information providing server 100 stores subject information corresponding to the subject name.

[0120] (2) Icon The icon used to set the composition can be any shape that allows the subject to be roughly recognized. In other words, any shape that allows the rough composition to be set. Therefore, for example, a silhouette can be used.

[0121] (3) Composition In the above embodiment, the composition is set using a partial area (composition setting area) of the display screen, but the composition may be set using the entire screen, for example. In this case, the display may be switched to full screen display in response to an instruction from the user.

[0122] (4) Image capture position estimation process In the above embodiment, only the imaging position is estimated, but it is also possible to estimate the imaging direction in addition to the imaging position. Fig. 23 is a diagram showing an example of the display of the result of the estimation process including the imaging direction. As shown in the figure, direction When estimating the image capturing direction including the image capturing position, the estimated image capturing direction is displayed on the display 15. In the example shown in the figure, a mark M indicating the image capturing position is displayed. x or The imaging direction is indicated by an arrow Dx extending from the arrow.

[0123] Furthermore, the imaging position can be estimated by taking into account the altitude. This allows imaging that reflects the vertical position. When estimating the imaging position by taking into account the altitude, a sphere is set instead of a circle, and a position where each subject can be imaged is estimated. That is, when the lens is set to a predetermined focal length fx [mm], a position where each subject can be imaged at the set composition size is represented by a sphere, and the intersection of the sphere is found to estimate the imaging position.

[0124] Furthermore, when setting the composition, the orientation of the subject can also be set, and a position where the subject can be imaged in the set orientation can be estimated. For example, for a subject that faces forward, the orientation of the subject can be set, and a position where the subject can be imaged can be estimated. The orientation of the subject can be set by using, for example, a three-dimensional icon or the like so that the orientation can be adjusted.

[0125] Furthermore, it may be determined whether the estimated imaging position is an accessible location. In this case, for example, information on inaccessible locations is stored in the information database 100D, and it is determined whether the estimated imaging position corresponds to an inaccessible location. If the estimated imaging position is an inaccessible location, a predetermined warning is issued. For example, a warning message is displayed on the display 15. Inaccessible locations include private land, the sea, rivers, lakes, etc., as well as dangerous areas and areas where entry is difficult. For areas where entry is difficult, such as mountains, for example, information on people's passage history can be collected, and areas with no passage history can be determined as areas where entry is difficult. Alternatively, a separately generated heat map of entry history can be referenced to determine whether entry is possible.

[0126] Furthermore, when estimating the imaging position including the altitude, it may be further determined whether the estimated imaging position is a reachable position. For example, if the altitude from the ground surface exceeds 2 m, it may be determined that the position is unreachable.

[0127] If the estimated imaging position is an unreachable position, a predetermined warning is issued, for example, a warning message is displayed on the display 15.

[0128] (5) Providing information on backlighting The device may further include a function to calculate the time period when backlighting occurs when capturing images of the first and second subjects from a location estimated as the imaging position and to present information about the calculated time period to the user. The information about the time period when backlighting occurs is displayed in information display area 15B together with information about the imaging position, for example.

[0129] Note that instead of the information about the time periods when backlighting occurs, information about the time periods when backlighting does not occur may be presented. In this case, the time periods excluding the time periods when backlighting occurs are presented.

[0130] Furthermore, the device may have a function of calculating a time period when the amount of light is insufficient and presenting information about the calculated time period to the user. The time period when the amount of light is insufficient may be calculated as a time period when the brightness is below a threshold value, for example.

[0131] The information on the time period when there is backlight and the information on the time period when there is insufficient light may be presented together, or only one of the information may be presented.

[0132] (6) Camera In the above embodiment, an example has been described in which an image is captured using the camera 18 provided in the user terminal 10. However, an image can also be captured using a camera separate from the user terminal. In this case, it is necessary to separately obtain camera information for the camera to be used. The method for obtaining the camera information is not particularly limited, and for example, the camera information can be manually entered into the user terminal. Alternatively, the camera information can be automatically obtained by communicating with the camera. Furthermore, the camera information can be obtained by obtaining identification information for the camera to be used. In this case, for example, the camera information for each camera is stored in the information database 100D of the information providing server 100, and the camera information for the corresponding camera is obtained from the information providing server 100.

[0133] In the above embodiment, the lens has a zoom function, but the camera lens may be a so-called fixed focal length lens. In this case, the imaging position is estimated with the focal length kept constant.

[0134] Furthermore, the camera can be mounted on an unmanned vehicle, an unmanned aerial vehicle (a so-called drone), an unmanned ship, etc. to capture images. In this case, it becomes possible to capture images in places where people cannot enter, and the range of selectable image capture positions can be expanded. In particular, when a camera is mounted on an unmanned aerial vehicle to capture images, altitude restrictions can be lifted.

[0135] (7) Equipment configuration In the above embodiment, the functions of the information processing device are realized in the user terminal, but for example, the functions of the information processing device may be provided in a camera.

[0136] Furthermore, in the above embodiment, various types of information are provided from the information providing server 100, but the functions of the information providing server 100 may be provided in the user terminal. In this case, the information providing server is not required.

[0137] [Second embodiment] In the above embodiment, a case has been described in which an immobile subject, i.e., a subject (fixed object) whose geographical position is fixed, is the subject of imaging. The subject of imaging is not limited to this. Any object whose geographical position can be identified at least at the time of imaging can be included in the subject of imaging. For example, an object that moves with regularity, such as a celestial body, can be included in the subject of imaging.

[0138] An example in which a celestial body is included in the subject candidates will be described below. Note that the basic configuration of the system is the same as in the above embodiment, so only the subject selection method and the imaging position estimation method will be described here.

[0139] [Subject Selection] In this embodiment, the subject to be imaged is selected by selecting the object on a map or by inputting the name of the object.

[0140] FIG. 24 is a diagram showing an example of a screen for performing a subject selection operation.

[0141] As shown in the figure, a map for selecting a subject is displayed in a map display area 15 A. This is the same as in the first embodiment.

[0142] On the other hand, when selecting a subject by inputting the name of an object, search boxes B1 and B2 displayed in information display area 15B are used. As shown in FIG. 24, a first search box B1 is displayed in first icon display area 15b1 set in information display area 15B. A second search box B2 is displayed in second icon display area 15b2 set in information display area 15B. When selecting a first subject by name, the name is input in first search box B1. When selecting a second subject by name, the name is input in second search box B2. Note that a character input function provided in user terminal 10 is used to input characters. After inputting the name, a search button is touched to search for the subject. If the subject can be selected as a subject, its icon is displayed in the icon display area as a result.

[0143] 25 is a diagram showing an example of a subject selection result. The figure shows an example in which a first subject is selected using a map, and a second subject is selected using a name. The figure also shows a case in which an object with a fixed geographical location (a fixed object) is selected as the first subject, and a celestial body, specifically the moon, is selected as the second subject.

[0144] After this, the icon displayed in each icon display area is dragged to the composition setting area 15C to set the composition. The composition setting operation is the same as in the first embodiment. Note that if a celestial body is selected as the subject to be imaged, its size cannot be adjusted. Therefore, if a subject is selected as the subject to be imaged, only the position can be adjusted.

[0145] FIG. 26 is a diagram showing an example of a set composition.

[0146] The user selects the icon I in the composition setting area 15C. c 1. Adjust the position and size of Ic2 to set the desired composition. Then, touch the Check & Search button Bt1 displayed in the information display area 15B to instruct the start of the imaging position estimation process.

[0147] [Estimation of imaging position] Fig. 27 is a diagram showing an example of a set composition, and Fig. 28 is a conceptual diagram of a method for estimating an imaging position.

[0148] 28, point P1 is the geographical location of the first subject. Circle C with a radius r and centered at point P1 indicates the position where the first subject can be captured at the set composition size when the lens is set to a predetermined focal length fx [mm].

[0149] As shown in Figure 27, in the set composition, the height of the first subject is h1 and the height of the second subject is h2. The distance between the first subject and the second subject is w. Meanwhile, the actual height of the first subject is H1. In this case, the elevation angle θ (altitude of the second subject) required to capture an image of the second subject at the set height of the composition at a focal length fx [mm] can be calculated using the following formula: tanθ=(H1 / r)×(h2 / h1)

[0150] Using the above formula, distance within the composition can be converted into angle. Therefore, the distance w between the first and second subjects within the composition can also be converted into the actual angle φ using the following formula. φ=θ×(w / h2)

[0151] At the location where the first subject is located, the date and time when the second subject will have an elevation angle of θ is searched for. The direction in which the second subject can be observed (imaged) at the searched date and time is found.

[0152] For celestial bodies, if the location and date and time of observation can be identified, the position (elevation angle and azimuth) at which the celestial body is observed can be identified. Information on the position of the celestial body may be obtained from the information providing server 100 or may be calculated.

[0153] In Fig. 28, the orientation indicated by arrow D2 is an example of a determined orientation. That is, it indicates the orientation in which the second subject should be. As shown in Fig. 28, the imaging position PX is identified as a point on circle C when the first subject is located in a direction that is shifted by angle φ from the orientation in which the second subject should be (arrow D2). In other words, the imaging position PX is the point at which a half ray is drawn from the first subject in a direction that is shifted by angle φ from the orientation in which the second subject should be (arrow D2) and the circle C intersect.

[0154] The imaging position is estimated through the above series of steps. As described above, when a celestial body is the subject of imaging, the imaging date and time are also estimated in addition to the imaging position.

[0155] FIG. 29 is a diagram showing an example of a display of the results of the estimation process.

[0156] As shown in the figure, if the imaging position can be estimated, a mark Mx indicating the imaging position is displayed on the map displayed in the map display area 15A. In addition, information such as the latitude and longitude of the estimated imaging position, the focal length when capturing an image, and the imaging date and time is displayed in the information display area 15B. By looking at this display, the user can understand the position where the image can be captured with the composition they have set, and the imaging conditions (focal length and imaging date and time).

[0157] [Variations] In the above embodiment, the imaging date and time are estimated in addition to the imaging position, but the imaging date and time may be set by the user and the possibility of imaging may be determined. That is, the possibility of imaging may be determined by determining whether an image with a set composition can be captured at the set date and time. The possibility of imaging may be determined, for example, by determining whether the date and time at the location where the first subject is present, when the second subject has an elevation angle θ, matches the specified date and time. If the specified date and time do not match, it is determined that imaging is not possible. The date and time may be specified within a range.

[0158] [Third embodiment] According to the imaging position estimation systems of the first and second embodiments, it is possible to estimate a position where a desired composition can be captured. However, when an attempt is made to actually capture an image from the estimated position, an obstacle may be present and the desired image may not be captured. In this embodiment, a case will be described in which the presence or absence of an obstacle is determined to determine whether or not imaging is possible.

[0159] FIG. 30 is a block diagram of the main functions realized by the user terminal.

[0160] As shown in the figure, the user terminal 10 of this embodiment further has the function of an imaging possibility determination unit 10G. Other functions are substantially the same as those of the first embodiment. Therefore, the function of the imaging possibility determination unit 10G will be described here. The function of the imaging possibility determination unit 10G is realized by the CPU 11 executing a predetermined program.

[0161] The imaging feasibility determination unit 10G acquires information on the imaging position and focal length estimated by the imaging position estimation unit 10E, and determines whether or not there is an object (obstacle) that will hinder imaging based on the acquired information on the imaging position and focal length and the information on the positions of the first and second subjects. That is, when imaging the first and second subjects from the estimated imaging position, it determines whether or not an obstacle is present within the angle of view. Specifically, it detects obstacles between the imaging position and the first and second subjects. Information on obstacles is stored in the information database 100D of the information providing server 100. For example, an object that exceeds a specified size is registered as an obstacle along with information on its position and size. The imaging feasibility determination unit 10G detects obstacles based on the information stored in the information database 100D. If an obstacle is detected, it determines that imaging is not possible.

[0162] If the imaging possibility determining unit 10G determines that imaging is not possible, a message indicating that imaging is not possible due to the presence of an obstacle is displayed on the display 15. If it determines that imaging is possible, information on the imaging position and focal length is displayed on the display 15 as usual (see FIG. 18).

[0163] In this way, according to this embodiment, the image capturing position is estimated taking into consideration obstacles, thereby making it possible to more reliably present a position where an image with a desired composition can be captured.

[0164] [Fourth embodiment] According to the imaging position estimation systems of the first and second embodiments, if an imaging position cannot be estimated, it is determined that imaging is impossible. However, imaging may become possible by making a simple correction to the composition set by the user. For example, if an imaging cannot be performed with the set composition because the front-to-back positional relationship of two subjects is inappropriate, imaging may become possible by swapping the front-to-back positional relationship of the two subjects. Furthermore, if an imaging cannot be performed with the set composition because the two subjects are too close, imaging may become possible by increasing the distance between the two subjects. Similarly, if an imaging cannot be performed with the set composition because the two subjects are too far apart, imaging may become possible by narrowing the distance between the two subjects. In this embodiment, a case will be described in which, if an imaging position cannot be estimated with the composition set by the user, the composition is automatically corrected and presented to the user.

[0165] FIG. 31 is a block diagram of the main functions realized by the user terminal.

[0166] As shown in the figure, the user terminal 10 of this embodiment further has the function of a composition correction unit 10H. Other functions are substantially the same as those of the first embodiment. Therefore, the function of the composition correction unit 10H will be described here. The function of the composition correction unit 10H is realized by the CPU 11 executing a predetermined program.

[0167] The composition correction unit 10H performs processing to correct the composition set by the user when the imaging position estimation unit 10E determines that imaging is impossible. As described above, when the imaging position estimation unit 10E cannot estimate the imaging position, it determines that imaging is impossible with the set composition. When the imaging position cannot be estimated, the composition correction unit 10H corrects the composition set by the user in a predetermined procedure.

[0168] (1) Correcting the size of the subject If the imaging position cannot be estimated because the sizes of the first and second subjects are inappropriate in the set composition, the sizes of the first and second subjects are corrected to correct the composition. The imaging position estimation unit 10E performs processing to estimate the imaging position again for the corrected composition.

[0169] 32 is a diagram showing an example of size correction, in which (A) shows the composition before correction, (B) and (C) show the composition during the correction process, and (D) shows the composition after correction is complete.

[0170] As shown in the figure, the size correction is performed for both the first and second subjects, for example, in stages at a certain ratio. If the imaging position cannot be estimated after the size correction is performed up to a predetermined ratio, it is determined that imaging is impossible.

[0171] In cases where the first and second subjects are arranged overlapping each other in the set composition, the composition may be corrected by further changing the front-to-back positional relationship between the first and second subjects. In this case, the front-to-back positions of the first and second subjects are swapped.

[0172] If the imaging position can be estimated with the corrected composition, the corrected composition is displayed in the composition setting area 15C.

[0173] (2) Correcting the front-to-back positional relationship Set Ta When the imaging position cannot be estimated due to an inappropriate positional relationship between the first and second subjects in the composition, the positional relationship between the first and second subjects is changed to correct the composition. An example of an inappropriate positional relationship between the first and second subjects is when a subject that should be captured large is captured small and in a position closer to the subject.

[0174] 33A and 33B are diagrams showing an example of correction of the front-to-back positional relationship. Fig. 33A shows the composition before correction, and Fig. 33B shows the composition after correction.

[0175] As shown in the figure, the composition correction unit 10H corrects the composition by swapping the front-to-back positions of the first and second subjects. The imaging position estimation unit 10E performs processing to estimate the imaging position again for the corrected composition. If the imaging position cannot be estimated for the corrected composition, it is determined that imaging is impossible.

[0176] If the imaging position can be estimated with the corrected composition, the corrected composition is displayed in the composition setting area 15C.

[0177] (3) Correcting the distance between subjects If the imaging position cannot be estimated because the distance (interval) between the subjects is inappropriate in the set composition, the distance between the subjects is corrected to correct the composition.

[0178] 34 is a diagram showing an example of correcting the distance between subjects. The figure shows an example of correcting the distance when the distance between subjects is too close. Note that FIG. 34(A) shows the composition before correction, FIG. 34(B) and FIG. 34(C) show the composition during the correction process, and FIG. 34(D) shows the composition after correction is complete.

[0179] As shown in the figure, if the imaging position cannot be estimated because the distance between the subjects is too close, the distance between the subjects is gradually increased to correct the composition. The distance is gradually increased at a fixed ratio. Correction ends when the imaging position can be estimated. On the other hand, if the imaging position cannot be estimated even after correction up to a predetermined ratio, it is determined that imaging is impossible.

[0180] If the imaging position cannot be estimated because the distance between the subjects is too far, the distance between the subjects is corrected to be closer.

[0181] The distance is corrected, for example, by first correcting the distance in a direction to shorten the distance. If the imaging position cannot be estimated by correcting the distance in a direction to shorten the distance, the distance is corrected in a direction to increase the distance. Alternatively, the opposite is performed.

[0182] If the imaging position can be estimated with the corrected composition, the corrected composition is displayed in the composition setting area 15C.

[0183] In this way, according to this embodiment, if an image cannot be captured with the composition set by the user, the image can be automatically corrected to a similar composition and the image capturing position can be presented to the user, thereby improving convenience.

[0184] [Variations] The above corrections can be combined as needed. In this case, for example, the corrections are performed in a predetermined order, and the process ends when the imaging position is estimated.

[0185] Similarly, if it is determined that an image cannot be captured due to the presence of an obstacle, the composition may be corrected. That is, the composition is corrected to avoid the obstacle and capture an image of the first and second subjects. In this case, the composition is corrected by, for example, correcting the position of the subject blocked by the obstacle.

[0186] 35 is a diagram showing an example of correction when an obstacle is present. The figure shows an example of correction when an obstacle OX is present in front of the first subject. Figure 35(A) shows the composition before correction, and Figure 35(B) shows the composition after correction.

[0187] As shown in the figure, if it is determined that imaging is impossible due to the presence of an obstacle OX in front of the first subject, the position of the first subject is moved to correct the composition, i.e., the first subject is moved to a position where it is not hidden by the obstacle.

[0188] In addition, if the imaging position cannot be estimated because the front-to-back positional relationship between the first and second subjects is inappropriate, methods can be used such as swapping the front-to-back positional relationship or correcting the composition so that the depth relationship is unclear.

[0189] [Fifth embodiment] According to the imaging position estimation system of the first embodiment, it is possible to estimate a position where a desired composition can be captured. However, there are cases where the estimated position is a place where people cannot enter. In this case, it is necessary to change the composition or the imaging position. When changing the imaging position, it is possible to improve convenience if it is possible to know in advance how the composition will change. In this embodiment, a case will be described in which the system is provided with a function to correct the imaging direction in addition to the estimated imaging position, and a function to present compositions that can be captured when the imaging position and / or imaging direction are corrected.

[0190] FIG. 36 is a block diagram of the main functions realized by the user terminal.

[0191] As shown in the figure, the user terminal 10 of this embodiment further has the functions of a correction receiving unit 10I and a composition estimation unit 10J. Other functions are substantially the same as those of the first embodiment. Therefore, the functions of the correction receiving unit 10I and the composition estimation unit 10J will be mainly described here. The functions of the correction receiving unit 10I and the composition estimation unit 10J are realized by the CPU 11 executing a predetermined program.

[0192] The correction receiving unit 10I performs processing to receive corrections to the imaging position and imaging direction. Correction of the imaging position is performed by moving a mark Mx indicating the imaging position displayed on the display 15. Correction of the imaging direction is performed by changing the direction of an arrow Dx indicating the imaging direction displayed on the display 15.

[0193] 37 is a diagram showing an example of a screen for accepting corrections to the imaging position and imaging direction. As shown in the figure, a predetermined button (Move button) Bt2 is displayed superimposed on the map in the map display area 15A. By touching this Move button Bt2, it becomes possible to correct the imaging position and imaging direction. To correct the imaging position, touch and move the mark Mx. To correct the imaging direction, touch and move (rotate) the arrow Dx. Note that the arrow Dx rotates around the mark Mx.

[0194] When the imaging position is corrected, the composition estimation unit 10J performs processing to estimate the composition of an image captured from the corrected imaging position and imaging direction.

[0195] The result output unit 10F displays the composition estimated by the composition estimation unit 10J in the composition setting area 15C.

[0196] As described above, according to the present embodiment, when the imaging position and / or imaging direction is corrected, the composition of the image to be captured after the correction can be presented, thereby enabling the user to easily capture an image with the desired composition.

[0197] [Variations] In the above embodiment, the case where the imaging position and imaging direction are corrected has been described, but the focal length may also be corrected.

[0198] [Sixth embodiment] In the above series of embodiments, the case where an image of a subject whose position and size are known has been described. In this embodiment, the case where an image of an object whose position and / or size is unknown will be described.

[0199] When the position and / or size of the subject to be imaged is unknown, this information is obtained by actual measurement. In this embodiment, a range sensor is used to three-dimensionally scan the subject to be imaged and measure its position and size.

[0200] FIG. 38 is a block diagram of the main functions realized by the user terminal.

[0201] As shown in the figure, the user terminal 10 of this embodiment further has the functions of a measurement control unit 10K and an icon acquisition unit 10L. Other functions are substantially the same as those of the first embodiment. Therefore, the functions of the measurement control unit 10K and the icon acquisition unit 10L will be mainly described here. The functions of the measurement control unit 10K and the icon acquisition unit 10L are realized by the CPU 11 executing a predetermined program.

[0202] The measurement control unit 10K controls the sensor unit 23 to perform processing to measure the position and size of the subject to be imaged. In this embodiment, a range sensor is used to three-dimensionally scan the subject to be imaged and measure its position and size.

[0203] Figure 39 is a conceptual diagram of spatial measurement.

[0204] As shown in the figure, the position and size of the subject to be imaged are measured using a predetermined position PO in real space as a reference.

[0205] The icon acquisition unit 10L performs a process of acquiring information about an icon of a subject to be imaged. In this embodiment, the icon is acquired by capturing an image of the subject to be imaged. Specifically, an image obtained by extracting a subject portion from an image of the subject to be imaged is used as the icon.

[0206] The composition setting unit 10D uses the icon acquired by the icon acquiring unit 10L to accept the input of the composition.

[0207] The imaging position estimation unit 10E estimates a position where an image with the composition set by the composition setting unit 10D can be captured, based on information on the position and size of the subjects (first subject and second subject) acquired by measurement by the measurement control unit 10K.

[0208] As described above, according to this embodiment, even when capturing an image of a subject whose position and / or size is unknown, it is possible to estimate a position where an image with a desired composition can be captured.

[0209] [Variations] In the above embodiment, an icon is generated and acquired from an image of an actual subject, but the method of acquiring an icon of the subject is not limited to this. Alternatively, for example, icons of a plurality of objects (subject candidates) may be prepared in advance and the user may select one. Furthermore, the icon may be substituted with, for example, a rectangular frame. In this case, it is preferable to have the user input the aspect ratio and use it as the icon of the target subject.

[0210] [Other embodiments] (1) Navigation function Since the imaging position can be estimated, the device may further include a function of guiding the user to the estimated imaging position. That is, the device may further include a navigation function. In this case, the user terminal 10 is further provided with a function of acquiring current location information, a function of generating guidance information that leads the user to the imaging position based on the acquired current location information, and a function of providing guidance based on the generated guidance information. The guidance function is performed, for example, by displaying the generated guidance information on the display 15. These functions can be realized using known navigation technology.

[0211] (2) Imaging guide function When the imaging direction is estimated, the device may further include a function for guiding the imaging direction. That is, the device may include a function for indicating the imaging direction to the user at the actual imaging position. In this case, the user terminal is provided with a function for acquiring camera orientation information (information on the orientation of the optical axis), a function for determining the direction to point the camera based on the acquired camera orientation information, and a function for notifying the determined direction. The determined direction is notified, for example, by displaying the orientation information on a display. In this case, it is more preferable to display the orientation information superimposed on the live view image.

[0212] Furthermore, an image with a composition set by the user may be displayed overlaid on the live view image. In this case, the image with the composition may be displayed, for example, semi-transparently. This makes it easier to capture an image with the set composition.

[0213] (3) Automatic imaging When the imaging position and imaging direction are estimated, an imaging configuration can be adopted in which imaging is performed automatically based on the estimated imaging position and imaging direction. For example, when imaging is performed using a camera mounted on an unmanned vehicle, an unmanned aerial vehicle, an unmanned ship, etc., information on the estimated imaging position and imaging direction can be provided to these devices, and imaging can be performed automatically. Note that unmanned vehicles, unmanned aerial vehicles, unmanned ships, etc. equipped with cameras, and the technology for automatically imaging using these are publicly known, and therefore detailed description thereof will be omitted.

[0214] (4) Information server In the above embodiment, the information providing server is configured to provide only information necessary for estimating the imaging position, but the information providing server may also be configured to perform the imaging position estimation process, etc. In other words, the user terminal is used as an interface, and the imaging position estimation process, etc. is performed by the information providing server. In this case, the information providing server functions as an information processing device. In addition, in this case, the user terminal constitutes another display device.

[0215] Furthermore, the functions of the information providing server may be provided in the user terminal, that is, the functions of the user terminal and the functions of the information providing server may be realized in one device.

[0216] (5) Hardware configuration The functions of the user terminal are realized by various processors. The various processors include a CPU and / or a GPU (Graphic Processing Unit), which are general-purpose processors that execute programs and function as various processing units, a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture, and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically to execute specific processing. A program is synonymous with software.

[0217] A single processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types. For example, a single processing unit may be configured with multiple FPGAs, or a combination of a CPU and an FPGA. Alternatively, multiple processing units may be configured with a single processor. Examples of multiple processing units configured with a single processor include, first, a configuration in which a single processor is configured with a combination of one or more CPUs and software, as typified by computers used as clients or servers, and this processor functions as multiple processing units. Second, a configuration in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by a system on chip (SoC). In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure. [Explanation of symbols]

[0218] 1. Imaging position estimation system 2 Network 10 User terminal 10A Camera information acquisition unit 10B Subject selection section 10C Subject information acquisition section 10D Composition Settings 10E Imaging position estimation unit 10F Result output section 10G Imaging availability determination unit 10H Composition Correction Section 10I Correction Reception Department 10J Composition Estimation Unit 10K Measurement control unit 10L Icon Acquisition Section 11 CPU 12 ROM 13 RAM 14 EEPROM 15 Display 15A Map display area 15B Information display area 15C Composition Setting Area 15b1 First icon display area 15b2 Second icon display area 16 Touch Panel 17 GPS receiver 18 Camera 19 Audio input section 20 Audio output section 21 Communications Department 22 Near Field Communication Department 23 Sensor section 100 Information Server 100A Request Reception Department 100B Request processing unit 100C Request processing result output section 100D Information Database 101 CPU 102 ROM 103 RAM 104 HDD 105 Display 106 Communications Department 107 keyboard 108 Mouse 110 Optical Drive B1 First search box B2 Second search box Bt1 Check&Search button Bt2 Move button C. A circle of radius r centered at point P1 C11: A dashed circle centered at point P1 C12 Solid circle centered at point P1 C13: A dashed circle centered at point P1 C21: A dashed circle centered at point P2 C22 Solid circle centered at point P2 C23: A dashed circle centered at point P2 D2 Arrow indicating the direction in which the second subject should be Dx Arrow indicating the imaging direction F frame Ic1 First subject icon Ic2 Second subject icon M Map Me Settings Menu Mx Imaging position mark OX Obstacle Ob subject candidate P1 Position of first subject P2 Position of the second subject PX imaging position X1 Intersection of solid circles C21 and C22 X2 Intersection of solid circles C21 and C22 f1 first focal length f2 second focal length f3 third focal length fx focal length r Radius of circle C H1 Actual height of the first subject h1 Height of the first subject in the composition h2 Height of the second subject in the composition w Distance between the first and second subject in the composition θ elevation angle φ angle S1 to S9: Imaging position estimation process S11~S16 User terminal processing operations

Claims

1. A system comprising an imaging device, a display device, and a processor, The processor: acquiring information about the imaging device and information about the geographical positions and sizes of the first and second subjects to be imaged; changing a display state of the first image and the second image on the display device in response to an input operation by a user that inputs an arrangement of a first image representing the first subject and a second image representing the second subject on the display device; and and performing control to estimate an imaging position where the first subject and the second subject can be imaged in a composition that matches the display state of the first image and the second image on the display device, by calculating a positional relationship between the imaging device and the first subject and the second subject that can reproduce the display state of the first image and the second image on the display device by imaging, based on information about the imaging device and information about the geographical positions and sizes of the first and second subjects. Information processing device.

2. the processor controls the display device to display information about the estimated imaging position. The information processing device according to claim 1 .

3. the processor accepts an input of a composition of the first image and the second image on the display device; performing control to estimate the imaging position based on the input composition; 3. The information processing device according to claim 1 or 2.

4. the composition is a position and a size of the first image and a position and a size of the second image on the display device; The information processing device according to claim 3 .

5. A system comprising an imaging device, a display device, and a processor, The processor: acquiring information about the imaging device and information about the geographical positions of the first and second subjects to be imaged; changing a display state of the first image and the second image on the display device in response to an input operation by a user that inputs an arrangement of a first image representing the first subject and a second image representing the second subject on the display device; and a positional relationship between the imaging device and the first and second subjects that can reproduce the display states of the first and second images on the display device by imaging, based on information about the imaging device and information about the geographical positions of the first and second subjects, thereby estimating a direction in which the first and second subjects can be imaged in a composition that matches the display states of the first and second images on the display device, and performing control to determine the direction in which the imaging device should be pointed; Information processing device.

6. the processor controls the display device to display and notify information about the direction in which the imaging device is pointed. The information processing device according to claim 5 .

7. the information on the geographical position of the first subject and / or the information on the geographical position of the second subject is information on the altitude and direction of the first subject and / or information on the altitude and direction of the second subject; 7. The information processing device according to claim 1, 5 or 6.

8. The processor: accepting an input of information identifying the first subject and the second subject; and performing control to acquire information on the geographical position and size of the identified first subject and information on the geographical position and size of the identified second subject from a database.

7. The information processing device according to claim 1, 5 or 6.

9. the processor controls acquisition of information measuring the geographical position and size of the first object and the geographical position and size of the second object; 7. The information processing device according to claim 1, 5 or 6.

10. The processor: Get your current location information, generating guidance information that leads to the imaging position based on the acquired information on the current location; Controlling the display of the generated guidance information on the display device.

3. The information processing device according to claim 1 or 2.

11. The processor: calculating a time period during which backlight and / or insufficient light will occur when the first subject and the second subject are imaged at the imaging position; Controlling to display the calculated time period or a time period excluding the calculated time period on the display device.

3. The information processing device according to claim 1 or 2.

12. when the imaging position cannot be estimated, the processor performs control to correct the imaging position to the composition that allows estimation. The information processing device according to claim 3 .

13. the processor controls the display device to display a frame having the same aspect ratio as the image captured by the imaging device. The information processing device according to claim 3 .

14. when the imaging position cannot be estimated because the sizes of the first image and the second image in the composition for which the input has been received are inappropriate, the processor changes the sizes of the first image and the second image within the frame and / or the front-to-back positional relationship, thereby performing control to correct the composition so that the imaging position can be estimated. The information processing device according to claim 13.

15. When the imaging position cannot be estimated because the front-to-back positional relationship between the first image and the second image in the composition for which the input has been received is inappropriate, the processor performs control to change the size of the first image and the size and / or the front-to-back positional relationship of the second image within the frame to correct the composition so that the imaging position can be estimated. The information processing device according to claim 13.

16. when the imaging position cannot be estimated because a distance between the first image and the second image in the composition for which an input has been received is inappropriate, the processor changes the distance between the first image and the second image within the frame to correct the composition so that the imaging position can be estimated. The information processing device according to claim 13.

17. The processor: detecting an obstacle when the first subject and the second subject are imaged at the estimated image capturing position; When the obstacle is detected, control is performed to correct the composition so that the first subject and the second subject can be imaged while avoiding the obstacle. The information processing device according to claim 13.

18. The processor: determining whether the estimated imaging position is an accessible location; Control to warn when an area is inaccessible, 3. The information processing device according to claim 1 or 2.

19. The imaging position includes altitude information.

3. The information processing device according to claim 1 or 2.

20. The processor: determining whether the estimated imaging position is a reachable position; Control to warn in case of unreachable position, 3. The information processing device according to claim 1 or 2.

21. The processor further performs control to accept input of orientations of the first image and the second image within the frame. The information processing device according to claim 13.

22. The information processing device is capable of outputting an image to a second display device provided in an external device, The processor: changing a display state of the first image and the second image on the second display device in response to an input operation by a user that inputs an arrangement of the first image and the second image on the second display device; and and performing control to estimate an imaging position where the first subject and the second subject can be imaged in a composition that matches the display state of the first image and the second image on the second display device by calculating a positional relationship between the imaging device and the first subject and the second subject, which can reproduce the display state of the first image and the second image on the second display device by imaging, based on information about the imaging device and information about the geographical positions and sizes of the first subject and the second subject.

3. The information processing device according to claim 1 or 2.

23. acquiring information about the imaging device and information about the geographical positions and sizes of the first and second objects to be imaged; changing a display state of the first image and the second image on the display device in response to an input operation by a user that inputs an arrangement of a first image representing the first subject and a second image representing the second subject on the display device; calculating a positional relationship between the imaging device and the first and second subjects, which allows the display states of the first and second images on the display device to be reproduced by imaging, based on information about the imaging device and information about the geographical positions and sizes of the first and second subjects, and thereby estimating an imaging position at which the first and second subjects can be imaged in a composition that matches the display states of the first and second images on the display device; An information processing method including:

24. a function of acquiring information about the imaging device and information about the geographical positions and sizes of the first and second subjects to be imaged; changing a display state of the first image and the second image on the display device in response to an input operation by a user that inputs an arrangement of a first image representing the first subject and a second image representing the second subject on the display device; a function of estimating an imaging position at which the first subject and the second subject can be imaged in a composition that matches the display state of the first image and the second image on the display device, by calculating a positional relationship between the imaging device and the first subject and the second subject that can reproduce the display state of the first image and the second image on the display device by imaging, based on information about the imaging device and information about the geographical positions and sizes of the first subject and the second subject; An information processing program that enables a computer to achieve the above.

25. A non-transitory computer-readable recording medium having the program according to claim 24 recorded thereon.

Images