Information processing device, method and program

Abstract

[Problem] To easily measure the wall surface of an object. [Solution] An information processing device 10 according to the present disclosure comprises: a first acquisition unit 1131 for acquiring coordinate information, on an image displayed on a screen, of at least two reference points, each of which is displayed on the screen and has coordinate information in world coordinates; a first inference unit 1141 for inferring the location of a virtual wall surface in the world coordinates which passes through the at least two reference points, on the basis of the coordinate information of the at least two reference points in the world coordinates, the coordinate information of the at least two reference points on the image, and information pertaining to an imaging condition in which the image has been captured; a second acquisition unit 1132 for acquiring coordinate information of at least one input point on the image which has been input with respect to the screen; and a second inference unit 1142 for inferring world coordinates of the input point for a case where the input point is assumed to be located on the virtual wall surface.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, a method, and a program. [Background technology]

[0002] In recent years, aircraft such as drones or multicopters, which fly by rotating multiple propellers, have been used to observe objects from high altitudes, photograph the ground from above, or observe difficult-to-access areas. Images captured from high altitudes using such aircraft are sometimes used for inspecting, surveying, and the like. For example, Patent Document 1 discloses a technology for measuring the shape and dimensions of a roof from an image of the object captured by a camera mounted on the aircraft, and calculating the area of ​​the roof from the shape and dimensions. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2003-162552 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the technology disclosed in the above-mentioned patent document, the image of the object captured from a high place is subjected to perspective due to central projection, making it difficult to measure the wall surface of the object.

[0005] The present disclosure has been made in view of the above background, and its purpose is to provide an information processing device, method, and program that can easily measure the wall surface of an object. [Means for solving the problem]

[0006] According to the present disclosure, there is provided an information processing device comprising: a first acquisition unit that acquires coordinate information on an image displayed on a screen of at least two reference points that are displayed on the screen and each have coordinate information in world coordinates; a first estimation unit that estimates the position of a virtual wall surface in the world coordinates that passes through the at least two reference points based on the coordinate information in the world coordinates of the at least two reference points, their coordinate information on the image, and information related to the imaging conditions when the image was captured; a second acquisition unit that acquires coordinate information on the image of an input point that is input to the screen; and a second estimation unit that estimates the world coordinates of the input point when it is assumed that the input point is located on the virtual wall surface.

[0007] Furthermore, according to the present disclosure, a method is provided, including: a processor acquiring coordinate information on an image displayed on the screen of at least two reference points that are displayed on the screen and each have coordinate information in world coordinates; estimating the position of a virtual wall surface in the world coordinates that passes through the at least two reference points based on the coordinate information in the world coordinates of the at least two reference points, the coordinate information on the image, and information related to the imaging conditions when the image was captured; acquiring coordinate information on the image of an input point that was input to the screen; and estimating the world coordinate of the input point when it is assumed that the input point is located on the virtual wall surface.

[0008] Furthermore, according to the present disclosure, there is provided a program for causing a computer to function as a first acquisition unit that acquires coordinate information on an image displayed on a screen of at least two reference points that are displayed on the screen and each have coordinate information in world coordinates; a first estimation unit that estimates the position of a virtual wall surface in the world coordinates that passes through the at least two reference points based on the coordinate information in the world coordinates of the at least two reference points, their coordinate information on the image, and information related to the imaging conditions when the image was captured; a second acquisition unit that acquires coordinate information on the image of an input point that is input to the screen; and a second estimation unit that estimates the world coordinates of the input point when it is assumed that the input point is located on the virtual wall surface. [Effects of the Invention]

[0009] According to the present disclosure, it is possible to easily measure the wall surface of an object. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an overview of a system 1 according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a block diagram showing a configuration of an information processing terminal 10 according to the embodiment. [Figure 3] A block diagram showing an example of the functional configuration of the unmanned aerial vehicle 20 according to the embodiment. [Figure 4] 2 is a block diagram showing the functional configuration of a control unit 11 according to the embodiment. FIG. [Figure 5] 10 is a flowchart illustrating a series of processes in the information processing terminal 10 according to the embodiment. [Figure 6] FIG. 2 is a diagram showing an example of imaging processing by the unmanned aerial vehicle 20. [Figure 7] 10 is a display example on screen V1 of touch panel 12. [Figure 8] FIG. 10 is a diagram showing an example of setting a reference point on an image D11. [Figure 9] 10A and 10B are diagrams for explaining a process of estimating the position of a virtual wall surface; [Figure 10] 10A and 10B are diagrams for explaining processing relating to setting of input points and setting of areas; [Figure 11] 10A and 10B are diagrams for explaining processing relating to area estimation according to a modified example of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0012] FIG. 1 is a diagram illustrating an overview of a system 1 according to an embodiment of the present disclosure. As illustrated, the system 1 includes an information processing terminal 10 (an example of an information processing device) and an unmanned aerial vehicle 20. The system 1 according to this embodiment can be used, for example, to survey or inspect a building S1, which is an object to be photographed by the unmanned aerial vehicle 20. In this system 1, a user U using the information processing terminal 10 operates the touch panel of the information processing terminal 10 and captures an image including a wall W1 of the building S1 using the unmanned aerial vehicle 20. The information processing terminal 10 then defines an area of ​​the wall W1 included in the image displayed on the touch panel of the information processing terminal 10, and acquires position information of the wall W1 in world coordinates (real space) based on the area and various information captured by the unmanned aerial vehicle 20 at the time of capturing the image. Based on this position information, it is possible to estimate, for example, the length between any two points on the wall W1, the area of ​​any area, and the like.

[0013] The information processing terminal 10 according to this embodiment is implemented as a so-called small tablet-shaped computer. In other embodiments, the information processing terminal 10 may be implemented as a portable information processing terminal such as a smartphone or a game console, or as a stationary information processing terminal such as a personal computer. The information processing terminal 10 may also be implemented as a plurality of pieces of hardware, with functions distributed among them.

[0014] 2 is a block diagram showing the configuration of the information processing terminal 10 according to this embodiment. As shown in the figure, the information processing terminal 10 includes a control unit 11 and a touch panel 12, which is an example of a display unit.

[0015] The processor 11a is an arithmetic unit that controls the operation of the control unit 11, controls the transmission and reception of data between each element, and performs processes necessary for executing programs, etc. In this embodiment, the processor 11a is, for example, a CPU (Central Processing Unit), and performs various processes by executing programs stored in the storage 11c (described later) and expanded in the memory 11b.

[0016] The memory 11b includes a main storage device that is configured with a volatile storage device such as a DRAM (Dynamic Random Access Memory), and an auxiliary storage device that is configured with a nonvolatile storage device such as a flash memory or an HDD (Hard Disc Drive). This memory 11b is used as a working area for the processor 11a, and also stores a boot loader that is executed when the control unit 11 starts up, various setting information, etc.

[0017] The storage 11c stores programs, information used for various processes, etc. For example, when a user operates an aircraft for capturing image information of the wall surface W1 via the information processing terminal 10, the storage 11c may store a program for controlling the flight of the aircraft.

[0018] The transceiver 11d connects the control unit 11 to a network such as the Internet, and may include a local area network (LAN), a wide area network (WAN), infrared, wireless, WiFi, a point-to-point (P2P) network, a telecommunications network, cloud communication, or a short-range communication interface such as Bluetooth (registered trademark) or BLE (Bluetooth Low Energy).

[0019] The input / output unit 11e is an interface to which an input / output device is connected, and in this embodiment, the touch panel 12 is connected.

[0020] The bus 11f transmits, for example, address signals, data signals, and various control signals between the connected processor 11a, memory 11b, storage 11c, transmission / reception unit 11d, and input / output unit 11e.

[0021] The touch panel 12 is an example of a display unit and includes a display surface on which acquired videos and images are displayed. In this embodiment, the display surface accepts information input by touching the display surface, and is implemented using various technologies such as a resistive film system or a capacitive system.

[0022] For example, an image captured by the unmanned aerial vehicle 20 may be displayed on the display surface of the touch panel 12. Buttons, objects, etc. for controlling the flight of the unmanned aerial vehicle 20 or for controlling the imaging device may also be displayed on the display surface. A user may input information via the touch panel 12 to the images, buttons, etc. displayed on the display surface. Operations for inputting such input information include, for example, touching (tapping), sliding, or swiping buttons, objects, etc.

[0023] Fig. 3 is a block diagram showing an example of the functional configuration of the unmanned aerial vehicle 20 according to this embodiment. As shown in Fig. 3, the unmanned aerial vehicle 20 according to one embodiment includes a main body 21, a transceiver 22, a flight controller 23, a battery 24, an ESC 25, a motor 26, a propeller 27, and a camera 28. The unmanned aerial vehicle 20 is an example of an aerial vehicle. The type of the aerial vehicle is not particularly limited, and may be, for example, a multi-rotor drone as shown in Fig. 3.

[0024] The flight controller 23 may have one or more processors 23A, such as programmable processors (e.g., central processing units (CPUs)).

[0025] Flight controller 23 has and has access to memory 23B, which stores logic, code, and / or program instructions that the flight controller can execute to perform one or more steps.

[0026] The memory 23B may include a separable medium such as an SD card or random access memory (RAM) or an external storage device. Data acquired from the sensors 23C may be directly transmitted to and stored in the memory 23B. For example, still image and video data captured by the camera 28 is recorded in the built-in memory or an external memory.

[0027] The flight controller 23 includes a control module configured to control the state of the air vehicle. For example, the control module may have six degrees of freedom (translational x, y, and z, and rotational θ x , θ y and θ z The flight controller 23 controls the propulsion mechanism (motor 26, etc.) of the air vehicle via an ESC (Electric Speed ​​Controller) 25 to adjust the spatial arrangement, speed, and / or acceleration of the air vehicle having a control module 23C. The control module can control one or more of the camera 28, sensors 23C, etc. The flight controller 23 can also generate and store information related to the state of the air vehicle. The information related to the state of the air vehicle includes, for example, data acquired by the camera 28 and sensors 23C. The data acquired by the camera 28 includes, for example, image information generated by capturing an image by the camera 28, and information related to the imaging situation of the camera 28 (for example, the imaging position of the camera 28, the imaging direction of the camera 28). The imaging direction can include the angles of the pan direction and tilt direction of the camera 28, etc.

[0028] The flight controller 23 can communicate with a transceiver 22 configured to transmit and / or receive data from one or more external devices (e.g., a terminal such as the information processing terminal 10, a display device, a transmitter or other remote controller that remotely operates the unmanned air vehicle 20). For example, the transceiver 22 can utilize one or more of a local area network (LAN), a wide area network (WAN), infrared, wireless, WiFi, a point-to-point (P2P) network, a telecommunications network, cloud communication, etc.

[0029] The transmitter / receiver 22 can transmit and / or receive one or more of data acquired by the camera 28 or sensors 23C, processing results generated by the flight controller 23, predetermined control data, and user commands from the information processing terminal 10 or a remote controller. The transmitter / receiver 22 can, for example, receive input to the information processing terminal 10 and receive control related to flight and imaging via a transmitter (not shown). The transmitter / receiver 22 can also transmit data acquired by the camera 28 or the like to the information processing terminal via a transmitter (not shown).

[0030] The sensors 23C according to this embodiment may include an inertial sensor (acceleration sensor, gyro sensor), a GPS sensor, a proximity sensor (e.g., lidar), or a vision / image sensor (e.g., camera).

[0031] The battery 24 may be a known battery such as a lithium polymer battery. The power that drives the unmanned aerial vehicle 20 is not limited to the electric power supplied from the battery 24, but may be powered by an internal combustion engine, for example.

[0032] Camera 28 is an example of an imaging device. The type of camera 28 is not particularly limited, and may be, for example, a normal digital camera, a spherical camera, an infrared camera, a thermography image sensor, or the like. Camera 28 may be connected to main body 21 by a gimbal (not shown) or the like so as to be independently displaceable.

[0033] Fig. 4 is a block diagram showing the functional configuration of the control unit 11 according to this embodiment. As shown in Fig. 4, the control unit 11 includes an input information acquisition unit 111, a display control unit 112, an acquisition unit 113, an estimation unit 114, a calculation unit 115, and an image information DB (database) 116. Each of these functional units can be realized by the processor 11a reading a program stored in the storage 11c into the memory 11b and executing the program.

[0034] The input information acquisition unit 111 has a function of acquiring input information generated based on an operation on an image displayed on the touch panel 12. The input information here includes, for example, information about a position on the image displayed on the touch panel 12. The position on the image is, for example, the position of a pixel that constitutes the image. In other words, the input information includes information indicating at which position on the image the user has performed an operation. More specifically, the input information may include information about designation of a point on the image displayed on the touch panel 12 and information about a touch operation on a button object displayed on the touch panel 12.

[0035] The display control unit 112 has a function of displaying the acquired image on the touch panel 12. The display control unit 112 also has a function of displaying information such as buttons, objects, and text, etc., included in the image, for providing information to a user who uses the present system 1 and for acquiring input information based on an operation by the user. The display control unit 112 may also have a function of displaying, on the touch panel 12, an image based on the input information acquired by the input information acquisition unit 111.

[0036] The acquisition unit 113 has a function of acquiring input information and image information. For example, the acquisition unit 113 may acquire image information obtained by capturing an image of an object from the image information DB 116. The acquisition unit 113 may also acquire image information from the unmanned aerial vehicle 20 that is performing image capture processing in real time.

[0037] Furthermore, the acquiring unit 113 may acquire other information based on various types of information. Specifically, the acquiring unit 113 includes a first acquiring unit 1131 and a second acquiring unit 1132.

[0038] The first acquisition unit 1131 acquires coordinate information on an image displayed on the screen of the touch panel 12 of at least two reference points each having coordinate information in world coordinates. The reference points here may be assigned information related to coordinate information (e.g., latitude information, longitude information, altitude information, etc.) in world coordinates (coordinates in real space). The reference points may be set on the screen by, for example, an operation by the user U on the touch panel 12. The coordinate information in world coordinates may be assigned when setting the reference points on the screen, as described below, or may be assigned to a predetermined reference point. Furthermore, the coordinate information on the image may be, for example, coordinate information in an XY coordinate system determined based on the pixels of the image. Specific behavior will be described later.

[0039] The reference point may have, for example, coordinate information in a predetermined height direction in world coordinates. The coordinate information in the predetermined height direction may be coordinate information in the height direction of the ground. Furthermore, coordinate information in a planar direction of the reference point in world coordinates may be acquired based on coordinate information in the predetermined height direction, coordinate information of the reference point on the image, and information related to the imaging conditions when the image was captured. In other words, coordinate information in a planar direction of the reference point in world coordinates can be obtained from the coordinate information of the reference point on the image by performing a coordinate transformation process based on a transformation formula obtained from information related to the imaging conditions and coordinate information in the predetermined height direction.

[0040] The second acquisition unit 1132 has a function of acquiring coordinate information on the image of at least one input point input to the screen. The input point is a point different from the reference point. The input point can be set on the screen by, for example, an operation on the touch panel 12 by the user U.

[0041] The estimation unit 114 has a function of estimating a position in world coordinates and the like based on the coordinate information acquired by the acquisition unit 113. Specifically, the estimation unit 114 includes a first estimation unit 1141 and a second estimation unit 1142.

[0042] The first estimation unit 1141 has a function of estimating the position of a virtual wall surface in world coordinates that passes through at least two reference points, based on coordinate information in world coordinates of at least two reference points, coordinate information on the image, and information related to the imaging conditions when the image was captured. For example, if it is assumed that the (actual) wall surface of the object is at a predetermined angle (e.g., vertical) with respect to the ground, it can be assumed that the virtual wall surface is also at a similar predetermined angle. In this case, the first estimation unit 1141 can estimate the position of the virtual wall surface in world coordinates by, for example, performing a coordinate conversion process as described below. The position of the virtual wall surface in world coordinates means, for example, a group of coordinates in real space of an area corresponding to the virtual wall surface.

[0043] The second estimation unit 1142 has a function of estimating the world coordinates of the input point when the input point is assumed to be located on a virtual wall surface. For example, the second estimation unit 1142 can estimate the world coordinates of the input point by performing coordinate conversion processing for when the input point is assumed to be located on a virtual wall surface, based on coordinate information of the input point on an image.

[0044] The calculation unit 115 has a function of calculating the distance between a reference point and an input point on a virtual wall surface in world coordinates. The calculation unit 115 may also calculate the distance between an input point and another input point on the virtual wall surface in world coordinates. The calculation unit 115 may also calculate the area of ​​a region on the virtual wall surface, with points on the virtual wall surface in world coordinates that correspond to at least one of the reference point and the input point as vertices. Such a region may be a region with vertices corresponding to the reference point and the input point, or a region with vertices corresponding to multiple input points. A region with vertices corresponding to multiple input points may be, for example, a region formed by vertices consisting only of points corresponding to the input points. Note that such a region may be formed by at least one of straight lines and curves. That is, the region may have a polygonal shape or a shape that can be drawn freehand. The setting of such a region is performed, for example, by the calculation unit 115.

[0045] The image information DB 116 is a database that stores information (image information) about images captured by the unmanned aerial vehicle 20 or the like. Such images may be captured images obtained by the unmanned aerial vehicle 20, but are not limited to such examples. For example, such images may be captured images obtained by capturing images from a high place or the like using a digital camera, smartphone, tablet, or the like. In this case, if the capturing position, capturing direction, etc. can be obtained by any method, such information can be used as information related to the capturing situation.

[0046] Next, an example of a method for measuring the wall surface of an object using the information processing terminal 10 according to this embodiment will be described with reference to a flowchart. Fig. 5 is a flowchart illustrating a series of processes in the information processing terminal 10 according to this embodiment.

[0047] First, the acquisition unit 113 acquires image information and information related to the imaging situation (step SQ101). Here, it is assumed that the unmanned aerial vehicle 20 captures and acquires an image of the wall surface W1 of the building S1.

[0048] FIG. 6 is a diagram showing an example of imaging processing by the unmanned aerial vehicle 20. As shown in FIG. 6, the unmanned aerial vehicle 20 is flying near a building S1, and the camera 28 is facing in the direction PV1 of the wall surface W1. The camera 28 performs imaging processing so that the wall surface W1 is captured, and obtains an image. At this time, the unmanned aerial vehicle 20 acquires information related to the imaging situation, such as the height H1 of the camera 28, the imaging direction Dir1 in the horizontal direction of the camera 28, and the imaging angle Ang1 of the camera 28. The acquired image information and information related to the imaging situation are transmitted to the information processing terminal 10 via the unmanned aerial vehicle 20. The acquisition unit 113 acquires this information. The image information may be stored once in the image information DB 116. Furthermore, the information related to the imaging situation may be linked to the image information.

[0049] Next, the display control unit 112 displays an image on the screen of the touch panel 12 (step SQ103). FIG. 7 is an example of a display on screen V1 of the touch panel 12. Screen V1 displays image D11 captured by the unmanned aerial vehicle 20. Image D11 includes an image of a building S1. Building S1 includes a wall W1, a window W2, and a window W3. Screen V1 may also be provided with a button 101 for generating a virtual wall object, a button 102 for generating an area object corresponding to the window, and a button 103 for deleting an area that has been created.

[0050] Next, the input information acquisition unit 111 acquires an operation by the user U, and a reference point is set (step SQ105). The reference point is a point set to acquire the coordinates of the wall W1 in the world coordinate system, and at least two reference points are set. The reference points may be set on the image D11, for example, by an operation by the user U. FIG. 8 is a diagram showing an example of setting the reference points on the image D11. As shown in FIG. 8, two reference points 51 and 52 may be set on the image D11. For example, the two reference points 51 and 52 are set so as to correspond to the positions of the lower vertices of the wall W1 to be surveyed.

[0051] Next, the first acquisition unit 1131 associates the coordinates of the reference points on the image with the coordinates in the world coordinate system, and acquires coordinate information of the reference points in the world coordinate system (step SQ107). As shown in FIG. 8, the height coordinate (i.e., height from the ground) of the lower vertex of the wall W1 can be assumed to be 0 in the world coordinate system. The height coordinate of the lower vertex of the wall W1 may be a value other than 0, such as the actual altitude. The horizontal coordinates of the reference points 51 and 52 in the world coordinate system can be obtained using a known coordinate transformation method from the coordinates of the reference points 51 and 52 on the image, their height coordinates in the world coordinate system, and information related to the imaging situation of the camera 28. Specifically, the world coordinates of the reference points 51 and 52 can be obtained by converting the coordinates of the reference points 51 and 52 on the image to the film coordinates of the camera 28, converting the film coordinates to the camera coordinates of the camera 28, and converting the camera coordinates to the world coordinates. At this time, by substituting the height coordinates of reference points 51 and 52 in the world coordinate system into the height component of the coordinate conversion formula, the horizontal coordinates of reference points 51 and 52 in the world coordinate system can be obtained. The conversion formula between the respective coordinate systems can be determined based on the specifications of camera 28 and information related to the imaging conditions of camera 28. Note that in such coordinate conversion, a method may be used that takes into account information such as the specifications of camera 28, such as the F-number and distortion of the lens.

[0052] Next, the first estimation unit 1141 estimates the position of the virtual wall in world coordinates (step SQ109). FIG. 9 is a diagram for explaining the process of estimating the position of the virtual wall. As shown in FIG. 9, a virtual wall 60 passing through reference points 51 and 52 is virtually provided on the image D11. The virtual wall 60 may or may not actually be displayed on the screen V1. The virtual wall 60 may be calculated, for example, as follows. Here, the virtual wall 60 is assumed to be perpendicular to the ground, but the present technology is not limited to this example. First, assuming a virtual wall passing through reference points 51 and 52, the normal direction (yaw direction) of the virtual wall is calculated. Then, a coordinate system consisting of a line segment connecting reference points 51 and 52 and the calculated normal direction is defined as the coordinate system of the virtual wall. As a result, the coordinates of the virtual wall 60 on the image and the world coordinates are linked, and the position of the virtual wall 60 in the world coordinates is associated with the position of the actual wall W1 in real space.

[0053] Next, the input information acquisition unit 111 acquires an operation by the user U, and an input point is set (step SQ111). The second acquisition unit 1132 acquires coordinate information of the input point on the image. Then, the second estimation unit 1142 estimates the world coordinates of the input point when it is assumed that the input point is located on the virtual wall surface 60 (step SQ113). Then, an area having the reference point and the input point as vertices is set (step SQ115).

[0054] 10 is a diagram for explaining the processing related to setting of input points and setting of areas. As shown in FIG. 10, in addition to reference points 51 and 52, input points 53 and 54 are set on image D11. Here, the input points 53 and 54 are processed as being located on a virtual wall surface 60. The coordinates of the input points 53 and 54 in the world coordinate system are obtained by the second estimation unit 1142 converting the coordinates of the input points 53 and 54 on image D11 via the coordinate system of the virtual wall surface 60.

[0055] Also, an area 61 is set that is surrounded by the reference points 51 and 52 and the input points 53 and 54. This area 61 is an area that corresponds to the virtual wall surface 60, and is an area that corresponds to the wall surface W1 that appears in the image D11.

[0056] Then, the calculation unit 115 calculates the area surrounded by the region 61 (step SQ117). The calculation unit 115 calculates the area in real space of the region surrounded by the reference points 51 and 52 and the input points 53 and 54 using coordinate information in world coordinates of the points. The calculation unit 115 may also calculate the distance between the two points in real space. Information related to the area and distance may be output to the screen V1 or the like in any manner by the display control unit 112, for example.

[0057] The above has described the method for measuring the wall surface of an object using the information processing terminal 10 according to this embodiment. As a modified example, it is also possible to set (estimate) another area inside the area 61 set on the virtual wall surface 60.

[0058] FIG. 11 is a diagram illustrating processing related to area estimation according to a modified example of this embodiment. As shown in FIG. 11, an area 62 surrounded by four other input points 55, 56, 57, and 58 is set inside an area 61 (corresponding to wall W1) surrounded by reference points 51 and 52 and input points 53 and 54. This area 62 corresponds to a window W2 provided on wall W1 of building S1. The calculation unit 115 calculates the areas of the areas 61 and 62, and subtracts the area of ​​the area 62 from the area of ​​the area 61 to calculate the area of ​​wall W1 excluding window W2. Although not shown in FIG. 11, an area surrounded by multiple input points can also be set for window W3, similar to window W2.

[0059] As described above, the information processing terminal 10 according to this embodiment can estimate the position of a virtual wall surface corresponding to the world coordinates using at least two reference points whose world coordinates are known for an image of the wall surface W1 of a building S1 captured from the sky by the unmanned aerial vehicle 20, set an input point on the virtual wall surface, and estimate the position coordinates of the reference point and the input point in world coordinates. In particular, if the world coordinates of at least two reference points in the height direction (e.g., height from the ground) are determined, coordinate information in the world coordinates of the reference points can be obtained from information related to the imaging situation by coordinate transformation. By setting this reference point, for example, as a part of the building S1 that corresponds to the ground (i.e., a point with a height of zero), it is possible to automatically and uniquely determine the coordinate information in the world coordinates of the reference point set at a position corresponding to the installation part of the wall surface W1 of the building S1.

[0060] By setting a virtual wall surface from at least two reference points, it is possible to uniquely determine the coordinate information of an input point set on an image in world coordinates when the input point is located on the virtual wall surface. This makes it possible to calculate, for example, the area of ​​the wall surface in real space or the distance between two points. In this way, by utilizing the relationship between the shape of a building and its position in real space, it is possible to more accurately calculate the actual position, length, and area in real space even for a wall surface captured from an oblique direction.

[0061] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0062] The devices described in this specification may be realized as a single device, or may be realized by multiple devices, some or all of which are connected via a network. For example, the control unit and storage of the information processing terminal 10 may be realized by different servers connected to each other via a network.

[0063] The series of processes performed by the device described in this specification may be realized using software, hardware, or a combination of software and hardware. A computer program for realizing each function of the information processing terminal 10 according to this embodiment may be created and installed on a PC or the like. A computer-readable recording medium on which such a computer program is stored may also be provided. Examples of the recording medium include a magnetic disk, an optical disk, a magneto-optical disk, and a flash memory. The computer program may also be distributed, for example, via a network, without using a recording medium.

[0064] Furthermore, the processes described herein using flowchart diagrams do not necessarily have to be performed in the order shown. Some process steps may be performed in parallel. Additional process steps may be employed, and some process steps may be omitted.

[0065] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0066] The following configurations also fall within the technical scope of the present disclosure. (Item 1) a first acquisition unit that acquires coordinate information on an image displayed on the screen of at least two reference points that are displayed on the screen and each have coordinate information in world coordinates; a first estimation unit that estimates a position of a virtual wall surface in the world coordinate system that passes through the at least two reference points based on coordinate information of the at least two reference points in the world coordinate system, coordinate information on the image, and information related to an imaging situation when the image was captured; a second acquisition unit that acquires coordinate information on an image of an input point that is input to the screen; a second estimation unit that estimates world coordinates of the input point when the input point is assumed to be located on the virtual wall surface; An information processing device comprising: (Item 2) Item 1. The information processing device according to item 1, the reference point has coordinate information in a predetermined height direction in the world coordinate system, An information processing device, wherein coordinate information of the reference point in the planar direction in the world coordinate system is obtained based on coordinate information in the specified height direction, coordinate information of the reference point on the image, and information related to the imaging conditions when the image was captured. (Item 3) Item 2. The information processing device according to item 2, An information processing device, wherein the coordinate information in the predetermined height direction in the world coordinate system is coordinate information in the height direction of the ground. (Item 4) The information processing device according to any one of items 1 to 3, an information processing device further comprising a calculation unit that calculates a distance on the virtual wall surface in the world coordinate system between the reference point and the input point. (Item 5) The information processing device according to any one of items 1 to 4, The information processing device further includes a calculation unit that calculates an area of ​​a region on the virtual wall surface having vertices that are points on the virtual wall surface in the world coordinate system that correspond to at least one of the reference point and the input point. (Item 6) Item 5. The information processing device according to item 5, The information processing device, wherein the area includes an area having vertices at points corresponding to the reference point and the input point. (Item 7) Item 5 or 6, the information processing device according to item 5 or 6, The information processing device, wherein the area includes an area having vertices at points corresponding to the plurality of input points. (Item 8) The information processing device according to any one of items 1 to 7, An information processing device, wherein the information relating to the imaging situation includes information relating to an imaging position and an imaging direction of an imaging device. (Item 9) The information processing device according to any one of items 1 to 8, An information processing device, wherein the image includes a captured image obtained by capturing an image using an unmanned aerial vehicle. (Item 10) The processor: acquiring coordinate information on an image displayed on the screen of at least two reference points that are displayed on the screen and each have coordinate information in world coordinates; estimating a position of a virtual wall surface in the world coordinate system that passes through the at least two reference points based on coordinate information in the world coordinate system, coordinate information on the image, and information related to an imaging situation when the image was captured, of the at least two reference points; acquiring coordinate information on an image of an input point input to the screen; estimating world coordinates of the input point when the input point is assumed to be located on the virtual wall surface; A method comprising: (Item 11) Computer, a first acquisition unit that acquires coordinate information on an image displayed on the screen of at least two reference points that are displayed on the screen and each have coordinate information in world coordinates; a first estimation unit that estimates a position of a virtual wall surface in the world coordinate system that passes through the at least two reference points based on coordinate information of the at least two reference points in the world coordinate system, coordinate information on the image, and information related to an imaging situation when the image was captured; a second acquisition unit that acquires coordinate information on an image of an input point that is input to the screen; a second estimation unit that estimates world coordinates of the input point when the input point is assumed to be located on the virtual wall surface; A program to function as a [Explanation of symbols]

[0067] 1 System 10 Information processing terminal 11 Control section 12 Touch Panel 20 Unmanned Aerial Vehicles 28 Camera 111 Input information acquisition unit 112 Display control unit 113 Acquisition Department 114 Estimation Department 115 Calculation Unit 1131 First Acquisition Department 1132 Second Acquisition Department 1141 1st Estimation Department 1142 Second Estimation Department

Claims

1. a first acquisition unit that accepts designation of at least two reference points in a captured image displayed on a screen and acquires coordinate information of the at least two reference points on the captured image; a first estimation unit that estimates a position of a virtual wall surface in the world coordinate system that passes through the at least two reference points based on coordinate information in the world coordinate system that is predetermined and assigned to the at least two reference points or that is assigned when the at least two reference points are designated on the screen, coordinate information on the captured image, and information related to an imaging situation when the captured image is captured; a second acquisition unit that acquires coordinate information on the captured image of an input point that is input to the screen; a second estimation unit that estimates world coordinates of the input point when the input point is assumed to be located on the virtual wall surface; An information processing device comprising:

2. 2. The information processing device according to claim 1, the reference point has coordinate information in a predetermined height direction in the world coordinate system, An information processing device, wherein coordinate information of the reference point in the planar direction in the world coordinates is obtained based on coordinate information in the specified height direction, coordinate information of the reference point on the captured image, and information related to the imaging conditions when the captured image was captured.

3. 3. The information processing device according to claim 2, An information processing device, wherein the coordinate information in the predetermined height direction in the world coordinate system is coordinate information in the height direction of the ground.

4. The information processing device according to any one of claims 1 to 3, an information processing device further comprising a calculation unit that calculates a distance on the virtual wall surface in the world coordinate system between the reference point and the input point.

5. The information processing device according to any one of claims 1 to 4, The information processing device further includes a calculation unit that calculates an area of ​​a region on the virtual wall surface having vertices that are points on the virtual wall surface in the world coordinate system that correspond to at least one of the reference point and the input point.

6. 6. The information processing device according to claim 5, The information processing device, wherein the area includes an area having vertices at points corresponding to the reference point and the input point.

7. 7. The information processing device according to claim 5, The information processing device, wherein the area includes an area having vertices at points corresponding to the plurality of input points.

8. The information processing device according to any one of claims 1 to 7, An information processing device, wherein the information relating to the imaging situation includes information relating to an imaging position and an imaging direction of an imaging device.

9. The information processing device according to any one of claims 1 to 8, An information processing device, wherein the captured image includes an image obtained by capturing an image using an unmanned aerial vehicle.

10. The processor: Accepting designation of at least two reference points on a captured image displayed on a screen, and acquiring coordinate information of the at least two reference points on the captured image; estimating a position of a virtual wall surface in the world coordinate system that passes through the at least two reference points based on coordinate information in the world coordinate system that is predetermined and assigned to the at least two reference points or that is assigned when the at least two reference points are designated on the screen, coordinate information on the captured image, and information related to an imaging situation when the captured image is captured; acquiring coordinate information on the captured image of an input point input to the screen; estimating world coordinates of the input point when the input point is assumed to be located on the virtual wall surface; A method comprising:

11. Computer, a first acquisition unit that accepts designation of at least two reference points in a captured image displayed on a screen and acquires coordinate information of the at least two reference points on the captured image; a first estimation unit that estimates a position of a virtual wall surface in the world coordinate system that passes through the at least two reference points based on coordinate information in the world coordinate system that is predetermined and assigned to the at least two reference points or that is assigned when the at least two reference points are designated on the screen, coordinate information on the captured image, and information related to an imaging situation when the captured image is captured; a second acquisition unit that acquires coordinate information on the captured image of an input point that is input to the screen; a second estimation unit that estimates world coordinates of the input point when the input point is assumed to be located on the virtual wall surface; A program to function as a

Images