Methods, systems, and programs

Abstract

[Problem] To make it easy to set the flight path of a flight vehicle on site. [Solution] A method relating to the control of a flight vehicle, said method including: acquiring information about an image captured by an uncrewed flight vehicle 20; displaying the image on a touch panel 12; acquiring input information generated on the basis of an operation on the image displayed on the touch panel 12; and outputting to the uncrewed flight vehicle 20 information about flight control for flying to a position in actual space corresponding to a position on the image, the input information including information relating to the position on the image.

Description

Technical Field

[0001] The present disclosure relates to a method, a system, and a program.

Background Art

[0002] Needs for inspections, photography, etc. using unmanned aerial vehicles such as drones are increasing. Therefore, the development of technologies for easily operating unmanned aerial vehicles is underway. For example, Patent Document 1 discloses a technology for operating an unmanned aerial vehicle using a touch panel.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In order to perform photography using an aerial vehicle at a site such as an inspection or construction management, it is required to set an appropriate flight path on site.

[0005] The present disclosure has been made in view of such a background, and an object thereof is to provide a method, a system, and a program capable of easily setting a flight path of an aerial vehicle on site.

Means for Solving the Problems

[0006] According to the present disclosure, there is provided a method for controlling an aerial vehicle, including: acquiring information on an image captured by the aerial vehicle; displaying the image on a display unit; acquiring input information generated based on an operation on the image displayed on the display unit, the input information including information on a position on the image; and outputting to the aerial vehicle flight control information for flying to a position in real space corresponding to the position on the image.

[0007] Furthermore, according to this disclosure, a system for controlling an aircraft is provided, comprising: an image acquisition unit that acquires information of an image captured by the aircraft; a display control unit that displays the image on a display unit; an input information acquisition unit that acquires input information including information about a position on the image, which is generated based on an operation on the image displayed on the display unit; and an output control unit that outputs flight control information to the aircraft for flying at a position in real space corresponding to the position on the image.

[0008] Furthermore, according to this disclosure, a program is provided for causing a computer to function as a control device for an aircraft, the program comprising: an image acquisition unit that acquires information of an image captured by the aircraft; a display control unit that displays the image on a display unit; an input information acquisition unit that acquires input information including information about a position on the image, generated based on operations on the image displayed on the display unit; and an output control unit that outputs flight control information to the aircraft for flying at a position in real space corresponding to the position on the image. [Effects of the Invention]

[0009] According to the present invention, the flight path of an aircraft can be easily set on-site. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows a schematic diagram of System 1 according to one embodiment of the present disclosure. [Figure 2] This block diagram shows the configuration of the information processing terminal 10 according to the same embodiment. [Figure 3] This is a block diagram showing an example of the functional configuration of the unmanned aerial vehicle 20 according to the same embodiment. [Figure 4] This is a block diagram showing the functional configuration of the control unit 11 according to the same embodiment. [Figure 5] This is a flowchart illustrating a series of control operations in System 1 according to the same embodiment. [Figure 6]It is a diagram showing an example of a first situation related to the control method by the system 1 according to the same embodiment. [Figure 7] It is a diagram showing an example of a first screen displayed on the touch panel 12 according to the same embodiment. [Figure 8] It is a diagram showing an example of a second screen displayed on the touch panel 12 according to the same embodiment. [Figure 9] It is a diagram showing an example of a third screen displayed on the touch panel 12 according to the same embodiment. [Figure 10] It is a diagram showing an example of a fourth screen displayed on the touch panel 12 according to the same embodiment. [Figure 11] It is a diagram showing a modified example of the example of the fourth screen displayed on the touch panel 12 according to the same embodiment. [Figure 12] It is a diagram showing an example of a fifth screen displayed on the touch panel 12 according to the same embodiment. [Figure 13] It is a diagram showing an example of a sixth screen displayed on the touch panel 12 according to the same embodiment. [Figure 14] It is a diagram showing an example of a second situation related to the control method by the system 1 according to the same embodiment. [Figure 15] It is a diagram showing an example of a seventh screen displayed on the touch panel 12 according to the same embodiment. [Figure 16] It is an example of a control method by the system 1 according to a modified example of the same embodiment.

Mode for Carrying Out the Invention

[0011] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted.

[0012] <System Overview> FIG. 1 is a diagram showing an overview of a system 1 according to an embodiment of the present disclosure. As shown in the figure, the system 1 includes an information processing terminal 10 and an unmanned aerial vehicle 20. The system 1 according to the present embodiment can be used, for example, for construction management and inspection of a building S1, which is an object to be photographed by the unmanned aerial vehicle 20. In such a system 1, a user U using the information processing terminal 10 operates on the touch panel of the information processing terminal 10 to trace the flight path of the unmanned aerial vehicle 20. Such an operation can be a continuous or intermittent operation such as a slide or a swipe. The unmanned aerial vehicle 20 can be controlled to fly at a position in real space estimated from the position information on the image obtained by such tracing operation.

[0013] The information processing terminal 10 according to the present embodiment is implemented by a so-called tablet-sized small computer. In other embodiments, the information processing terminal 10 may be realized by a portable information processing terminal such as a smartphone or a game machine, or may be realized by a stationary information processing terminal such as a personal computer. Further, the information processing terminal 10 may be realized by a plurality of hardware and have a configuration in which functions are distributed among them.

[0014] FIG. 2 is a block diagram showing the configuration of the information processing terminal 10 according to the present embodiment. As shown in the figure, the information processing terminal 10 includes a control unit 11 and a touch panel 12 which is an example of a display unit.

[0015] The processor 11a is an arithmetic device that controls the operation of the control unit 11 and performs processing such as control of data transmission and reception between each element and processing necessary for program execution. In the present embodiment, this processor 11a is, for example, a CPU (Central Processing Unit), and executes a program stored in a storage 11c described later and developed in a memory 11b to perform each processing.

[0016] Memory 11b comprises a main memory consisting of a volatile storage device such as DRAM (Dynamic Random Access Memory), and an auxiliary storage device consisting of a non-volatile storage device such as flash memory or an HDD (Hard Disk Drive). This memory 11b is used as a workspace for the processor 11a, while also storing the BIOS (Basic Input / Output System) executed when the control unit 11 starts up, and various setting information.

[0017] Storage 11c stores information used for programs and various processes. For example, if a user operates an aircraft for capturing image information of the roof 101 via an information processing terminal 10, storage 11c may store a program that controls the flight of such an aircraft.

[0018] The transmitting / receiving unit 11d connects the control unit 11 to a network such as the Internet, and may be equipped with a short-range communication interface such as Bluetooth® or BLE (Bluetooth Low Energy).

[0019] The input / output unit 11e is an interface to which input / output devices are connected, and in this embodiment, the touch panel 12 is connected.

[0020] The bus 11f transmits, for example, address signals, data signals, and various control signals between the connected processor 11a, memory 11b, storage 11c, transceiver 11d, and input / output unit 11e.

[0021] The touch panel 12 is an example of a display unit and includes a display surface on which acquired video and images are displayed. In this embodiment, this display surface accepts information input by contact with the display surface and is implemented using various technologies such as resistive touch or capacitive touch.

[0022] The display surface of the touch panel 12 may, for example, display an image captured by the unmanned aerial vehicle 20. The display surface may also display buttons, objects, etc., for controlling the flight of the unmanned aerial vehicle 20 or the imaging device. Furthermore, the user can input information via the touch panel 12 to the images, buttons, etc., displayed on the display surface. Details of such input information will be described later, but operations related to inputting such information include, for example, touch (tap) operations on buttons and objects, and tap, slide, and swipe operations to determine the flight path of the unmanned aerial vehicle 20.

[0023] Figure 3 is a block diagram showing an example of the functional configuration of the unmanned aerial vehicle 20 according to this embodiment. As shown in Figure 3, the unmanned aerial vehicle 20 according to one embodiment includes a main body 21 comprising a transmitting / receiving unit 22, a flight controller 23, a battery 24, an ESC 25, a motor 26, a propeller 27, and a camera 28. Note that the unmanned aerial vehicle 20 is just one example of an aircraft. The type of aircraft is not particularly limited, and for example, it may be a multi-rotor type so-called drone as shown in Figure 3.

[0024] The flight controller 23 may have one or more processors 23A, such as a programmable processor (e.g., a central processing unit (CPU)).

[0025] The flight controller 23 has a memory 23B, which is accessible. The memory 23B stores logic, code, and / or program instructions that the flight controller can execute to perform one or more steps.

[0026] Memory 23B may include, for example, a separable medium such as an SD card or random access memory (RAM), or an external storage device. Data acquired from sensors 23C may be directly transmitted to and stored in memory 23B. For example, still images and video data captured by camera 28 are recorded in the internal memory or external memory.

[0027] The flight controller 23 includes a control module configured to control the state of the aircraft. For example, the control module has 6 degrees of freedom (translational motion x, y, and z, and rotational motion θ). x θ y and θ z The propulsion mechanism (motor 26, etc.) of the aircraft is controlled via an ESC (Electric Speed ​​Controller) 25 to adjust the spatial arrangement, speed, and / or acceleration of the aircraft having the following: The control module can control one or more of the cameras 28 and sensors 23C, etc.

[0028] The flight controller 23 can communicate with a transceiver 22 configured to transmit and / or receive data from one or more external devices (e.g., terminals such as the information processing terminal 10, display devices, or other remote controllers). For example, the transceiver 22 can utilize one or more of the following: local area networks (LANs), wide area networks (WANs), infrared, wireless, Wi-Fi, point-to-point (P2P) networks, telecommunications networks, cloud communication, etc.

[0029] The transmitting / receiving unit 22 can transmit and / or receive one or more of the following: data acquired by the camera 28 and sensors 23C, processing results generated by the flight controller 23, predetermined control data, and user commands from the information processing terminal 10 or a remote controller.

[0030] The sensors 23C according to this embodiment may include inertial sensors (accelerometers, gyroscopes), GPS sensors, proximity sensors (e.g., LiDAR), or vision / image sensors (e.g., cameras).

[0031] The battery 24 may be a known battery, such as a lithium polymer battery. The power source for driving the unmanned aerial vehicle 20 is not limited to electricity supplied from the battery 24, but may also be powered by, for example, an internal combustion engine.

[0032] Camera 28 is an example of an imaging device. The type of camera 28 is not particularly limited and may be, for example, a regular digital camera, a 360-degree camera, an infrared camera, a thermographic image sensor, etc. Camera 28 may be connected to the main body 21 so as to be independently displaceable by a gimbal or the like (not shown).

[0033] Figure 4 is a block diagram showing the functional configuration of the control unit 11 according to this embodiment. As shown in Figure 4, the control unit 11 includes an image acquisition unit 111, a display control unit 112, an input information acquisition unit 113, a flight control information generation unit 114, a camera control information generation unit 115, and an output control unit 116. Each of these functional units can be realized by the processor 11a reading a program stored in the storage 11c into the memory 11b and executing it.

[0034] The image acquisition unit 111 has the function of acquiring information from images captured by the unmanned aerial vehicle 20. The image acquisition unit 111 appropriately acquires images captured by the camera 28 mounted on the unmanned aerial vehicle 20. The acquired images may be moving images obtained in real time, or still images captured at any given time. The information from the acquired images is output to the display control unit 112.

[0035] The display control unit 112 has the function of displaying the acquired image on the touch panel 12. The display control unit 112 also has the function of including information such as buttons, objects, and text in the image for providing information to the user of the system 1 and for acquiring input information based on user operations. Furthermore, it may also have the function of displaying information obtained by the input information acquisition unit 113 (described later) on the touch panel 12.

[0036] The input information acquisition unit 113 has the function of acquiring input information generated based on operations performed on an image displayed on the touch panel 12. The input information here includes, for example, information about the position on the image displayed on the touch panel 12. The position on the image is, for example, the position of a pixel that makes up the image. In other words, the input information includes information indicating which position on the image the user performed an operation on.

[0037] Information regarding positions on the image includes, for example, information about line segments consisting of a continuous set of positions on the image. Such line segment information is obtained when the user performs an operation such as tracing on the touch panel 12. For example, when an image of a space captured from above by an unmanned aerial vehicle 20 is displayed on the touch panel 12, the user performs an operation such as tracing the position where they want the unmanned aerial vehicle 20 to fly. The trajectory obtained by this tracing can become the flight route of the unmanned aerial vehicle 20. Note that such tracing operations do not necessarily have to be continuous; they may be intermittent. That is, even if the user performs tracing operations at multiple locations, a flight route can be formed based on these trajectories. In the parts where the tracing operation is interrupted, a path corresponding to the flight route can be appropriately supplemented.

[0038] Furthermore, the input information may include information regarding the target imaging position on the image for the camera 28 mounted on the unmanned aerial vehicle 20. The target imaging position on the image for the camera 28 is the position used to determine the imaging direction of the camera 28 during the flight of the unmanned aerial vehicle 20. The user can select at least one location in the image of the space displayed on the touch panel 12 as the target imaging position.

[0039] Furthermore, the input information may include information regarding the vertical position (altitude) of the unmanned aerial vehicle 20 in real space during flight controlled by flight control. The position of the camera 28 during flight can be determined from this vertical position, the horizontal position of the unmanned aerial vehicle 20 in real space during flight, and the target position of the camera 28.

[0040] Furthermore, the input information may include information regarding the speed of the unmanned aircraft 20 during flight controlled by flight control. Other information input via the touch panel 12 or the like may be acquired as input information by the input information acquisition unit 113 as appropriate.

[0041] The flight control information generation unit 114 has the function of generating information (flight control information) for controlling the flight of the unmanned aerial vehicle 20 based on the input information. The flight control information includes, for example, information regarding the flight path of the unmanned aerial vehicle 20, the start and end points of the flight path, the orientation of the unmanned aerial vehicle 20 (around the pitch axis, roll axis, and yaw axis), the flight altitude of the unmanned aerial vehicle 20, and the flight speed of the unmanned aerial vehicle 20. Flight control as used here means controlling the flight of the unmanned aerial vehicle 20 according to the flight control information generated based on the input information according to this embodiment that is input to the touch panel 12.

[0042] The flight control information generation unit 114 calculates, for example, the position in real space corresponding to the position on the image included in the input information. The position information in real space is, for example, latitude information and longitude information. Such latitude information and longitude information can be calculated based on the imaging position information of the camera 28 of the unmanned aerial vehicle 20 (or the unmanned aerial vehicle 20). Specifically, the flight control information generation unit 114 can calculate the position in real space corresponding to each pixel of the image, using the pixel at the center of the captured image as a reference, based on the field of view of the camera 28 and the height position of the unmanned aerial vehicle 20 at the time the camera 28 captured the image while hovering.

[0043] The flight control information generation unit 114 calculates the flight position (flight path) in real space, then determines the flight speed and orientation of the unmanned aerial vehicle 20 at that flight position, and generates it as flight control information. The flight speed may be determined, for example, by input based on user operation. The orientation during flight may be determined, for example, according to the orientation of the camera 28 determined by the camera control information generation unit 115 described later, or the orientation during flight may be constant if the camera 28 is connected to the main body 21 by a gimbal or the like. Control of the behavior of the unmanned aerial vehicle 20 during flight can be achieved by known technologies.

[0044] The camera control information generation unit 115 has the function of generating information (camera control information) for controlling the operation of the camera 28 based on the input information. The camera control information includes, for example, information regarding the orientation of the camera 28, the timing of image capture, the image capture process, etc.

[0045] The camera control information generation unit 115 generates information for controlling the orientation (imaging direction) of the camera 28 based on its position in real space corresponding to the imaging target position, for example, during the flight of the unmanned aerial vehicle 20 which flies based on flight control information. The information for controlling the orientation of the camera 28 includes, for example, information regarding the orientation of the camera 28 at the flight position calculated based on the imaging target position acquired by the input information acquisition unit 113 and the position of the unmanned aerial vehicle 20 in flight. The orientation of the camera 28 in at least the horizontal direction can be calculated based on the relationship between the imaging target position and the flight position. If there are multiple imaging target positions, for example, an imaging target position corresponding to the flight position at a certain timing may be separately defined. Also, the imaging target position may specify only the position in the horizontal direction, for example. In this case, the angle of the camera 28 around the pitch axis can be adjusted as appropriate, as will be described later. In addition, the orientation of the camera 28 can be controlled as appropriate during flight other than flight control, for example, in response to input from the user to the touch panel 12. Also, the imaging target position may include the position in the height direction in real space. If such a height position is set, the angle of the camera 28 around the pitch axis can be automatically determined.

[0046] The output control unit 116 has the function of outputting various information generated by the flight control information generation unit 114 and the camera control information generation unit 115 to the unmanned aerial vehicle 20. Based on the acquired flight control information and camera control information, the unmanned aerial vehicle 20 controls its own flight and the operation of the camera 28.

[0047] Next, an example of a control method for the unmanned aerial vehicle 20 using the system 1 according to this embodiment will be described with reference to a flowchart. Figure 5 is a flowchart diagram of a series of control operations in the system 1 according to this embodiment.

[0048] First, the image acquisition unit 111 acquires image information from the unmanned aerial vehicle 20 (step SQ101). At this time, the unmanned aerial vehicle 20 is hovering above the object to be imaged.

[0049] Figure 6 shows an example of a first situation relating to the control method by System 1 according to this embodiment. As shown in Figure 6, the unmanned aerial vehicle 20 is hovering above the building S1, which is the object to be imaged, at an altitude H1. The imaging direction of the camera 28 is directly downwards. Here, altitude H1 is the altitude of the camera 28, but altitude H1 may be the altitude of the unmanned aerial vehicle 20.

[0050] Figure 7 shows an example of a first screen displayed on the touch panel 12 according to this embodiment. As shown in Figure 7, the screen V1 of the touch panel 12 displays an information bar D11, a main screen D12, a sub-screen D13, and a number of buttons and objects 101 to 109. The information bar D11 is an area mainly for displaying information about the touch panel 12 and the unmanned aerial vehicle 20. In the example shown in Figure 7, the information bar D11 displays the radio wave status of the touch panel 12, the battery level, and the current altitude of the unmanned aerial vehicle 20. The main screen D12 may display images captured by the camera 28, etc. Other information may be displayed superimposed on the captured images on the main screen D12. The sub-screen D13 is an area for displaying images captured by the camera 28. In this embodiment, the sub-screen D13 may be used to supplementarily display images used to determine the flight path, for example. For example, the setting of the flight path can be started by tapping the sub-screen D13. Specific examples will be described later.

[0051] The following describes buttons 101-107 and objects 108 and 109. Button 101 is used to land the unmanned aerial vehicle 20. Button 102 is used, for example, to move the unmanned aerial vehicle 20 to the starting point of a set flight path. Button 103 is used to start control of the flight of the unmanned aerial vehicle 20 along the said flight path. Button 104 is used to perform imaging processing by the camera 28. Button 105 is used to adjust the altitude of the unmanned aerial vehicle 20. Button 106 is used to adjust the horizontal position of the unmanned aerial vehicle 20. Button 107 is used to display and edit the set path. Object 108 is used to adjust the horizontal orientation (around the yaw axis) of the unmanned aerial vehicle 20. Object 109 is used to adjust the angle of the camera 28 around the pitch axis. The captured image displayed on the main screen D12 includes images of the building S1, the plaza S2, and the road S3, which are the target areas for imaging.

[0052] Next, the user taps sub-screen D13 to start the process related to inputting the flight path (step SQ103). Figure 8 is a diagram showing an example of a second screen displayed on the touch panel 12 according to this embodiment. The screen shown in Figure 8 is for inputting the flight path of the unmanned aerial vehicle 20. This screen V1 displays the captured image and buttons 201 to 204. Button 201 is a button for setting the flight path. Button 202 is a button for setting the center point (target position for imaging) of the captured image. Button 203 is a button for setting the flight speed of the unmanned aerial vehicle 20. Button 204 is a button for saving the settings.

[0053] In the screen V1 shown in Figure 8, selecting button 201 allows the user to input points and lines corresponding to the flight path onto screen V1. For example, the user can operate screen V1 using their finger or a stylus pen to specify the position on the image corresponding to the flight path. Such operations could be intuitive sliding operations, such as drawing with a pen.

[0054] Figure 9 shows an example of a third screen displayed on the touch panel 12 according to this embodiment. As shown in Figure 9, a trajectory 210 can be drawn on screen V1 by user operation. Points 211 indicating the start and end points of the trajectory 210 may be displayed in other ways. These points 211 indicating the start and end points may, for example, be the start and end points in the flight control of the unmanned aerial vehicle 20. The start and end points of the trajectory (line segment) may or may not coincide. Also, if the start and end points are in different positions when the trajectory 210 is drawn, one of the start and end points may be adjusted to be in the same position as the other. By making the start and end points coincide, the images (frames) taken during the flight of the unmanned aerial vehicle 20 will coincide at the start and end points. This makes it possible to obtain more useful images. Also, the location of either the start or end point on the trajectory 210 may be changeable. Furthermore, the trajectory 210, once drawn, may be modified as needed.

[0055] Next, when the user taps button 202, processing related to inputting the image target position is started (step SQ105). Figure 10 is a diagram showing a fourth screen example displayed on the touch panel 12 according to this embodiment. As shown in Figure 10, when the user taps button 202, they can select an image target position (also called a center point) 212 on the image. Such an image target position 212 on the image can be adjusted, for example, by tapping or sliding. The image target position 212 determined here corresponds to at least a horizontal position in real space. That is, during flight control, the camera 28 of the unmanned aircraft 20 will continue to face the direction in the horizontal direction where the image target position in real space corresponding to the image target position 212 on the image exists.

[0056] In the example shown in Figure 10, the imaging target position 212 was a single point, but this technology is not limited to such an example. Figure 11 shows a modified example of a fourth screen displayed on the touch panel 12 according to this embodiment. As shown in Figure 11, the imaging target position may be selected not as a point, but as a line segment or the like consisting of multiple points, such as the imaging target position 213. Also, similar to the point 211 that indicates the start and end points of the trajectory 210, the start and end points of the imaging target position 213 may be set as point 214. This makes it possible to dynamically change the direction in which the camera 28 of the unmanned aerial vehicle 20 faces the imaging target during flight control.

[0057] Next, when the user taps button 203, the process for inputting the flight speed during flight controlled by the flight control of the unmanned aerial vehicle 20 is initiated (step SQ107). Figure 12 is a diagram showing a fifth example screen displayed on the touch panel 12 according to this embodiment. As shown in Figure 12, when the user taps button 203, they can select the flight speed at which the unmanned aerial vehicle 20 flies according to the flight path corresponding to the trajectory 210. In the example shown in Figure 12, for example, the flight speed can be selected by selecting one of the low-speed button 215, the medium-speed button 216, and the high-speed button 217. The method of setting the flight speed is not limited to this example; for example, the flight speed can also be specified numerically.

[0058] Tapping button 204 completes the setting of the flight path and the camera's target area.

[0059] Next, returning to screen V1 as shown in Figure 7, the user selects to start the process of inputting the flight altitude during flight controlled by the flight control of the unmanned aerial vehicle 20 (step SQ109). Figure 13 is a diagram showing a sixth example screen displayed on the touch panel 12 according to this embodiment. Screen V1 shown in Figure 13 displays a form 120 for setting the flight altitude of the unmanned aerial vehicle 20 during flight controlled by the flight control when the user taps button 105 on screen V1 shown in Figure 7. By inputting an altitude into this form 120, the flight altitude controlled by the flight control is determined. The flight altitude may be fixed or variable during flight. If the altitude is variable, the altitude may be set appropriately using a form other than form 120. The flight altitude may also be predetermined.

[0060] In steps SQ103 to SQ109, the information obtained through the user's operation on the touch panel 12 can be output as input information to the flight control information generation unit 114 and the camera control information generation unit 115 as appropriate.

[0061] Next, the flight control information generation unit 114 generates flight control information for flying at a real-space position corresponding to the trajectory 210 based on the obtained input information (step SQ111). The camera control information generation unit 115 also generates camera control information for the camera 28 to continuously face and capture an image at a real-space position corresponding to the image target position 212 on the image, based on the obtained input information (step SQ113). This information is output to the unmanned aerial vehicle 20 by the output control unit 116.

[0062] Next, when the user taps button 102, the unmanned aerial vehicle 20 starts moving to the starting point of the flight path based on the flight control information (steps SQ115, SQ117). Figure 14 is a diagram showing an example of a second situation related to the control method by system 1 according to this embodiment. Here, the unmanned aerial vehicle 20 flies in a circle above the building S1 at an altitude H2 according to the flight control information. At that time, the unmanned aerial vehicle 20 can control its attitude during flight according to the camera control information so that the image target position C1 is in the image direction of the camera 28.

[0063] Next, when the user taps button 103, the unmanned aerial vehicle 20 starts flying based on flight control information (step SQ119). At that time, the camera 28 takes an image of the unmanned aerial vehicle in flight while facing the direction of the image target position C1 (step SQ121). Figure 15 is a diagram showing an example of a seventh screen displayed on the touch panel 12 according to this embodiment. On the main screen D12 of screen V1 shown in Figure 15, images captured by the camera 28 of the unmanned aerial vehicle 20, which is flying based on flight control, are displayed as appropriate. Such images may be moving images or still images taken at predetermined intervals. In addition, the sub-screen D13 may display an image taken during hovering. Furthermore, the sub-screen D13 may also display a trajectory 130 corresponding to the flight path previously set by user input. In addition, an object 131 corresponding to the unmanned aerial vehicle 20 may be displayed on the sub-screen D13 at a position corresponding to the current position of the unmanned aerial vehicle 20 as it flies along the flight path. This makes it possible to check in real time (or when viewing images after the flight) which point along the flight path the image displayed on the main screen D12 was captured from.

[0064] The above describes an example of a control method for an unmanned aerial vehicle 20 using System 1 according to this embodiment. Although the above embodiment describes an example of a control method for the actual flight of the unmanned aerial vehicle 20, this technology is not limited to such examples. For example, this technology can be used for simulations when a virtual drone is flown in a VR (Virtual Reality) or AR (Augmented Reality) space. By simulating the flight behavior of such an unmanned aerial vehicle 20 through flight control using VR or AR, it is possible to confirm in advance the imaging direction of the camera 28 when the unmanned aerial vehicle 20 is actually flown. The simulation technology using VR or AR space can be one of known technologies.

[0065] As described above, according to the system 1 of this embodiment, position information corresponding to the flight path can be intuitively input to the user interface via the touch panel 12. Furthermore, by simply inputting the position information on the screen of the touch panel 12, the unmanned aerial vehicle 20 can be made to fly along the desired flight path. This makes it easy to set the flight path of the aircraft on site. In addition, by setting the position of the target to be imaged for determining the camera's orientation on the touch panel 12, the orientation of the camera during flight can also be easily determined. This makes it easy to image objects such as buildings using the unmanned aerial vehicle. The flight control information and camera control information that have been set may be stored in the storage 11c or the like. This makes it possible to repeatedly fly the unmanned aerial vehicle under the same conditions during inspections, etc., and to image the target object from the same position and angle.

[0066] In the above embodiment, the image captured by the unmanned aerial vehicle 20 during hovering was obtained by setting the imaging direction of the camera 28 mounted on the unmanned aerial vehicle 20 to the ground direction. However, this technology is not limited to this example. Figure 16 shows an example of a control method by system 1 according to one modification of this embodiment. As shown in Figure 16, in such system 1, the imaging direction of the camera 28 during hovering of the unmanned aerial vehicle 20 may be horizontal with respect to the structure S4. In this case, the unmanned aerial vehicle 20 may be controlled to image the structure S4 with the camera 28 while flying on a plane parallel to the height direction, with height H3 as the reference. In this case, the imaging direction of the camera 28 may be determined, for example, by setting the position at height H3 on the structure S4 as the imaging target position. In this way, system 1 according to this embodiment can be applied regardless of the positional relationship between the unmanned aerial vehicle 20 and the structure that is the imaging target.

[0067] While preferred embodiments of the present disclosure have been described in detail above with reference to the attached drawings, the technical scope of the present disclosure is not limited to such examples. It is clear to any person with ordinary skill in the art of the present disclosure that various modifications or alterations may be conceived within the scope of the technical idea set forth in the claims, and these will naturally also fall within the technical scope of the present disclosure.

[0068] The devices described herein may be implemented as a single device, or as a group of devices, some or all of which are connected via a network. For example, the control unit and storage of the information processing terminal 10 may be implemented as different servers connected to each other via a network.

[0069] The series of processes performed by the apparatus described herein may be implemented using software, hardware, or a combination of software and hardware. Computer programs for implementing each function of the information processing terminal 10 according to this embodiment can be created and implemented on a PC or the like. Furthermore, a computer-readable recording medium containing such a computer program can also be provided. Examples of recording media include magnetic disks, optical disks, magneto-optical disks, and flash memory. Alternatively, the computer program may be distributed without using a recording medium, for example, via a network.

[0070] Furthermore, the processes described using flowcharts in this specification do not necessarily have to be executed in the order shown. Some processing steps may be executed in parallel. Additional processing steps may be adopted, and some processing steps may be omitted.

[0071] Furthermore, the effects described herein are merely descriptive or illustrative and not limiting. In other words, the technology relating to this disclosure may produce other effects that will be apparent to those skilled in the art from the description herein, in addition to or in lieu of the effects described herein.

[0072] Furthermore, the following configurations also fall within the technical scope of this disclosure. (Item 1) A method relating to the control of an aircraft, To acquire information from images captured by the aircraft, The aforementioned image is displayed on the display unit, The system acquires input information generated based on operations performed on the image displayed on the display unit, and the input information includes information about the position on the image. The aircraft outputs flight control information for flying to a position in real space corresponding to the position on the aforementioned image, A method that includes this. (Item 2) The method described in item 1, The input information includes information about line segments consisting of a continuous set of positions on the image, A method for outputting flight control information to an aircraft for flying along a route in real space corresponding to a line segment on the aforementioned image. (Item 3) The method described in item 2, The information of the line segment includes information about the start and end points of the line segment. A method for adjusting the position of the endpoint so that it matches the position of the starting point when the position of the endpoint obtained by the operation on the image differs from the position of the starting point. (Item 4) A method described in any one of items 1 to 3, The input information includes information relating to the imaging target position on the image of the imaging device mounted on the aircraft, A method further comprising outputting to the aircraft information for controlling the orientation of the imaging device based on the imaging target position in real space corresponding to the imaging target position, while the aircraft is flying based on the flight control information. (Item 5) The method described in item 4, The method wherein the imaging target positions are specified in multiple or consecutive ways on the image. (Item 6) A method described in item 4 or 5, The method wherein the imaging target position includes a position in the height direction in real space. (Item 7) A method described in any one of items 1 to 6, Information regarding the height position of the aircraft at the time the aforementioned image was captured is obtained, A method in which the position in real space corresponding to the position in the image is determined based on the position of the aircraft in the height direction at the time the image was captured. (Item 8) A method described in any one of items 1 to 7, The input information includes information relating to the height position of the aircraft in real space during flight controlled by the flight control method. (Item 9) A method described in any one of items 1 to 8, A method further comprising displaying an image captured by an imaging device mounted on the aircraft, which is flying based on the aforementioned flight control, on the display unit. (Item 10) A method described in any one of items 1 to 9, A method further comprising displaying an image previously displayed on the display unit in another area of ​​the display unit while the aircraft is flying based on the aforementioned flight control. (Item 11) The method described in item 10, A method further comprising displaying an object corresponding to the aircraft in the other region at an image position in the other region that corresponds to the aircraft's flight position in real space, based on the flight control. (Item 12) A method described in any one of items 1 to 11, A method further comprising displaying the results of a simulation of the flight behavior of the aircraft under the flight control on the display unit. (Item 13) A method described in any one of items 1 to 12, The flight control information includes information relating to the speed of the aircraft while it is flying. A method for performing operations on the image displayed on the display unit, including an operation for determining the speed. (Item 14) A method described in any one of items 1 to 13, The image captured by the aircraft while it is hovering is an image obtained by setting the imaging direction of the imaging device mounted on the aircraft to the ground direction. (Item 15) A system relating to the control of an aircraft, An image acquisition unit that acquires information from images captured by the aircraft, A display control unit that displays the aforementioned image on a display unit, An input information acquisition unit that acquires input information including information about the position on the image, which is generated based on an operation on the image displayed on the display unit, An output control unit that outputs flight control information to the aircraft for flying to a position in real space corresponding to the position on the image, A system equipped with these features. (Item 16) A program that makes a computer function as a control system for an aircraft. An image acquisition unit that acquires information from images captured by the aircraft, A display control unit that displays the aforementioned image on a display unit, An input information acquisition unit that acquires input information including information about the position on the image, which is generated based on an operation on the image displayed on the display unit, An output control unit that outputs flight control information to the aircraft for flying to a position in real space corresponding to the position on the image, A program that makes it function as such. [Explanation of symbols]

[0073] 1 System 10 Information Processing Terminals 11 Control Unit 12 Touch panel 20 Unmanned aircraft 28 Cameras 111 Image acquisition unit 112 Display Control Unit 113 Input Information Acquisition Unit 114 Flight control information generation unit 115 Camera control information generation unit

Claims

1. A method relating to the control of an aircraft, To acquire information from images captured by the aircraft, The aforementioned image is displayed on the display unit, This includes obtaining input information generated based on operations performed on the image displayed on the display unit, The input information includes information relating to the position on the image and information relating to the imaging target position on the image of the imaging device mounted on the aircraft, The aforementioned imaging target position includes a position in the height direction in real space. The aforementioned target imaging positions are specified in multiple or consecutive ways on the image. The aircraft outputs flight control information for flying to a position in real space corresponding to the position on the aforementioned image, During the flight of the aircraft, which is flying based on the flight control information, information for controlling the orientation of the imaging device based on the imaging target position in real space corresponding to the imaging target position is output to the aircraft. Methods that further include this.

2. The method according to claim 1, The input information includes information about line segments consisting of a continuous set of positions on the image, A method for outputting flight control information to an aircraft for flying along a route in real space corresponding to a line segment on the aforementioned image.

3. The method according to claim 2, The information of the line segment includes information about the start and end points of the line segment. A method for adjusting the position of the endpoint so that it matches the position of the starting point when the position of the endpoint obtained by the operation on the image differs from the position of the starting point.

4. A method according to any one of claims 1 to 3, Information regarding the height position of the aircraft at the time the aforementioned image was captured is obtained, A method in which the position in real space corresponding to the position in the image is determined based on the position of the aircraft in the height direction at the time the image was captured.

5. A method according to any one of claims 1 to 4, The input information includes information relating to the height position of the aircraft in real space during flight controlled by the flight control method.

6. A method according to any one of claims 1 to 5, A method further comprising displaying an image captured by an imaging device mounted on the aircraft, which is flying based on the aforementioned flight control, on the display unit.

7. A method according to any one of claims 1 to 6, A method further comprising displaying an image previously displayed on the display unit in another area of ​​the display unit while the aircraft is flying based on the aforementioned flight control.

8. The method according to claim 7, A method further comprising displaying an object corresponding to the aircraft in the other region at an image position in the other region that corresponds to the aircraft's flight position in real space, based on the flight control.

9. A method according to any one of claims 1 to 8, A method further comprising displaying the results of a simulation of the flight behavior of the aircraft under the flight control on the display unit.

10. A method according to any one of claims 1 to 9, The flight control information includes information relating to the speed of the aircraft while it is flying. A method for performing operations on the image displayed on the display unit, including an operation for determining the speed.

11. A method according to any one of claims 1 to 10, The image captured by the aircraft while it is hovering is an image obtained by setting the imaging direction of the imaging device mounted on the aircraft to the ground direction.

12. A system relating to the control of an aircraft, An image acquisition unit that acquires information from images captured by the aircraft, A display control unit that displays the aforementioned image on a display unit, An input information acquisition unit that acquires input information including information about the position on the image, which is generated based on an operation on the image displayed on the display unit, and information about the target position on the image of an imaging device mounted on the aircraft, An output control unit outputs flight control information to the aircraft for flying to a position in real space corresponding to the position on the image, and during the flight of the aircraft flying based on the flight control information, outputs information to the aircraft for controlling the orientation of the imaging device based on the imaging target position in real space corresponding to the imaging target position. Equipped with, The aforementioned imaging target position includes a position in the height direction in real space. The system specifies multiple or sequential imaging target positions on the image.

13. A program that makes a computer function as a control system for an aircraft. An image acquisition unit that acquires information from images captured by the aircraft, A display control unit that displays the aforementioned image on a display unit, An input information acquisition unit that acquires input information including information about the position on the image, which is generated based on an operation on the image displayed on the display unit, and information about the target position on the image of an imaging device mounted on the aircraft, An output control unit outputs flight control information to the aircraft for flying to a position in real space corresponding to the position on the image, and during the flight of the aircraft flying based on the flight control information, outputs information to the aircraft for controlling the orientation of the imaging device based on the imaging target position in real space corresponding to the imaging target position. To make it function as, The aforementioned imaging target position includes a position in the height direction in real space. The program specifies multiple or sequential imaging target positions on the image.

Images