Method, system and program

Abstract

To easily confirm a flight restricted area of a flying object at a site.SOLUTION: This method for setting and utilizing a flight restricted area includes steps of: displaying a pickup image obtained by being imaged by a camera 28, etc. on a touch panel 12, the pickup image being associated with information concerning a position of the camera 28, etc. in a three-dimensional space and a posture of the camera 28, etc.; acquiring information concerning a position of the flight restricted area of an unmanned flying object 20 set in the three-dimensional space; and superposing an object concerning the flight restricted area with the pickup image obtained from the unmanned flying object flying in the flight restricted area to be displayed on the touch panel 12.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to methods, systems and programs. [Background technology]

[0002] There is a growing need for inspections, photography, and the like using unmanned aerial vehicles such as drones. Therefore, development of technologies for easily operating unmanned aerial vehicles is underway. For example, Patent Document 1 discloses a technology for operating an unmanned aerial vehicle using a touch panel. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-85041 Summary of the Invention [Problem to be solved by the invention]

[0004] At inspection and photography sites, it is sometimes necessary to restrict the area in which an aircraft can fly as much as possible. Even if the restricted area is set when setting the flight path of an unmanned aircraft, it is difficult to visually grasp it in real space.

[0005] The present disclosure has been made in light of this background, and its purpose is to provide a method, system, and program that can easily check flight restriction areas for aircraft at the site. [Means for solving the problem]

[0006] According to the present disclosure, there is provided a method including: displaying an image obtained by capturing an image using an imaging device on a display device; linking the captured image to information relating to the position of the imaging device in three-dimensional space and the attitude of the imaging device; acquiring information relating to the position of a flight restriction zone of an aircraft set in the three-dimensional space; and superimposing an object relating to the flight restriction zone on the captured image obtained from the aircraft flying within the flight restriction zone and displaying the image on the display device.

[0007] Furthermore, according to the present disclosure, there is provided a system related to display control, the system including: a display control unit that causes a captured image obtained by an imaging device to be displayed on a display device; an input information acquisition unit that acquires information related to the position of a flight restricted zone of an aircraft set in the three-dimensional space, to which information related to the position of the imaging device in three-dimensional space and the attitude of the imaging device is linked; and an output control unit that superimposes an object related to the flight restricted zone on the captured image obtained from the aircraft flying within the flight restricted zone, and displays the image on the display device.

[0008] Furthermore, according to the present disclosure, there is provided a program that causes a computer to function as a display control unit that causes a captured image obtained by an imaging device to be displayed on a display device; an input information acquisition unit that acquires information related to the position of a flight restricted zone of an aircraft set in three-dimensional space, to which information related to the position of the imaging device in three-dimensional space and the attitude of the imaging device is linked, and an output control unit that superimposes an object related to the flight restricted zone on a captured image obtained from the aircraft flying within the flight restricted zone, and displays the image on the display device. [Effects of the Invention]

[0009] According to the present invention, it is possible to easily check the flight restriction zone of an aircraft on-site. [Brief explanation of the drawings]

[0010] [Figure 1]1 is a diagram illustrating an overview of a system 1 according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a block diagram showing a configuration of an information processing terminal 10 according to the embodiment. [Figure 3] A block diagram showing an example of the functional configuration of the unmanned aerial vehicle 20 according to the embodiment. [Figure 4] 2 is a block diagram showing the functional configuration of a control unit 11 according to the embodiment. FIG. [Figure 5] 4 is a flowchart illustrating a series of controls in the system 1 according to the embodiment. FIG. [Figure 6] 10 is a diagram showing a first example screen of the touch panel 12 relating to the method for setting a flight restriction zone by the system 1 according to the embodiment. FIG. [Figure 7] 10 is a diagram showing a second example of a screen of the touch panel 12 relating to the method for setting a flight restriction zone by the system 1 according to the embodiment. FIG. [Figure 8] 10 is a flowchart showing a series of controls in a method for outputting information related to flight restriction zones using the system 1 according to the embodiment. FIG. [Figure 9] 10 is a diagram showing a first example of a screen of the touch panel 12 relating to a method for outputting information relating to flight restriction zones by the system 1 according to the embodiment. FIG. [Figure 10] 10 is a diagram showing a second example of a screen of the touch panel 12 relating to the method of outputting information relating to flight restriction zones by the system 1 according to the embodiment. FIG. [Figure 11] 10 is a diagram showing a third example of a screen of the touch panel 12 relating to the method of outputting information relating to flight restriction zones by the system 1 according to the embodiment. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0011] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0012] <System Overview> FIG. 1 is a diagram illustrating an overview of a system 1 according to an embodiment of the present disclosure. As illustrated, the system 1 includes an information processing terminal 10 and an unmanned aerial vehicle 20. The system 1 according to this embodiment can be used, for example, to photograph an area FRA1 including a building S1, etc. In this system 1, a user U using the information processing terminal 10 performs operations on the touch panel of the information processing terminal 10 to control the flight of the unmanned aerial vehicle 20. At that time, an image including the above-mentioned area, captured by a camera 28 mounted on the unmanned aerial vehicle 20, is displayed on the touch panel. In this embodiment, for example, information about the flight restriction area of ​​the unmanned aerial vehicle 20 can be displayed superimposed on the image displayed on the touch panel. This display allows the user U to understand the areas in which the unmanned aerial vehicle 20 is / is not flyable by looking at the touch panel.

[0013] The information processing terminal 10 according to this embodiment is implemented as a so-called small tablet-shaped computer. In other embodiments, the information processing terminal 10 may be implemented as a portable information processing terminal such as a smartphone or a game console, or as a stationary information processing terminal such as a personal computer. The information processing terminal 10 may also be implemented as a plurality of pieces of hardware, with functions distributed among them.

[0014] 2 is a block diagram showing the configuration of the information processing terminal 10 according to this embodiment. As shown in the figure, the information processing terminal 10 includes a control unit 11 and a touch panel 12, which is an example of a display unit.

[0015] The processor 11a is an arithmetic unit that controls the operation of the control unit 11, controls the transmission and reception of data between each element, and performs processes necessary for executing programs, etc. In this embodiment, the processor 11a is, for example, a CPU (Central Processing Unit), and performs various processes by executing programs stored in the storage 11c (described later) and expanded in the memory 11b.

[0016] The memory 11b includes a main storage device that is configured with a volatile storage device such as a DRAM (Dynamic Random Access Memory), and an auxiliary storage device that is configured with a nonvolatile storage device such as a flash memory, an HDD (Hard Disc Drive), etc. This memory 11b is used as a working area for the processor 11a, and also stores a BIOS (Basic Input / Output System) that is executed when the control unit 11 starts up, various setting information, etc.

[0017] The storage 11c stores programs, information used for various processes, etc. For example, when a user operates an aircraft for capturing images of an area including a building S1, etc., via the information processing terminal 10, the storage 11c may store a program for controlling the flight of the aircraft.

[0018] The transmitter / receiver 11d connects the controller 11 to a network such as the Internet, and may include a short-range communication interface such as Bluetooth (registered trademark) or BLE (Bluetooth Low Energy).

[0019] The input / output unit 11e is an interface to which an input / output device is connected, and in this embodiment, the touch panel 12 is connected.

[0020] The bus 11f transmits, for example, address signals, data signals, and various control signals between the connected processor 11a, memory 11b, storage 11c, transmission / reception unit 11d, and input / output unit 11e.

[0021] The touch panel 12 is an example of a display unit and includes a display surface on which acquired videos and images are displayed. In this embodiment, the display surface accepts information input by touching the display surface, and is implemented using various technologies such as a resistive film system or a capacitive system.

[0022] For example, an image captured by the unmanned aerial vehicle 20 may be displayed on the display surface of the touch panel 12. Information regarding the flight restriction zone of the unmanned aerial vehicle 20 may be displayed on the display surface superimposed on the image captured by the unmanned aerial vehicle 20. Buttons, objects, etc. for controlling the flight of the unmanned aerial vehicle 20 and the imaging device may also be displayed on the display surface. A user may input information via the touch panel 12 to the images, buttons, etc. displayed on the display surface.

[0023] Note that, although the present embodiment cites the touch panel 12 as an example of a display unit, the present technology is not limited to such an example. For example, the display unit may be realized by a display device such as another display, a monitor, or a smartphone. Furthermore, an input device for acquiring input information may be realized by a method other than the touch panel 12. For example, instead of the touch panel 12, the input device may be realized by various input devices such as a mouse, a keyboard, a voice recognition device, or an eye gaze recognition device. Furthermore, the information processing terminal 10 may be realized independently of at least one of a display device such as the touch panel 12 and an input device. In this case, for example, the information processing terminal 10 may be realized by one or more servers such as a cloud server, and may be provided to be able to communicate with the touch panel 12.

[0024] Fig. 3 is a block diagram showing an example of the functional configuration of the unmanned aerial vehicle 20 according to this embodiment. As shown in Fig. 3, the unmanned aerial vehicle 20 according to one embodiment includes a main body 21, a transceiver 22, a flight controller 23, a battery 24, an ESC 25, a motor 26, a propeller 27, and a camera 28. The unmanned aerial vehicle 20 is an example of an aerial vehicle. The type of the aerial vehicle is not particularly limited, and may be, for example, a multi-rotor drone as shown in Fig. 3.

[0025] The flight controller 23 may have one or more processors 23A, such as programmable processors (e.g., central processing units (CPUs)).

[0026] Flight controller 23 has and has access to memory 23B, which stores logic, code, and / or program instructions that the flight controller can execute to perform one or more steps.

[0027] The memory 23B may include a separable medium such as an SD card or random access memory (RAM) or an external storage device. Data acquired from the sensors 23C may be directly transmitted to and stored in the memory 23B. For example, still image and video data captured by the camera 28 is recorded in the built-in memory or an external memory.

[0028] The flight controller 23 includes a control module configured to control the state of the air vehicle. For example, the control module may have six degrees of freedom (translational x, y, and z, and rotational θ x , θ y and θ z The control module controls the propulsion mechanism (motor 26, etc.) of the aircraft via an ESC (Electric Speed ​​Controller) 25 to adjust the spatial arrangement, speed, and / or acceleration of the aircraft. The control module can control one or more of the camera 28, sensors 23C, etc.

[0029] The flight controller 23 can communicate with a transceiver 22 configured to transmit and / or receive data from one or more external devices (e.g., a terminal such as the information processing terminal 10, a display device, or other remote control). For example, the transceiver 22 can utilize one or more of a local area network (LAN), a wide area network (WAN), infrared, radio, WiFi, a point-to-point (P2P) network, a telecommunications network, cloud communications, etc.

[0030] The transceiver unit 22 can transmit and / or receive one or more of the following: data acquired by the camera 28 or sensors 23C, processing results generated by the flight controller 23, predetermined control data, user commands from the information processing terminal 10 or a remote controller, etc.

[0031] The sensors 23C according to this embodiment may include an inertial sensor (acceleration sensor, gyro sensor), a GPS sensor, a proximity sensor (e.g., lidar), or a vision / image sensor (e.g., camera).

[0032] The battery 24 may be a known battery such as a lithium polymer battery. The power that drives the unmanned aerial vehicle 20 is not limited to the electric power supplied from the battery 24, but may be powered by an internal combustion engine, for example.

[0033] Camera 28 is an example of an imaging device. The type of camera 28 is not particularly limited, and may be, for example, a normal digital camera, a spherical camera, an infrared camera, a thermography image sensor, or the like. Camera 28 may be connected to main body 21 by a gimbal (not shown) or the like so as to be independently displaceable.

[0034] Fig. 4 is a block diagram showing the functional configuration of the control unit 11 according to this embodiment. As shown in Fig. 4, the control unit 11 includes a map acquisition unit 111, an image acquisition unit 112, a display control unit 113, an input information acquisition unit 114, a flight restriction zone determination unit 115, a coordinate conversion unit 116, and an output control unit 117. Each of these functional units can be realized by the processor 11a reading a program stored in the storage 11c into the memory 11b and executing the program. Furthermore, the map information 118 can be stored in, for example, the storage 11c, an external storage, a cloud server, or the like.

[0035] The map acquisition unit 111 has a function of acquiring map information 118. The map information 118 includes information related to a map. The map information is, for example, information in which location information (including at least one of longitude information, latitude information, and altitude information) is associated with a map or an aerial photograph (which may include one captured by the unmanned aerial vehicle 20). The map information 118 may include not only geographical information but also information on the type of structure or area (such as a residence, park, public facility, commercial facility, industrial facility, or river). The map information 118 may be acquired using, for example, an API (Application Programming Interface) provided by a map-related service. In this case, the map information 118 may be updated as needed by, for example, a user performing an operation to search for a location where a flight restriction zone should be set by inputting information to the touch panel 12 on which the map is displayed. The acquired map information 118 is output to the display control unit 113.

[0036] The image acquisition unit 112 has a function of acquiring information about images captured by the unmanned aerial vehicle 20. The image acquisition unit 112 appropriately acquires images captured by the camera 28 mounted on the unmanned aerial vehicle 20. The acquired images may be moving images acquired in real time or still images captured at any timing. Information about the acquired images is output to the display control unit 113. Note that the image information may include position information (including at least one of longitude information, latitude information, and altitude information) of the unmanned aerial vehicle 20 at the time of capture and attitude information (information such as the attitude angle and direction of the camera 28 and / or the unmanned aerial vehicle 20). Note that the image does not necessarily need to include such information. For example, the system 1 may acquire information about flight parameters including position information, attitude information, etc. when a certain captured image was captured. In this case, the system 1 may link the captured image to the information about the flight parameters.

[0037] The display control unit 113 has a function to display the acquired images and information on the touch panel 12. The display control unit 113 also has a function to display, in the image, information such as buttons, objects, and text for providing information to the user of the present system 1 and for acquiring input information based on operations by the user. The display control unit 113 may also have a function to display, on the touch panel 12, an image based on input information acquired by an input information acquisition unit 114, which will be described later.

[0038] For example, the display control unit 113 may have a function to display a map related to the acquired map information 118. This causes the map to be displayed on the touch panel 12. The display control unit 113 may also have a function to display the captured image acquired by the image acquisition unit 112.

[0039] The input information acquisition unit 114 has a function of acquiring input information generated based on an operation on an image displayed on the touch panel 12. The input information here includes, for example, information about a position on the image displayed on the touch panel 12. The position on the image is, for example, the position of a pixel that constitutes the image. In other words, the input information includes information indicating at which position on the image the user performed an operation. Such operations include, for example, tapping, touching, swiping, sliding, and the like.

[0040] Such input information may include, for example, input information related to the setting of a flight restricted area. As will be described in detail later, input information related to the setting of a flight restricted area is, for example, information (designated area information) generated by an operation to set a designated area corresponding to the flight restricted area of ​​the unmanned aerial vehicle 20 on a map when the area in which the unmanned aerial vehicle 20 may fly is displayed on the touch panel 12. Such input information may include, for example, information related to the registration, change, or deletion of a flight restricted area.

[0041] The input information may also include input information for controlling the flight of the unmanned aerial vehicle 20. For example, if buttons, objects, etc. for operating the flight of the unmanned aerial vehicle 20 are displayed on the touch panel 12 by the display control unit 113, the input information for controlling the flight of the unmanned aerial vehicle 20 is information generated by operating the buttons or objects. Such input information may include, for example, information for controlling the takeoff, landing, ascent, descent, translation, rotation, speed, etc. of the unmanned aerial vehicle 20, as well as information for controlling the imaging process, zoom, and imaging direction of the camera 28.

[0042] The flight restricted area determination unit 115 has a function of determining a flight restricted area in real space based on input information related to the setting of the flight restricted area. For example, the flight restricted area determination unit 115 determines a flight restricted area in real space based on designated area information included in the input information. The flight restricted area can be identified, for example, by position information (latitude information and longitude information) of the designated area on a map.

[0043] The coordinate conversion unit 116 has a function of converting the flight restricted zone into position coordinates in an image captured by the unmanned aerial vehicle 20. For example, the coordinate conversion unit 116 obtains the coordinates of the flight restricted zone on the captured image based on the latitude information and longitude information of the flight restricted zone determined by the flight restricted zone determination unit 115. This makes it possible to recognize, for example, which part of the captured image is the flight restricted zone on the touch panel 12 described below.

[0044] The output control unit 117 has a function of outputting information related to the flight restriction zone in real space. The manner in which the output control unit 117 outputs the information is not particularly limited. For example, the output control unit 117 may output the information to the touch panel 12. The information may be displayed, for example, superimposed on a captured image captured by the unmanned aerial vehicle 20. More specifically, the output control unit 117 may display the information on the touch panel 12 in the form of an AR (augmented reality) display that indicates a position (e.g., an area) corresponding to at least either the inside or the outside of the flight restriction zone, superimposed on the captured image.

[0045] Furthermore, the output control unit 117 may control the display of an object for moving the unmanned aerial vehicle 20 when the object is displayed superimposed on a captured image captured by the unmanned aerial vehicle 20. Specifically, when the object is located outside the flight restriction zone in the captured image (i.e., a no-fly zone), the output control unit 117 may control the display of the object in a manner corresponding to the object being unable to be set as a movement target position. The manner corresponding to the object being unable to be set may include, for example, changing the color or shape of the object, or making it impossible to select an object for completing the setting of a movement target. Furthermore, the output control unit 117 may control the display of the object for setting a movement target position so that the object can be moved based on an operation on the touch panel 12 only to a position inside the flight restriction zone in the captured image.

[0046] The output control unit 117 may also output information related to flight restriction zones in real space to the unmanned aerial vehicle 20. For example, the output control unit 117 may control the unmanned aerial vehicle 20 to fly only within the area inside the flight restriction zone. This allows the unmanned aerial vehicle 20 to fly only within the area corresponding to the flight restriction zone, for example.

[0047] Next, an example of a method for setting a flight restricted area using the system 1 according to this embodiment will be described with reference to a flowchart. Fig. 5 is a flowchart showing a series of controls in the method for setting a flight restricted area using the system 1 according to this embodiment.

[0048] First, the map acquisition unit 111 acquires the map information 118 (step SQ101). Then, the acquired map is displayed on the touch panel 12 by the display control unit 113 (step SQ103).

[0049] FIG. 6 is a diagram showing an example of a first screen of the touch panel 12 relating to the method for setting a flight restricted zone by the system 1 according to this embodiment. As shown in FIG. 6, a map M1 is displayed on the screen V1 of the touch panel 12. Also, a button 101 for modifying the designated area is displayed on the screen V1. The user can set a designated area on the map M1 by operating the touch panel 12. Also, the area displayed on the map M1 can be changed or enlarged or reduced by performing inputs such as sliding or pinching on the map M1.

[0050] Next, the input information acquisition unit 114 acquires information related to the designated area through input to the touch panel 12 (step SQ105). FIG. 7 is a diagram showing a second example screen of the touch panel 12 relating to the method for setting a flight restriction zone by the system 1 according to this embodiment. As shown in FIG. 7, a user can set multiple pins 201 on the map M1 through operation on the touch panel 12. Then, a designated area 202 can be set with each pin 201 as a vertex. Note that the method for setting the designated area 202 is not particularly limited; for example, the designated area may be set freehand. Furthermore, if a similar area has been set in the past, the designated area may be set by retrieving information set in the past. The designated area 202 may be confirmed, for example, when the designation of the pins 201 is completed and an enclosed area is generated.

[0051] Next, the flight restricted area determination unit 115 determines a flight restricted area in real space based on the confirmed designated area 202 (step SQ107). Information related to such flight restricted area may be temporarily stored in the storage 13, for example.

[0052] Next, an example of a method for outputting information related to flight restriction zones using the system 1 according to this embodiment will be described with reference to a flowchart. Fig. 8 is a flowchart showing a series of controls in the method for outputting information related to flight restriction zones using the system 1 according to this embodiment.

[0053] First, the unmanned aerial vehicle 20 captures an image of the flight restriction zone and the surrounding area using the onboard camera 28 (step SQ201). The captured image obtained by this imaging process is acquired by the image acquisition unit 112 (step SQ203). At that time, the position of the unmanned aerial vehicle 20, etc., when the captured image was captured is also acquired. Information related to the position can be used in the conversion process between coordinates on the captured image and coordinates in real space, which will be described later.

[0054] Next, the display control unit 113 displays the captured image on the touch panel 12 (step SQ205). Note that the captured image may be a moving image received in real time from the unmanned aerial vehicle 20, or may be a still image.

[0055] Next, the coordinate conversion unit 116 associates the coordinates on the captured image with real-space coordinates (step SQ207). This process is for displaying an area corresponding to a flight restriction zone on the captured image, which will be described later. The association between the coordinates on the captured image (local coordinates) and real-space coordinates (global coordinates) can be performed using a known technique such as mapping transformation.

[0056] Next, the coordinate conversion unit 116 converts the flight restriction zone into coordinates on the captured image based on the above association (step SQ209). Through this process, an area on the captured image corresponding to the flight restriction zone is uniquely determined.

[0057] Next, the output control unit 117 displays the area outside the flight restriction zone on the touch panel 12, superimposed on the captured image (step SQ211).

[0058] FIG. 9 is a diagram showing a first example screen of the touch panel 12 relating to the method of outputting information related to the flight restricted zone by the system 1 according to this embodiment. As shown in FIG. 9, a captured image P1 captured by the unmanned aerial vehicle 20 is displayed on the screen V1 of the touch panel 12. This captured image P1 is obtained by capturing an image with the camera 28 of the unmanned aerial vehicle 20 pointing directly downward. Then, an area FRA10 inside the flight restricted zone and an area FRA11 outside the flight restricted zone are superimposed on the captured image P1. outside The left and right areas FRA11 are displayed on either side of the boundary line 203 of the flight restriction zone. Screen V1 also displays a button 103 for landing the unmanned aerial vehicle 20, a button 104 for moving the unmanned aerial vehicle 20 up and down, a button 105 for rotating the unmanned aerial vehicle 20 around the yaw axis, an object 106 for adjusting the shooting direction of the camera 28 mounted on the unmanned aerial vehicle 20, an object 107 for adjusting the degree of wide-angle / telephoto of the camera 28, an object 108 for positioning the target position for movement of the unmanned aerial vehicle 20, and a button 109 for starting movement of the unmanned aerial vehicle 20.

[0059] As shown in FIG. 9, the area FRA10 inside the flight restricted area and the area FRA11 outside the flight restricted area may be displayed in other manners. In the example shown in FIG. 9, no image processing is performed on the area FRA10 inside the flight restricted area, whereas an object may be displayed on the area FRA10 outside the flight restricted area. In this way, it becomes possible to visually recognize the inside and outside of the flight restricted area from the viewpoint of the unmanned aerial vehicle 20 on the touch panel 12. In other words, it becomes possible to visually recognize the position in real space corresponding to the flight restricted area that the user set on the interface of the touch panel 12 while the unmanned aerial vehicle 20 is flying. Note that the display positions of the areas inside and outside the flight restricted area may also change depending on the movement of the unmanned aerial vehicle 20.

[0060] FIG. 10 is a diagram illustrating a second example screen of the touch panel 12 relating to a method for outputting information related to flight restricted zones by the system 1 according to this embodiment. In the example illustrated in FIG. 10, the captured image P2 is obtained by capturing an image with the camera 28 of the unmanned aerial vehicle 20 facing obliquely, rather than directly downward. Even when the capturing direction is oblique, it is possible to display information related to the flight restricted zone on the captured image P2, distinguishing between an area FRA10 inside the flight restricted zone and an area FRA11 outside the flight restricted zone. In this case, for example, information related to the capturing angle of the camera 28 may be included as position information contained in the captured image. Based on this information related to the capturing angle, the coordinate conversion unit 116 associates the coordinates of the flight restricted zone with the coordinates of the captured image, thereby displaying the area on the captured image P2. In this case, the position information of the flight restricted zone may include predetermined altitude information. This allows the position of the flight restricted zone to be superimposed on the captured image P2 to be uniquely determined.

[0061] FIG. 11 is a diagram showing a third example screen of the touch panel 12 relating to a method for outputting information related to flight restricted zones by the system 1 according to this embodiment. The captured image P1 shown in FIG. 11 is the same as the captured image P1 shown in FIG. 9. Here, a case will be described in which an operation is performed to move an object 108, which is used to position the movement target position of the unmanned aerial vehicle 20, into an area FRA11 outside the flight restricted zone. As shown in FIG. 11, when an attempt is made to move the object 108 into the area FRA11 outside the flight restricted zone, the appearance of the object 108 changes, and a display such as object 108a appears on the screen V1. In particular, object 108a may be displayed in a manner that makes it impossible to set it in this position. This allows the user to intuitively understand that the unmanned aerial vehicle 20 cannot be moved into that area. As another example, when the object 108 is located in a position included in the area FRA11 outside the flight restricted zone, the button 109 for starting the movement of the unmanned aerial vehicle 20 may be changed and displayed in a manner that makes it unselectable. This prevents the user from moving the unmanned aerial vehicle 20 outside the flight restriction area.

[0062] An example of a control method for the unmanned aerial vehicle 20 by the system 1 according to this embodiment has been described above. Note that in the above embodiment, an example of a control method for the actual flight of the unmanned aerial vehicle 20 has been described, but the present technology is not limited to such an example. For example, the present technology can be used for a simulation of flying a virtual drone in a VR (Virtual Reality) or AR (Augmented Reality) space. By using VR or AR to simulate the flight behavior of the unmanned aerial vehicle 20 through flight control, it is possible to confirm in advance the imaging direction of the camera 28 when the unmanned aerial vehicle 20 is actually flown. Note that known simulation techniques using a VR space or an AR space can be used.

[0063] As described above, according to the system 1 of this embodiment, a flight restricted zone can be set on the touch panel 12, and information about the flight restricted zone can be obtained via the touch panel 12. For example, the flight restricted zone can be displayed in an AR-like manner by being superimposed on an image captured by the unmanned aerial vehicle 20. This allows the user operating the touch panel 12 to set the flight restricted zone within which the unmanned aerial vehicle 20 can fly and intuitively grasp its location. For example, even if the shooting angle of the camera 28 of the unmanned aerial vehicle 20 changes, such a display can be displayed by superimposing an object indicating the flight restricted zone at an appropriate position on the captured image.

[0064] In the above embodiment, the inside and outside of the flight restricted zone are displayed in an AR-like manner using objects or the like, but the present technology is not limited to such an example. For example, when the unmanned aerial vehicle 20 is approaching the boundary of the flight restricted zone, an alert indicating that the unmanned aerial vehicle is approaching the boundary may be output by the touch panel 12. Furthermore, when it is determined that the unmanned aerial vehicle 20 has flown outside the flight restricted zone or that flight control is being performed in such a way that the unmanned aerial vehicle 20 will exceed the boundary of the flight restricted zone within a predetermined time, the unmanned aerial vehicle 20 may be controlled to automatically head inside the flight restricted zone.

[0065] A modified example of the above embodiment will be described below. In the above embodiment, when setting the destination and flight route for the flight of the unmanned aerial vehicle 20, a predetermined flight restriction zone (or set by the function of the above embodiment) is superimposed and displayed on an image captured by the unmanned aerial vehicle 20. In this modified example, while the unmanned aerial vehicle 20 flies along such a flight route, etc., the area corresponding to the above flight restriction zone can be continuously displayed by the unmanned aerial vehicle 20 as superimposed on an image captured by the unmanned aerial vehicle 20.

[0066] The system 1 according to this modification may have the configuration of the control unit 11 shown in FIG. 4, for example. Furthermore, the system 1 according to this modification repeatedly performs a series of processes shown in steps SQ201 to SQ211 in FIG. 8. That is, the system 1 displays on the touch panel 12 an image captured by the camera 28 of the unmanned aerial vehicle 20, acquires information related to the position of the flight restriction zone of the unmanned aerial vehicle 20 set in three-dimensional space (real space), and, based on the information related to the position and attitude of the camera 28 (or the unmanned aerial vehicle 20) in three-dimensional space and the information related to the position of the flight restriction zone in three-dimensional space, displays on the touch panel 12 information indicating the flight restriction zone (e.g., an object indicating the boundary of the flight restriction zone) at a position corresponding to the three-dimensional space on the captured image. The captured image is linked to information related to the position and attitude of the camera 28 (or the unmanned aerial vehicle 20) in three-dimensional space. In this way, while the unmanned aerial vehicle 20 is flying (including at least one of while moving and while hovering), the system 1 acquires captured images obtained by continuous or intermittent imaging processing by the unmanned aerial vehicle 20 and position information of the unmanned aerial vehicle 20, and executes this series of processes. As a result, even while the unmanned aerial vehicle 20 is flying, the flight restriction zone is displayed superimposed on the captured images.

[0067] An object indicating the position of the unmanned aerial vehicle 20 may be displayed on the superimposed screen of the captured image and flight restricted zone displayed by the touch panel 12. Such an object may, for example, indicate the position of the unmanned aerial vehicle 20 in a planar coordinate system (e.g., a coordinate system defined by latitude and longitude). This allows the user to visually grasp the position of the unmanned aerial vehicle 20 itself and the distance from the flight restricted zone. Furthermore, if the captured image is taken at an angle that satisfies predetermined conditions (e.g., from an oblique angle), the flight restricted zone may be displayed three-dimensionally. In this case, information related to the flight restricted zone includes altitude information on the boundary, and the object indicating the boundary of the flight restricted zone may be displayed in the form of a wall. This allows the user to grasp the flight restricted zone in the height direction as well.

[0068] Furthermore, if it is determined that the unmanned aerial vehicle 20 flies outside the flight restricted zone or that flight control is being performed in a way that will cause the unmanned aerial vehicle 20 to exceed the boundary of the flight restricted zone within a predetermined time, the unmanned aerial vehicle 20 may be controlled to automatically head toward the inside of the flight restricted zone. That is, the system 1 (specifically, the output control unit 117) may determine the distance between the unmanned aerial vehicle 20's own position and the flight restricted zone based on information about the unmanned aerial vehicle's own position and information about the flight restricted zone, and perform control of the unmanned aerial vehicle 20 based on this distance. Such control may be, for example, control to output an alert via the touch panel 12 (auditory information such as voice, tactile information such as vibration, or visual information such as a warning screen), or flight control of the unmanned aerial vehicle 20 (hovering, returning, route change, etc.). The type of control to be performed may be based, for example, on the fact that the position of the unmanned aerial vehicle 20 is inside the flight restricted area and within a specified distance, or on information on the unmanned aerial vehicle 20's approach to the boundary of the flight restricted area (for example, information on the time to escape the flight restricted area) estimated from distance information and speed information of the unmanned aerial vehicle 20.

[0069] The system 1 according to this modification may be configured to be able to perform the process of setting the flight restricted area and / or the process of setting the flight position based on the flight restricted area.

[0070] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0071] The devices described in this specification may be realized as a single device, or may be realized by multiple devices, some or all of which are connected via a network. For example, the control unit and storage of the information processing terminal 10 may be realized by different servers connected to each other via a network.

[0072] The series of processes performed by the device described in this specification may be realized using software, hardware, or a combination of software and hardware. A computer program for realizing each function of the information processing terminal 10 according to this embodiment may be created and installed on a PC or the like. A computer-readable recording medium on which such a computer program is stored may also be provided. Examples of the recording medium include a magnetic disk, an optical disk, a magneto-optical disk, and a flash memory. The computer program may also be distributed, for example, via a network, without using a recording medium.

[0073] Furthermore, the processes described herein using flowchart diagrams do not necessarily have to be performed in the order shown. Some process steps may be performed in parallel. Additional process steps may be employed, and some process steps may be omitted.

[0074] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0075] The following configurations also fall within the technical scope of the present disclosure. (Item 1) Displaying a map on a display unit; acquiring information relating to a designated area on the map input based on an operation on the map displayed on the display unit; outputting information related to a flight restriction zone of the aircraft in real space based on the designated area on the map; A method comprising: (Item 2) The method according to item 1, A method for outputting information related to the flight restriction zone in the real space to the display unit. (Item 3) The method according to item 2, Displaying a captured image carried on the aircraft flying within the flight restriction area on the display unit; and displaying the information related to the flight restriction zone on the display unit by superimposing the information on the captured image. (Item 4) The method according to item 3, a display indicating a position in the captured image that corresponds to at least one of the inside and outside of the flight restriction zone, the display being superimposed on the captured image; (Item 5) The method according to item 4, A method for displaying an area corresponding to at least either the inside or the outside of the flight restriction zone superimposed on the captured image based on information relating to the position in the real space when the captured image was captured. (Item 6) The method according to any one of items 2 to 5, displaying on the display unit an image captured by the aircraft flying within the flight restriction area; an object for setting a movement target position of the flying object is displayed on a display unit that displays the captured image; When the position of the object is a position corresponding to outside the flight restriction zone in the captured image, the object is displayed on the display unit in a manner corresponding to the object being unable to be set as the movement target position. (Item 7) The method according to any one of items 2 to 6, displaying on the display unit an image captured by the aircraft flying within the flight restriction area; an object for setting a movement target position of the flying object is displayed on a display unit that displays the captured image; A method in which display of the object is controlled so that the object can be moved based on an operation on the display unit only to a position inside the flight restriction zone in the captured image. (Item 8) The method according to any one of items 1 to 7, A method for outputting information relating to a flight restriction zone in real space to the flying object. (Item 9) Item 8. The method according to item 8, The method, wherein the flying vehicle is controlled to fly only within an area inside a flight restriction zone in the real space. (Item 10) The method according to any one of items 1 to 9, A method, wherein the information relating to the flight restricted zone in the real space includes information relating to a no-fly zone outside the flight restricted zone. (Item 11) A system for flight control of an aircraft, a display control unit that displays a map on a display unit; an input information acquisition unit that acquires information related to a designated area on the map that is input based on an operation on the map displayed on the display unit; an output control unit that outputs information related to a flight restriction zone in real space based on the designated area on the map; A system comprising: (Item 12) Computer, a display control unit that displays a map on a display unit; an input information acquisition unit that acquires information related to a designated area on the map that is input based on an operation on the map displayed on the display unit; an output control unit that outputs information related to a flight restriction zone of an aircraft in real space based on the designated area on the map; A program that functions as a (Item 13) a display device displays a captured image obtained by capturing an image using an imaging device, and the captured image is associated with information relating to the position of the imaging device in a three-dimensional space and the attitude of the imaging device; acquiring information relating to the position of a flight restriction zone of the aircraft set in the three-dimensional space; superimposing an object related to the flight restricted area on a captured image obtained from the aircraft flying within the flight restricted area and displaying the image on the display device; A method comprising: (Item 14) Item 14. The method according to item 13, A method for displaying on the display device an indication of the position of the aircraft in the captured image, superimposed on the captured image. (Item 15) 15. The method according to item 13 or 14, A method for outputting information relating to a flight restriction zone in real space to the flying object. (Item 16) Item 15. The method according to item 15, The method, wherein the flying vehicle is controlled to fly only within an area inside a flight restriction zone in the real space. (Item 17) The method according to any one of Items 13 to 16, A method for outputting control information for the aircraft based on position information of the aircraft and information relating to its position in the flight restricted area. (Item 18) Item 17. The method according to item 17, A method for outputting control information for the aircraft based on the distance between the aircraft and the boundary of the flight restricted area obtained from the position information of the aircraft and information relating to its position regarding the flight restricted area. (Item 19) 19. The method according to item 17 or 18, A method for outputting control information for the flying object based on speed information of the flying object. (Item 20) 19. The method according to any one of items 17 to 19, The method, wherein controlling the flying object includes controlling the output of visual information, tactile information, and / or auditory information by the display device. (Item 21) The method according to any one of Items 17 to 20, The method, wherein control relating to the air vehicle includes flight control of the air vehicle. (Item 22) The method according to any one of Items 13 to 21, The method, wherein the imaging device is on board the air vehicle or is the air vehicle. (Item 23) A system for display control, comprising: a display control unit that displays on a display device a captured image obtained by capturing an image using an imaging device; the captured image is associated with information relating to the position of the imaging device in a three-dimensional space and the attitude of the imaging device; an input information acquisition unit that acquires information related to the position of a flight restriction zone of the aircraft set in the three-dimensional space; an output control unit that superimposes an object related to the flight restricted area on a captured image obtained from the aircraft flying within the flight restricted area and displays the superimposed image on the display device; A system comprising: (Item 24) Computer, a display control unit that displays on a display device a captured image obtained by capturing an image using an imaging device; the captured image is associated with information relating to the position of the imaging device in a three-dimensional space and the attitude of the imaging device; an input information acquisition unit that acquires information related to the position of a flight restriction zone of the aircraft set in the three-dimensional space; an output control unit that superimposes an object related to the flight restricted area on a captured image obtained from the aircraft flying within the flight restricted area and displays the superimposed image on the display device; A program that functions as a [Explanation of symbols]

[0076] 1 System 10 Information processing terminal 11 Control section 12 Touch Panel 20 Unmanned Aerial Vehicles 28 Camera 111 Map Acquisition Unit 112 Image acquisition unit 113 Display control unit 114 Input information acquisition unit 115 Flight Restriction Zone Determination Department 116 Coordinate conversion unit 117 Output control section 118 Map Information

Claims

1. A captured image obtained by capturing an image using an imaging device is displayed on a display device, and the captured image is associated with flight parameters including information related to the position of the imaging device in three-dimensional space and the attitude of the imaging device; a setting of a position of a flight restriction zone of the aircraft on a map displayed on the display device is accepted, and information relating to the position of the flight restriction zone in the three-dimensional space is acquired based on the setting; and the setting of the position of the flight restriction zone on the map can be accepted while the aircraft is flying; converting the acquired information about the position of the flight restriction zone into coordinates of the flight restriction zone in the captured image based on information about the position of the image capture device in three-dimensional space and the attitude of the image capture device; superimposing an object related to the flight restricted area on the converted coordinates of the flight restricted area in a captured image obtained from the aircraft flying within the flight restricted area, and displaying the image on the display device; A method comprising:

2. 10. The method of claim 1, A method for displaying on the display device an indication of the position of the aircraft in the captured image, superimposed on the captured image.

3. 3. The method of claim 1 or 2, A method for outputting information relating to a flight restriction zone in real space to the flying object.

4. 4. The method of claim 3, The method, wherein the flying vehicle is controlled to fly only within an area inside a flight restriction zone in the real space.

5. The method according to any one of claims 1 to 4, A method for outputting control information for the aircraft based on position information of the aircraft and information relating to its position in the flight restricted area.

6. 6. The method of claim 5, A method for outputting control information for the aircraft based on the distance between the aircraft and the boundary of the flight restricted area obtained from the position information of the aircraft and information relating to its position regarding the flight restricted area.

7. 7. The method of claim 5 or 6, A method for outputting control information for the flying object based on speed information of the flying object.

8. The method according to any one of claims 5 to 7, The method, wherein controlling the flying object includes controlling the output of visual, tactile, and / or auditory information by the display device.

9. The method according to any one of claims 5 to 8, The method, wherein control relating to the air vehicle includes flight control of the air vehicle.

10. The method according to any one of claims 1 to 9, The method, wherein the imaging device is on board the air vehicle or is the air vehicle.

11. A system for display control, comprising: A display control unit that displays an image captured by an imaging device on a display device, and the captured image is linked to flight parameters including information related to the position of the imaging device in three-dimensional space and the attitude of the imaging device, an input information acquisition unit that receives a setting of a position of a flight restriction zone of the aircraft on a map displayed on the display device and acquires information related to the position of the flight restriction zone in the three-dimensional space based on the setting; and the setting of the position of the flight restriction zone on the map is configured to be able to be received while the aircraft is flying; a conversion unit that converts the acquired information about the position of the flight restriction zone into coordinates of the flight restriction zone in the captured image based on information about the position of the image capture device in three-dimensional space and the attitude of the image capture device; an output control unit that superimposes an object related to the flight restricted zone on the converted coordinates of the flight restricted zone in a captured image obtained from the aircraft flying within the flight restricted zone and displays the superimposed object on the display device; A system comprising:

12. Computer, A display control unit that displays an image captured by an imaging device on a display device, and the captured image is linked to flight parameters including information related to the position of the imaging device in three-dimensional space and the attitude of the imaging device, an input information acquisition unit that receives a setting of a position of a flight restriction zone of the aircraft on a map displayed on the display device and acquires information related to the position of the flight restriction zone in the three-dimensional space based on the setting; and the setting of the position of the flight restriction zone on the map is configured to be able to be received while the aircraft is flying; a conversion unit that converts the acquired information about the position of the flight restriction zone into coordinates of the flight restriction zone in the captured image based on information about the position of the image capture device in three-dimensional space and the attitude of the image capture device; an output control unit that superimposes an object related to the flight restricted zone on the converted coordinates of the flight restricted zone in a captured image obtained from the aircraft flying within the flight restricted zone and displays the superimposed object on the display device; A program that functions as a

Images