Apparatus, method and software to assist a human operator in flying a drone using a remote controller

Abstract

An apparatus (100), method, and software for assisting a human operator (120) in flying a drone (160) using a remote controller (150), the apparatus (100) includes an internal data communications interface (108) for receiving data from the remote controller (150), an augmented reality display (112) for displaying the data, one or more memories (104) containing computer program code (106), and one or more processors (102) for causing the apparatus (100) to superimpose a target symbol (200) indicative of a position of the drone (160) on the augmented reality display (112) while the human operator (120) is looking towards the drone (160), to superimpose an orientation symbol (202) indicative of a direction of the drone (160) on the augmented reality display (112) while the human operator (120) is looking towards the drone (160), to obtain a geographic location for the drone (160), and to set a world marker at the obtained geographic location.

Description

[Technical field]

[0001] Various embodiments relate to an apparatus for assisting a human operator in flying a drone using a remote controller, a method for assisting a human operator in flying a drone using a remote controller, and a computer readable medium including computer program code for carrying out the method, which when executed by one or more processors reduces performance of the method. [Background technology]

[0002] A human operator (on the ground) flies the drone (or Unmanned Aerial Vehicle (UVA)) using a remote controller (in some cases, assisted at least in part by an autopilot).

[0003] A human operator must simultaneously orient the drone in the air, operate a handheld remote controller, and sometimes even point the remote controller's display, which can lead to reduced situational awareness and potentially dangerous situations.

[0004] As a legal requirement, a human operator must maintain visual contact (line of sight) with the drone while in the air, which can be extremely difficult due to long distances, low ambient light, physical obstructions, etc. that may make the drone invisible.

[0005] These problems can be mitigated by having another person, the so-called spotter, maintain visual contact with the drone even when using binoculars, allowing the human operator to concentrate on operating the remote controller (although he or she may still need to check the remote controller display from time to time). Naturally, such a setup requires good communication skills from the human operator and spotter. Moreover, it increases the cost of operating the drone, as it doubles the manual effort.

[0006] U.S. Patent Application No. 2018 / 0196425, U.S. Patent Application No. 2019 / 0077504, and U.S. Patent Application No. 2019 / 0049949 disclose various aspects relating to the use of head mounted displays in flying drones. Summary of the Invention

[0007] According to one aspect, the subject matter of the independent claims is provided. The dependent claims define several embodiments.

[0008] One or more examples of implementations are described in more detail in the accompanying drawings and description of embodiments. [Brief description of the drawings]

[0009] [Figure 1A] 1A and 1B show an embodiment of an apparatus that assists a human operator in flying a drone using a remote controller. [Figure 1B] 1A and 1B show an embodiment of an apparatus that assists a human operator in flying a drone using a remote controller. [Diagram 2] 2 and 3 show embodiments of views provided by the augmented reality display of the device. [Diagram 3] 2 and 3 show embodiments of views provided by the augmented reality display of the device. [Figure 4] 4, 5, 6, 7, 8 and 9 show embodiments of drone orientation. [Diagram 5] 4, 5, 6, 7, 8 and 9 show embodiments of drone orientation. [Figure 6] 4, 5, 6, 7, 8 and 9 show embodiments of drone orientation. [Figure 7] 4, 5, 6, 7, 8 and 9 show embodiments of drone orientation. [Figure 8]4, 5, 6, 7, 8 and 9 show embodiments of drone orientation. [Figure 9] 4, 5, 6, 7, 8 and 9 show embodiments of drone orientation. [Figure 10] 10, 11 and 12 show an embodiment for visualizing obstacles for a drone. [Figure 11] 10, 11 and 12 show an embodiment for visualizing obstacles for a drone. [Figure 12] 10, 11 and 12 show an embodiment for visualizing obstacles for a drone. [Figure 13] FIG. 13 illustrates an embodiment for visualizing waypoints for a drone. [Figure 14] 14 and 15 show an embodiment for visualizing data captured by a drone. [Figure 15] 14 and 15 show an embodiment for visualizing data captured by a drone. [Figure 16] 16 and 17 show an embodiment for visualizing a map for a drone flight. [Figure 17] 16 and 17 show an embodiment for visualizing a map for a drone flight. [Figure 18] 18 and 19 show an embodiment for visualizing the menu structure of a device. [Figure 19] 18 and 19 show an embodiment for visualizing the menu structure of a device. [Figure 20] 20, 21 and 22 show an embodiment for visualizing external data related to the drone's physical environment. [Figure 21] 20, 21 and 22 show an embodiment for visualizing external data related to the drone's physical environment. [Figure 22] 20, 21 and 22 show an embodiment for visualizing external data related to the drone's physical environment. [Diagram 23]23, 24, 25, 26 and 27 show an embodiment for visualizing the line of sight to a drone during different visibility. [Figure 24] 23, 24, 25, 26 and 27 show an embodiment for visualizing the line of sight to a drone during different visibility. [Diagram 25] 23, 24, 25, 26 and 27 show an embodiment for visualizing the line of sight to a drone during different visibility. [Figure 26] 23, 24, 25, 26 and 27 show an embodiment for visualizing the line of sight to a drone during different visibility. [Figure 27] 23, 24, 25, 26 and 27 show an embodiment for visualizing the line of sight to a drone during different visibility. [Figure 28] 28 and 29 show an embodiment of a system comprising two devices. [Figure 29] 28 and 29 show an embodiment of a system comprising two devices. [Diagram 30] FIG. 30 illustrates a flow chart of an embodiment of a method for assisting a human operator in flying a drone using a remote controller. [Diagram 31] 31, 32 and 33 show an embodiment relating to world markers. [Diagram 32] 31, 32 and 33 show an embodiment relating to world markers. [Diagram 33] 31, 32 and 33 show an embodiment relating to world markers. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Several embodiments will now be described with reference to the accompanying drawings.

[0011] The following embodiments are exemplary. Although the specification may refer to "an" embodiment in multiple places, this does not necessarily mean that each such reference is to the same embodiment or that the feature applies only to that particular embodiment. Individual features of different embodiments may be combined to provide other embodiments. Furthermore, the terms "comprising" and "including" should not be understood to limit the described embodiment to only the features referenced, and such embodiments may also include features / structures not specifically mentioned.

[0012] Reference signs in both the description of the embodiments and in the claims refer to the drawings and not to the embodiments alone, but to the examples.

[0013] The embodiments and features disclosed in the following description, which do not fall within the scope of the independent claims, should be interpreted as examples helpful for understanding various embodiments of the invention.

[0014] 1A shows a simplified block diagram of an apparatus 100 that assists a human operator (or pilot) 120 in flying a drone 160 using a remote controller 150. The drone 160 is also known as an unmanned aerial vehicle (UAV). An unmanned aircraft system (UAS) may be defined as including the drone (or UAV) 160, a (ground-based) remote controller 150, and a wireless communication system 152 between the remote controller 150 and the drone 160.

[0015] 30 shows a flowchart of an embodiment of a method for assisting a human operator 120 in flying a drone 160 using a remote controller 150.

[0016] The method starts at 3000 and ends at 3010. Note that the method can run for as long as necessary (after power-up of the device 100 until it is switched off) by looping back to step 3002.

[0017] In Figure 30, the steps are not in strict chronological order. Some steps may be performed simultaneously and certain steps may be performed in a different order. Other functions may also be performed between or within the steps, as well as other data exchanged between steps. Some steps or portions of steps may be omitted or replaced by corresponding steps or portions of steps. Note that no particular ordering of steps is required except as required by logical requirements of the processing order.

[0018] The device 100 includes an internal data communication interface 108 for receiving 3002 flight related data from the remote controller 150. The flight related data may include drone 160 telemetry data. The flight related data may include, but is not limited to, sensor values ​​such as gyroscopes and magnetometers, fusion data such as angular velocity, speed, altitude and earth position, aircraft information such as battery, gimbal, flight status, etc. Depending on the drone environment, data may also be received by the device 100 directly from the drone 160.

[0019] The internal data communication interface 108 may be implemented using a wireless transceiver that communicates with the wireless transceiver of the remote controller 150. The technology of the internal data communication interface 108 includes, but is not limited to, one or more of a wireless local area network (WLAN) implemented using the IEEE 802.11ac standard or the Wi-Fi protocol suite, a short-range wireless network such as Bluetooth or Bluetooth LE (Low Energy), a cellular wireless network using a subscriber identity module (SIM) or eSIM (embedded SIM), or other standard or proprietary wireless connection means. It is noted that in some use cases, the internal data communication interface 108 may additionally or alternatively utilize a standard or proprietary wired connection, such as, for example, an applicable bus. In an embodiment, a wired connection according to the USB (Universal Serial Bus) standard is utilized.

[0020] The device 100 also includes an augmented reality (AR) display 112 that displays 3004 data about the flight to the human operator 120. Figures 2 to 29 show specific embodiments, but in addition to these, various notifications and status about the flight may also be displayed on the augmented reality display 112.

[0021] In the figures, the augmented reality display 112 is implemented as a head-mounted display attached to a headband (or attached to a helmet) and worn as a visor in front of the eyes by a human operator 120. In the figures, the augmented reality display 112 is implemented as a see-through display on which holographic images are displayed. In alternative embodiments, the augmented reality display 112 can intercept a view of the real world using a camera and display an augmented view of the real world as a projection.

[0022] In an embodiment, the device 100 is implemented using Microsoft® HoloLens® 2 (or later version) mixed reality smart glasses that employ see-through holographic lenses as the augmented reality display 112, providing a complete development environment. The head-mounted device 100 includes the necessary processor (including system-on-chip, custom-made holographic processing unit, and co-processor) 102, memory 104 and software 106, depth camera, video camera, projection lens, inertial measurement unit (including accelerometers, gyroscopes, and magnetometers), wireless connection devices 108, 110, and a rechargeable battery. Note that some of these components are not shown in FIG. 1. Such a ready-made environment provides an augmented reality engine 144 that provides basic operations related to, for example, blending the real world with augmented reality and tracking the head and eye movements of a human operator 120.

[0023] However, other applicable implementations of the augmented reality display 112 may also be used, including, but not limited to, glasses, head-up displays, contact lenses with augmented reality imaging, etc. For purposes of the present embodiment, the augmented reality display 112 provides an interactive real-time experience of the real-world flight environment 210 and drone 160, enhanced with computer-generated perceptual information. In addition to the natural environment 210 and drone 160, data about the flight is superimposed (or overlaid).

[0024] The device 100 also comprises one or more memories 104 containing computer program code 106, and one or more processors 102 that execute the computer program code 106 to cause the device 100 to perform the necessary data processing. The data processing performed by the device 100 may be interpreted as a method or algorithm 130.

[0025] The term "processor" 102 refers to a device capable of processing data. In an embodiment, the processor 102 is implemented as a microprocessor that implements the functionality of a central processing unit (CPU) on an integrated circuit. The CPU is a logic machine that executes computer program code 106. The CPU may include a set of registers, an arithmetic logic unit (ALU), and a control unit (CU). The control unit is controlled by a sequence of computer program code 106 transferred from the (working) memory 104 to the CPU. The control unit may include a number of microinstructions for basic operations. The implementation of the microinstructions may vary depending on the CPU design. One or more processors 102 may be implemented as individual processor cores and / or as separate processors.

[0026] The term "memory" 104 refers to a device that can store data either at runtime (= working memory) or persistently (= non-volatile memory). Working memory and non-volatile memory can be implemented by Random Access Memory (RAM), Dynamic RAM (DRAM), Static RAM (SRAM), Flash memory, Solid State Disks (SSD), PROM (Programmable Read Only), suitable semiconductors, or other means of implementing electrical computer memory.

[0027] The computer program code 106 is implemented in software. In an embodiment, the software may be written in a suitable programming language such that the resulting executable code may be stored in the memory 104 and executed by the one or more processors 102.

[0028] The computer program code 106 implements the method / algorithm 130. The computer program code 102 may be coded as a computer program (or software) using a programming language, which may be a high-level programming language such as C, C++, or Rust. The computer program code 106 may be in source code form, object code form, an executable file, or some intermediate form, but is in an executable form as an application 140 for use by one or more processors 102. There are many ways to structure the computer program code 106. Depending on the software design methodology and the programming language used, the steps may be divided into modules, subroutines, methods, classes, objects, applets, macros, etc. In modern programming environments, software libraries, i.e., compilations of ready-made functions, exist that may be utilized by the computer program code 106 to perform a wide variety of standard tasks. Additionally, an operating system (such as a general-purpose operating system) may provide system services to the computer program code 106.

[0029] As shown in FIG. 30, an embodiment provides a computer readable medium 170 storing computer program code 106, which when loaded into and executed by one or more processors 102, causes the one or more processors 102 to execute the method / algorithm 130. The computer readable medium 170 may include at least any entity or device capable of carrying the computer program code 106 to one or more processors 102, a recording medium, a computer memory, a read-only memory, an electrical carrier signal, an electrical communication signal, and a software distribution medium. In some jurisdictions, depending on the law and patent practice, the computer readable medium 170 may not be an electrical communication signal. In an embodiment, the computer readable medium 170 may be a computer readable medium. In an embodiment, the computer readable medium 170 may be a non-transitory computer readable medium.

[0030] 1A and 1B, computer readable medium 170 may carry computer program code 160 as executable application 140 for device 100 and as executable application 142 for remote controller 150 for transmitting data regarding flight to device 100. In a typical drone environment, such as DJI®, a software development kit may be used for application 142 to interface with remote controller 150.

[0031] FIG. 1A shows the device 100 as an integrated unit comprising an augmented reality display 112, one or more memories 104 containing computer program code 106, and one or more processors 102.

[0032] However, as shown in FIG. 1B, the device 100 may be implemented as a distributed device 100 such that the human operator 120 is provided with an augmented reality display 112 but with a separate processing unit 180. The device 100 is communicatively coupled to the augmented reality display 112 and a remote controller 150. The device 100 also comprises one or more memories 104 including computer program code 106 and one or more processors 102. This may be implemented such that the processing unit 180 is a user device carried by the human operator 120, such as a smartphone, tablet computer, or portable computer, and the communication connection may be wired or wireless. Other embodiments may be a networked computer server, etc., where the processing unit 180 interoperates with the augmented reality display 112 according to a client-server architecture, a cloud computing architecture, a peer-to-peer system, or other applicable distributed computing architecture.

[0033] Figures 2 and 3 show embodiments of views provided by the augmented reality display 112 of the device 100. Figures 2 to 29 show each use case as a combination of two different viewing angles.

[0034] 2 and 3 in more detail. A first viewing angle shows flight. A human operator 120 operates a remote controller 150 and observes (or looks towards) 204 a drone 160 in the air 210 through the device 100, or more precisely, via the augmented reality display 112 of the device 100. As shown, a second viewing angle shows elements 200, 202 shown on the augmented reality display 112.

[0035] This convention is used in all figures from Figures 2 to 29. The dotted arrow 204 indicates the direction in which the human operator 120 is looking, typically the direction of the drone 160 in the air. However, in some use cases, the human operator 120 looks in another direction, such as towards the ground, as indicated by dotted arrows such as 600, 1404, 1706, 1804, 2602, etc. However, in the alternative embodiment described above, a camera is used to block the view of the real world and display an augmented view of the real world as a projection. The direction of gaze towards the augmented reality display 112 may be different from the capture direction of the camera. For example, to make the flying position more comfortable, the human operator 120 does not need to tilt his head to look at the sky, but the camera is tilted upwards.

[0036] It should be noted that in all the described embodiments, the human operator 120 stands on the ground 500 and the drone 160 flies in the air 210. However, the embodiments are also applicable to flying the drone 160 underground, in caves, in man-made structures (such as buildings or tunnels), or in other types of environments where the drone 160 flies below the human operator 120, i.e., the human operator 120 looks 240 towards the drone 160, but not up. In such use cases, the human operator 120 may be standing on a high platform (such as a high-rise building or a mountain) and the drone 160 may be flying below (such as above a road or in a valley). The embodiments are also applicable to flying the drone 160 submerged, i.e., the drone 160 becomes an unmanned underwater vehicle (UUV), and the human operator 120 may operate the drone 160 from a land or a watercraft, for example, when the drone is underwater, such as in a river, lake, ocean, or a mine or tunnel filled with water.

[0037] In a sense, all of the figures in Figures 2 through 29 show a hybrid representation of augmented reality on top of the real world, where the real world is shown from an external perspective (such as the perspective of another person observing the use case from outside the real world) while the augmented reality display 112 is shown from the first person's perspective of a human operator 120.

[0038] Returning now to Figures 2 and 3, in an embodiment, the device 100 superimposes 3006 a target symbol 200 indicating the position of the drone 160 (in the air 210, if a UAV) on the augmented reality display 112 while the human operator 120 looks 204 in the direction of the drone 160 (in the air 210, if a UAV). In an embodiment, the device also superimposes 3008 an orientation symbol 202 indicating the direction of the drone 160 (in the air 210, if a UAV) on the augmented reality display 112 while the human operator 120 looks 204 in the direction of the drone 160 (in the air 210, if a UAV).

[0039] The use of the augmented reality display 112 allows the human operator 120 to look 204 towards the drone 160 in the air 210 during flight, thereby improving the situational awareness of the operator 120 regarding the flight without the need for a spotter. The human operator maintains visual contact (line of sight) with the drone 160 in the air 210, but is simultaneously shown the aerial data in its actual correct world position, as described below.

[0040] The target symbol 200 indicates the location of the drone 160 in the air 210, making it easier for the human operator 120 to track the drone 160 during flight. In an embodiment, the target symbol 200 is a reticle as shown. Reticles 200 are commonly used in telescopic sights on firearms. The reticle 200 may include a combination of a circle 300 and partial crosshairs 302, as shown in FIG. 3. However, other patterns such as dots, posts, chevrons, etc. may also be used.

[0041] The orientation symbol 202 indicates the orientation of the drone 160 in the air 210, which may help the operator 120 understand the effect of control commands given to the drone 160 by the remote controller 150 during flight. In an embodiment, the orientation symbol 202 is an arrow, as shown. As shown in FIG. 3, the arrow 202 may be augmented by an arc 304 that indicates a portion of a 360 degree circle around the human operator 120. The arrow 202 may point in the direction of travel of the drone 160, as will be described later.

[0042] In the augmented reality display 112 , target symbols 200 and orientation symbols 202 from the digital world blend into the human operator's 120 perception of the real world through an immersive integration that is perceived as a natural part of the flight environment 210 .

[0043] Consider Figures 4, 5, 6, 7, 8 and 9, which show embodiments of drone 160 orientation.

[0044] In the embodiment, the orientation symbol 202 points to a fixed predetermined direction relative to the orientation of the drone 160 in the air 210. Since the human operator 120 knows the predetermined direction, it is possible for the human operator 120 to understand how the steering commands given by the remote controller 150 will affect the flight. As shown in FIG. 13, the remote controller 150 can include, for example, two joysticks 310, 312 to give steering commands. Of course, other types of steering devices are also compatible with the described embodiments. However, the remote controller 150 can control the drone 160 with various degrees of freedom, namely roll, which tilts the drone 160 left and right, pitch, which tilts the drone 160 forward or backward, and yaw, which rotates the drone 160 clockwise or counterclockwise. Additionally, an altitude control device controls the drone 160 to fly higher or lower. It should be noted that some user interface elements of the remote controller 150 may be programmed to interact with the device 100 such that user interface operations of the device 100 can be performed by user interface elements of the (physical) remote controller 150 in addition to being performed in the augmented reality environment.

[0045] 4, the predetermined direction is fixed relative to the heading 400 of the drone 160. In navigation, the heading 400 of the drone 160 is the compass direction that the nose of the drone 160 is pointing in. Note that a drone 160 that is, for example, a quadcopter (= a helicopter with four rotors) may not have a "natural" nose, in which case one direction of the drone 160 is simply defined as the nose.

[0046] 5 shows various coordinate systems 502, 504, 506 that need to relate to each other to enable the embodiment. The world coordinate system 502 defines a three-dimensional world model visualization that is mapped to the coordinate system 504 of the device 100 and the coordinate system 506 of the drone 160. The device 100 then uses its own coordinate system 504 to show the augmented reality, but also the position of the drone 160 and the position of the human operator 120 in the world coordinate system 502.

[0047] In the embodiment shown in Figures 5 and 6, the device 100 performs the following: Obtaining the position of the drone 160 on the ground 500 in the world coordinate system 502. Obtaining the position of the drone 160 on the ground 500 in the augmented reality coordinate system 504 of the device 100. Locking the position of the drone 160 in the augmented reality coordinate system 504 with the position of the drone 160 in the world coordinate system 502. - Obtaining 400 directions for 160 drones on the ground (500). Setting the heading 400 as the direction of the calibration orientation symbol in the augmented reality coordinate system 504 of the device 100.

[0048] In this way, the augmented reality coordinate system 504, which constantly tracks the head movements of the human operator 120, is firmly based on world coordinates 502 and also follows the actual compass heading 602. In this way, the coupling of world latitude and longitude (x and z in world coordinate system 502) and compass heading information 602 into the augmented reality presentation is achieved.

[0049] In a more particular embodiment, the apparatus 100 performs the following. Obtaining (from the remote controller 150 or the drone 160) the position of the drone 160 on the ground 500 in the world coordinate system 502. · Superimposition of calibration position symbols on the augmented reality display 112. Receipt of a first user confirmation after the calibration location symbol has been placed on the drone 160 (such as at the center of the drone 160 or another predefined point on the drone 160). Obtaining (from the augmented reality engine 144) the position of the drone 160 on the ground 500 in the augmented reality coordinate system 504 of the device 100. Locking the position of the drone 160 in the augmented reality coordinate system 504 with the position of the drone 160 in the world coordinate system 502. Obtaining the direction of travel 400 of the drone 160 on the ground 500 (from the remote controller 150 or the drone 160). · Overlay of a calibrated orientation symbol onto the augmented reality display 112. Receiving a second user confirmation after the calibration orientation symbol is aligned with the drone 160 (e.g., with the tail-nose line of the drone 160 or another predetermined orientation of the drone 160). Set heading 400 as the direction of the calibration orientation symbol in the augmented reality coordinate system 504 of the device 100.

[0050] First, the augmented reality system is shown the location of the drone 160 in the world coordinate system 502 and the location of the drone 160 relative to the augmented reality coordinate system 504. The augmented reality indicator is used to indicate that the center of the drone 160 is located at a precise spot in the augmented reality view 112, which spot is known in both the real world coordinate system 502 and the augmented reality coordinate system 504. This combination results in a fixed common location, along with world latitude and longitude information. This latitude and longitude are obtained from the drone 160 since it currently knows its precise coordinates (provided by GPS or another global navigation satellite system, or another positioning technology such as cellular radio-based positioning). An augmented reality pointer stick, or another type of calibrated position symbol, can indicate the location in the augmented reality display 112 to the human operator 120. When indicating the location of the drone 160, this stick, which moves a certain distance in front of the human operator 160 and points down, is guided to be over the center of the drone 160. It is held steady to confirm its position and then the coordinate systems 502, 504 are locked together. Alternatively, this can be done using machine vision to look at the drone 160, decipher its position in the augmented reality coordinate system 504, and then lock the latitude, longitude, and even orientation of the drone 160. Positioning the drone 160 can be done in many ways, but needs to be done reliably to ensure a reliable lock between the world and the augmented reality coordinate systems 502, 504.

[0051] Second, drone 160 knows where its nose is pointing, i.e. drone 160 communicates its compass heading in degrees, which can be used to ultimately determine the conjunction of coordinate systems 502, 504. An augmented reality system is used to align the displayed line, or another type of calibration orientation symbol, with the tail line of drone 160, and once this is achieved, the compass heading of this displayed line in world coordinate system 502 is known. Thus, the world compass heading of any direction, e.g. north, can be calculated from there.

[0052] As an optional step, when the world position (latitude, longitude) is obtained from the drone 160, the precise altitude (y in the world coordinate system 502) can also be queried from the map system based on the precise world coordinates. Thus, the altitude of this point in space can also be calibrated (using a drone-specific offset of the top surface of the drone 160 from the ground 500 if precise accuracy is required), so that the map data is used in the future to precisely determine the terrain altitudes of other world points. In summary, latitude, longitude, possibly altitude, and compass heading may be required to achieve world lock.

[0053] After this coupling, everything else in the entire system is built on the knowledge of where drone 160 actually is in world coordinates 502 and what is exactly around it in the world. It should be noted that the described embodiment of the coupling can operate as a standalone embodiment, independent of all other embodiments, as well as the embodiments described in relation to the independent claims and other dependent claims.

[0054] The flight data is mapped to world coordinates 502 and then displayed 3004, 3006, 3008 such that its visualization takes advantage of knowing its three-dimensional position expressed in world coordinate system 502, which is locked to augmented reality coordinate system 504.

[0055] In the embodiment shown in FIG. 7, situational awareness may be further enhanced by numerical information. The device 100 superimposes the cruising altitude 704 of the drone 160 on the augmented reality display 112 using numerical values ​​and a scale 700 visually coupled with the target symbol 200 while the human operator 120 is looking 204 towards the drone 160 in the air 210. As shown in FIG. 7, the scale 700 may include horizontal lines each indicating a particular altitude. The device also superimposes the heading 706 of the drone 160 in degrees 702 visually coupled with the orientation symbol 202 on the augmented reality display 112 while the human operator 120 is looking 204 towards the drone 160 in the air 210. This may be useful for a skilled human operator 120.

[0056] 8, the device 100 superimposes on the augmented reality display 112, while the human operator 120 looks 204 towards the drone 160 in the air 210, an indirect line of sight guideline 800 that extends horizontally to the geographic location of the drone 160 on the ground 500, from which an indirect line of sight guideline 802 extends vertically to the target symbol 200 at the cruising altitude of the drone 160 in the air 210. This can further enhance situational awareness, as the human operator 120 can first observe the horizontal guideline 800 to ascertain the geographic location of the drone 160 on the ground 500, and then observe the vertical guideline 802 to understand where the drone 160 is in the air 210.

[0057] In the embodiment shown in Figure 9, the device 100 superimposes a wake symbol 900 showing the wake 902 and speed of the drone 160 in the air 210 on the augmented reality display 112 while the human operator 120 looks 204 towards the drone 160 in the air 210. In navigation, the wake 902 is the path that the drone 160 actually travels. The difference between the heading 400 and the wake 902 is caused by the movement of the air 210 (such as air currents). By showing the wake 902 and speed, the human operator 120 predicts the effect of the current control, which can be adjusted if necessary.

[0058] Consider now the embodiment of visualizing obstacles in relation to a drone 160 shown in FIGS.

[0059] In the embodiment shown in Figure 10, the device 100 superimposes an obstacle indicator symbol 1000 depicting a distance 1002 from the drone 160 to a real object 1004 on the augmented reality display 112 while the human operator 120 looks 204 in the direction of the drone 160. The distance 1002 may be the shortest distance between the drone 160 and the real object 1004. As shown in Figure 10, the obstacle indicator symbol 1000 may mark the distance using an arrow, possibly augmented by a numerical value indicating the distance 1002. The real object 1004 may be a man-made object such as a building, a bridge, etc., or a natural object such as a hill or a forest.

[0060] 11 illustrates an additional embodiment, where the obstacle indication symbol 1000 includes a visual indication 1100 at least partially superimposed on the real object 1004. As shown in FIG. 11, the visual indication 1100 may be shading or a similar visual effect superimposed on the real object 1004. In this way, the human operator 120 immediately recognizes the risk of a collision when the drone 160 approaches the object 1004.

[0061] Figure 12 shows a further embodiment applicable to either the embodiment of Figure 10 or the embodiment of Figure 11. An obstacle indication symbol 1200 includes elements representing the shortest horizontal distance 1202 and vertical distance 1204 from the drone 160 to a real object 1206. In this way, the effects of both vertical and horizontal movement of the drone 160 can be recognized to avoid a collision with the real object 1206.

[0062] Consider now Fig. 13, which illustrates an embodiment of visualizing waypoints with respect to a drone 160. The device 100 overlays on the augmented reality display 112 a map 1300 showing the geographic location 1302 of the human operator 120 and the geographic location 1304 of the human operator 120. In this way, the human operator 120 can better intuitively understand the surroundings with respect to the flight. As shown in Fig. 13, the map 1300 and the drone 160 can be simultaneously in the field of view of the human operator 120, and the gaze can alternately be directed 204 to the drone or 1310 to the map 1300. While the human operator 120 looks in the direction 204 of the drone 160, the device 100 overlays on the augmented reality display 112 a vertical waypoint symbol 1308 starting from the geographic location of the waypoint 1306 above the ground 500 and extending toward the predetermined altitude of the waypoint 1306. A narrow portion of the waypoint symbol 1308 may indicate a precise geographic location on Earth 500, while a wider portion of the waypoint symbol 1308 may indicate the set altitude of the waypoint in the air 210. The waypoint symbol 1308 is shown in the correct location in the real world.

[0063] 14 and 15 next illustrate an embodiment for visualizing data captured by a drone 160. The device 100 superimposes one or more visual elements 1400 captured in real time using one or more sensors 1402 mounted on the drone 160 near the target symbol 200 on the augmented reality display 112 while the human operator 120 looks 204 toward the drone 160 in the air 210, and positions the one or more visual elements 1400 on the augmented reality display 112 so that there is no line of sight obstructed while the human operator 120 looks 204 toward the drone 160 in the air 210. The visual elements 1400 can be positioned on either side of the target symbol 200 as shown, but can be positioned anywhere around the target symbol 200. The human operator 120 can glance 1404 at the visual elements 1400 while primarily looking 204 toward the drone 160 and simultaneously piloting the drone 170. In the illustrated embodiment, the image sensor 1402 captures images or video feeds as data. In this manner, the human operator 120 can maneuver the drone 120 so that the image sensor 1402 captures a desired view. It should be noted that the image sensor can operate as a (regular) visible light camera, such as a photo camera or video camera. In addition to this, the image sensor can operate as, for example, a thermal (or infrared) camera, a multispectral camera, a hyperspectral camera, or a corona discharge camera. The one or more sensors 1402 on board the drone 160 can include, but are not limited to, one or more of a LIDAR (light detection and ranging, or laser imaging, detection and ranging, or 3D laser scanning) sensor, a SONAR (voice navigation and ranging) sensor, a RADAR (radio detection and ranging) sensor, a chemical sensor, a biological sensor, a radiation sensor, a particle sensor, a magnetic sensor, a network signal strength sensor, and the like. The drone 160 can carry any combination of these sensors 1402 as payloads, whose data is visualized as described with one or more dynamically positioned visual elements 1400.

[0064] 16 and 17 show an embodiment of visualizing a map for the flight of a drone 160. The human operator 120 may select the layout of the maps 1600, 1700 or the device 100 may automatically determine which layout to use depending on the flight situation. The device 100 superimposes on the augmented reality display 112 a map 1600 in a vertical layout showing the geographical position 1602 of the human operator 120 and the geographical position 1604 of the drone 160 in the vicinity of the target symbol 200 while the human operator 120 looks 204 towards the drone 160 in the air 210. Alternatively, the device 100 superimposes on the augmented reality display 112 a map 1700 in a horizontal layout showing the geographical position 1702 of the human operator 112 and the geographical position 1704 of the drone 160 while the human operator 120 looks 1706 towards the ground 500. By using a vertical layout map 1600, situational awareness may be maintained at all times when the human operator gazes 204 towards the drone 160 and sees the map 1600 to the side. By using a horizontal layout map 1700, the human operator 120 must look 1706 towards the ground 500, but the map 1700 may be shown larger and more intuitively because the surface of the map 1700 is parallel to the ground surface 500 as shown.

[0065] FIG. 17 also shows that the map 1700 used may be a three-dimensional topographical map that also shows elevation data represented by three-dimensional buildings 1708 .

[0066] 18 and 19 show an embodiment of visualizing the menu structure of the device 100. The device 100 superimposes a menu structure 1800 around the human operator 120 on the augmented reality display 112 while the human operator 120 looks 1804 towards the ground 500. The device 100 detects gestures 1802 from the operator 120 as commands related to the menu structure 1800 and controls the display 1900 of flight related data on the augmented reality display 112 based on the commands. In this way, the human operator 120 can quickly operate the device 100. As shown, FIG. 18 shows a basic display of a target symbol 200 and an orientation symbol 202, and in FIG. 19, the human operator 120 selects from the menu structure 1800 to display the cruising altitude of the drone 160 using a numerical value and a scale 700 visually coupled with the target symbol 200, as described above with reference to FIG. 7.

[0067] 20, 21 and 22 show an embodiment for visualizing external data related to the physical environment of the drone 160.

[0068] As shown in FIGS. 1A and 1B, the device 100 includes an external data communication interface 110 that receives external data 114 related to the physical environment of the drone 160. It should be noted that the external data communication interface 110 may be implemented using the internal data communication interface 108 in an embodiment. The device 100 superimposes one or more visualizations 2000 of the external data 114 onto the augmented reality display 112. In this manner, the device 100 may enhance the situational awareness of the human operator 120 by incorporating external data sources into the augmented reality display 112. As described above, the external data 114 is displayed as visualized utilizing knowledge of its three-dimensional position expressed in the world coordinate system 502, which is mapped to the world coordinate system 502 and, as a result, locked to the augmented reality coordinate system 504. In addition to obtaining external data from various sources, the external data communication interface 110 may also be used to communicate data related to the flight to an external receiver 116. The data may include, but is not limited to, the position of the drone 160, audio from the human operator 120, one or more video feeds from the drone 160, etc.

[0069] As shown in FIG. 20, the external data 114 may include weather data and one or more visualizations 2000 depicting the weather data. In an embodiment, the weather data includes information about wind speed and direction. The direction is indicated by an arrow, and the speed as shown is indicated by a scale or numerical value on the arrow. Additionally or alternatively, the weather data may include one or more of turbulence (predicted or known), humidity, cloud visualization, rain alert, hail alert, snow alert, storm alert, lightning alert, lighting conditions (time of day, sun and / or moon position), fog, temperature and pressure, visibility, dew point (important for airline pilots), “feels like” temperature. And all of these may be time-related, i.e., weather forecast visualizations such as incoming cloud fronts and wind changes.

[0070] As shown in FIG. 21 , the external data may include 114 air traffic control data including airspace classifications and one or more visualizations 2100, 2102 showing the airspace classifications that match the location of the drone 160 in the air 210. As shown, the free airspace 2100 may be marked with an "I" and the restricted airspace 2102 may be marked with a "II" and may be a shaded rectangle as shown, or another three-dimensional shape (e.g., a polygon mesh), or a two-dimensional shape (e.g., a polygon). In general, the airspace classifications may include, but are not limited to, drone no-fly zones (areas, volumes), reservations and notifications of airspace for drones and / or other aerial operations, airfield controlled areas, airspace controlled areas, power lines and other obstructions, border areas, all of the above at various altitudes, warning / danger / restricted areas, UVA reserved areas, UAS reserved areas, model aircraft reserved areas. The aerial map can be visualized using a three-dimensional polygon mesh with various walls, roofs, flight levels, etc., all in the correct place as seen on the augmented reality display 112.

[0071] 22 , the external data may include air traffic control data 114 including the position of the aircraft 2200 in the air 210, and one or more visualizations 2202, 2204 show the position of the aircraft 2200 in the air 210. An embodiment shows further visualizations 2202, 2204 of the aircraft 2200 flying within a predetermined distance (e.g., within a radius of 3, 5, or 10 kilometers, etc.) from the position of the drone 160 in the air 210. The visualizations may be implemented with arrows 2202, 2204 indicating the position of the aircraft 2200, and additionally or alternatively, a simulation of the aircraft 2200 may be shown.

[0072] 23, 24, 25, 26 and 27 show embodiments for visualizing the line of sight to the drone 160 during different visibility.

[0073] 23 embodiment, the device 100 superimposes data about the flight of the drone 160 on the augmented reality display 112 with a visual line of sight while the human operator 120 looks 204 toward the drone 160 in the air 210 when visibility is good, which is an ideal flying situation.

[0074] 24, the device 100 superimposes data about the flight of the drone 160 on the augmented reality display 112 with an augmented line of sight while the human operator 120 views 204 the drone 160 in the air 210 when visibility is reduced. The augmented line of sight may be achieved by guiding the human operator 120 to look in the correct direction of the target symbol 200. Optionally, a simulated drone 160 may be shown in the correct position. The reduced visibility may be caused by low light conditions, clouds, fog, smog, rain, snowfall, or other physical phenomena.

[0075] In the embodiment of FIG. 25, the device 100 superimposes data about the flight on the augmented reality display 112 with an augmented and simulated line of sight to the drone 160 while the human operator 120 sees 204 the drone 160 in the air 210 when the view is obstructed. The obstruction of the view can be caused by an obstacle 2502, i.e., when the drone 160 is behind the obstacle 2502. The obstacle 2502 can be the real object 1004 of FIG. 10 and FIG. 11, i.e., the obstacle 2502 can be a man-made object such as a building, a bridge, or a natural object such as a hill, a forest, etc. The augmentation is achieved by guiding the human operator 120 to look in the right direction with the target symbol 200 and pointing the simulated drone 160 to the right position.

[0076] In an embodiment, the device 100 superimposes flight related data onto the augmented reality display 112 with an extended line of sight to the drone while the human operator 120 looks 204 towards the drone 160 in the air 210 when the human operator 120 is in long range vision. This is not shown in any of the drawings, but essentially the drone 160 is, for example, high up in the sky or near the horizon and the human operator 120 is guided to look in the right direction with the target symbol 200, such that the human operator 120 only sees the drone 160 as a small object in the distance.

[0077] In the embodiment shown in Figures 26 and 27, the device 100 adjusts 2700 the display 2600 of flight-related data on the augmented reality display 112 so that the line of sight 2602 is unobstructed while the human operator 120 looks 204 toward the drone 160 in the air 210. As shown in Figure 26, the human operator 120 looks 204 toward the drone 160 with a free line of sight 2602. However, as the drone descends 2604, the map 2600 will eventually obstruct the line of sight 2602. As shown in Figure 27, the drone 160 is currently flying relatively low, but the line of sight 2602 remains free due to the movement 2700 of the map 2600 to the left.

[0078] Finally, consider FIGS. 28 and 29, which show an embodiment of a system comprising two devices 100,2800.

[0079] The first device 100 is used to assist a first human operator 120 in flying 210 a drone 160 in the air using a remote controller 150.

[0080] A first geographic location 2814 of the first human operator 120 relative to the position of the drone 160 in the air 210 is used to adjust a first viewpoint to render flight-related data including the first target symbol 200 and the first orientation symbol 202 superimposed on the first augmented reality display 112 of the first device 100.

[0081] As shown in FIG. 28, a second device 2800 is used to provide information to a second human operator 2802 regarding the flight of the drone 160 in the air 210.

[0082] A second geographic location 2804 of the second human operator 2802 relative to the position of the drone 160 in the air 210 is used to adjust a second viewpoint to render flight-related data including a second target symbol 2806 and a second orientation symbol 2808 superimposed on a second augmented reality display 2810 of the second device 2800.

[0083] In this manner, the second human operator 2802 can at least observe 2812 the flight of the drone 160 in the air 210. This may be useful, for example, simply for fun, for educational purposes, to pass a flight license test, for surveillance, to track missing persons, or to assist the first human operator 120. As discussed above, one or both of the operators 120, 2802 may be provided with one or more visual elements based on data captured in real time using one or more sensors 1402 onboard the drone 160.

[0084] In the embodiment shown in FIG. 29 , the second device 2800 is used to assist a second human operator 2802 in controlling 2902 one or more sensors 1402 mounted on the drone 160, while the first human operator 120 controls the flight direction 2908 and speed of the drone 160.

[0085] For example, if the sensor 1402 is an image sensor as described above, the second geographic location 2804 of the second human operator 2802 is used to adjust a second viewpoint that renders data about the flight, which is superimposed on a second augmented reality display 2810 of the second device 2800. The data about the flight also includes one or more video feeds captured in real time from one or more video cameras 2900 mounted on the drone 160. As shown in FIG. 29, the one or more video feeds 2904 are superimposed on the second augmented reality display 2810.

[0086] Note that the use case of Figure 29 is that because both operators 120, 2802 may be shown the same information on the augmented reality display 112, 2810, and both have a remote controller 150, 2906, responsibility for the flight may be seamlessly transferred between operators 120, 2802 during flight. This may be particularly useful during training sessions or long missions. It is also envisioned that, in particularly dangerous or restricted airspace, a licensed pilot 2802 may safely pilot the drone 160, after which the (original) operator 120 may regain control of the drone 160.

[0087] It should be noted that the scenarios shown in Figs. 28 and 29 in which the second human operator 2802 is physically present near the drone 160 and the first human operator 120 are not limiting. As previously mentioned, the external data communication interface 110 can communicate data regarding the flight to an external receiver 116. The receiver 116 may be a networked computer server that interoperates with the first device 100 and the second device 2800 according to a client-server architecture, a cloud computing architecture, a peer-to-peer system, or another applicable distributed computing architecture. In this way, the second human operator 120 can still observe or assist as described above, even if far away, even if in another city, country, or continent. Of course, data transmission delays should be minimized and taken into account, especially when, for example, the remote second human operator 2802 is controlling 2902 one or more sensors 1402.

[0088] Finally, embodiments relating to world markers are considered with reference to Figures 31, 32 and 33.

[0089] First, the device 100 obtains a geographic location for the drone 160. Then, the device 100 sets a world marker on the obtained geographic location. The world marker may be set by a user interface operation performed by the human operator 120, the world marker may be set automatically by the device 100 (e.g., when a predetermined condition is met), or the world marker may be set semi-automatically with the device 100 suggesting to the human operator 120 to set a world marker in the user interface (and the human operator 120 can confirm or cancel the world marker). In this way, the world marker links the drone 160 and the geographic location together. This linking can be done in several different ways. As previously described with reference to FIG. 5, a spot can be known in both real world coordinates 502 and augmented reality system coordinates 504. The "geographic location" referenced with respect to the "world marker" may refer to the real world coordinates 502, but in some cases it may also refer to the augmented reality system coordinates 504. The difference is that the real world coordinates 502 define a precise location (in a map system, as provided by GPS or another global navigation satellite system, or another positioning technology), whereas the augmented reality system coordinates 504 define a precise location within the augmented reality (such as within a building or other man-made structure that exists in the real world and is modeled within the augmented reality). Indeed, when the coordinate systems 502, 504 are locked together as described above, world markers are defined in both coordinate systems 502, 504.

[0090] However, even when GNSS positioning is not available, such as when flying inside a building, a locking relationship between the augmented reality system coordinate system 504 and the drone coordinate system 506 can be formed, and the positioning and orientation of the drone 160 is performed in the augmented reality model of the world. If a known model is available for the physical structures in the flight environment of the drone 160, the position represented in the real world coordinate system 502 of the entire structure can be mapped to the augmented reality coordinate system 504. Thus, even without GNSS, the position of the drone 160 in the real world coordinate system 502 is known. Of course, if indoor positioning information is available, it can be fused to further improve the accuracy of the position and orientation of the drone 160 and its mapping in the augmented reality coordinate system 504.

[0091] In the embodiment shown in FIG. 31, the geographic location for the drone 160 is the geographic location of the drone 160. This may be implemented such that the geographic location for the drone 160 is obtained by determining the geographic location 3100 and a world marker of the drone 160. The obtained geographic location 3100 is set by setting a world marker 3102 on the geographic location of the drone 160 (latitude and longitude coordinates, altitude above ground / above sea level). As shown in FIG. 31, the human operator 120 observes 204 the drone 160 in the real-world flight environment 210 through the device 100 and sets the world marker 3102 on the geographic location 3100 of the drone 160. With the assistance of the use of an augmented reality environment, the human operator 120 actually sees the setting of the world marker 3102 on the augmented reality display 112 caused by a user interface operation performed by the human operator 120 (in the augmented reality environment 112 controlled by the remote controller 150).

[0092] Note that the world marker 3102 may take a variety of graphic shapes, such as a pole or another symbol that models a real-world object (such as a park bench). The human operator 120 may add a comment ("missing person's backpack") or some predefined text label regarding the world marker 3102. The drone 160 may capture image data with an on-board sensor 1402 from the area around the world marker 3102 and bind to the world marker 3102 so that the image data can be later viewed by referencing the world marker 3102.

[0093] 32, the device 100 also superimposes on the augmented reality display 112 a map 2600 showing the geographic location 3106 of the world marker 3102. Additionally, the map 2600 may also show the geographic location 3104 of the human operator 120. Additionally, the map 3600 may also show a drone 3108 flying above the geographic location 3106 of the world marker 3102. In this manner, the operator 120 can compare the real world 210 with the map 2600 and identify the location under the drone 160 as a point of interest.

[0094] In the embodiment shown in FIG. 1, as shown in FIG. 32, the geographical location for the drone 160 is obtained from image data captured by the drone 160 itself. This may be implemented such that the geographical location for the drone 160 is obtained by: obtaining image data from one or more sensors 1402 mounted on the drone 160, receiving a user interface selection 3204 from a human operator 120 marking a location shown in the image data, and determining a real-world geographical location 3200 of the location shown in the image data. Furthermore, a world marker is set on the obtained geographical location 3200 by setting a world marker 3202 on the real-world geographical location 3200. As shown in FIG. 32, the human operator 120 observes a video feed 2904 obtained by an on-board video camera 1402 and marks a location 3204. However, as previously described, the one or more sensors 1402 may comprise one or more of (regular) visible light cameras, such as, for example, a photo camera, a thermal (or infrared) camera, a multispectral camera, a hyperspectral camera, a corona discharge camera, etc., in addition to or instead of a video camera. All of these may generate measurement data in the form of image data. Additionally, and as previously described, the one or more sensors 1402 on board the drone 160 may include, but are not limited to, one or more of the following technologies: LIDAR (light detection and ranging, or laser imaging, detection, and ranging, or 3D scanning) sensors, SONAR (voice navigation and ranging) sensors, RADAR (radio detection and ranging) sensors, chemical sensors, biological sensors, radiation sensors, particle sensors, magnetic sensors, network signal strength sensors. All of these can measure different types of data and visualize it as image data. In this way, the human operator 120 can mark a world location by pointing to a location 3204 in the video 2904 from the drone 160, and then extrapolate this point in the video frame to a world location based on the known drone 160 location and height, the known drone 160 heading, the known camera 1402 line of sight, and the known height above ground at the indicated location.This embodiment can be used to accurately identify a possible person within the video view of the drone 160 and obtain the person's latitude and longitude for use in emergency services, surveillance operations, or another application.

[0095] In the embodiment shown in FIG. 33, the geographical location for the drone 160 is specified as a view from one or more sensors 1402 mounted on the drone 160. This may be implemented such that the geographical location for the drone 160 is obtained by acquiring image data from one or more sensors 1402 mounted on the drone 160 and defining a view 3310 in the image data based on the geographical location and altitude of the drone 160, the measurement direction of the one or more sensors 1402, and the view frustum 3304 of the one or more sensors 1402. The view frustum 3304 may be formed by a pyramid cut as shown in FIG. 14, which in 3D computer graphics defines a three-dimensional area displayed on the screen. The pyramid is an adaptation of the (usually cone-shaped) vision of the camera 1402. A world marker on the acquired geographical location is set by setting the view 3310 as a world marker. Furthermore, a map 2600 showing the world marker as a view 3310 extending from the geographical location of the drone 160 is superimposed on the augmented reality display 112. Using this embodiment, a view 3310 of the image feed 2904 (captured by the image feed 2904) from the drone 160 can be overlaid on the map 2600 to better understand where a particular feature, e.g., a person, is shown on the map 2600. In this manner, the person 3300 and the real world location 3302 of the person 3300 can be estimated. FIG. 33 also shows an image feed 2904 including a person 3306 shown on the augmented reality display 112. The human operator 120 can observe the drone 160 in the augmented reality display 112 and simultaneously observe at least one of the image feed 2904 and the map 2600 on the augmented reality display 112.

[0096] In an embodiment, the device 100 uses the external data communications interface 110 described above to communicate data about the world markers to an external receiver 116. The device 100 uses the external data communications interface 110 to cause data about the world markers 3102, 3202, 3312 to be transmitted to an external receiver 116, such as emergency personnel, monitoring personnel, an operations center, etc.

[0097] Although the present invention has been described with reference to one or more embodiments according to the accompanying drawings, it is clear that the present invention is not limited thereto and can be modified in several ways within the scope of the appended claims. All words and expressions should be interpreted broadly and are intended to illustrate, not to limit, the embodiments. With the advancement of technology, it will be obvious to those skilled in the art that the concept of the present invention can be realized in various ways.

Claims

1. An apparatus (100) for assisting a human operator (120) in flying a drone (160) using a remote controller (150), comprising: an internal data communications interface (108) for receiving flight-related data from the remote controller (150); an augmented reality display (112) for displaying data relating to the flight to the human operator (120); one or more memories (104) containing computer program code (106); one or more processors (102) that execute the computer program code (106) to cause the device (100) to at least superimpose a target symbol (200) indicating a position of the drone (160) on the augmented reality display (112) while the human operator (120) is looking (204) toward the drone (160), superimpose an orientation symbol (202) indicating a direction of the drone (160) on the augmented reality display (112) while the human operator (120) is looking (204) toward the drone (160), obtain a geographic position for the drone (160), and set a world marker at the obtained geographic position; Equipped with Obtaining the geographical position of the drone (160) comprises: acquiring image data from one or more sensors (1402) mounted on said drone (160); receiving a user interface selection (3204) from the human operator (120) marking a location shown in the image data; and determining a real-world geographic location (3200) of the location shown in the image data; Setting a world marker (3202) at the acquired geographic location includes setting a world marker (3200) at a geographic location in the real world. Device.

2. Obtaining a geographic location for the drone (160) includes determining a geographic location (3100) of the drone (160); Setting the world marker at the acquired geographic location (3100) includes setting a world marker (3102) at the geographic location of the drone (160).

2. The apparatus of claim 1.

3. Obtaining a geographical position for the drone (160) includes obtaining image data from one or more sensors (1402) mounted on the drone (160) and defining a view (3310) for the image data based on the geographical position and altitude of the drone (160), measurement directions of the one or more sensors (1402), and a view frustum (3304) of the one or more sensors (1402); setting a world marker at the acquired geographic location includes setting the view (3310) as a world marker; overlaying on the augmented reality display (112) a map (2600) showing the world markers as a view (3310) extending from the geographic location of the drone (160); 2. The apparatus of claim 1.

4. overlaying on the augmented reality display (112) a map (2600) showing the geographic location (3106) of the world marker (3102); 4. Apparatus according to any one of claims 1 to 3.

5. The map (2600) also shows the geographic location (3104) of the human operator (120).

5. The apparatus of claim 4.

6. An external data communication interface (110) for transmitting data regarding the flight of the drone (160); Using the external data communication interface (110), data regarding the world markers (3102, 3202, 3312) is transmitted to an external receiver (116).

6. Apparatus according to any one of claims 1 to 5.

7. While a human operator (120) looks (204) toward said drone (160) in the air (210), superimposing on the augmented reality display (112) an indirect line of sight guideline (800) that extends horizontally to the geographic location of the drone (160) on the ground (500) and from there, an indirect line of sight guideline (802) that extends vertically to a target symbol (200) at the cruising altitude of the drone (160) in the air (210).

7. Apparatus according to any one of claims 1 to 6.

8. A map (1300) showing the geographic location (1302) of the human operator (120), the geographic location (1304) of the drone (160), and waypoints (1306) is superimposed on the augmented reality display (112). superimposing a vertical waypoint symbol (1308) on the augmented reality display (112) starting from the geographic location of the waypoint (1306) on the ground (500) and extending to a predetermined altitude of the waypoint (1306) while the human operator (120) is looking (204) towards the drone (160) in the air (210); 8. Apparatus according to any one of claims 1 to 7.

9. superimposing on the augmented reality display (112) one or more visual elements (1400) based on data captured in real time using one or more sensors (1402) mounted on the drone (160) in the vicinity of the target symbol (200) while the human operator (120) is looking (204) towards the drone (160) in the air (210); positioning the one or more visual elements (1400) on the augmented reality display (112) such that the human operator (120) has an unobstructed line of sight while looking (204) toward the drone (160) in the air (210); 9. Apparatus according to any one of claims 1 to 8.

10. superimposing on the augmented reality display (112) a map (1600) in a vertical layout showing a geographical position (1602) of the human operator (120) and a geographical position (1604) of the drone (160) in the vicinity of the target symbol (200) while the human operator (120) is looking (204) towards the drone (160) in the air (210); or superimposing on the augmented reality display (112) a map (1700) in a horizontal layout showing the geographic location (1702) of the human operator (112) and the geographic location (1704) of the drone (1706) while the human operator (120) is looking (204) towards the drone (160) in the air (210); 10. Apparatus according to any one of claims 1 to 9.

11. While a human operator (120) looks (204) towards a drone (160) in the air (210), When visibility is good, a line of sight (2300) to said drone (160), When visibility is reduced, with an extended line of sight (2400) to the drone (160), With an augmented and simulated line of sight to the drone (160) when visibility is obstructed, or When long-range visibility is ensured, with an extended line of sight to the drone (160), Overlaying flight-related data on an augmented reality display (112); 11. Apparatus according to any one of claims 1 to 10.

12. 1. A method for assisting a human operator in flying a drone using a remote controller, comprising: receiving data relating to the flight from the remote controller (3002); displaying (3004) the data regarding the flight to the human operator on an augmented reality display; superimposing (3006) a target symbol indicating the location of the drone on the augmented reality display while the human operator is looking towards the drone; superimposing (3008) an orientation symbol on the augmented reality display indicating the direction of the drone while the human operator is looking towards the drone; obtaining a geographic location for the drone (160); 1. A method for setting a world marker at an acquired geographic location, comprising: Obtaining the geographical position of the drone (160) comprises: acquiring image data from one or more sensors (1402) mounted on said drone (160); receiving a user interface selection (3204) from the human operator (120) marking a location shown in the image data; and determining a real-world geographic location (3200) of the location shown in the image data; Setting a world marker (3202) at the acquired geographic location includes setting a world marker (3200) at a geographic location in the real world. method.

13. A computer readable medium (170) including computer program code (106) that, when executed by one or more processors (102), implements a method for assisting a human operator in flying a drone using a remote controller, the method comprising: The method comprises: receiving data relating to the flight from the remote controller (3002); displaying (3004) the data relating to the flight to the human operator on an augmented reality display; and superimposing (3006) on the augmented reality display a target symbol indicating the location of the drone while the human operator is looking towards the drone; superimposing (3008) on the augmented reality display an orientation symbol indicating the direction of the drone while the human operator is looking towards the drone; Obtaining a geographical position of the drone (160); Setting a world marker at the acquired geographic location; Obtaining the geographical position of the drone (160) comprises: acquiring image data from one or more sensors (1402) mounted on said drone (160); receiving a user interface selection (3204) from the human operator (120) marking a location shown in the image data; and determining a real-world geographic location (3200) of the location shown in the image data; Setting a world marker (3202) at the acquired geographic location includes setting a world marker (3200) at a geographic location in the real world. Computer-readable medium.

Images