Apparatus, method, and software for assisting a human operator who flies a drone using a remote controller

Abstract

An apparatus (100), method, and software for assisting a human operator (120) in flying a drone (160) using a remote controller (150), the apparatus (100) includes an internal data communications interface (108) for receiving data from the remote controller (150), an augmented reality display (112) for displaying the data, one or more memories (104) containing computer program code (106), and one or more processors (102) for causing the apparatus (100) to superimpose a target symbol (200) indicative of the position of the drone (160) on the augmented reality display (112) while the human operator (120) is looking towards the drone (160), and to superimpose an orientation symbol (202) indicative of the direction of the drone (160) on the augmented reality display (112) while the human operator (120) is looking towards the drone (160) while the human operator (120) is looking towards the drone (160).

Description

Technical Field

[0001] Various embodiments relate to an apparatus for assisting a human operator who flies a drone using a remote controller, a method for assisting a human operator who flies a drone using a remote controller, and a computer-readable medium including computer program code for executing the method, which, when executed by one or more processors, results in a degradation of the performance of the method.

Background Art

[0002] A human operator (on the ground) uses a remote controller to fly a drone (or unmanned aerial vehicle (UAV)), optionally assisted at least in part by autopilot.

[0003] The human operator has to simultaneously face the direction of the drone in the air, operate the hand-held remote controller, and sometimes also look at the display of the remote controller. This leads to a decrease in situation awareness and can potentially cause dangerous situations.

[0004] As a legal requirement, the human operator has to maintain visual contact (line of sight) with the drone in the air. This is very difficult because the drone may not be visible due to long distances, low ambient light, physical obstacles, etc.

[0005] These problems can be alleviated by another person, a so-called assistant (spotter), maintaining visual contact with the drone using binoculars, but the human operator can then focus on operating the remote controller (although still sometimes having to check the display of the remote controller). Naturally, such a setup requires excellent communication skills between the human operator and the spotter. Furthermore, since the manual work is doubled, the operating cost of the drone increases.

[0006] U.S. Patent Application Publication No. 2018 / 0196425, U.S. Patent Application Publication No. 2019 / 0077504, and U.S. Patent Application Publication No. 2019 / 0049949 disclose various aspects related to the use of head-mounted displays in the flight of drones. SUMMARY OF THE INVENTION

[0007] According to one aspect, the subject matter of the independent claims is provided. The dependent claims define several embodiments.

[0008] One or more examples of the implementation are described in more detail in the accompanying drawings and the description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0009]

Figure 1A

Figure 1B

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Embodiments for Carrying Out the Invention

[0010] Next, several embodiments will be described with reference to the accompanying drawings.

[0011] The following embodiments are examples. Although the specification may refer to "one" embodiment in multiple places, this does not necessarily mean that each such reference is to the same embodiment, or that its features apply only to individual embodiments. Individual features of different embodiments may be combined to provide other embodiments. Furthermore, the terms "comprising" and "including" should be understood not to limit the described embodiments to consisting only of the recited features, and such embodiments may include features / structures not specifically recited.

[0012] The reference signs in both the description of the embodiments and the claims are for explaining the embodiments with reference to the drawings and are not limited to these examples only.

[0013] The embodiments and features disclosed in the following description that do not fall within the scope of the independent claims should be construed as examples useful for understanding the various embodiments of the present invention.

[0014] Figure 1A shows a simplified block diagram of an apparatus 100 that assists a human operator (or pilot) 120 in flying a drone 160 using a remote controller 150. The drone 160 is also known as an unmanned aerial vehicle (UAV). An unmanned aircraft system (UAS) can be defined as including the drone (or UAV) 160, the (ground-based) remote controller 150, and a wireless communication system 152 between the remote controller 150 and the drone 160.

[0015] Meanwhile, Figure 30 shows a flowchart of an embodiment of a method for assisting a human operator 120 in flying a drone 160 using a remote controller 150.

[0016] The method starts at 3000 and ends at 3010. Note that the method can be executed as long as necessary (after the device 100 is started until the switch is turned off) by looping back to step 3002.

[0017] In Figure 30, the steps are not strictly in chronological order. Some steps may be executed simultaneously, and certain steps may be executed in a different order. Other functions may also be executed between or within the steps, and other data that is exchanged between the steps may also be executed. Some steps or parts of steps may be omitted or replaced by corresponding steps or parts of steps. Note that a special order of steps is not necessary, except when required by the logical requirements of the processing sequence.

[0018] The device 100 includes an internal data communication interface 108 that receives 3002 flight-related data from the remote controller 150. The flight-related data may include telemetry data of the drone 160. The data related to flight may include, but is not limited to, sensor values such as gyroscopes and magnetometers, fused data such as angular velocity, speed, altitude, and earth position, aircraft information such as battery, gimbal, and flight status, etc. Depending on the environment of the drone, the data may also be received directly from the drone 160 by the device 100.

[0019] The internal data communication interface 108 can be implemented using a wireless transceiver that communicates with the wireless transceiver of the remote controller 150. The technology of the internal data communication interface 108 includes, but is not limited to, a wireless local area network (WLAN) implemented using the IEEE 802.11ac standard or Wi-Fi protocol suite, a short-range wireless network such as Bluetooth or Bluetooth LE (Low Energy), a cellular wireless network using a subscriber identity module (SIM) or eSIM (embedded SIM), or other standard or proprietary wireless connection means, including one or more. Note that in some use cases, the internal data communication interface 108 can utilize a standard or proprietary wired connection, such as a suitable bus, as an addition or alternative. In an embodiment, a wired connection according to the USB (Universal Serial Bus) standard is utilized.

[0020] The apparatus 100 also includes an augmented reality (AR) display 112 that displays 3004 flight-related data to the human operator 120. FIGS. 2 through 29 illustrate specific embodiments, but in addition to these, various flight-related notifications and statuses can also be displayed on the augmented reality display 112.

[0021] In the drawings, the augmented reality display 112 is implemented as a head-mounted display with a headband attached (or attached to a helmet) and worn in front of the eyes of the human operator 120 as a visor. In the drawings, the augmented reality display 112 is implemented as a see-through display on which holographic images are displayed. In an alternative embodiment, the augmented reality display 112 can use a camera to intercept a view of the real world and display an augmented view of the real world as a projection.

[0022] In an embodiment, the apparatus 100 is implemented using a Microsoft (R) HoloLens (R) 2 (or later version) mixed reality smart glass that employs a see-through holographic lens as the augmented reality display 112, providing a complete development environment. The head-mounted apparatus 100 includes the necessary processors (including system-on-chip, custom-made holographic processing units, and coprocessors) 102, memory 104 and software 106, depth cameras, video cameras, projection lenses, inertial measurement devices (including accelerometers, gyroscopes, and magnetometers), wireless connection devices 108, 110, and a rechargeable battery. Note that some of these components are not shown in FIG. 1. Such an off-the-shelf environment provides, for example, an augmented reality engine 144 that fuses the real world and augmented reality and provides basic operations related to tracking the movements of the head and eyes of the human operator 120.

[0023] However, other applicable implementations of the augmented reality display 112 may be used, including but not limited to glasses, head-up displays, contact lenses with augmented reality imaging, etc. For the purposes of this embodiment, the augmented reality display 112 provides an interactive real-time experience of the real-world flight environment 210 and the drone 160 enhanced by computer-generated perceptual information. In addition to the natural environment 210 and the drone 160, flight-related data is superimposed (or overlaid).

[0024] The apparatus 100 also includes one or more memories 104 that contain computer program code 106, and one or more processors 102 that execute the computer program code 106 to cause the apparatus 100 to perform the necessary data processing. The data processing performed by the apparatus 100 may be interpreted as a method or algorithm 130.

[0025] The term "processor" 102 refers to a device capable of processing data. In an embodiment, the processor 102 is implemented as a microprocessor that implements the functions of a central processing unit (CPU) on an integrated circuit. The CPU is a logical machine that executes computer program code 106. The CPU may include a set of registers, an arithmetic logic unit (ALU), and a control unit (CU). The control unit is controlled by a sequence of computer program code 106 transferred from the (working) memory 104 to the CPU. The control unit can include a number of microinstructions for basic operations. The implementation of microinstructions may vary depending on the CPU design. One or more processors 102 may be implemented as cores of individual processors and / or as separate processors.

[0026] The term "memory" 104 refers to a device capable of storing data either temporarily (= working memory) or permanently (= non-volatile memory) during execution. Working memory and non-volatile memory may be implemented by random access memory (RAM), dynamic RAM (DRAM), static RAM (SRAM), flash memory, solid state disk (SSD), PROM (programmable read), suitable semiconductors, or other means implementing electrical computer memory.

[0027] The computer program code 106 is implemented by software. In an embodiment, the software may be written in a suitable programming language, and the resulting executable code may be stored in the memory 104 and executed by one or more processors 102.

[0028] The computer program code 106 implements the method / algorithm 130. The computer program code 102 can be coded as a computer program (or software) using a programming language, which can be a high-level programming language such as C, C++, or Rust. The computer program code 106 can be in source code form, object code form, executable file, or some intermediate form, but it needs to be in an executable form as an application 140 for use by one or more processors 102. There are many ways to structure the computer program code 106. Depending on the software design methodology and the programming language used, the process can be divided into modules, subroutines, methods, classes, objects, applets, macros, etc. In the latest programming environments, there are software libraries, that is, compilations of ready-made functions, which can be utilized by the computer program code 106 to perform a wide variety of standard operations. Furthermore, an operating system (such as a general-purpose operating system) can provide system services to the computer program code 106.

[0029] As shown in FIG. 30, the embodiment provides a computer-readable medium 170 that stores computer program code 106, which, when loaded onto one or more processors 102 and executed by the one or more processors 102, causes the one or more processors 102 to execute a method / algorithm 130. The computer-readable medium 170 may include at least any entity or device, recording medium, computer memory, read-only memory, electrical carrier signal, electrical communication signal, and software distribution medium that can carry the computer program code 106 to the one or more processors 102. In some jurisdictions, depending on laws and patent practices, the computer-readable medium 170 may not be an electrical communication signal. In an embodiment, the computer-readable medium 170 may be a computer-readable medium. In an embodiment, the computer-readable medium 170 may be a non-transitory computer-readable medium.

[0030] As shown in FIGS. 1A and 1B, the computer-readable medium 170 can carry computer program code 160 as an executable application 140 of the device 100 and also as an executable application 142 of the remote controller 150 for transmitting flight-related data to the device 100. In a typical drone environment such as DJI (registered trademark), a software development kit can be used for the application 142 to interface with the remote controller 150.

[0031] FIG. 1A shows the device 100 as an integrated unit including an augmented reality display 112, one or more memories 104 including computer program code 106, and one or more processors 102.

[0032] However, as shown in FIG. 1B, although the apparatus 100 provides the augmented reality display 112 to the human operator 120, it may be implemented as a distributed apparatus 100 such that a separate processing unit 180 is provided. The apparatus 100 is communicatively coupled to the augmented reality display 112 and the remote controller 150. The apparatus 100 also includes one or more memories 104 containing computer program code 106 and one or more processors 102. This may be implemented such that the processing unit 180 is a user device carried by the human operator 120, such as a smartphone, a tablet computer, or a portable computer, and the communication connection may be wired or wireless. Other embodiments may be a networked computer server or the like that interoperates with the augmented reality display 112 according to a client-server architecture, a cloud computing architecture, a peer-to-peer system, or other applicable distributed computing architectures.

[0033] FIGS. 2 and 3 show embodiments of views provided by the augmented reality display 112 of the apparatus 100. FIGS. 2 through 29 show each use case as a combination of two different perspectives.

[0034] Examine FIGS. 2 and 3 in detail. The first perspective shows flight. The human operator 120 operates the remote controller 150 and observes (or looks at) the drone 160 in the air 210 through the apparatus 100, or more precisely, through the augmented reality display 112 of the apparatus 100 (204). As shown in the illustration, the second perspective shows the elements 200, 202 shown on the augmented reality display 112.

[0035] This rule is used in all the drawings from Figure 2 to 29. The dotted arrow line 204 indicates the direction that the human operator 120 is looking at, usually the direction of the aerial drone 160. However, in some use cases, the human operator 120 looks in another direction such as towards the ground, and is indicated by the dotted arrow lines with reference numerals 600, 1404, 1706, 1804, 2602, etc. However, in the aforementioned alternative embodiments, a camera is used to block the view of the real world and, as a projection, display an extended view of the real world. The direction of the line of sight towards the augmented reality display 112 may be different from the capture direction of the camera. For example, in order to make the flight attitude easier, the human operator 120 does not need to tilt the head to look at the sky, but the camera can be tilted upwards.

[0036] It should be noted that in all the described embodiments, the human operator 120 stands on the ground 500 and the drone 160 is flying in the air 210. However, the embodiments are also applicable when the drone 160 is made to fly inside an underground cave, inside a man-made structure (such as a building or a tunnel), or when the drone 160 is flying below the human operator 120, that is, the human operator 120 is looking towards the drone 160 and looking down instead of up, in other types of environments. In such use cases, the human operator 120 may stand on a high platform (such as a high-rise building or a mountain) and the drone 160 may fly below it (such as above a road or in a valley). The embodiments can also be applied to making the drone 160 fly submerged in water, that is, the drone 160 becomes an unmanned underwater vehicle (UUV), and when the drone is in water such as a river, a lake, a sea, a mine or a tunnel filled with water, the human operator 120 may operate the drone 160, for example, from the land or a ship.

[0037] In a sense, all the drawings from Figure 2 to Figure 29 show a hybrid that represents augmented reality over the real world. The real world is shown from an external perspective (like the perspective of another person observing the use case from outside the real world), while the augmented reality display 112 is shown from the first-person perspective of the human operator 120.

[0038] Now, returning to Figures 2 and 3. In an embodiment, the device 100 superimposes 3006 a target symbol 200 indicating the position of the drone 160 (in the case of a UAV, in the air 210) on the augmented reality display 112 while the human operator 120 is looking in the direction of the drone 160 (in the case of a UAV, in the air 210) at 204. In an embodiment, the device also superimposes 3008 an azimuth symbol 202 indicating the direction of the drone 160 (in the case of a UAV, in the air 210) on the augmented reality display 112 while the human operator 120 is looking in the direction of the drone 160 (in the case of a UAV, in the air 210) at 204.

[0039] By using the augmented reality display 112, the human operator 120 can look in the direction of the drone 160 in the air 210 during flight at 204. This improves the operator 120's situation awareness regarding the flight without the need for a spotter. The human operator maintains visual contact (by line of sight) with the drone 160 in the air 210, but at the same time, as will be described later, the aviation data at the actual correct world position is also shown.

[0040] The target symbol 200 indicates the position of the drone 160 in the air 210, which makes it easier for the human operator 120 to track the drone 160 during flight. In an embodiment, the target symbol 200 is a reticle as shown. Reticles 200 are commonly used in firearm telescopic sights. The reticle 200 can include a combination of a circle 300 and a partial crosshair 302 as shown in Figure 3. However, other patterns such as dots, posts, chevrons, etc. may also be used.

[0041] The orientation symbol 202 indicates the orientation of the drone 160 in the air 210, thereby making it easier for the operator 120 to understand the effect of the control commands given to the drone 160 by the remote controller 150 during flight. In an embodiment, the orientation symbol 202 is an arrow as shown in the figure. As shown in FIG. 3, the arrow 202 can be enlarged by an arc 304 that represents a part of a 360-degree circle around the human operator 120. The arrow 202 can point in the direction of travel of the drone 160, as will be described later.

[0042] In the augmented reality display 112, the target symbol 200 and the orientation symbol 202 from the digital world blend into the human operator 120's perception of the real world through an immersive integration perceived as a natural part of the flight environment 210.

[0043] Consider FIGS. 4, 5, 6, 7, 8, and 9, which show embodiments of the orientation of the drone 160.

[0044] In the embodiment, the orientation symbol 202 indicates a predetermined direction fixed with respect to the direction of the drone 160 in the air 210. Since the human operator 120 recognizes the predetermined direction, the human operator 12 can understand how the steering commands given by the remote controller 150 affect the flight. As shown in FIG. 13, the remote controller 150 can include, for example, two joysticks 310, 312 for giving steering commands. Naturally, other types of steering devices are also compatible with the described embodiments. However, the remote controller 150 can control the drone 160 in various degrees of freedom, namely, roll that tilts the drone 160 left or right, pitch that tilts the drone 160 forward or backward, and yaw that rotates the drone 160 clockwise or counterclockwise. Further, the altitude control device controls the drone 160 to fly higher or lower. Note that some user interface elements of the remote controller 150 can be programmed to interact with the device 100 so that, in addition to the user interface operations of the device 100 being performed in the augmented reality environment, they can also be performed by the user interface elements of the (physical) remote controller 150.

[0045] In the embodiment shown in FIG. 4, the predetermined direction is fixed with respect to the traveling direction 400 of the drone 160. In navigation, the traveling direction 400 of the drone 160 is the compass bearing in which the nose of the drone 160 is pointing. Note that, for example, a drone 160 that is a quadcopter (= helicopter with four rotors) may not have a "natural" nose, in which case one direction of the drone 160 is simply defined as the nose.

[0046] FIG. 5 shows various coordinate systems 502, 504, 506 that need to be related to each other to enable the embodiments. The world coordinate system 502 defines a three-dimensional world model visualization that is mapped to the coordinate system 504 of the device 100 and the coordinate system 506 of the drone 160. The device 100 then uses its own coordinate system 504 to show augmented reality, but also shows the position of the drone 160 and the position of the human operator 120 in the world coordinate system 502.

[0047] In the embodiments shown in FIGS. 5 and 6, the device 100 performs the following. · Obtaining the position of the drone 160 on the ground 500 in the world coordinate system 502. · Obtaining the position of the drone 160 on the ground 500 in the augmented reality coordinate system 504 of the device 100. · Locking the position of the drone 160 in the augmented reality coordinate system 504 and the position of the drone 160 in the world coordinate system 502. · Obtaining the traveling direction 400 of the drone 160 on the ground 500. · Setting the traveling direction 400 to be the direction of the calibration azimuth symbol in the augmented reality coordinate system 504 of the device 100.

[0048] In this way, the augmented reality coordinate system 504 that always tracks the movement of the head of the human operator 120 is firmly based on the world coordinates 502 and also follows the actual compass azimuth 602. In this way, the combination of the latitude and longitude of the world (x and z of the world coordinate system 502) and the compass azimuth information 602 in the augmented reality presentation is achieved.

[0049] In a more specific embodiment, the device 100 performs the following. · Obtaining (from the remote controller 150 or the drone 160) the position of the drone 160 on the ground 500 in the world coordinate system 502. · Superimposing a calibration position symbol on the augmented reality display 112. · After the calibration position symbol is placed on the drone 160 (such as at the center of the drone 160 or another predetermined point on the drone 160), receiving a first user confirmation. · Obtaining (from the augmented reality engine 144) the position of the drone 160 on the ground 500 in the augmented reality coordinate system 504 of the device 100. · Locking the position of the drone 160 in the augmented reality coordinate system 504 and the position of the drone 160 in the world coordinate system 502. · Obtaining (from the remote controller 150 or the drone 160) the traveling direction 400 of the drone 160 on the ground 500. · Superimposing the calibration azimuth symbol onto the augmented reality display 112. · Receiving confirmation from the second user after the calibration azimuth symbol is aligned with the drone 160 (e.g., the line between the tail and the nose of the drone 160 or another predetermined direction of the drone 160). · Setting the traveling direction 400 as the direction of the calibration azimuth symbol in the augmented reality coordinate system 504 of the device 100.

[0050] First, in the augmented reality system, the position of the drone 160 in the world coordinate system 502 and the position of the drone 160 relative to the augmented reality coordinate system 504 are shown. By using the augmented reality indicator to show that the center of the drone 160 is located at an exact spot within the view 112 of the augmented reality, that spot is known in both the real-world coordinate system 502 and the augmented reality coordinate system 504. With this combination, a fixed common position is obtained together with the world latitude and longitude information. Since the latitude and longitude are known at this time as exact coordinates (provided by GPS or another global navigation satellite system, or another positioning technology such as cellular radio-based positioning), they are obtained from the drone 160. An augmented reality pointerstic, or another type of calibration position symbol, can indicate the position within the augmented reality display 112 to the human operator 120. When indicating the position of the drone 160, this stick, which moves a certain distance in front of the human operator 160 and points downward, is guided to come above the center of the drone 160. It is held steadily to confirm the position, and then the coordinate systems 502, 504 are locked together. Alternatively, this can also be done using machine vision, looking at the drone 160, decoding its position in the augmented reality coordinate system 504, and then even locking the latitude and longitude of the drone 160 and its shape. There are many ways to indicate the position of the drone 160, but it needs to be done reliably to securely lock the world and the augmented reality coordinate systems 502, 504.

[0051] Second, the drone 160 knows where its nose is pointing, that is, the drone 160 communicates its compass bearing in degrees, and this can be used to finally determine the alignment of the coordinate systems 502, 504. The augmented reality system is used to align with a displayed line, or another type of calibration bearing symbol with the tail fin line of the drone 160, and when this is achieved, the compass bearing of this displayed line in the world coordinate system 502 is known. Therefore, the bearing of the world compass in any direction, for example north, can be calculated from there.

[0052] As an optional step, when the world position (latitude, longitude) is obtained from the drone 160, the exact altitude (y in the world coordinate system 502) can also be queried from the map system based on the exact world coordinates. Thus, (if exact accuracy is required, using the drone-specific offset from the ground 500 to the top surface of the drone 160) the altitude of this point in space can also be calibrated, so that in the future map data can be used to accurately determine the terrain altitude of other world points. In summary, to achieve world lock, latitude, longitude, and possibly altitude, and the compass orientation may be required.

[0053] After this coupling, everything else in the system is built based on the knowledge of where the drone 160 actually is in the world coordinates 502 and what is exactly around it in the world. It should be noted that the described embodiments regarding the coupling can operate as independent embodiments, independent of all other embodiments and the embodiments described in connection with the independent claims and other dependent claims.

[0054] The data regarding the flight is mapped to the world coordinates 502, and as a result, it is displayed such that its visualization is performed using the knowledge of its three-dimensional position represented in the world coordinate system 502 that is locked to the extended reality coordinate system 504: 3004, 3006, 3008.

[0055] In the embodiment shown in FIG. 7, situation awareness can be further enhanced by numerical information. While a human operator 120 is looking at the drone 160 in the air 210 at 204, the device 100 uses a numerical value and scale 700 visually coupled with the target symbol 200 to superimpose the cruising altitude 704 of the drone 160 on the augmented reality display 112. As shown in FIG. 7, the scale 700 can include horizontal lines each indicating a specific altitude. The device also superimposes, on the augmented reality display 112, the direction of travel 706 of the drone 160 at degrees 702 visually coupled with the azimuth symbol 202 while a human operator 120 is looking at the drone 160 in the air 210 at 204. This can be useful for a skilled human operator 120.

[0056] In the embodiment shown in FIG. 8, while a human operator 120 is looking at the drone 160 in the air 210 at 204, the device 100 superimposes, on the augmented reality display 112, an indirect line-of-sight guideline 800 that horizontally extends to the geographical location of the drone 160 on the ground 500 and from there the indirect line-of-sight guideline 802 vertically extends to the target symbol 200 at the cruising altitude of the drone 160 in the air 210. This can further enhance situation awareness because the human operator 120 can first observe the horizontal guideline 800 to confirm the geographical location of the drone 160 on the ground surface 500 and then observe the vertical guideline 802 to grasp where the drone 160 is within the air 210.

[0057] In the embodiment shown in FIG. 9, while a human operator 120 is looking at the drone 160 in the air 210 at 204, the device 100 superimposes, on the augmented reality display 112, a track symbol 900 indicating the track 902 and speed of the drone 160 in the air 210. In navigation, the track 902 is the path along which the drone 160 actually moves. The difference between the direction of travel 400 and the track 902 is caused by the movement (such as air currents) in the air (Air) 210. By indicating the track 902 and speed, the human operator 120 can predict the effect of the current control that can be adjusted as needed.

[0058] Next, consider an embodiment of visualizing an obstacle in relation to the drone 160, shown in FIGS. 10, 11, and 12.

[0059] In the embodiment shown in FIG. 10, the apparatus 100 superimposes on the augmented reality display 112 an obstacle index symbol 1000 that depicts a distance 1002 from the drone 160 to a real object 1004 while a human operator 120 is looking in the direction of the drone 160. The distance 1002 can be the shortest distance between the drone 160 and the real object 1004. As shown in FIG. 10, the obstacle index symbol 1000 can mark the distance using an arrow and, in some cases, is extended by a numerical value indicating the distance 1002. The real object 1004 may be an artifact such as a building or a bridge, or a natural object such as a hill or a forest.

[0060] FIG. 11 shows an additional embodiment, where the obstacle index symbol 1000 includes a visual indicator 1100 that is at least partially superimposed on the real object 1004. As shown in FIG. 11, the visual indicator 1100 may be a shading or a similar visual effect superimposed on the real object 1004. In this way, the human operator 120 can immediately recognize the risk of collision as the drone 160 approaches the object 1004.

[0061] FIG. 12 shows a further embodiment applicable to either the embodiment of FIG. 10 or the embodiment of FIG. 11. The obstacle index symbol 1200 includes elements representing a shortest horizontal distance 1202 and a shortest vertical distance 1204 from the drone 160 to a real object 1206. In this way, in order to avoid a collision with the real object 1206, the influence of the movement of the drone 160 in both the vertical and horizontal directions can be recognized.

[0062] Next, consider FIG. 13 showing an embodiment of visualizing an intermediate point with respect to the drone 160. The apparatus 100 superimposes a map 1300 showing the geographical location 1302 of the human operator 120 and the geographical location 1304 of the human operator 120 on the augmented reality display 112. In this way, the human operator 120 can more intuitively and better understand the surrounding situation regarding the flight. As shown in FIG. 13, the map 1300 and the drone 160 can simultaneously exist within the field of view of the human operator 120, and the line of sight can alternately occur between being directed towards the drone 204 or towards the map 1300 1310. While the human operator 120 is looking in the direction 204 of the drone 160, the apparatus 100 superimposes on the augmented reality display 112 a vertical intermediate point symbol 1308 that starts from the geographical location of the intermediate point 1306 on the ground 500 and extends towards a predetermined altitude of the intermediate point 1306. The narrow portion of the intermediate point symbol 1308 can accurately indicate the geographical location on the Earth 500, while the wider portion of the intermediate point symbol 1308 can indicate the set altitude of the intermediate point in the air 210. The intermediate point symbol 1308 is shown at the correct position in the real world.

[0063] Next, FIGS. 14 and 15 illustrate an embodiment for visualizing data captured by drone 160. While human operator 120 is looking at drone 160 in the air 210 at 204, one or more visual elements 1400 captured in real time using one or more sensors 1402 mounted on drone 160 near target symbol 200 are superimposed on augmented reality display 112, and while human operator 120 is looking at drone 160 in the air 210 at 204, one or more visual elements 1400 are arranged on augmented reality display 112 so that the line of sight is not blocked. Visual elements 1400 can be arranged on either side of target symbol 200 as shown, but can be arranged anywhere around target symbol 200. Human operator 120 can quickly glance at visual elements 1400 at 1404, but mainly looks at drone 160 at 204 while simultaneously operating drone 170. In the illustrated embodiment, image sensor 1402 captures an image or video feed as data. In this way, human operator 120 can operate drone 120 so that image sensor 1402 captures a desired view. Note that the image sensor can operate as a (conventional) visible light camera such as a photographic camera or a video camera. In addition to this, the image sensor can operate as, for example, a thermal (or infrared) camera, a multispectral camera, a hyperspectral camera, or a corona discharge camera. One or more sensors 1402 mounted on drone 160 can include one or more of, but are not limited to, lidar (light detection and ranging, or laser imaging, detection and ranging, or 3D laser scanning) sensors, sonar (sound navigation and ranging) sensors, radar (radio detection and ranging) sensors, chemical sensors, biological sensors, radiation sensors, particle sensors, magnetic sensors, network signal strength sensors, and the like. Drone 160 can carry any combination of these sensors 1402 as a payload, and the data is visualized as described using one or more dynamically arranged visual elements 1400.

[0064] FIG. 16 and FIG. 17 show embodiments for visualizing maps related to the flight of drone 160. A human operator 120 may select the layout of maps 1600, 1700, or the device 100 may automatically determine which layout to use according to the flight situation. While the human operator 120 is looking at the drone 160 in the air 210, the device 100 superimposes on the augmented reality display 112 a vertically laid out map 1600 showing the geographical location 1602 of the human operator 120 and the geographical location 1604 of the drone 160 in the vicinity of the target symbol 200. Alternatively, while the human operator 120 is looking in the direction of the ground 500, the device 100 superimposes on the augmented reality display 112 a horizontally laid out map 1700 showing the geographical location 1702 of the human operator 112 and the geographical location 1704 of the drone 160. By using the vertically laid out map 1600, the human operator can always maintain situation awareness when gazing at the drone 160 and looking at the lateral map 1600. By using the horizontally laid out map 1700, although the human operator 120 needs to look at the ground 500, as shown, since the surface of the map 1700 is parallel to the ground surface 500, the map 1700 can be shown larger and more intuitively.

[0065] FIG. 17 also shows that the map 1700 used can be a three-dimensional topographic map that also shows altitude data indicated by three-dimensional buildings 1708.

[0066] Figures 18 and 19 show an embodiment of visualizing the menu structure of the apparatus 100. The apparatus 100 superimposes a menu structure 1800 around the human operator 120 on the augmented reality display 112 while the human operator 120 is looking towards the ground 500 between 1804. The apparatus 100 detects a gesture 1802 from the operator 120 as a command regarding the menu structure 1800, and controls the display 1900 of flight-related data on the augmented reality display 112 based on the command. In this way, the human operator 120 can quickly operate the apparatus 100. As shown, Figure 18 shows a basic display of the target symbol 200 and the azimuth symbol 202, and in Figure 19, the human operator 120 selects from the menu structure 1800 to display the cruising altitude of the drone 160 using the numerical value and scale 700 visually coupled to the target symbol 200 as described above with reference to Figure 7.

[0067] Figures 20, 21, and 22 show an embodiment of visualizing external data regarding the physical environment of the drone 160.

[0068] As shown in FIGS. 1A and 1B, device 100 includes an external data communication interface 110 that receives external data 114 regarding the physical environment of drone 160. Note that the external data communication interface 110 may be implemented using the internal data communication interface 108 in an embodiment. Device 100 superimposes one or more visualizations 2000 of the external data 114 onto the augmented reality display 112. In this way, device 100 can enhance the situation awareness of the human operator 120 by incorporating the external data source into the augmented reality display 112. As described above, the external data 114 is mapped to the world coordinates 502 and as a result, is locked to the augmented reality coordinate system 504, and is displayed to be visualized using its three-dimensional position represented in the world coordinate system 502. In addition to obtaining external data from various sources, the external data communication interface 110 can also be used to communicate flight-related data to the external receiver 116. The data includes, but is not limited to, the position of the drone 160, the voice from the human operator 120, one or more video feeds from the drone 160, etc.

[0069] As shown in FIG. 20, the external data 114 can include weather data and one or more visualizations 2000 indicating the weather data. In an embodiment, the weather data includes information regarding wind speed and wind direction. The direction is indicated by an arrow and the speed as illustrated is indicated by the scale of the arrow or a numerical value. Further, or alternatively, the weather data can include one or more of turbulence (predicted or known), humidity, cloud visualization, rain warning, hail warning, snow warning, storm warning, lightning warning, lighting conditions (time, position of the sun and / or moon), fog, temperature and pressure, visibility, dew point (important for airline pilots), "feels like" temperature. And all of these may be associated with time, i.e., weather forecasts such as approaching cloud fronts or wind changes may be visualized.

[0070] As shown in FIG. 21, the external data can include 114 air traffic control data including the classification of airspace, and one or more visualizations 2100, 2102 showing the classification of the airspace that coincides with the position of the drone 160 in the air 210. As shown in the figure, the free airspace 2100 may be marked with "I", and the restricted airspace 2102 may be marked with "II", and may be a shaded rectangle as shown in the figure, or another three-dimensional shape (such as a polygon mesh), or a two-dimensional shape (such as a polygon). Generally, the classification of airspace can include, but is not limited to, drone flight prohibited areas (areas, volumes), reservation and notification of airspace for drones and / or other aviation operations, airfield control areas, airspace control areas, power transmission lines and other obstacles, national border areas, all of the above at various altitudes, warning / danger / restricted areas, UVA reservation areas, UAS reservation areas, model aircraft reservation areas. The aviation map can be visualized using a three-dimensional polygon mesh with various walls, roofs, flight levels, etc., all of which are in the correct place as seen on the augmented reality display 112.

[0071] As shown in FIG. 22, the external data can include 114 air traffic control data including the position of the aircraft 2200 in the air 210, and one or more visualizations 2202, 2204 show the position of the aircraft 2200 in the air 210. Embodiments show additional visualizations 2202, 2204 of aircraft 2200 flying within a predetermined distance (e.g., within a radius of 3, 5, or 10 kilometers, etc.) from the position of the drone 160 in the air 210. The visualization may be implemented by arrows 2202, 2204 indicating the position of the aircraft 2200, and additionally or alternatively, a simulation of the aircraft 2200 may be shown.

[0072] FIGS. 23, 24, 25, 26 and 27 show embodiments that visualize the line of sight to the drone 160 between different visibility.

[0073] In the embodiment of FIG. 23, when the visibility is good, while the human operator 120 is looking at the drone 160 in the air 210 during 204, data regarding the flight of the drone 160 is superimposed on the augmented reality display 112 with a visual line of sight. This is an ideal flight situation.

[0074] In the embodiment shown in FIG. 24, when the visibility is reduced, while the human operator 120 is looking at the drone 160 in the air 210 during 204, data regarding the flight of the drone 160 is superimposed on the augmented reality display 112 with an extended line of sight. The extended line of sight can be achieved by guiding the human operator 120 to look in the correct direction of the target symbol 200. Optionally, the simulated drone 160 can be shown in the correct position. The reduction in visibility can be caused by low light conditions, clouds, fog, smog, rain, snowfall, or other physical phenomena.

[0075] In the embodiment of FIG. 25, when the visibility is blocked, while the human operator 120 is looking at the drone 160 in the air 210 during 204, data regarding the flight is superimposed on the augmented reality display 112 with an extended and simulated line of sight to the drone 160. The blocking of visibility can be caused by an obstacle 2502, i.e., when the drone 160 is behind the obstacle 2502. The obstacle 2502 can be the real object 1004 of FIGS. 10 and 11, i.e., the obstacle 2502 can be an artifact such as a building, a bridge, etc., or a natural object such as a hill, a forest, etc. The extension is achieved by guiding the human operator 120 to look in the correct direction with the target symbol 200 and by showing the simulated drone 160 in the correct position.

[0076] In an embodiment, when the human operator 120 is looking at the drone 160 in the air 210 at a long-distance view, during 204 when the human operator 120 is looking towards the drone 160, flight-related data is superimposed on the augmented reality display 112 with an extended line of sight to the drone. Although not shown in any drawing, basically, the drone 160 is, for example, in the sky or near the horizon, and the human operator 120 is guided to look in the correct direction with the target symbol 200, whereby the human operator 120 can only see the drone 160 as a small object far away.

[0077] In the embodiments shown in FIGS. 26 and 27, the device 100 adjusts 2700 the display 2600 of flight-related data on the augmented reality display 112 so that the line of sight 2602 is not blocked during 204 when the human operator 120 is looking towards the drone 160 in the air 210. As shown in FIG. 26, the human operator 120 looks towards the drone 160 with a free line of sight 2602. However, when the drone is descending 2604, the map 2600 will eventually block the line of sight 2602. As shown in FIG. 27, although the drone 160 is currently flying at a relatively low altitude, the line of sight 2602 remains free due to the leftward movement 2700 of the map 2600.

[0078] Finally, consider FIGS. 28 and 29 showing an embodiment of a system comprising two devices 100, 2800.

[0079] The first device 100 is used to assist the first human operator 120 in flying the drone 160 in the air 210 using the remote controller 150.

[0080] The first geographical position 2814 of the first human operator 120 with respect to the position of the drone 160 in the air 210 is used for adjusting the first viewpoint in order to render flight-related data including the first target symbol 200 and the first azimuth symbol 202 superimposed on the first augmented reality display 112 of the first device 100.

[0081] As shown in FIG. 28, the second device 2800 is used to provide information to a second human operator 2802 regarding the flight of the drone 160 in the air 210.

[0082] The second geographical location 2804 of the second human operator 2802 regarding the position of the drone 160 in the air 210 is used to adjust the second perspective in order to render flight-related data including a second target symbol 2806 and a second azimuth symbol 2808 overlaid on the second augmented reality display 2810 of the second device 2800.

[0083] In this way, the second human operator 2802 can at least observe 2812 the flight of the drone 160 in the air 210. This can be useful, for example, simply for entertainment, for educational purposes, to pass a flight license test, for surveillance, to track missing persons, or to assist the first human operator 120. As described above, one or more visual elements may be provided to one or both of the operators 120, 2802 based on data captured in real time using one or more sensors 1402 mounted on the drone 160.

[0084] In the embodiment shown in FIG. 29, the second device 2800 is used to assist the second human operator 2802 regarding the control 2902 of one or more sensors 1402 mounted on the drone 160, while the first human operator 120 controls the flight direction 2908 and speed of the drone 160.

[0085] For example, when the sensor 1402 is the image sensor described above, the second geographical location 2804 of the second human operator 2802 is used to adjust a second perspective that renders flight-related data superimposed on the second extended reality display 2810 of the second device 2800. The flight-related data also includes one or more video feeds captured in real time from one or more video cameras 2900 mounted on the drone 160. As shown in FIG. 29, one or more video feeds 2904 are superimposed on the second extended reality display 2810.

[0086] Note that in the usage example of FIG. 29, since both operators 120, 2802 can be shown the same information on the extended reality displays 112, 2810 and both have remote controllers 150, 2906, the responsibility for the flight can be seamlessly transferred between the operators 120, 2802 during flight. This can be particularly useful during training sessions or long missions. In particularly dangerous or restricted airspace, it is also envisioned that an authorized pilot 2802 safely operates the drone 160 and then the (original) operator 120 takes back control of the drone 160.

[0087] Note that the scenario where the second human operator 2802, shown in FIGS. 28 and 29, is physically present near the drone 160 and the first human operator 120 is not limited. As described above, the external data communication interface 110 can communicate flight-related data to the external receiver 116. The receiver 116 can be a networked computer server that interoperates with the first device 100 and the second device 2800 according to a client-server architecture, a cloud computing architecture, a peer-to-peer system, or another applicable distributed computing architecture. In this way, the second human operator 120 can still observe and assist as described above, even if they are far away, for example, in a different city, country, or continent. Of course, in particular, when, for example, a remote second human operator 2802 is controlling one or more sensors 1402, it is necessary to consider minimizing data transmission delays.

[0088] The present invention has been described with reference to one or more embodiments in accordance with the accompanying drawings, but it is obvious that the present invention is not limited thereto and can be modified in several ways within the scope of the appended claims. All words and expressions should be construed broadly and are not intended to limit the embodiments but to illustrate them. It will be apparent to those skilled in the art that the concept of the present invention can be implemented in various ways with the progress of technology.

Claims

1. An apparatus (100) for assisting a human operator (120) in flying a drone (160) using a remote controller (150), comprising: an internal data communication interface (108) for receiving flight-related data from the remote controller (150); a head-mounted display (112) for displaying the flight-related data to the human operator (120); one or more memories (104) including computer program code (106); one or more processors (102) for executing the computer program code (106) to cause the apparatus (100) to superimpose a target symbol (200) indicating the position of the drone (160) on the head-mounted display (112) at least while the human operator (120) is looking at the drone (160) (204), and to superimpose an azimuth symbol (202) indicating the direction of the drone (160) on the head-mounted display (112) while the human operator (120) is looking at the drone (160) (204); wherein the one or more processors are configured to superimpose on the head-mounted display (112) a vertically laid-out map (1600) showing the geographical location (1602) of the human operator (120) and the geographical location (1604) of the drone (160) in the vicinity of the target symbol (200) while the human operator (120) is looking at the drone (160) (204) in the air (210); superimpose on the head-mounted display (112) a horizontally laid-out map (1700) showing the geographical location (1702) of the human operator (112) and the geographical location (1704) of the drone (1706) while the human operator (120) is looking at the ground (500) (204); apparatus.

2. The azimuth symbol (202) indicates a predetermined direction fixed in relation to the direction of the drone (160) in the air (210). The apparatus according to claim 1.

3. The predetermined direction is fixed in relation to the traveling direction (400) of the drone (160). The apparatus according to claim 2.

4. acquiring the position of the drone (160) on the ground (500) in a world coordinate system (502); Obtain the position of the drone (160) projected onto the ground (500) in the augmented reality coordinate system (504) of the device (100). Lock the position of the drone (160) in the augmented reality coordinate system (504) and the position of the drone (160) in the world coordinate system (502). Obtain the traveling direction (400) of the drone (160) projected onto the ground (500). Set the traveling direction (400) to the direction of the calibration azimuth symbol in the augmented reality coordinate system (504) of the device (100). The device according to any one of claims 1 to 3.

5. While the human operator (120) is looking at the drone (160) in the air (210) (204), use a numerical value and scale (700) visually coupled to the target symbol (200) to superimpose the cruising altitude (704) of the drone (106) on the head-mounted display (112). While the human operator (120) is looking at the drone (160) in the air (210) (204), superimpose the traveling direction (400) of the drone (106) represented by an angle (degree) (702) visually coupled to the azimuth symbol (202) on the head-mounted display (112). The device according to any one of claims 1 to 4.

6. While the human operator (120) is looking at the drone (160) in the air (210) (204), superimpose an indirect line-of-sight guideline (800) that horizontally extends to the geographical position of the drone (160) projected onto the ground (500) and vertically extends to the target symbol (200) at the cruising altitude of the drone in the air (210) on the head-mounted display (112). The device according to any one of claims 1 to 5.

7. While the human operator (120) is looking at the drone (160) in the air (210) (204), superimpose a track symbol (900) indicating a track (902) and the speed of the drone (160) in the air (210) on the head-mounted display (112). The device according to any one of claims 1 to 6.

8. While the human operator (120) is looking at the drone (160) in the air (210) (204), a hazard indicator symbol (1000) representing the distance (1002) from the drone (160) to the real object (1004) is superimposed on the head-mounted display (112). The apparatus according to any one of claims 1 to 7.

9. The hazard indicator symbol (1000) includes a visual indicator (1100) superimposed at least on the real object (1004). The apparatus according to claim 8.

10. A map (1300) showing the geographical location (1302) of the human operator (120), the geographical location (1304) of the drone (160), and a waypoint (1306) is superimposed on the head-mounted display (112). While the human operator (120) is looking at the drone (160) in the air (210) (204), a vertical waypoint symbol (1308) starting from the geographical location of the waypoint (1306) projected on the ground (500) and extending to a predetermined altitude of the waypoint (1306) is superimposed on the head-mounted display (112). The apparatus according to any one of claims 1 to 9.

11. While the human operator (120) is looking at the drone (160) in the air (210) (204), one or more visual elements (1400) based on data captured in real time using one or more sensors (1402) mounted on the drone (160) in the vicinity of the target symbol (200) are superimposed on the head-mounted display (112). While the human operator (120) is looking at the drone (160) in the air (210) (204), the one or more visual elements (1400) are arranged on the head-mounted display (112) so that the line of sight is not blocked. The apparatus according to any one of claims 1 to 10.

12. While the human operator (120) is looking at the drone (160) in the air (210) (1804), a menu structure (1800) around the human operator (120) is superimposed on the head-mounted display (112). A gesture (1802) is detected from the human operator (120) as a command regarding the menu structure (1800). On the head-mounted display (112), controlling the display (1900) of flight-related data based on the command The apparatus according to any one of claims 1 to 11

13. Comprising an external data communication interface (110) for receiving external data (114) regarding the physical environment of the drone (160) Superimposing one or more visualizations (2000) of the external data (114) on the head-mounted display (112) The apparatus according to any one of claims 1 to 12

14. The external data (114) includes weather data and includes one or more visualizations (2000) showing the weather data The apparatus according to claim 13

15. The external data (114) includes air traffic control data having a classification of airspace, and the one or more visualizations (2100, 2102) show the classification of the airspace that coincides with the position of the drone (160) in the air (210) The apparatus according to claim 13 or 14

16. The external data (114) includes air traffic control data having the position of an aircraft (2200) in the air (210), and the one or more visualizations (2202, 2204) show the aircraft (2200) in the air (210) The apparatus according to any one of claims 13, 14 or 15

17. (i) When the visibility is good and the human operator can visually recognize the drone, superimposing the flight-related data on the head-mounted display (112) with a visual line of sight (2300) for the drone (160), (ii) When a physical phenomenon occurs between the human operator and the drone and the visibility decreases, superimposing the flight-related data on the head-mounted display with an extended line of sight (2400) for the drone (160), (iii) When there is an obstacle blocking the line of sight between the human operator and the drone, superimposing the flight-related data on the head-mounted display with an extended and simulated line of sight for the drone (160), or (iv) When the human operator has a long-distance line of sight where the drone can only be visually recognized as a small object, superimposing the flight-related data on the head-mounted display with an extended line of sight to the drone (160) The extended line of sight, or the extended and simulated line of sight, is a line of sight that prompts the operator to look in the correct direction. The apparatus according to any one of claims 1 to 16.

18. While the human operator (120) is looking at the drone (160) in the air (210), adjust (2700) the display (2600) of the flight-related data on the head-mounted display (112) so that the line of sight (2602) is not blocked. The apparatus according to any one of claims 1 to 17.

19. A system having two apparatuses according to any one of claims 1 to 18, The first apparatus (100) is used to assist a first human operator (120) in flying a drone (160) in the air (210) using a remote controller (150). The geographical position (2814) of the first human operator (120) with respect to the position of the drone (160) in the air (210) is utilized for adjusting a first perspective that renders flight-related data including a first target symbol (200) and a first azimuth symbol (202) superimposed on a first head-mounted display (112) of the first apparatus (100). The second apparatus (2800) is used to notify a second human operator (2802) about the flight of the drone (160) in the air (210). The geographical position (2804) of the second human operator (2802) with respect to the position of the drone (160) in the air (210), including a second target symbol (2806) and a second azimuth symbol (2808) superimposed on a second head-mounted display (2810) of the second apparatus (2800), is utilized for adjusting a second perspective that renders flight-related data. System.

20. The second apparatus (2800) is utilized to assist the second human operator (2802) regarding the control of one or more sensors (1402) mounted on the drone (160). The system according to claim 19.

21. A method for assisting a human operator who flies a drone using a remote controller, Receiving (3002) flight-related data from the remote controller, displaying the data regarding the flight to the human operator on a head-mounted display (3004), while the human operator is looking towards the drone, superimposing a target symbol indicating the position of the drone on the head-mounted display (3006), while the human operator is looking towards the drone, superimposing an azimuth symbol indicating the direction of the drone on the head-mounted display (3008), while the human operator (120) is looking towards the drone (160) in the air (210) (204), superimposing a vertically laid-out map (1600) showing the geographical position (1602) of the human operator (120) and the geographical position (1604) of the drone (160) near the target symbol (200) on the head-mounted display (112), while the human operator (120) is looking towards the ground (500) (204), superimposing a horizontally laid-out map (1700) showing the geographical position (1702) of the human operator (112) and the geographical position (1704) of the drone (1706) on the head-mounted display (112), A method.

22. A computer-readable medium (170) including computer program code (106) for implementing a method of assisting a human operator who flies a drone using a remote controller by causing one or more processors (102) to execute, The method includes receiving data regarding the flight from the remote controller (3002), displaying the data regarding the flight to the human operator on the head-mounted display (3004), while the human operator is looking towards the drone, superimposing on the head-mounted display a target symbol indicating the position of the drone (3006), while the human operator is looking towards the drone, superimposing on the head-mounted display an azimuth symbol indicating the direction of the drone (3008), The method includes While the human operator (120) is looking at the drone (160) in the air (210) (204), a map (1600) in a vertical layout showing the geographical location (1602) of the human operator (120) and the geographical location (1604) of the drone (160) near the target symbol (200) is superimposed on the head-mounted display (112), While the human operator (120) is looking at the ground (500) (204), a map (1700) in a horizontal layout showing the geographical location (1702) of the human operator (112) and the geographical location (1704) of the drone (1706) is superimposed on the head-mounted display (112), A computer-readable medium.

Images