Enhanced vision for unmanned vehicles

Abstract

The present invention is entitled “Augmentation of Line of Sight for Unmanned Vehicles.” The present invention relates to an augmented reality head-mounted display configured to display an indication of the position of an unmanned aerial vehicle relative to the field of view of the head-mounted display in order to assist a UAV operator in maintaining line of sight with the UAV.

Description

Technical Field

[0001] The present disclosure relates to unmanned aerial vehicles (UAVs). Background Art

[0002] Large industrial companies, particularly in the utility and oil and gas sectors, can own hundreds of miles of asset infrastructure (e.g., power lines, pipelines), which requires regular inspections to ensure high productivity. Due to the rapid collection of high-quality data by unmanned aerial vehicles (UAVs), some entities have recently begun utilizing small UAVs to perform these periodic inspections. Federal Aviation Administration (FAA) regulations require that UAV operators or pilots maintain line of sight (LOS) with the UAV at all times. Summary of the Invention

[0003] In general, the present disclosure relates to systems and techniques for determining and indicating the position of at least one unmanned aerial vehicle (UAV) relative to a pilot of the UAV to assist the pilot in maintaining line of sight (LOS) with the UAV. As described herein, an augmented reality (AR) head-mounted display (HMD) worn by a UAV pilot is configured to display an indication of the position of the UAV relative to a field of view (FOV) of the HMD.

[0004] In one example, the present disclosure describes a device that includes an AR HMD and processing circuitry configured to: determine a location of a UAV; determine an orientation of the AR HMD; and present an indication of the location of the UAV on the AR HMD based on the orientation of the AR HMD.

[0005] In another example, this disclosure describes a method that includes determining a location of a UAV; determining an orientation of an AR HMD; and presenting an indication of the location of the UAV on the AR HMD based on the orientation of the AR HMD.

[0006] The details of one or more examples of the present disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a conceptual diagram illustrating the use of an unmanned aerial vehicle (UAV) to inspect utility assets, as may be performed in accordance with one or more techniques of the present disclosure.

[0008] Figure 2 is a block diagram illustrating an exemplary system for determining the position of a UAV relative to a field of view (FOV) of a head-mounted display (HMD).

[0009] Figure 3Ais a conceptual diagram illustrating an HMD determining the horizontal position of a UAV relative to the HMD's FOV, in accordance with one or more techniques of this disclosure.

[0010] Figure 3B yes Figure 3A Schematic top view of .

[0011] Figure 3C is a conceptual diagram illustrating an HMD determining the vertical position of a UAV relative to the HMD's FOV, in accordance with one or more techniques of this disclosure.

[0012] Figure 3D yes Figure 3C Schematic side view of .

[0013] Figure 4 is an exemplary graphical user interface (GUI) that may be generated and displayed on an HMD in accordance with one or more techniques of this disclosure.

[0014] Figure 5 is another exemplary GUI that may be generated and displayed on an HMD in accordance with one or more techniques of this disclosure.

[0015] Figure 6 is another exemplary GUI that may be generated and displayed on an HMD in accordance with one or more techniques of this disclosure.

[0016] Figure 7 is a flow chart illustrating example operations for determining and displaying the position of a UAV relative to the FOV of an HMD, in accordance with the techniques of this disclosure.

[0017] Figure 8 is a flow chart illustrating example operations for determining and displaying the position of a UAV relative to the FOV of an HMD, in accordance with the techniques of this disclosure.

[0018] Figure 9 is a flow chart illustrating example operations for determining and displaying the position of a UAV relative to the FOV of an HMD, in accordance with the techniques of this disclosure. DETAILED DESCRIPTION

[0019] The use of unmanned aerial vehicles (UAVs) is becoming increasingly common in non-military operations, including but not limited to surveillance, search and rescue, shipping and delivery, and inspection. For example, Figure 1 1 is a conceptual diagram depicting an inspection of a utility asset using an unmanned aerial vehicle system (UAS) 10, including a UAV 14. The UAV 10 includes a UAV operator or pilot 12, the UAV 14, an augmented reality (AR) head mounted device (HMD) 16, a ground station 20, and a UAV controller 30.

[0020] The UAV pilot 12 uses the UAV controller 30 to control the flight path 114 of the UAV 14, for example, to conduct an inspection of the structures 22A through 22E (collectively, “structures 22 ”). Figure 1 In the depicted example, structure 22 is depicted as a transmission tower, but structure 22 may include any structure, including a pipeline, a flare chimney, a solar panel, a bridge, and the like.

[0021] For example, the pilot 12 may launch the UAV 14 from the launch location 18 and guide the UAV 14 around the structure 22 so that a camera or other sensor mounted on the UAV 14 can capture data. For example, the UAV 14 may capture photographic or video data using one or more camera-based sensors and may additionally or alternatively capture other types of data, such as temperature data, environmental data, or electromagnetic field data, using other types of sensors mounted on the UAV 14.

[0022] The pilot 12 may guide the UAV 14 along an extended and / or tortuous flight path 114 in order to capture a large amount of data during a single transmission. For example, at any time during an inspection, the UAV 14 may be as close as 1,600 meters from the pilot 12 while conducting an inspection mission. The distance of the UAV 14 from the pilot 12 may be limited only by the battery life of the UAV 14 and / or the radio communication range of the controller 30.

[0023] Federal Aviation Administration (FAA) Small-UAS Rule §107.31 1 The UAV pilot must always maintain line of sight (LOS) with the UAV during operation. Many factors, including but not limited to tortuous terrain, background (e.g., sky) color contrast, weather or other obstacles, may hinder the pilot 12's ability to maintain line of sight with the UAV 14.

[0024] In some examples of the present disclosure, a pilot 12 may wear an HMD 16 having processing circuitry therein configured to determine the position of a UAV 14 relative to a field of view (FOV) 26 of the HMD 16 and output an indication of the relative position of the UAV 14 for display on a transparent display screen of the HMD 16 to assist the pilot 12 in maintaining and / or regaining line of sight 24 with the UAV 14. In some examples, such as when the UAV 14 is positioned within the FOV 26 of the HMD 16, the HMD 16 may display a bounding box on the screen around the approximate position of the UAV 14. In other examples, such as when the UAV 14 is not positioned within the FOV 26 of the HMD 16, the HMD 16 may display information indicating the relative position of the UAV 14, such as a set of arrows guiding the pilot 12 to turn his or her head in order to bring the UAV 14 back within the FOV 26 of the HMD 16. As described in further detail below, the HMD 16 may be configured to display other relevant UAV flight information to the pilot 12 , including, but not limited to, UAV airspeed and / or direction, distance to the UAV, remaining UAV battery life, an expected UAV flight path, or an indication of controlled airspace 28 that the UAV 14 must avoid.

[0025] Figure 2 FIG. 1 shows a method for determining the position of the UAV 14 relative to the FOV of the HMD 16. Figure 1 1 is a block diagram of a system 10. The exemplary system 10 includes a pilot 12, a UAV 14, an HMD 16, a ground station 20, and a UAV controller 30.

[0026] The pilot 12 is the person who guides the UAV 14 along a flight path, such as to take aerial photographs, collect sensor data, or deliver a package. Figure 1 1 and 2. The UAV 14 is shown as a four-rotor "quadcopter," but the UAV 14 may be any type of UAV, including but not limited to rotorcraft, fixed-wing aircraft, compound aircraft (such as tiltrotors, X2s, and X3s), aerostats, or any other such type of UAV, including all vertical take-off and landing (VTOL) aircraft, stand-up aircraft, and the like. The UAV 14 may be configured to fly with varying degrees of autonomy. Although the technology of the present disclosure is not limited to any particular type of UAV, the UAV 14 may be, for example, a relatively small, low-altitude, low-airspeed UAV, where "small" in this case corresponds to less than 100 pounds, "low altitude" corresponds to an operating altitude of less than 3,000 feet above the ground, and "low airspeed" corresponds to an airspeed of less than 250 knots. Furthermore, it is contemplated that the UAV 14 may have a hovering capability, meaning that the UAV 14 may have the ability to maintain a generally constant position in the air.

[0027] The UAV 14 includes a positioning system (PS) 52, an inertial measurement unit (IMU) 54, an orientation unit (OU) 56, an altimeter 58, a processing circuit (PC) 60, a memory 61, a transceiver 62, a camera 116, and a sensor 118. Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but do not necessarily need to be implemented by different hardware units. Although for illustrative purposes, the various components, modules, or units are described in detail below. Figure 2 Although shown separately in the figures, many of the components of the UAV 14 may actually be highly integrated. For example, many of the components of the UAV 14 may be implemented as a single circuit or system-on-a-chip. The various units or modules may be combined into a single hardware unit or provided by a collection of interoperable hardware units (including one or more processors) in conjunction with appropriate software and / or firmware.

[0028] The positioning system (PS) 52 includes any hardware and / or software configured to determine the relative position (such as the geographic location) of the UAV 14. In some examples, the PS 52 may include a GPS system configured to determine the latitude and longitude of the UAV 14. In other examples, the PS 52 may be configured to determine the location of the UAV 14 based on nearby wireless internet signals, cellular tower signals, or transponder signals. In other examples, the positioning system 52 may include a camera-based positioning system. Generally speaking, the positioning system 52 may include any one or more types of positioning systems and is not limited to any one particular type of positioning system.

[0029] The inertial measurement unit (IMU) 54 is an electronic component or device configured to detect acceleration, motion, and / or orientation of the UAV 14. The IMU 54 may include one or more accelerometers, gyroscopes, and / or magnetometers. The IMU 54 (alone or in combination with the IMU 56) may be configured to output data indicating the flight direction of the UAV 14.

[0030] The orientation unit (OU) 56 includes one or more devices configured to determine the relative orientation of the UAV 14 with respect to the cardinal directions of the Earth. For example, the OU 56 may include a magnetometer configured to measure the strength and direction of the Earth's magnetic field to identify the direction of North. In other examples, the OU 56 includes a simple compass configured to identify the direction of magnetic North. In some examples, the processing circuit 60 may be configured to combine the magnetic-based orientation data from the OU 56 with the position data from the PS 52 to determine the orientation of the UAV 14 with respect to true North (as defined by the Earth's axis of rotation). In other examples, the OU 56 may be camera-based and common visual landmarks to determine the orientation of the UAV 14.

[0031] The altimeter 58 is a device or component for determining the height of the UAV 14 above the ground. In some examples, the altimeter 58 may include a device configured to determine the altitude of the UAV 14 based on atmospheric pressure. The altimeter 58 may include data from the positioning system 52 to determine the altitude based on the difference between the atmospheric pressure measured at a known height above the ground level at the local latitude and longitude and the expected atmospheric pressure, as indicated by the positioning system 52. In other examples, the altimeter 58 may include a signal transceiver configured to reflect a signal (such as an electromagnetic signal, a sonar signal, or other signal) off the ground and measure the time until the reflected signal is detected. In other examples, the altimeter 58 may use stereo images to determine the altitude.

[0032] The UAV 14 includes processing circuitry (PC) 60 and memory 61 configured to process and store data, such as data received from the transceiver 62 or from any other internal component of the UAV 14. The PC 60 may include one or more processors configured to execute instructions, such as one or more digital signal processors (DSPs), general-purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuits. The processing circuitry 60 may include analog and / or digital circuitry. As used herein, the terms "processor" or "processing circuitry" may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein.

[0033] The memory 61 may be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The memory 61 may include one or more of a hard drive, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data.

[0034] The transceiver 62 includes one or more electronic components configured to wirelessly transmit and receive data, such as from the ground station 20, the controller 30, or the HMD 16. The transceiver 62 may represent any one or more of the following: a wireless transmitter / receiver, a modem, a networking component, a wireless communication component that operates according to any of a variety of IEEE 802.11 standards, or other physical components. The transceiver 62 may be configured to operate according to a cellular communication standard such as 4G, 4G-LTE (Long Term Evolution), LTE Advanced, 5G, etc., or according to other wireless standards such as the IEEE 802.11 specifications, the IEEE 802.15 specifications (e.g., ZigBee TM ), Bluetooth TM Standards such as transmission data.

[0035] The UAV 14 includes a camera 116 and / or additional sensors 118 configured to capture photographs or other data of a designated target, such as a structure. The camera 116 may represent one or both of a monoscopic camera or a stereoscopic camera configured to acquire image and / or video data. The UAV 14 may store the images and video in the memory 61 or may additionally or alternatively stream the images and video to another device, such as the ground station 20.

[0036] HMD 16 includes a positioning system (PS) 32, an inertial measurement unit (IMU) 34, an orientation unit (OU) 36, a display 38, a camera 40, a processing circuit (PC) 42, a memory 43, a transceiver 44, and an altimeter 45. HMD 16 is a head-mounted device that may include various electronic components found in a computing system, including one or more processors 42 (e.g., microprocessors or other types of processing units) and a memory 43 that may be mounted on or within a frame. Furthermore, HMD 16 may include a transparent display screen 38 that is positioned at eye level when a user, such as UAV pilot 12, wears HMD 16. In some examples, display screen 38 may include one or more liquid crystal displays (LCDs) or other types of display screens, on which an image of UAV pilot 12 wearing or otherwise using HMD 16 may be perceived via display screen 38. Other display examples include organic light-emitting diode (OLED) displays. In some examples, HMD 16 may be operable to project a 3D image onto the user's retina using techniques known in the art.

[0037] In some examples, display 38 may include a see-through holographic lens (sometimes referred to as a waveguide) that allows a user to see real-world objects through (e.g., beyond) the lens and also see holographic images projected into the lens and onto the user's retina through a display, such as a liquid crystal on silicon (LCoS) display device, sometimes referred to as a light engine or projector, as an example of a holographic projection system in HMD 16. In other words, HMD 16 may include one or more see-through holographic lenses to present virtual images to the user. Thus, in some examples, HMD 16 may be operable to project a 3D image onto the user's retina via display 38, for example, formed by the holographic lenses. In this way, HMD 16 may be configured to present a 3D virtual image to the user within the real-world view observed through display 38, for example, such that the virtual image appears to form part of the real-world environment. In some examples, HMD 16 may be a Microsoft HOLOLENS available from Microsoft Corporation of Redmond, Washington, USA. TM Headset or similar device, such as a similar MR visualization device including a waveguide. HOLOLENS TM The device can be used to present 3D virtual objects via a holographic lens or waveguide, while allowing a user to view actual objects in a real-world scene (i.e., in a real-world environment) through the holographic lens.

[0038] In other examples, the display screen 38 may include an opaque (e.g., non-transparent) digital screen configured to display rendered images. Some non-limiting examples of devices with opaque displays include the RealWear HMT-1, the Vuzix M300, or any standard smartphone or tablet. In the case of an AR HMD with an opaque display screen 38, the user's optical axis (e.g., the line of sight or the center of the field of view) may be slightly different from the optical axis of the camera 40. In some examples, the AR HMD may have a display screen 38 configured to cover only one eye ("monocular") or both eyes ("binocular") of the user.

[0039] like Figures 4 to 6 As shown, the HMD 16 may also generate a graphical display or user interface (UI) visible to the user, for example, as a holographic image projected into a see-through holographic lens as described above. The image presented by the HMD 16 may include, for example, one or more 3D virtual objects. Details of an exemplary UI are described elsewhere in this disclosure. The HMD 16 may also include speakers or other sensing devices that may be positioned adjacent to the user's ears. The sensing devices may convey audible information or other perceptible information (e.g., vibrations) to assist the user of the HMD 16.

[0040] The HMD 16 may also include a transceiver 44 to connect the HMD 16 to a second processing device such as an intermediate ground station 20, or directly to the UAV 14, and / or to a network and / or computing cloud, such as via a wired communication protocol or a wireless protocol (e.g., Wi-Fi, Bluetooth, etc.). The transceiver 44 may represent any one or more of the following: a wireless transmitter / receiver, a modem, a networking component, a wireless communication component that operates according to any of a variety of IEEE 802.11 standards, or other physical components. The transceiver 44 may be configured to operate in accordance with a cellular communication standard such as 4G, 4G-LTE (Long Term Evolution), LTE Advanced, 5G, etc., or in accordance with other wireless standards such as the IEEE 802.11 specifications, the IEEE 802.15 specifications (e.g., ZigBee TM ), Bluetooth TM Standards such as transmission data.

[0041] HMD 16 also includes various sensors for collecting sensor data, such as one or more optical cameras 40 (or other optical sensors) and one or more depth cameras (or other depth sensors) mounted to, on, or within the frame. In some examples, optical sensor 40 is operable to scan the geometry of the physical environment in which the user of HMD 16 is located and collect two-dimensional (2D) optical image data (monochrome or color). Depth sensor 40 is operable to provide 3D image data, such as by employing time-of-flight, stereo, or other known or future-developed techniques to determine depth, thereby generating three-dimensional image data. Camera 40 may represent one or both of a monoscopic camera or a stereoscopic camera configured to acquire image and / or video data. HMD 16 may store images and video in memory 43 or, in addition or alternatively, may stream the images and video to another device, such as ground station 20. Other sensors may include motion sensors (e.g., inertial mass unit (IMU) sensor 34, accelerometers, etc.) to help track movement.

[0042] The system 10 processes the sensor data so that geometric, environmental, and textural landmarks (e.g., corners, edges or other lines, walls, floors, objects) in the user's environment or "scene" can be defined, and movement within the scene can be detected. For example, various types of sensor data can be combined or fused so that the user of the HMD 16 can perceive a 3D image that can be positioned or fixed and / or moved within the scene. When fixed within the scene, the user can walk around the 3D image, view the 3D image from different perspectives, and manipulate the 3D image within the scene using gestures, voice commands, gaze lines (or directions), and / or other control inputs. For another example, the sensor data can be processed so that the user can position a 3D virtual object (e.g., a bounding box) on a physical object observed in the scene (e.g., the UAV 14) and / or orient the 3D virtual object using other virtual images displayed in the scene.

[0043] The HMD 16 may include, for example, one or more processors or processing circuits (PCs) 42 and memory 43 within the frame of the HMD. The PCs 42 may include one or more processors configured to execute instructions, such as one or more digital signal processors (DSPs), general-purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuits. The processing circuits 42 may include analog and / or digital circuits. As used herein, the terms "processor" or "processing circuit" may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein.

[0044] Memory 43 may be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. Memory 43 may include one or more of a hard drive, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data.

[0045] In some examples, instead of or in addition to the internal PC 42 and memory 43, one or more external computing resources process and store information, such as sensor data. In this way, data processing and storage can be performed by one or more PCs 42 and memory 43 within the HMD 16, and / or some of the processing and storage requirements can be offloaded from the HMD 16. Thus, in some examples, one or more PCs that control the operation of the HMD 16 can be located within the HMD, such as PC 42. Alternatively, in some examples, at least one of the processors that control the operation of the HMD 16 can be located external to the HMD. Likewise, in some examples, the operation of the HMD 16 can be controlled in part by a combination of one or more PCs 42 within the visualization device and one or more processors external to the visualization device.

[0046] For example, in some examples, when the HMD 16 is in Figure 2 In the context of FIG, processing of sensor data may be performed by the PC 42 in conjunction with the memory 43 or storage device. In some examples, the PC 42 and memory 43 mounted to the frame may provide sufficient computing resources to process the sensor data collected by the camera 40 and the motion sensor 34. In some examples, the sensor data may be processed using a simultaneous localization and mapping (SLAM) algorithm or other known or future developed algorithms for processing and mapping 2D and 3D image data and tracking the position of the HMD 16 in a 3D scene. In some examples, the sensor data may be processed using algorithms developed by Microsoft HOLOLENS TM system (e.g., by the HMD 16 substantially compliant with Microsoft HOLOLENS TM Image tracking is performed using sensor processing and tracking functionality provided by one or more sensors and a processor 42 of a device or similar mixed reality (MR) visualization device.

[0047] The HMD 16 includes a positioning system (PS) 32. The PS 32 includes any hardware and / or software configured to determine the relative position (such as the geographic location) of the HMD 16. In some examples, the PS 32 may include a GPS system configured to determine the latitude and longitude of the HMD 16. In other examples, the PS 32 may be configured to determine the location of the HMD 16 based on nearby wireless internet signals, cellular tower signals, or transponder signals. In other examples, the positioning system 32 may include a camera-based positioning system. Generally speaking, the positioning system 32 may include any one or more types of positioning systems and is not limited to any particular type of positioning system.

[0048] The HMD 16 includes an inertial measurement unit (IMU) 34. The IMU 34 is an electronic component or device configured to detect acceleration, motion, and / or orientation (such as relative to gravity) of the HMD 16. The IMU 34 may include one or more accelerometers, gyroscopes, and / or magnetometers. The IMU 34 may be configured to output data indicating the angle of the HMD 16 relative to gravity, which correspondingly indicates the angle of the field of view of the HMD 16 relative to the ground, which may be assumed to be locally perpendicular to the direction of gravity.

[0049] The HMD 16 includes an orientation unit (OU) 36. The OU 36 includes one or more devices configured to determine the relative orientation of the HMD 16 relative to the Earth's cardinal directions. For example, the OU 36 may include a magnetometer configured to measure the strength and direction of the Earth's magnetic field to identify the direction of north. In other examples, the OU 36 includes a simple compass configured to identify the direction of magnetic north. In some examples, the processing circuit 42 may be configured to combine the magnetic-based orientation data from the OU 36 with the position data from the PS 32 to determine the orientation of the HMD 16 relative to true north (as defined by the Earth's axis of rotation). In other examples, the OU 36 may be camera-based and use common visual landmarks to determine the orientation of the HMD 16.

[0050] In some examples, HMD 16 includes an altimeter 45. Altimeter 45 includes hardware and / or software for determining the height of HMD 16 above the ground. In some examples, when worn on the head of pilot 12, the height of HMD 16 is substantially negligible relative to the height of UAV 14 above the ground in flight. In these examples, PC 42 determines the height of HMD 16 to be zero for purposes of determining the relative position of UAV 14. In other examples, altimeter 45 may include a device configured to determine the altitude of HMD 16 based on atmospheric pressure. Altimeter 45 may include data from positioning system 32 to determine the altitude based on the difference between atmospheric pressure measured at a known altitude above ground level at the local latitude and longitude and the expected atmospheric pressure, as indicated by positioning system 32. In other examples, the altimeter may include a signal transceiver configured to reflect a signal (such as an electromagnetic signal, sonar signal, or other signal) off the ground and measure the time until the reflected signal is detected. Because HMD 16 is worn on the head of pilot 12, in other examples, altimeter 45 may include a data input device for pilot 12 to enter his or her height, which generally corresponds to the height of HMD 16 above the ground when pilot 12 is standing upright. In other examples, altimeter 45 may use stereoscopic images to determine altitude.

[0051] Ground station (GS) 20 is a ground-based computing device that includes a transceiver 46, controls 48, and a display 50. In some examples, GS 20 is a fixed communication station, such as a radio tower. In other examples, GS 20 is a mobile computing device, such as a laptop, tablet, smartphone, or other computing device. Figure 2 As shown, the GS 20 is configured to send and receive data from both the UAV 14 and the HMD 16 via the transceiver 46. In some examples, the UAV 14 is configured to communicate with the HMD 16 directly.

[0052] The GS 20 includes one or more controls 48, such as a keyboard, touch screen, buttons, throttles, or other similar user input devices. The GS 20 further includes a display 50, such as a screen configured to display camera, sensor, and / or flight data from the UAV 14. In some examples, the display 50 is configured to display a live video feed captured by the camera 116 on the UAV 14. In some examples, the ground station 20 includes a transceiver 46 that is configured to receive telemetry data from the sensors 118 of the UAV 14 and wirelessly transmit the telemetry data to the transceiver 44 of the HMD 16.

[0053] In some examples, the system 10 may also include one or more user-operated control devices (controllers) 30 that allow the user to operate the UAV 14, interact with the UI, and / or otherwise provide commands or requests to a processing device or other system connected to the network. For example, the controller 30 may include a microphone, a touchpad, a control panel, a motion sensor, or other type of control input device with which the user can interact. In some examples, the controller 30 may be incorporated into the GS 20. In other examples, the controller 30 may be incorporated into the HMD 16. In other examples, the controller 30 is a different handheld device having one or more buttons, switches, a touch screen, or other similar control inputs configured to control the movement of the UAV 14.

[0054] In some examples according to the present disclosure, and as with respect to Figures 3A to 3D As described in further detail below, the processing circuitry 42 of the HMD 16 is configured to receive position data from the UAV 14 and at least orientation data from the HMD 16, and use the received data to determine the position of the UAV 14 relative to the FOV of the display screen 38 of the HMD 16. The FOV of the HMD 16 between the HMD 16 and the UAV 14 may have the shape of a rectangular frustum having an extended horizontal width dimension and an extended height dimension, both of which are taken perpendicular to a line extending from the HMD 16 to the center “C” of the field of view at a distance from the UAV 14. For example, the processing circuitry 42 within the HMD 16 may be configured to determine the horizontal position of the UAV 14 relative to the center “C” of the FOV of the HMD 16 based on the received position and orientation data ( Figures 3A to 3B ) and vertical position ( Figures 3C to 3D )Both. Figures 3A to 3D The exemplary methods and processes (eg, specifically disclosed calculations) are described by way of example only and are not intended to be limiting.

[0055] Figure 3Ais a conceptual diagram illustrating an exemplary process for determining the horizontal position of a UAV 14 relative to a FOV 26 of an HMD 16, in accordance with one or more techniques of this disclosure. To determine whether the UAV 14 is within the horizontal FOV of the HMD 16, the HMD 16 (e.g., processing circuitry 42 within the HMD 16) determines the magnitude of two horizontal distances: (1) the distance at which the center "C" of the field of view 26 of the HMD 16 is located from the left edge of the FOV 26 at the distance of the UAV 14; and (2) the distance at which the center "C" of the field of view 26 of the HMD 16 is located from the left edge of the FOV 26 at the distance of the UAV 14. 左 (or equivalently, the right edge FOV 右 ) width FOV 宽度 ; and (2) UAV 14 and FOV 中心 The horizontal distance d hor Once the HMD 16 determines the FOV 宽度 and d hor , the HMD 16 compares the two values ​​to determine whether the UAV 14 is within the horizontal width of the FOV 26. For example, if d hor Smaller than FOV 宽度 , then UAV 14 is within the horizontal width of FOV 26. If d hor Larger than FOV 宽度 , then UAV 14 is not within the horizontal width of FOV 26.

[0056] Equivalently (e.g., additionally or alternatively), HMD 16 may determine a horizontal angle α between FOV center C, HMD 16, and UAV 14. Δ Is it greater than or less than 1 / 2f? θ , where f θ represents the fixed angle of the horizontal field of view of the HMD 16. If α Δ Less than 1 / 2f θ , then UAV 14 is within the horizontal width of FOV 26.

[0057] Figure 3B yes Figure 3A In one exemplary process, HMD 16 determines α based on a known value and three measured values. Δ d hor and / or FOV 宽度 The value of horizontal angle f θ (which defines the horizontal component of the field of view 26) is the display screen 38 and / or camera 40 ( Figure 2) are fixed values ​​defined by the size and relative position of ). For example, the display screen 38 and the camera 40 may each define a field of view, which in some cases may be substantially aligned with each other. In some examples, the fields of view of the display screen 38 and the camera 40 may be slightly different, for example, may not be perfectly aligned, but may be similar enough to be interchangeable for the exemplary techniques described herein. Therefore, the HMD 16 (e.g., the PC 42 and the memory 43) knows that 1 / 2f θ fixed value.

[0058] The HMD 16 (eg, transceiver 44 and / or PC 42) receives three other values ​​measured by various sensors and / or detectors. The positioning system 32 ( Figure 2 ) determines the relative position of the HMD 16. For example, the positioning system 32 may include a GPS device configured to determine the latitude and longitude (HMD 16) of the HMD 16. lat ,H lon ). Similarly, the positioning system 52 of the UAV 14 ( Figure 2 ) to determine the relative position of the UAV 14. For example, the positioning system 52 may include a GPS device configured to determine the latitude and longitude (U lat ,U lon ). Orientation unit 36 ​​in HMD 16 ( Figure 2 ) determines the relative orientation of the HMD 16. For example, the orientation unit 36 ​​may include a HMD A compass or magnetometer that measures the horizontal angle between the FOV center C, the HMD 16, and North 66 (e.g., magnetic North and / or true North). The transceiver 44 and / or PC 42 within the HMD 16 ( Figure 2 )Receive indication (H lat ,H lon )、(U lat ,U lon ) and α HMD data.

[0059] Based on (H lat ,H lon ) and (U lat ,U lon ), the HMD 16 determines the distance d 地面 , which is the horizontal "ground distance" along a straight line between the HMD 16 and the UAV 14. For example, the HMD 16 may implement the Pythagorean theorem a 2 +b 2 =c 2 , where "a" is U lat and H lat The difference between "ΔLat", "b" is U lon and Hlon The difference between "ΔLon", and "c" is equal to d 地面 .

[0060] Use d 地面 , ΔLat, and ΔLon, the HMD 16 may determine α UAV , which is the horizontal angle between the UAV 14, the HMD 16, and the north 66. For example, using the known mathematical relationship of sine(θ) = (opposite side / hypotenuse), the HMD 16 can determine the angle α UAV Equal to (ΔLon / d 地面 ) is the arcsine of .

[0061] Using alpha HMD The measured value and α UAV The HMD 16 can determine the value of α Δ The value of α is determined HMD and α UAV Then, the HMD 16 can calculate the difference between α Δ The determined value of 1 / 2f θ to determine the fixed value of α Δ Is it less than 1 / 2f? θ , and thus determine whether the UAV 14 is within the horizontal field of view 26 of the HMD 16. Using additional trigonometric ratios, the HMD 16 can similarly determine the FOV 宽度 and d hor d to determine the approximate horizontal position of the UAV 14 within the FOV 26 (e.g., the approximate horizontal position of the UAV 14 on the display screen 38). hor Equal to d 地面 *sine(α Δ ). The horizontal distance d from the HMD 16 to the FOV center C CH Equal to d 地面 *cosine(α Δ ). FOV 宽度 Equal to d CH *tangent(1 / 2f θ ).

[0062] If the HMD 16 determines that the UAV 14 is within the horizontal field of view, the PC 42 may use (d hor / FOV 宽度 ) to determine the approximate location on the display screen 38 (as a percentage of the screen width) to place a bounding box indicating the location of the UAV 14.

[0063] Figure 3C14 is a conceptual diagram illustrating an HMD determining the vertical position of a UAV relative to the HMD's FOV, in accordance with one or more techniques of this disclosure. The HMD 16 (e.g., processing circuitry within the HMD 16) determines the magnitude of two vertical distances: the distance between the center "C" of the field of view 26 of the HMD 16 at the distance of the UAV 14 and the top edge of the FOV ... 顶部 (or equivalently, the bottom edge FOV 底部 ) between the height FOV 高度 ; and the vertical distance d between the UAV 14 and the center of the FOV vert Once the HMD 16 determines the FOV 高度 and d vert , the HMD 16 compares the two values ​​to determine whether the UAV 14 is within the vertical height of the FOV 26. For example, if d vert Smaller than FOV 高度 , then the UAV 14 is within the vertical height of the FOV 26. If d vert Larger than FOV 高度 , then UAV 14 is not within the vertical height of FOV 26.

[0064] Equivalently, the HMD 16 may determine the vertical angle β between the FOV center C, the HMD 16, and the UAV 14. Δ Is it greater than or less than in represents the fixed angle of the vertical field of view of the HMD 16. If β Δ Less than The UAV 14 is then located within the vertical height of the FOV 26 .

[0065] Figure 3D yes Figure 3C The HMD 16 determines β based on one or more known values ​​and measured values. Δ d vert and / or FOV 高度 The vertical angle of the vertical field of view 26 The display screen 38 ( Figure 2 ) are fixed values ​​defined by the size and relative position of ). Therefore, the HMD 16 knows value.

[0066] The HMD 16 receives three other values ​​input or measured by various sensors and / or detectors. The HMD 16 determines the height or altitude A of the HMD 16 above the ground. HMD For example, since the HMD 16 is intended to be worn on the head of the pilot 12, if the pilot 12 stands upright while controlling the HMD 16, then A HMDA may be approximately equal to the height of pilot 12. System 10 may include means for receiving input indicating the height of pilot 12. In some examples, HMD 16 may include its own altimeter, similar to altimeter 58 of UAV 14. In some examples, for computational simplicity, system 10 may assume that A HMD is significantly smaller (e.g., negligible) than the altitude of the UAV 14 in flight, and therefore A HMD Set equal to zero.

[0067] UAV 14's altimeter 58 ( Figure 2 ) determines the height or altitude A of the UAV 14 above the ground UAV . The transceiver 44 ( Figure 2 ) Receive instruction A UAV Inertial measurement unit (IMU) 34 in HMD 16 ( Figure 2 ) determines the relative orientation of the HMD 16 with respect to gravity. For example, the IMU 34 may include a sensor configured to determine the angle β HMD The accelerometer or other sensor measures the value of the vertical angle between a horizontal axis extending from the HMD 16 (e.g., parallel to the ground) and the center of the field of view. The HMD 16 determines the horizontal ground distance d between the HMD 16 and the UAV 14. 地面 , as mentioned above Figure 3B described.

[0068] The HMD 16 sets the distance d CV (the vertical distance between the horizontal axis and the FOV center C) is determined to be equal to d 地面 *tan(β HMD ). Use d CV , the HMD 16 moves the distance d vert (the vertical distance between the UAV 14 and the center of the field of view C) is determined to be equal to (A UAV -A HMD -d CV ).

[0069] Use d vert , the HMD 16 adjusts the angle β UAV (the angle between the horizontal axis, HMD 16 and UAV 14) is determined to be equal to [(d vert +d CV ) / d 地面 ] is the inverse tangent of . Using β UAV , HMD 16 will be beta Δ Determined as β UAV and β HMD The difference between the two. HMD 16 will f 高度 Determined to be equal

[0070] The HMD 16 may then Δ The determined value of to determine the fixed value of β Δ Is it less than and thus respectively determine whether the UAV 14 is within the vertical field of view 26 of the HMD 16. The HMD 16 may similarly use the FOV 高度 and d vert The approximate vertical position of the UAV 14 within the FOV 26 (e.g., the approximate vertical position of the UAV 14 on the display screen 38) is determined using the values ​​of . Additionally, the HMD 16 may use the determined values ​​to further determine AC (the altitude of the field of view center “C” at the distance of the UAV 14), dC (the distance between the HMD 16 and the field of view center C), and d H-U (the distance between the HMD 16 and the UAV 14 along the direct line of sight 24).

[0071] If the HMD 16 determines that the UAV 14 is within the vertical field of view, the PC 42 may determine (d vert / FOV 高度 ) to determine an approximate location on the display screen 38 (as a percentage of the screen height) to place a graphical object, such as a bounding box, indicating the location of the UAV 14.

[0072] Figure 4 is a device that can be used on the HMD 16 ( Figure 2 ) is generated and displayed on the display screen 38 of the exemplary user interface (UI) or graphical user interface (GUI) 72. Figure 4 In the depicted exemplary GUI 72, the processing circuitry 42 ( Figure 2 ) has determined that the position of the UAV 14 is within the field of view of the display screen 38. In this case, the processing circuit 42 generates and outputs a graphical object (such as a bounding box 64) indicating the position of the UAV 14 relative to the screen for display on the display screen 38. Although Figure 4 The graphical object is depicted as a rectangular bounding box 64, but the graphical object may take the form of any graphical indication of the UAV's location, such as any other geometric shape, such as a circle or triangle, a general outline around the UAV 14, or a highlight, shadow, flashing, or other visually identifiable indication around the image of the UAV 14 on the display screen 38.

[0073] In some examples, the HMD 16 (e.g., the transceiver 44 within the HMD 16) may receive orientation data and inertial data from the OU 56 and IMU 54, respectively, so that the PC 42 can "predict" the subsequent relative position of the UAV 14 and update the bounding box 64 accordingly. By determining the location of the UAV 14 before the UAV 14 reaches that location, the HMD 16 may reduce the "lag time" between the UAV 14 arriving at a particular location and the display of the bounding box 64 over that location.

[0074] The GUI 72 further includes a plurality of virtual elements indicating data obtained by sensors within the UAV 14, the HMD 16, or both. For example, the GUI 72 includes a screen indicating the distance 70 between the UAV 14 and the HMD 16, the airspeed and orientation 74 of the UAV 14, the position of the navigation system components 32, 52, and the like. Figure 2 )'s number of satellites 76 and elements of the estimated remaining battery life 78 of the UAV 14.

[0075] Figure 5 is a device that can be used on the HMD 16 ( Figure 2 ) is generated and displayed on the display screen 38 of the exemplary UI or GUI 80. Figure 5 In the depicted exemplary GUI 80, the processing circuitry 42 ( Figure 2 ) has determined that the position of the UAV 14 is not within the field of view of the display screen 38. In this case, the processing circuit 42 generates and outputs a text alert 82 for display on the display screen 38 indicating that the UAV pilot 12 has lost line of sight with the UAV 14. In some examples, the HMD 16 may additionally or alternatively output an audio alert indicating this situation. The GUI 80 further includes a graphical object such as a set of arrows 84 that indicate to the pilot 12 the direction to turn their head in order to bring the UAV 14 back into the field of view of the display screen 38. Although Figure 5 The graphical object is depicted as a set of arrows 84, but the graphical object may take any other form of directional indication, such as a single arrow or a flashing light along the respective edge of the display screen 38. In some examples, in addition to or in lieu of the graphical objects, the HMD 16 may output an audible indication, such as a tone or sound, in the wearer's respective ears indicating the direction in which the wearer should turn his or her head.

[0076] Figure 6 is a device that can be used on the HMD 16 ( Figure 2 ) is generated and displayed on the display screen 38 of the exemplary UI or GUI 86. Figure 6In the depicted exemplary GUI 86, the processing circuitry 42 ( Figure 2 ) has determined that the position of UAV 14 is within the field of view of display screen 38 and has generated a bounding box 64 around the position of the UAV. However, transceiver 44 of HMD 16 has received data from altimeter 58 of UAV 14 indicating that UAV 14 is flying too low and may be at risk of impacting the ground. In this case, processing circuitry 42 generates and outputs a text alert 88 for display on display screen 38, which advises UAV pilot 12 to operate controller 30 to increase the altitude of UAV 14. In some examples, HMD 16 may additionally or alternatively output an audio alert indicating this situation.

[0077] In some other examples, processing circuitry 42 may generate and output a number of other virtual or graphical elements ( Figure 6 14) for display on screen 38. For example, screen 38 may display the planned flight path of the UAV 14, including indications of any potential obstacles within the flight path, such as bounding boxes. In another example, screen 38 may display an indication of a controlled airspace or "no-fly zone." For example, screen 38 may receive data indicating an oil refinery and generate a virtual boundary indicating a threshold distance (such as 100 meters) that the UAV 14 must leave the refinery. In some examples, screen 38 may display a live video feed from the camera 116 of the UAV 14, displayed as a "picture-in-picture" within the larger field of view of screen 38. In some examples, screen 38 may display a graphical element indicating the field of view of the camera 116 of the UAV 14, such as a triangular graphical element with the UAV 14 at its vertices. In some examples, screen 38 may display a number of other flight indications or warnings, such as indications of low UAV battery power, loss of GPS signal, close proximity of obstacles, or completion of one or more mission milestones. For example, the processing circuitry 42 may perform image processing on image data captured by the camera 40 of the HMD 16, the camera 116 of the UAV 14, or both, in order to identify one or more obstacles, such as birds or nearby aircraft, that are proximate to the UAV 14. In other examples, the processing circuitry may determine the presence of a nearby aircraft when the transceiver 44 receives an Automatic Dependent Surveillance-Broadcast (ADS-B) signal from an aircraft.

[0078] Figure 7 is a flow chart illustrating exemplary operations for determining and displaying the position of a UAV relative to the FOV of an HMD according to the techniques of this disclosure. Figure 1 and Figure 2 System 10 Figure 7The exemplary technique of FIG. 1 is provided, but the technique may be performed by any suitable computing system. The processing circuit 42 receives data indicating the geographic location of the UAV 14, such as from the GPS device 52 and the altimeter 58 installed within the UAV, via the transceivers 62 and 44. The processing circuit 42 further receives data indicating the orientation of the UAV 14, such as from the compass 56 and the IMU 54 installed within the UAV (120).

[0079] Processing circuitry 42 further receives data indicating the geographic location of HMD 16, such as from GPS device 32 mounted within the HMD. Processing circuitry 42 further receives data indicating the orientation of HMD 16, such as from compass 36 and IMU 34 mounted within the HMD (122).

[0080] Processing circuitry 42 determines 124 the relative position of UAV 14 relative to the field of view of display screen 38 of HMD 16 based on the position and orientation data for both UAV 14 and HMD 16. For example, processing circuitry 42 determines whether UAV 14 is within or outside the field of view of the screen.

[0081] The processing circuitry 42 generates and outputs an indication of the location of the UAV 14 for display on the screen 38 (126). For example, if the UAV 14 is within the field of view of the display screen 38, the processing circuitry 42 generates and outputs a rectangular-shaped bounding box around the approximate location of the UAV 14 relative to the screen. In examples where the UAV 14 is not within the field of view of the display screen 38, the processing circuitry generates and outputs an indication, such as an arrow or set of arrows, indicating the location of the UAV 14 relative to the screen 38 so that the user of the HMD 16 can turn his or her head to bring the UAV 14 back into the field of view.

[0082] Figure 8 is a flow chart illustrating exemplary operations for determining and displaying the position of a UAV relative to the FOV of an HMD according to the techniques of this disclosure. Figure 1 and Figure 2 System 10 Figure 8 , but the technique may be performed by any suitable computing system. In some examples, the HMD 16 establishes a data communication connection (90) with a ground station 20, such as a laptop or other data transmission device. In other examples, the HMD 16 communicates directly with the UAV 14. The HMD 16 receives UAV position data (92), such as from the GPS sensor 52 and the altimeter 58 within the UAV 14. The HMD 16 receives HMD position data (94), such as from the GPS sensor 32 within the HMD 16.

[0083] The HMD 16 receives UAV orientation data (96), such as from the compass 56 and the IMU 54 within the UAV 14. The HMD 16 receives HMD position data (98), such as from the compass 36 and the IMU 34 within the HMD 16.

[0084] Using the position and orientation data for both HMD 16 and UAV 14, HMD 16 determines a distance between HMD 16 and UAV 14 (100). Processing circuitry 42 within HMD 16 generates a spherical coordinate plane centered within HMD 16 such that the boundaries of the field of view of display screen 38 extend along a set of radii of the sphere (102). Based on the spherical coordinate plane, processing circuitry 42 determines a frustum bounded by the field of view of camera 40 (104). Processing circuitry 42 determines a distance between the approximate position of UAV 14 and the center of the determined frustum (e.g., the center of the spherical coordinate plane) (106). Based on the distance, processing circuitry 42 determines whether the position of UAV 14 is within the determined frustum (108).

[0085] If the UAV 14 is outside the frustum size, the processing circuit 42 determines and instructs the direction to move the frustum (e.g., by the wearer of the HMD 16 turning his or her head) so as to decrease the distance between the frustum and the UAV 14 (110), and then repeats the previous steps as needed until the UAV 14 is within the frustum size.

[0086] If the UAV 14 is within the frustum dimensions, the processing circuitry 42 generates and outputs a graphical indication, such as a rectangular bounding box, around the UAV location for display (112).

[0087] Figure 9 is a flow chart illustrating exemplary operations for determining and displaying the position of a UAV relative to the FOV of an HMD according to the techniques of this disclosure. Figure 1 and Figure 2 System 10 Figure 9 The exemplary techniques are described herein, but the techniques may be performed by any suitable computing system. The unmanned aerial vehicle system 10 includes an augmented reality (AR) head mounted display (HMD) device 16 having a processing circuit (PC) 42 configured to determine at least a position (900) of an unmanned aerial vehicle (UAV) 14. For example, the UAV 14 may include a positioning system (PS) 52, such as a global positioning system (GPS), an altimeter 58, and a transceiver 62 configured to position the UAV 14 above the ground at a UAV latitude U. lat 、UAV longitude U lon and altitude A UAV Transmitted to the HMD 16 .

[0088] The PC 42 of the HMD 16 may also be configured to determine 902 the position and orientation of the HMD 16. For example, the HMD 16 may include its own positioning system (PS) 32 and orientation unit (OU) 36 configured to: determine the HMD latitude H lat 、HMD longitude H lon , cardinal direction heading or bearing α HMD and the angle β relative to gravity HMD .

[0089] An indication of the location of the UAV is presented on the AR HMD based on the orientation of the AR HMD.

[0090] Using at least this data, PC 42 of HMD 16 is configured to determine whether the position of the UAV (including geolocation and altitude) is within the field of view (FOV) of HMD 16 (904). In some examples, PC 42 may construct a grid-based coordinate system to determine whether UAV 14 is positioned within both the horizontal and vertical fields of view of HMD 16. In other examples, PC 42 may construct a spherical coordinate system centered on HMD 16 and determine whether the angle between the center C of the FOV and UAV 14 is greater than or less than a fixed angle of the FOV of HMD 16. In some examples, PC 42 may use a combination of both grid-based distances and spherical angles to determine the relative position of UAV 14.

[0091] In response to (e.g., based on) determining whether the UAV 14 is within the FOV of the HMD 16, the PC 42 is configured to generate and present an indication of the relative position of the UAV on the display screen 38 of the AR HMD 16. For example, in response to determining that the UAV 14 is within the FOV of the HMD 16, the PC 42 is configured to present a first graphical object (906) on the HMD 16. The first graphical object may include, for example, a bounding box indicating a possible location of the UAV, or any other visually based indication of the UAV's position.

[0092] PC 42 is also configured to present a second graphical object on HMD 16 in response to determining that UAV 14 is not within the FOV of HMD 16 (908). The second graphical object may include, for example, a set of arrows or other visually based indication of the relative position of UAV 14 with respect to HMD 16.

[0093] In some examples, the PC 42 is further configured to determine the location of obstacles near the flight path of the UAV 14 and present an indication of the location of the obstacle on the AR HMD based on the orientation of the AR HMD. For example, the PC 42 may determine the location of the obstacle by processing image data from the HMD camera 40 or the UAV camera 116 and then display a bounding box around the obstacle on the display screen 38. Examples of flight path obstacles include structures (e.g., buildings, radio towers), birds, manned aircraft (such as airplanes, helicopters, etc.), second UAVs, or terrain (such as rocks, trees, hills, etc.).

[0094] In some examples, the PC 42 is further configured to determine the location of controlled airspace and present an indication of the location of the controlled airspace on the AR HMD based on the orientation of the AR HMD. For example, the nearby controlled airspace may include an airport, an oil refinery, a stadium, or other regulated airspace with a surrounding no-fly threshold distance.

[0095] In some examples, PC 42 is configured to determine the field of view of camera 116 of UAV 14 and output an indication of the field of view of camera 116 for display on HMD 16 so that a wearer of HMD 16 can visually determine the approximate direction of camera 116 and its target capture window.

[0096] In some examples, the PC 42 may be further configured to determine and output the remaining battery life of the UAV 14, the altitude A of the UAV 14, and the UAV , one or more of the airspeed of the UAV 14, the number of satellites in communication with the UAV 14, or the compass heading of the UAV 14 are displayed.

[0097] In some examples, PC 42 may be further configured to output an alert for display on HMD 16, such as in response to determining and indicating one or more of a low UAV battery, a loss of GPS signal, an obstacle approach warning, or reaching a UAV mission milestone.

[0098] In some examples, the transceiver 44 and the PC 42 may be configured to receive video data from the UAV camera 116 and output the video data for display on the HMD 16. For example, the display screen 38 of the HMD 16 may include a “picture-in-picture” window that features a live feed of recorded video data from the camera 116 of the UAV 14.

[0099] In some examples, transceiver 44 and PC 42 may be configured to receive a planned flight path for UAV 14 and output an indication of the flight path for display on HMD 16 , such as relative to a current position or location of UAV 14 .

[0100] In some examples, PC 42 may be further configured to determine that line of sight is lost between HMD 16 and UAV 14 and output an indication of the loss of line of sight for display on HMD 16 .

[0101] In one or more examples, the functions described may be implemented by hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or codes and executed by a hardware-based processing unit. The computer-readable medium may include computer-readable storage media 43, 61 corresponding to tangible media (such as data storage media), or communication media including, for example, any medium that facilitates the transfer of a computer program from one place to another according to a communication protocol. Thus, the computer-readable medium may generally correspond to: (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0102] By way of example and not limitation, such computer-readable storage media 43, 61 may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures and that can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves are used to transmit instructions from a website, server, or other remote source, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are directed to non-transient tangible storage media. As used herein, disks and optical disks include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks typically copy data magnetically, while optical discs use lasers to copy data optically. Combinations of the above should also be included within the scope of computer-readable media.

[0103] Instructions may be executed by one or more processors 42, 60, such as one or more DSPs, general-purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuits. Thus, as used herein, the term "processor" may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. Furthermore, these techniques may be implemented entirely in one or more circuits or logic elements.

[0104] The techniques of the present disclosure can be implemented in a variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or a set of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize the functional aspects of devices configured to perform the disclosed techniques, but they do not necessarily need to be implemented by different hardware units. Instead, as described above, the various units may be combined in a codec hardware unit or provided by a collection of interoperable hardware units (including one or more processors as described above) in combination with appropriate software and / or firmware.

[0105] The cloud technology used to automatically save images on a web server is not limited to a local or global internet cloud. It can be a private cloud and / or a public cloud protected by a user ID and password. The password may not be limited to one or two.

[0106] Various examples have been described. These and other embodiments are within the scope of the following claims.

Claims

1. A device comprising: Augmented reality AR head-mounted display HMD; and a processing circuit, the processing circuit being configured to: Receiving and determining a planned flight path of an unmanned aerial vehicle (UAV); determining a location of the UAV; determining a position and orientation of the AR HMD; determining, based on the position of the UAV and the planned flight path of the UAV, a first graphical object indicating the planned flight path of the UAV; outputting a first graphical object indicating the planned flight path of the UAV for display on the AR HMD; determining a location of an obstacle within the field of view of the AR HMD that presents a loss of line of sight between the AR HMD and the UAV along the planned flight path; as well as A second graphical object indicating the position of the obstacle is output for display on the AR HMD.

2. The apparatus of claim 1 , wherein the processing circuit is further configured to: determining a position and orientation of the AR HMD; determining, based on the position of the UAV and the position and the orientation of the AR HMD, that the UAV is outside a field of view of the ARHMD; In response to determining that the UAV is outside the field of view of the AR HMD, presenting a third graphical object on the AR HMD that indicates movement that brings the UAV into the field of view of the AR HMD; determining that the UAV is within the field of view of the AR HMD based on the position of the UAV and the position and the orientation of the AR HMD; as well as In response to determining that the UAV is within the field of view of the AR HMD, presenting a fourth graphical object on the AR HMD indicating the position of the UAV.

3. The apparatus of claim 1 , wherein the fourth graphical object indicating the location of the UAV comprises a bounding box surrounding the UAV.

4. The apparatus of claim 1 , wherein the processing circuit is configured to determine the location of the obstacle by processing image data from a camera located on the HMD or the UAV.

5. The apparatus of claim 2 , wherein the processing circuit is further configured to: determining a position of controlled airspace relative to the field of view of the AR HMD based on the position and the orientation of the AR HMD; and A fifth graphical object is presented on the AR HMD that indicates the position of the controlled airspace relative to the field of view of the AR HMD.

6. The apparatus of claim 2, the processing circuit further configured to determine a field of view of a camera of the UAV; and A fifth graphical object indicating a boundary of the field of view of the camera relative to the field of view of the AR HMD is output for display on the AR HMD.

7. The apparatus of claim 1 , the processing circuitry being further configured to display, on the AR HMD, a third graphical object indicating: the remaining battery life of the UAV; the altitude of the UAV; the airspeed of the UAV; the number of satellites in communication with the UAV; a compass heading of the UAV; Low UAV battery alert; GPS signal loss alarm; Obstacle proximity warning; or UAV mission milestone notifications.

8. The apparatus of claim 1 , wherein the processing circuit is further configured to: receiving video data from the UAV; and The video data is output for display on the AR HMD. 9 . The device of claim 1 , wherein the AR HMD comprises an opaque display screen, the AR HMD being configured to display an image on the display screen.

10. The device of claim 1, wherein the AR HMD comprises a see-through holographic lens, the AR HMD being configured to display one or more virtual elements on the lens.

Images