Methods and apparatus for augmented reality

Abstract

A method for generating virtual augmentation relating to a drone comprising the steps of providing the location of a drone and determining the location and orientation of a mobile device. Generating v

Description

Background of the Invention It is known to use camera enabled drones for inspection and search and rescue missions. 30 04 24 30 04 24 Brief Summary of Invention In accordance with the present invention, there is provided a method for generating virtual augmentation relating a drone and to a corresponding system and mobile device for the same, as claimed in the accompanying claims. In particular, such a method comprises the steps of: providing the location of a drone; determining the location and orientation of a mobile device (especially when not the not the primary controller of the drone); generating virtual augmentations relating to the drone based on the location and orientation of the mobile device; and displaying the virtual augmentations to the user aligned to a point-of-view of the mobile device as based on the location and orientation of the mobile device. The virtual augmentations may be defined as 3D objects in a 3D model aligned with the point-of-view of the mobile device and rendered in 2D therefrom, including by using a commercial, off-the-shelf gaming engine. The virtual augmentations may be initially generated based on an assumed orientation of the mobile device in the direction of the drone, and then regenerated based on the determined orientation of the mobile device. The virtual augmentations may be displayed over real-time images from a camera of the mobile device. Where this is the case, image stabilization algorithms applied to real-time images from the camera may also be applied to the virtual augmentations. Also, the virtual augmentations may be displayed over images generated from a 3D model of buildings and / or terrain and, similarly, image stabilization algorithms applied to images generated from the 3D model of buildings and / or terrain may also be applied to the virtual augmentations. Indeed, virtual augmentations may be defined as 3D objects in the same 3D model as any used for buildings and / or terrain. Virtual augmentations may be displayed over a combination of real-time images from a camera of the mobile device and images generated from a 3D model of buildings and / or terrain. In respect of determining the orientation of the mobile device, this may be done by a sensor in the mobile device, by comparing a 3D model of buildings and / or terrain around the mobile device with real-time images from a camera of the mobile device or first by the former and then the latter. If the orientation of the mobile device is not towards the drone such that the drone is located off-screen, an indicator may be displayed indicating the direction of the drone in relation to the mobile device. In respect of specific augmentations, the virtual augmentation may comprise an indication of the present location of the drone as shown in the display of the mobile device including when the drone is obscured by an object in the foreground of the display relative to the drone. Also, the virtual augmentation may comprise an indication of the present location of the drone when not in the present field of view. 30 04 24 In respect of the flight path of the drone, the virtual augmentation may comprise an indication of the future or past flight path of the drone (including when the drone is obscured by an object in the foreground relative to the past or future flight path of the drone). The virtual augmentation may comprise an indication of an obscured object which is relevant to the drone (for example, by virtue of either being in the foreground relative to the drone, adjacent the drone, adjacent the future flight path of the drone or identified by mission parameters). The virtual augmentation may comprise an indication of the destination of the drone and / or an indication of timing or distance metrics from the present location of the drone to objects adjacent the past or future flight path of the drone and / or the destination of the drone. The virtual augmentation may comprise an indication of any nearby controller of the drone, information specifically for the user of the mobile device, information relating to the status or mission of the drone and instructions describing actions to be taken in respect of the drone. The virtual augmentation may be selected, created or modified by a user of the mobile device and, in particular, within boundaries defined by the parameters of a drone's mission. The virtual augmentation may identify a 2D plane or 3D region (including one which can be selected, created or modified by a user). Part of such a region may indicate an area or region which is to be or has been inspected. Virtual augmentation identifying a 2D plane or 3D region may include a line array, at least some of which lines correspond to paths that the drone has or will take when inspecting. Virtual augmentation may identify the destination of a delivery from a drone (including where the delivery is suspended below the drone in which case the virtual augmentation may further identify the future trajectory of the delivery when so suspended). In such a delivery scenario, the virtual augmentation may be selected and moved (for example, by dragging and dropping) to modify the destination of the delivery. A predefined location may be displayed by the mobile device for the user to position themselves at. Where this is the case, virtual augmentations may be generated based on and / or limited to those predefined locations including so as to minimise or prioritise computational effort and / or bandwidth requirements. Pre-defined location(s) may be defined by a mission parameter(s). 30 04 24 Brief Description of Drawings The present invention will now be described, by way of example only, with reference to the following figures in which: Figure 1 is a schematic of apparatus used to implement the present invention; Figure 2 is a schematic of a drone of the apparatus of figure 1; Figure 3 is a schematic of a mobile device of the apparatus of figure 1; Figures 4A to 4E illustrate virtual augmentation for inspection / search and rescue scenarios; Figures 5A to 5G illustrate virtual augmentation for an inspection / security scenario; and Figure 6 illustrates virtual augmentation for a delivery scenario. Detailed Description of Invention Referring to figure 1, the present invention is illustrated by a system composed primarily of a drone, a drone mission control centre and an observer mobile device. Optionally, additional drones may be employed, multiple observer mobile devices may be used, mobile drone controller(s) may be employed, and the mission control aspect of the invention may be fragmented, i.e., employing distributed processing. Figures 2 and 3 illustrate the communication requires of a drone DI and a mobile device Ml for use with the present invention. Both drone DI and mobile device Ml have a GPS antenna 23, 33 for receiving GPS satellite SV signals, a GPS receiver front-end 24, 34 for pre-processing GPS signal (downconversion, amplification etc.) and a GPS microprocessor 25, 35 for processing the GPS signals to obtain a position fix (for example, by despreading GPS, resolving clock ambiguity and triangulating). Drone DI and mobile device Ml each have communications antennas 20, 30, communications frontends 21, 31 and communications microprocessors 22, 32 for effecting two-way communications with nearby cellular network base-stations BS1 and thus respective providing two-way data channels. Such data channels can be used to acquire aiding information for rapid GPS signal acquisition (i.e., Assisted-GPS). A limitation of conventional GPS is the difficulty of determining device orientation. As such, mobile device Ml contains an orientation sensor which can be a geomagnetic field sensor in combination with an accelerometer. Such drone and mobile device technology is conventional and commercially available, and it is expected that a person skilled in the art seeking to implement the present invention would start from such conventional and commercially available technology. 30 04 24 In accordance with the present invention, a computer system is configured as a drone mission control centre for co-ordinating drone missions with drones and observer devices as will now be described. The underlying observer experience facilitated by the present invention is the virtual augmentation of a mobile devices display, smart glasses or head-up projection for an observer in relation to drone and a corresponding drone mission. Whilst it is conceivable that some degree of drone control may be exercised by the observer, it is contemplated that the mobile device is not the primary controller of the drone. As such, the augmentation for the user is typically related to the nature of a drone mission per se and only indirectly related to drone control. In particular, in accordance with the present invention, virtual augmentations generated relating to a drone are done so based on the location and orientation of the mobile device. This enables virtual augmentations to be aligned to a point-of-view of the observer mobile device and thus the observer, and there are several ways in which this can be done. In the first instance, the virtual augmentations must be defined and one way this can be done is to create corresponding objects in a 3D model and align that model to the real world. For example, if a desired virtual augmentation is a trail of a drone (for example, for conveying the path history of a drone), that virtual augmentation may be defined in a 3D model as a flexible cylinder for which the centre line may be defined by a high order polynomial and the diameter of the cylinder defined. Thus, when aligned with real-time images from a camera of the mobile device, rendered in 2D and overlapped on those real-time images, a drone viewed in those images may be shown with virtual augmentation in the form of a trail emphasizing its historic path. Commercial, off-the-shelf gaming engines can be modified to readily provide the 2D rendering of a 3D modelled environment. For example, Unity3D provides APIs (Application Programming Interfaces) that allow developers to access and use a mobile device's sensors, including any accelerometer(s) and gyroscope, to determine the device's orientation. Moreover, using these APIs, it is possible to create scripts that track the device's orientation and use that information to position and orient virtual augmentation over the camera feed. Alignment from a determined orientation of a mobile device is preferred. However, it is possible that virtual augmentations are initially generated based on an assumed orientation of the mobile device in the direction of the drone in which case one would expect an observer to orientate the mobile device to align in the direction of the drone. This is especially the case if the virtual augmentation includes a representation of the drone itself. Moreover, this could be used as a temporary measure until a determined orientation of the mobile device becomes available. If a 3D model of buildings and / or terrain around the mobile device is available, it may be possible to determine and / or refine the orientation of the mobile device by comparing the model with real-time 30 04 24 images from a camera of the mobile device. If the same model is used to define the virtual augmentations, this may ensure precise alignment between the virtual augmentations and real-time images, and also facilitate the use of common image stabilization for both the real-time images and the virtual augmentations. Figures 4A to 4E illustrate virtual augmentation in a display of a mobile device for inspection and search and rescue scenarios. In the following inspection scenario, it is presupposed that prior to a drone mission, a controller at a mission control centre has programmed a mission for a drone to inspect an oil storage facility by capturing drone images in a predetermined direction from a virtual 2D plane running alongside the oil storage facility. Aground-based operative carrying a camera enabled mobile device (such as one of the type illustrated in figure 2) is able to view a display of real-time images from the camera of the mobile device. As shown in figure 4A, this includes displaying the real drone 40 as it approaches the scene as filmed by the camera, together with a virtual augmentation being an icon 41 confirming the presence of the drone and identifying the drone by its corresponding identifier 'DI' included in the icon 41. For the purposes of illustration, the icon 41 is shown adjacent the drone 40 in the scene but it could alternatively be wholly coincident. The virtual augmentation enables a ground-based operative to discern the presence of the drone in circumstances when it might otherwise not be possible, for example, when the drone is obscured by clouds or when the drone is beyond the range of the ground-based operative's visual acuity. Further presupposing that the mission is underway, and the drone has left the scene, figure 4B shows the display of real-time images from the camera of the mobile device on to which is superimposed a virtual 2D plane corresponding to the 2D plane of the mission. To do this, the virtual augmentation is rendered in 2D from 3D elements (vertical 42 and horizontal 43 being thin cylinders as shown or alternatively a line of spheres) to create the array. This requires alignment of real-time images from the camera of the mobile device with those 3D elements, and thus determination of the position and orientation of the mobile device. To indicate to the ground-based operative which side of the 2D plane is being inspected, the virtual augmentation further includes a direction pointer 44. As shown in figure 4C and 4D, the virtual augmentation can include distinct array sections 45,46 which may respectively be designate 'for inspection' and 'inspected'. Such array sections may be selectable by the ground-based operative, for example, in order to select a particular array sections 46 that has been inspected in order to retrieve and view corresponding images taken by the drone on the mobile device MSI. 30 04 24 In the following search and rescue scenario, it is presupposed that prior to a drone mission, a controller at a mission control centre has programmed a mission for a drone to take images of a region of coastline. Figure 4E illustrates virtual augmentation for this scenario being a 2D horizontally oriented planar array 47 positioned above a corresponding region of coastline. Although it is possible that perpendicular elements of the array may define a path to be taken or a path previously taken by a drone, the height of the horizontally oriented plane 46 may also arbitrary, located a particular height above the Earth's surface so as to illustrate search and rescue regions on the Earth's surface, beyond the horizon. As with the augmentation shown in figure 4C and 4D, the virtual augmentation in figure 4E can include distinct array sections which may respectively be designated as 'to be searched' and 'searched' and, similarly, where such array sections may be selectable in order to retrieve and view corresponding search images taken by the drone. Figures 5A to 5E illustrate virtual augmentation for an inspection / security scenario about a power station 50. In this scenario, it is presupposed that prior to a drone mission, a controller at a mission control centre has programmed a mission for a drone 'D2' to repeatedly inspect the perimeter of a power station. As with the scenarios shown in figures 4A to 4E, a ground-based operative carrying a camera enabled mobile device is able to view a display of real-time images from the camera of the mobile device. As illustrated in figure 5A, the drone is located on the far side of the power station relative to the ground-based operative and thus is obscured from view being behind by the power station. However, the virtual augmentation 51, 52 superimposed on the real-time images from the camera of the mobile device is able to illustrate the hidden position of drone 'D2' and, in addition, the future and past paths of that drone (the direction of perimeter inspection / drone trajectory being indicated by the arrow). Drones and path trajectories are provided in one format (dashed lines) when obscured and in another format (solid line) when not obscured. In addition to indicating the direction of perimeter inspection / drone trajectory, formats distinguishing between future and past drone paths could also be implemented. A variation to this is shown in figure 5B where a virtual representation 53 of the power station is superimposed on the real-time images from the camera of the mobile device and could be generated from commercially available 3D building models and the virtual representation of the power station rendered in to 2D at the same as rendering 3D elements corresponding to the drone and drone path. This variation would be particularly useful at night where the power station might not otherwise be clearly visible to the ground-based operative. A related variation is shown in figure 5C wherein the real-time images from the camera of the mobile device are temporarily disabled and only the virtual augmentation shown (still aligned to the orientation of the mobile device), supplemented with a 30 04 24 representation of the horizon 54. Also, a further related variation is shown in figure 5D wherein such virtual augmentation remains visible, even when obscured by an object in the foreground. Lastly, a yet further related variation is shown in figure 5E wherein the real-time images from the camera of the mobile device are only disabled in respect of the obscuring object in the foreground. Such displays convey to the ground-based operative mission information about the drone and adjacent / relevant buildings that might not otherwise be discernible. As illustrated in figure 5F, in this inspection / security scenario, the mission specifies a preferred location 55 for the ground-based worker to locate themselves, visible on a plan view map accessible from the mobile device. The virtual augmentation or computation thereof may be tailored and / or pre-processed based on such a location. In figures 5A to 5E, the camera of the mobile device is located in sufficient proximity of the power station so that the visibility of the location of the drone in respect of the horizon is not a problem. Figure 5G illustrates a more distant view of the power station 50 with the virtual augmentation 56, 57 including an indication of the path of the drone 56 originating from over the horizon. A further augmentation 58 in the form of a ribbon is located directly above the virtual augmentation 56 illustrating the path of the drone which continues to indicate the past path 56 of the drone in relation to the surface of the Earth beyond the horizon. The width of the ribbon 58 and its array construction enables the perspective and thus relationship with the path 56 of the drone to be readily discerned. Moreover, to further emphasize both the future and past path of the drone, a corresponding augmentation 59 is projected on the ground located directly below the path 56 of the drone. In figure 5G, visible on the beach above which the drone is travelling and further enables the perspective and thus relationship with the actual path 56 of the drone to be readily discerned. Figure 6 illustrates virtual augmentation in a display of a mobile phone 60 for a delivery scenario in which a drone (not shown) is lowering a package for delivery 61 to the user of the mobile phone. The delivery scene is potentially problematic since the cluttered scene includes posts 62, boats 63 and water 64 which could adversely affect the delivery. The desired destination is on the jete 65. When the delivery is being executed, the user is able to view augmentations 66, 67 superimposed on the realtime images from the camera of the mobile phone such that 'X' 66 marks the projected delivery spot and chevrons indicate the trajectory of the package as it is lowered. Were the projected delivery spot over water, it could be stopped or changed, for example, by the user of the mobile phone dragging and dropping the 'X' 66 as it appears in the display. Other variations on the presently described invention will suggest themselves to persons skilled in the art. 30 04 24

Claims

1. A method for generating virtual augmentation relating to a drone comprising the steps of:providing the location of a drone;determining the location and orientation of a mobile device;generating virtual augmentations relating to the drone based on the location and orientation of the mobile device; anddisplaying the virtual augmentations to the user aligned to a point-of-view of the mobile device based on the location and orientation of the mobile device.

2. A method according to claim 1 wherein the mobile device is not the primary controller of the drone.

3. A method according to claim 1 wherein the virtual augmentations are defined as 3D objects in a 3D model aligned with the point-of-view of the mobile device and rendered in 2D therefrom.

4. A method according to claim 3 wherein the rendering is done using a commercial, off-the-shelf gaming engine.

5. A method according to any preceding claim wherein the virtual augmentations relating to the drone are initially generated based on an assumed orientation of the mobile device in the direction of the drone, and then regenerated based on the determined orientation of the mobile device.

6. A method according to any preceding claim wherein the virtual augmentations are displayed over real-time images from a camera of the mobile device.

7. A method according to claim 6 wherein image stabilization algorithms are applied to the realtime images from the camera and corresponding image stabilization algorithms are also applied to virtual augmentations.

8. A method according to any of claims 1 to 4 wherein the virtual augmentations are displayed over images generated from a 3D model of buildings and / or terrain.

9. A method according to claim 8 wherein image stabilization algorithms are applied to images generated from the 3D model of buildings and / or terrain and corresponding image stabilization algorithms are also applied to virtual augmentations.

10. A method according to claim 8 or claim wherein the virtual augmentations are defined as 3D objects in the same 3D model as used for the buildings and / or terrain.

11. A method according to any of claims 1 to 4 wherein the virtual augmentations are displayed over a combination of real-time images from a camera of the mobile device and images generated from a 3D model of buildings and / or terrain.

12. A method according to any preceding claims wherein the orientation of the mobile device is determined by a sensor in the mobile device.

13. A method according to any of claims 1 to 11 wherein the orientation of the mobile device is determined by comparing a 3D model of buildings and / or terrain around the mobile device with realtime images from a camera of the mobile device.

14. A method according to any of claims 1 to 11 wherein the orientation of the mobile device is first determined by a sensor in the mobile device and is then refined by comparing a 3D model of buildings and / or terrain around the mobile device with real-time images from a camera of the mobile device.

15. A method according to any preceding claim further comprising the step of, if the orientation of the mobile device is not towards the drone such that the drone is located off-screen, displaying an indicator indicating the direction of the drone in relation to the mobile device.

16. A method according to any preceding claim wherein the virtual augmentation comprises an indication of at least one of the present location of the drone as shown in the display of the mobile device, the present location of the drone when the drone is obscured by an object in the foreground relative to the drone or the present location of the drone when not in the present field of view.

17. A method according to any preceding claim wherein the virtual augmentation comprises an indication of at least one of the future flight path of the drone, the future flight path of the drone when the drone is obscured by an object in the foreground relative to the future flight path of the drone, the past flight path of the drone or the past flight path of the drone when the drone is obscured by an object in the foreground relative to the past flight path of the drone and at least one of an obscured object which is relevant to the drone.

18. A method according to any preceding claim wherein the virtual augmentation comprises an indication of the destination of the drone and / or timing or distance metrics from the present location of the drone to objects adjacent the past or future flight path of the drone and / or the destination of the drone.

19. A method according to any preceding claim wherein the virtual augmentation comprises an indication of any nearby controller of the drone, provides information specifically for the user of the mobile device, provides information relating to the status or mission of the drone and / or provides instructions describing actions to be taking in relation to the drone.

20. A method according to any preceding claim wherein the virtual augmentation can be selected, created or modified by a user of the mobile device including within boundaries defined by parameters of a drone's mission.

21. A method according to any preceding claim wherein at least one of the following applies:the virtual augmentation identifies a 2D plane or 3D region;a part of either the 2D plane or 3D region can be selected, created or modified by a user of the mobile device;a part of either the 2D plane or 3D region indicates an area or region which is to be or has been inspected; andthe virtual augmentation identifying a 2D plane or 3D region includes a line array, at least some of which lines correspond to paths that the drone has or will take when inspecting.

22. A method according to any preceding claim wherein the virtual augmentation identifies the destination of a delivery from a drone including when suspended below the drone, identifies the future trajectory of the delivery when suspended below the drone and / or can be selected and moved to modify the destination of the delivery.

23. A method according to any preceding claim further comprising the step of displaying a predefined location for the user using the mobile device to position themselves including wherein the generation of virtual augmentations is based on and / or limited to those pre-defined locations, including to minimise or prioritise computational effort and / or bandwidth requirements and including wherein the pre-defined location is defined by a mission parameter.

24. A system comprising at least a drone, a drone controller and a mobile device for performing a method as claimed in any preceding claim.

25. A mobile device of the system of claim 24.

Images