Touch control for drones

Abstract

Touch control of an unmanned aerial vehicle. The present invention relates to a computer-implemented method for controlling a UAV, the method comprising the following steps in a target selection mode: displaying a 3D view of an environment of the UAV on a touch-sensitive display; superimposing a movable target indicator symbol to the 3D view of the environment, wherein the target indicator symbol is movable in the 3D view by touch input; while moving the target indicator symbol, continuously determining the position of the target indicator symbol in the 3D view and dynamically changing the appearance of the target indicator symbol so as to produce the effect that the target indicator symbol is displayed in an orientation that matches the orientation of a face located below the target indicator symbol, wherein the orientation of the corresponding face is derived from stored 3D data or from the 3D view; and selecting a target based on the position of the target indicator symbol.

Description

Technical Field

[0001] The present disclosure relates to a computer-implemented method for touch control of an unmanned aerial vehicle (UAV). Background Art

[0002] UAVs (particularly UAVs of the rotary wing drone type) have flooded the consumer and service markets and are being developed to manage a wide variety of tasks in both technical and non-technical fields. Typical tasks involve, for example, recording movie scenes, freight, inspection of buildings / technical facilities, surveying / digitizing the physical environment, etc. The functionality of UAVs is increasing rapidly, and the complexity of manageable tasks is also increasing rapidly. A key factor in the success of UAVs in the consumer market is their remote controllability, which must be feasible for almost anyone. In particular, with the increasing complexity of the tasks that UAVs must perform, it may be challenging to provide intuitive, simple and reliable controllability. Remotely controlling UAVs is typically based on the use of control systems, environmental sensor systems and motion generation systems, wherein these systems are interconnected at least in communication. The control system controls at least the propulsion unit of the UAV. The environmental sensor system is typically arranged on the UAV and senses the environment of the UAV in flight. Sensing typically involves detecting obstacles in the environment of the UAV to avoid collisions. The motion generation system generates control commands based on user input and sensing signals of the environmental sensor system, wherein these control commands are received by the control system. User input is usually received through a graphical user interface (GUI), wherein the GUI presents high-level control commands that can be triggered by the user. Triggering high-level control commands usually causes the control system to execute numerous complex control commands, ultimately guiding the movement of the UAV. In order to present an easy-to-grasp GUI with intuitive top-level control commands, the GUI usually shows a three-dimensional view (3D view) of the physical environment of the UAV from the perspective generated by the onboard camera of the UAV in a real-time stream, such as a three-dimensional representation. The GUI then displays a 3D view of the physical environment of the UAV, for example, on a two-dimensional display plane of the screen of a computing device. Thus, a simple image generated by the camera of the UAV represents a 3D view of the physical environment-because it provides a stereogram and thus includes depth information-such as a 3D model of the environment, which includes model points of the environment with three-dimensional coordinate information. The computing device can be, for example, a tablet computer, a mobile device / phone, etc. Summary of the invention

[0003] UAVs typically perform missions in which they interact exclusively with physical objects in their environment, which may be densely built and inhabited. Therefore, UAVs need to be reliably and intuitively remotely controlled in order to efficiently perform their missions and avoid collisions with obstacles and personal accidents. Methods are needed that allow for reliable and intuitive remote control of UAVs based on displaying a 3D view of the UAV's physical environment.

[0004] The object of the present invention is to provide such a UAV control method.

[0005] This paper presents a computer-implemented UAV control method for making a UAV more reliably and intuitively remotely controllable in its three-dimensional (3D) physical environment. This method is particularly important for controlling UAVs that rely on interacting with physical objects in their physical environment to complete their assigned tasks.

[0006] A typical UAV can move along multiple axes in a 3D space above the ground. These axes are usually defined by controlling the propulsion units of the UAV and the inclination of the UAV. Depending on the type of UAV, the inclination can be directly controlled by controlling the propulsion units. Then, the axis along which the UAV moves can be controlled by controlling only the propulsion units. Moving the UAV along the axis is a complex operation, which is made more complicated by the fact that the movement is remotely controlled from a remote location. Therefore, remote control of UAVs has been simplified to a level where the user can indicate a waypoint or target to the UAV. The UAV then flies autonomously to the indicated waypoint or target. Thereby, a motion generation system generates control commands based on the actual position of the UAV and the indicated waypoint or target. The control commands are received by the control system of the UAV, and the movement of the UAV is triggered based on the control commands. In flight, the motion generation system continuously updates the control system using the control commands based on the input received from the environmental sensor system of the UAV.

[0007] Navigating a UAV based on indicating a target or waypoint to which the UAV can autonomously move is an intuitive principle and is commonly used in remotely controlled UAVs. Thus, indicating a target or waypoint to the UAV is one of the key actions to be taken by the user. In order to make such UAV navigation intuitive, reliable and simple, indicating waypoints or targets to the UAV must be intuitive, reliable and simple.

[0008] A computer-implemented UAV control method is disclosed herein that allows a user to remotely control a UAV based on making a target or waypoint intuitively and reliably selectable and indicated to the UAV. Whereby, a target or waypoint can be selected in a displayed 3D view of the UAV's environment within a target selection mode, wherein the 3D view is displayed on a touch-sensitive screen.

[0009] The present invention relates to a computer-implemented UAV control method, the method comprising the following steps in a target selection mode:

[0010] ο displaying a 3D view of the UAV's environment on a touch-sensitive display,

[0011] o superimposing a movable target indicating symbol to the 3D view of the environment, the target indicating symbol being movable within the 3D view by touch input, and

[0012] While moving the target indicator

[0013] o continuously determining the position of the target indication symbol in the 3D view, and

[0014] o dynamically changing the appearance of the target indicator so as to produce the effect that the target indicator is displayed in an orientation that matches the orientation of a face located beneath the target indicator, the orientation of the face being derived from stored 3D data or from the 3D view, and

[0015] o Selecting a target based on the position of the target indicator symbol.

[0016] The computer-implemented UAV control method enables control of the UAV after being executed by a computing device. The steps of the UAV control method can be advantageously implemented within the general framework of a GUI or in other words as part of a GUI. The GUI typically presents an interface for receiving user input and is part of a UAV control device setup including a control system, an environmental sensor system, and a motion generation system, wherein these systems are at least communicatively interconnected. This typical UAV control device setup works based on the above principles.

[0017] According to the present invention, the steps included in the UAV control method can be implemented in a target selection mode. The target selection mode can be an optional control mode in addition to a plurality of other control modes. The target selection mode provides functions specifically related to the selection of a target or waypoint and provides information about the selected target or waypoint to the UAV.

[0018] The touch sensitive display is part of a computing device such as a mobile phone, tablet, laptop, etc. The computing device is communicatively connected to the UAV. The communication connection is typically a wireless connection enabling data transmission using air / vacuum as the transmission medium.

[0019] The 3D view relates to a three-dimensional representation of the physical environment around the UAV. The position of the UAV is referenced to the 3D view. Thus, the location or position of objects, targets or waypoints in the 3D view can be directly converted into their corresponding location or position in the physical environment of the UAV. The three-dimensional representation is displayed on a two-dimensional display plane of the computing device screen. The 3D view can be, for example, a camera image provided by a camera on the UAV, or can be generated, for example, based on a real-time video stream from a camera on the UAV or based on a 3D point cloud carrying three-dimensional information about the environment around the UAV, such as in the form of three-dimensional coordinates of points of the environment around the UAV. This has the following advantages: touch inputs (such as touch gestures) that are already common, such as those known in GPS or digital map navigation applications, can be used to interact with the display environment of the UAV.

[0020] In the target selection mode, a target indicator symbol is superimposed on the displayed 3D view of the environment of the UAV. The target indicator symbol indicates (as further described below) the optional target or waypoint located below by its appearance and position in the 3D view. The target indicator symbol is superimposed on the 3D view, which means that it is arranged on the display to appear above the displayed 3D view. The target indicator symbol can be moved within the 3D view by touch input. The touch input can involve, for example, a touch gesture in which the target indicator symbol is touched with a finger and dragged from a first position within the 3D view to a second position. The touch input can also involve touching and thereby activating a displayed virtual control button, thereby enabling movement of the target indicator symbol within the 3D view.

[0021] For example, a physical environment represented in a 3D view generated based on images of a real-time video stream from an onboard camera of a UAV can generally include static objects (such as natural landscape elements, buildings, technical facilities, roads, monuments, etc.), as well as moving objects (such as moving vehicles or aircraft, people, animals, etc.). By applying common feature recognition algorithms, various object types in the 3D view can be automatically recognized and identified. Based on this, the geometric features of the object are also determinable. In the case of static objects such as buildings, technical facilities or monuments, the determinable geometric features can involve, for example, contours, faces forming the object, orientations of such faces and objects, textures of faces, etc. For example, by using common feature recognition algorithms, a house including a roof and walls with windows can be automatically recognized and identified. Then, the orientation of each identified wall within the 3D view, as well as the orientation of the house, for example with respect to the roof ridge, can be determined.

[0022] Alternatively, a 3D view may be generated based on stored 3D data (e.g., based on a 3D point cloud including 3D point cloud data, wherein each point includes at least three-dimensional coordinate information). In this case, geometric features of objects in the 3D view may be derived directly from the point cloud data stored with the point cloud, or a feature recognition algorithm may be applied to the 3D point cloud data to automatically recognize and identify these geometric features.

[0023] During the movement of the target indicator symbol, its position in the 3D view is continuously determined and recorded by the computing device. Thus, continuously determining the position may mean determining the position at a given rate / frequency (e.g., a frequency in the range of 1 to 500 Hz). The determination itself is a discrete event, but is repeated continuously. This may also mean that the position is determined coherently, wherein the intervals between different determination events are not constant. The disclosed meaning of "continuous" is valid throughout the disclosure. Based on the determined position of the target indicator symbol, an object located below the target indicator symbol can be identified. Thus, the geometric features of the object located below the target indicator symbol can be obtained to dynamically change the appearance of the target indicator symbol based on the geometric features. According to the present invention, the appearance of the target indicator symbol changes dynamically so as to produce the effect of adapting its orientation to or matching the orientation of the surface temporarily covered by the target indicator symbol.

[0024] Based on the position of the target indicator symbol in the 3D view and its appearance, a target or waypoint associated with the position of the target indicator symbol can be selected. The target can be, for example, a part of an object, such as a wall of a building or a roof of a house. The target can also be, for example, a point / area of ​​an object or a point / area of ​​a part of an object, such as a point / area on the ground.

[0025] Known principles for indicating a target or waypoint to a UAV based on displaying a 3D view of the UAV's environment in a GUI rely on a UAV control method that recognizes user input and selects a target or waypoint based on the recognized user input. As a result, the user does not receive feedback regarding selectable targets or waypoints during the execution of the input. Therefore, performing user input on a 3D view (representing a true three-dimensional physical environment on a two-dimensional display plane) according to known principles typically results in multiple repetitions of user input being performed until the correct target or waypoint is selected.

[0026] It has been recognized that it is disadvantageous if the selection of a target or waypoint is presented to the user only after the input for selecting the target or waypoint is completed. In order to overcome this disadvantage, the method according to the present invention provides feedback to the user about the optional target or waypoint during the execution of the input (e.g., during the execution of a touch gesture on the touch screen of the computing device) so as to select the target or waypoint. The method disclosed herein provides feedback to the user by superimposing a movable target indicator symbol to the 3D view, continuously determining the position of the target indicator symbol, and dynamically changing the appearance of the target indicator symbol according to the geometric features of the object to which the target indicator symbol is to be superimposed. Thus, during the execution of the touch input, feedback related to identifying the optional target or waypoint is used to continuously provide updates to the user so as to select the target or waypoint. This helps the user select the correct target or waypoint on the first try, thereby improving efficiency, while improving the intuitiveness and reliability of selecting a target or waypoint and indicating the target or waypoint to the UAV.

[0027] According to an embodiment of the present invention, the computer-implemented UAV control method comprises the following steps: dynamically changing the appearance of the target indication symbol so that the target indication symbol also changes its shape and / or color based on the texture of the surface.

[0028] For example, in the case where the geometric feature is related to the texture of the surface, the appearance of the target indicator symbol may be changed based on the texture of the surface located below the target indicator symbol. Thus, the target indicator symbol may produce the effect of adapting or matching its orientation to the orientation of the surface temporarily covered by the target indicator symbol and changing its shape and / or color according to the texture of the surface located below. Thus, it may be indicated to the user whether the target indicator symbol is superimposed on, for example, a roof or a facade of a building.

[0029] According to a specific embodiment of the present invention, the computer-implemented UAV control method further comprises the following steps:

[0030] o superimposing an actuatable target confirmation symbol onto the 3D view while the target indication symbol is statically positioned above the surface,

[0031] After actuating the target confirmation symbol

[0032] ■ selecting the target based on the position of the target indicator, and

[0033] ■ Indicating that the UAV comprises a ranging module having a horizontally oriented field of view

[0034] Flying to a target position having a predetermined relative position relationship with the target, and

[0035] • Orienting the UAV towards the target so that the measurement field of view faces the target.

[0036] The control method disclosed herein appears to be particularly advantageous for application in controlling such UAVs: these UAVs rely on interaction with physical objects in their environment to complete their designated tasks. Such a UAV may have, for example, a ranging module arranged at its front, and has, for example, a task of digitizing a scene or landscape or survey object. The ranging module may be, for example, a light detection and ranging (lidar) module, and enables the measurement of coordinate information for determining the coordinates (e.g., 3D coordinates) of these points by determining the distance and direction to the points of the physical environment of the UAV. The ranging module has a measurement field of view, so that the points of the physical environment of the UAV within the measurement field of view are measurable. The ranging module is arranged so that the measurement field of view is horizontally oriented. The arrangement of the ranging module at the UAV affects the orientation of the UAV relative to, for example, an object or scene to be surveyed or digitized during flight. Using the method disclosed herein to control such a UAV allows a user to intuitively and reliably select a target or waypoint that may be related to, for example, the wall of a building that must be measured or digitized. In order to further improve the reliability of selecting the correct target or waypoint, the target indication symbol can be statically located above the face representing the optional target, while the actuable target confirmation symbol is superimposed on the 3D view. The actuable target confirmation symbol can be, for example, a check mark symbol, and the selection of the previously indicated optional target or waypoint is made effective by being touched. After selecting the target or waypoint (for example, located on the face representing the wall of the building in the 3D view), the motion generation system of the UAV will indicate that it flies to the target position, which has a predetermined position relationship with the selected target or waypoint. For example, if the selected target involves the wall of the building to be measured, the target position can be located at a position in front of the wall, for example, at a specific distance from the wall, and at a specific vertical position related to the height above the ground. Thus, the predetermined relative position relationship with the target can be predetermined based on user input, or the predefined relative position relationship with the target can be automatically predetermined, for example, by the motion generation system, based on the parameters / standards related to the task to be completed by the UAV, for example, in the case where the UAV has the task of measuring the face of the object, the predetermined relative position relationship can be predetermined based on the expected resolution of the obtained measurement result. In addition, the UAV is oriented to face the target so that the horizontally oriented measurement field of view also faces the target. The situation where the UAV is positioned and oriented in front of the target at the target location can present the starting point of the UAV flying along the surface (e.g., along the wall of a building) by following the automatically generated flight path pattern. Thus, the wall of the building can be autonomously digitized or surveyed.

[0037] According to a particular embodiment of the invention, the step of generating the effect of being displayed with an orientation matching the orientation of a face beneath the target indicator is based on projecting the target indicator onto the face and displaying the projection of the target indicator.

[0038] According to another embodiment of the present invention, the target indication symbol may be a ring symbol. Depending on the orientation of the surface to which the target indication symbol is superimposed, the appearance of the ring symbol may change from an undeformed circular ring symbol to any deformed elliptical ring symbol.

[0039] According to another specific embodiment of the present invention, the 3D data is derived from a two-dimensional image provided by a camera of the UAV in a real-time view of the environment. The UAV is usually equipped with a camera that generates a two-dimensional image of the physical environment of the UAV at a given frame rate. The position of the UAV and its orientation and / or the viewing angle of the camera are determinable and trackable. In flight, the same area of ​​the physical environment is imaged from different positions and different viewing angles. For each two-dimensional image, the actual position and actual viewing angle of the UAV / camera can be assigned. Based on this, the 3D data of the physical environment can be determined. Alternatively, an algorithm called a visual simultaneous localization and mapping (vSLAM) algorithm can be applied to generate 3D data from the physical environment of the UAV based on the two-dimensional image.

[0040] According to another specific embodiment of the present invention, the 3D data is based on a 3D point cloud, which includes at least the measured 3D coordinates of object points of the environment of the UAV. The 3D coordinates of the object points can be obtained by measuring using common geodetic survey equipment (such as a total station, theodolite, laser scanner, etc.).

[0041] According to another specific embodiment, the 3D coordinates of the object points are measured by the range finding module of the UAV. For example, in a first exploratory flight, the UAV may pass through an area that must be surveyed or digitized. During the exploratory flight, the 3D coordinates of the environmental points are measured with a coarse resolution in order to mainly capture the outlines of objects or landscape elements. Based on the captured outlines of the objects in the environment, 3D data for controlling the UAV according to the invention may be stored.

[0042] According to another embodiment of the present invention, selecting a target based on the position of the target indication symbol refers to selecting a surface located below the target indication symbol as the target. Then, the control method further includes the following steps:

[0043] o determining a flight path extending along the selected surface and at a predetermined distance from the selected surface, and

[0044] o Further instructing the UAV to fly along the flight path while the measurement field of view faces the selected surface.

[0045] Thus, the flight path may extend along the selected face, for example at a constant predetermined distance from the selected face. The distance may be predefined based on a user input. Alternatively, the distance of the flight path to the face may vary along the flight path based on predetermined parameters / criteria. The varying distance to the face may then be automatically predefined based on criteria / parameters related to the task to be performed by the UAV, for example, in the case where the UAV has a task to measure the face of an object, the varying distance may be automatically predefined (e.g., by a motion generation system) based on the desired resolution of the resulting measurements.

[0046] Another aspect of the present invention relates to another control method, the steps of which can be implemented within another optional UAV maneuvering mode.

[0047] Thus, the present invention relates to a computer-implemented UAV control method, the method comprising the following steps in a UAV maneuvering mode:

[0048] ο displaying a 3D view of the UAV's environment on a touch-sensitive display,

[0049] o displaying a UAV symbol representing the UAV and the position and orientation of the UAV in the environment in the 3D view,

[0050] o providing mobility of the UAV symbol in the 3D view in at least two of three mobility modes,

[0051] ■ a first mobility mode which provides limited mobility of the UAV symbol in the horizontal plane,

[0052] ■ a second mobility mode providing limited mobility of the UAV symbol along a vertical axis perpendicular to the horizontal plane, and

[0053] ■ a third mobility mode which provides limited mobility of the UAV symbol around the yaw axis of the UAV symbol,

[0054] in

[0055] ■ In each of the movement modes, the UAV symbol can be moved by touch input,

[0056] ■ Each of the movement modes can be individually selected by touch input, and

[0057] ■ provides switchability between the movement modes, and

[0058] οWhile moving the UAV symbol, continuously determine the position and orientation of the UAV symbol in the 3D view.

[0059] The steps of the control method implementable within the UAV maneuvering mode may advantageously be implemented within the general framework of the same GUI providing implementability of the control method in relation to the target selection mode (or in other words, as part of the same GUI).

[0060] The UAV symbol is displayed in the displayed 3D view of the UAV's environment / superimposed on the displayed 3D view of the UAV's environment. The UAV symbol represents the UAV with a position and orientation in the physical environment by its position and orientation (especially, by its shape and appearance) in the displayed 3D view. The position of the UAV is referenced to the 3D view. Thus, the positioning or position of the UAV symbol or object in the 3D view can be directly converted into their corresponding positioning or position in the physical environment of the UAV. The user can select three mobile modes, for example, by actuating the GUI or superimposing the virtual mode selection button of the 3D view via touch input, and providing limited mobility to the UAV symbol in each mobile mode. The first mobile mode provides the mobility of the UAV symbol in the 3D view limited to the horizontal plane. For example, based on the touch input, the UAV symbol can only be dragged to a position located in the horizontal plane. Based on this, the intuitive controllability of the UAV in the simulated depth plane of the two-dimensional display plane can be provided. The second mobile mode provides the mobility of the UAV symbol limited to moving along the vertical axis. The vertical axis is arranged perpendicular to the horizontal plane. For example, based on touch input, the UAV symbol can only be dragged to a position located on the vertical axis. Based on this, intuitive controllability of the vertical position of the UAV related to its height above the ground can be provided. The third movement mode provides the mobility of the UAV symbol limited to the rotation of the UAV symbol around its yaw axis. For example, based on touch input, the orientation of the UAV symbol can be changed by rotating the UAV symbol only around its yaw axis. Based on this, intuitive controllability of the orientation of the UAV in the horizontal plane can be provided. During all movements of the UAV symbol, its positioning and orientation within the 3D view will be determined and tracked. The tracked positioning / position and orientation of the UAV symbol can be directly provided to the motion generation system of the UAV so as to indicate that the UAV moves in its physical environment according to the movement of the UAV symbol in the 3D view.

[0061] The known principle of navigating the UAV in its physical environment based on displaying a 3D view of the UAV's environment on a GUI relies on a UAV control method that recognizes user input and instructs the UAV to move based on the recognized user input. After the input is completed, the actual movement of the UAV is triggered. Thus, during the execution of the input, the user does not receive feedback related to the position / positioning to which the UAV is instructed to move or the orientation to which the UAV will take. Therefore, performing user input on a 3D view representing a true three-dimensional physical environment on a two-dimensional display plane according to known principles typically results in performing multiple repeated user inputs until the UAV is moved to the desired position / positioning or oriented in the desired direction. In addition, such a principle has a high risk of collision.

[0062] It has been recognized that it is disadvantageous if no visual feedback is provided to the user during the execution of user input to move the UAV.

[0063] Thus, the method provides visual feedback to the user about the location / position and orientation of the UAV symbol during the execution of input to move the UAV (e.g., during the execution of a touch gesture on the touch screen of a computing device). Feedback is provided to the user by displaying a movable UAV symbol / superimposing the movable UAV symbol on the 3D view, by continuously determining the location of the UAV symbol and dynamically changing the appearance of the UAV symbol according to the location / position and orientation of the UAV symbol in the 3D view. Thus, during the execution of touch input to move the UAV, feedback related to the location / position and orientation of the UAV symbol in the 3D view is used to continuously provide updates to the user. This helps the user to safely navigate the UAV in the three-dimensional environment of the UAV. Thus, the risk of collision is reduced, wherein the efficiency of remote control of the UAV is improved, while the intuitiveness and reliability of remote control of the UAV are improved.

[0064] According to an alternative embodiment of the present invention, the UAV symbol is actuatable, and the method further comprises the following steps:

[0065] o After actuating the UAV symbol, another UAV symbol is displayed in the 3D view, which statically represents the UAV and the position and orientation of the UAV in the environment in the 3D view, and the UAV symbol is able to move within each of the three movement modes.

[0066] The UAV symbol can then be actuated, for example, based on a touch input (e.g., by touching the UAV symbol). The UAV symbol can then take the form of a UAV ghost symbol, and typically exhibits a degree of transparency to keep the 3D view uncluttered. The other UAV symbol remains static at its position / location in the displayed 3D view, while the UAV ghost symbol can be moved to a desired position / location and orientation in the 3D view. For example, based on the use of the UAV ghost symbol and another static UAV symbol, the user is enabled to position and orient the ghost symbol in the 3D view before the motion generation system indicates the corresponding movement to the UAV.

[0067] According to an alternative embodiment of the present invention, the method further comprises the following steps:

[0068] o superimposing an actuatable UAV position and orientation confirmation symbol onto the 3D view when the UAV symbol is not moving, and

[0069] οAfter actuating the UAV position and orientation confirmation symbol, instruct the UAV to change its position and orientation based on the determined position and orientation of the UAV symbol in the 3D view.

[0070] To further improve the reliability of locating and orienting the UAV, an actuatable UAV position and orientation confirmation symbol may be superimposed on the 3D view while the UAV symbol is statically located in the 3D view. The actuatable UAV position and orientation confirmation symbol may be, for example, a check mark-shaped symbol, and based on a touch input (e.g., by being touched), enables the motion generation system to instruct the UAV to move to a position and orient itself according to the position and orientation of the UAV symbol.

[0071] According to a specific embodiment, the method further comprises: displaying two activatable symbols in the shape of arrows, wherein upon activation of the activatable symbols, the UAV symbol is capable of moving in the first movement mode in the horizontal plane along the corresponding arrow directions.

[0072] According to another specific embodiment, the method further comprises the step of superimposing an actuatable vertical yaw axis of the UAV symbol as a vertical axis to the 3D view, wherein upon actuation of the vertical yaw axis, the UAV symbol is capable of moving along the vertical yaw axis within the second movement mode.

[0073] According to an alternative embodiment, the method further comprises the following steps: providing an actuatable front portion of the UAV symbol, wherein after the front portion is actuated, the UAV symbol is capable of rotating about its yaw axis within the third movement mode, and / or providing the UAV symbol with an actuatable ring-shaped symbol surrounding the UAV symbol, wherein after the ring-shaped symbol is actuated, the UAV symbol is capable of rotating about its yaw axis within the third movement mode, wherein the UAV symbol rotates about its yaw axis based on the actuated positioning / position on the ring-shaped symbol.

[0074] According to another embodiment, the method further comprises the following steps within the second movement mode:

[0075] o displaying a horizontal grid plane at the vertical position of the UAV symbol in the 3D view, the horizontal grid plane having a horizontal grid, the grid plane dynamically adjusting its vertical position to the vertical position of the UAV symbol, in particular, the grid plane gradually disappearing after a predetermined distance from the UAV symbol,

[0076] and / or

[0077] o dynamically highlighting in the 3D view the points where the horizontal grid meets the object,

[0078] and / or

[0079] o displaying and dynamically adjusting the object's intersecting curves intersecting the horizontal grid plane,

[0080] The highlighted points and / or the displayed through curve enable the vertical position of the UAV symbol to be indicated to the user when the UAV symbol moves along the vertical axis.

[0081] Thus, during movement of the UAV symbol, feedback is provided to the user about the vertical position of the UAV symbol. This enables simple control of the vertical position related to the height of the UAV, particularly in a 3D view provided on a two-dimensional display plane of a touch-sensitive display.

[0082] The invention also relates to a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to perform the UAV control method according to the invention.

[0083] Thus, a computer may include multiple computing subunits that cooperate in executing instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Embodiments of the present invention will now be described more fully hereinafter, by way of example only, with reference to the accompanying drawings, in which:

[0085] Figure 1A possible GUI is shown and the inventive concepts related to the method steps of the target selection mode and the UAV maneuvering mode are integrated therein;

[0086] Figure 2 A possible GUI is shown and in which the inventive concepts related to the method steps of the UAV maneuvering mode are integrated; and

[0087] Figure 3 A possible GUI is shown and in which the inventive concepts related to the method steps of the target selection mode and the UAV maneuvering mode are integrated. DETAILED DESCRIPTION

[0088] Figure 1 A mobile computing device in the form of a tablet computer is shown. The tablet computer has a touch-sensitive display 1 for receiving user input / touch input based on touching the display with one or more fingers or a stylus. A 3D view of the physical environment of the UAV Figure 2 Displayed on the two-dimensional display plane of the touch-sensitive display. 3D View Figure 2 It is a rendering of the physical environment of the UAV. The rendering is usually based on point cloud data, which includes at least three-dimensional coordinate information of points of the physical environment of the UAV. Alternatively, a 3D view can be provided by displaying images recorded by an onboard camera of the UAV.

[0089] A ring-shaped target indicator symbol 3 is superimposed on the 3D view. The target indicator symbol 3 exhibits a certain degree of transparency so that the features of the 3D view on which the target indicator symbol 3 is superimposed remain visible. The target indicator symbol 3 can be moved in the displayed 3D view based on user input. The user input is generally related to touching the target indicator symbol with a finger or a stylus and dragging it to a desired position in the 3D view. During the dragging of the target indicator symbol to the desired position, its appearance will be dynamically adjusted so as to produce an effect that the orientation of the target indicator symbol matches the orientation of the face (e.g., face 4) temporarily located below. After the target indicator symbol 3 is placed at the desired position in the 3D view, a target confirmation symbol 5 can be superimposed on the 3D view, for example, near the target indicator symbol, such as Figure 1 The user is then offered the possibility to confirm or reject the target selection by actuating a target confirmation symbol or, for example, a target rejection symbol. Figure 1In the case shown, the target selection is confirmed based on touch input, for example by touching the target confirmation symbol with a finger or a stylus, and the wall 4 of the house is selected as the target. The target selection is then indicated to the motion generation system of the UAV. The motion generation system (which is communicatively connected to the tablet computer, the control system of the UAV, and the environmental sensor system of the UAV) can then automatically determine the target position based on predetermined criteria. The determined target position is then indicated to the control system of the UAV by the motion generation system. The control system (which receives and executes commands from the motion generation system) causes the UAV to move to the determined target position. According to Figure 1 In the illustrated scenario, the target position is determined to be in front of the house wall 4 and at a predetermined distance from the house wall 4 and above the ground. The UAV equipped with the ranging module (e.g., at its front) is moved to the target position and oriented toward the house wall 4 so that the measurement field of view faces the house wall 4. This scenario can then become the starting point for performing a survey / scanning / digitization task on the house wall 4. A flight path along the house wall 4 can then be automatically determined based on the 3D data used to render the scene. The UAV (and the measurement field of view) facing the house wall 4 can autonomously measure the house wall 4 by following the determined flight path.

[0090] UAV symbol 6 is superimposed on Figure 1 3D view of the UAV. The UAV symbol 6 represents the UAV to be controlled by its positioning / location and appearance. The UAV symbol also indicates the front part including the ranging module 10. The position of the UAV symbol 6 within the 3D view represents the actual position of the UAV in its physical environment in a true-to-scale manner. In addition, the position / location in the 3D view is referenced to their actual corresponding positions in the physical environment. Figure 1Features related to the UAV manipulation mode are further illustrated, which make the control of the UAV intuitive and simple. The user can select and switch between one of three different movement modes. In the first movement mode, the user can only move the UAV symbol within the horizontal plane 7 where the UAV symbol is located based on touch input. In the second movement mode, the user can only move the UAV symbol similarly along the vertical axis 8, which can be the yaw axis 9 of the UAV symbol. In the third movement mode, the user can only rotate the UAV symbol around the yaw axis 9 of the UAV symbol. The positioning / position and orientation of the UAV are continuously determined and tracked, and can be processed by the motion generation system to instruct the UAV to move according to the movement of the UAV symbol through the control system. The user can manipulate the UAV symbol by changing the position / position and orientation of the UAV symbol. In addition, once the UAV symbol is statically located in the 3D view, an actuatable position and orientation confirmation symbol can be superimposed on the 3D view, for example, near the UAV symbol. By actuating the position and orientation confirmation symbol, the position and orientation of the UAV symbol are transmitted to the motion generation system, instructing the UAV to fly to the corresponding position in its physical environment and orient itself according to the orientation of the UAV symbol.

[0091] Figure 2 A 3D view of the physical environment with a UAV is shown. Figure 2 3D view of the physical environment Figure 2 Displayed on the two-dimensional display plane of the touch-sensitive display 1. 3D view Figure 2 It is a three-dimensional representation of the physical environment of the UAV and is based on point cloud data, which includes at least three-dimensional coordinate information of points of the physical environment of the UAV.

[0092] UAV symbol 6 is displayed on Figure 2 3D view. The UAV symbol 6 represents the UAV to be controlled by its appearance. The UAV symbol 6 also indicates the front part including, for example, a ranging module. In the 3D view, another UAV symbol 6' is displayed. This other UAV symbol 6' within the 3D view represents the actual position of the UAV in its physical environment in a true-to-scale manner. In addition, the position / positioning in the 3D view is with reference to their actual corresponding positions in the physical environment.

[0093] Figure 2Features related to a second movement mode are further illustrated, in which the user can only move the UAV symbol 6 along a vertical axis 8, which can be a yaw axis 9 of the UAV symbol 6. The location / position and orientation of the UAV 6 are continuously determined and tracked, and can be processed by the motion generation system to instruct the UAV to move in accordance with the movement of the UAV symbol 6 through the control system. The user can manipulate the UAV symbol by changing the location / position of the UAV symbol along the axes 8, 9. In addition, once the UAV symbol is statically located in the 3D view, an actuatable position confirmation symbol can be superimposed on the 3D view, for example, near the UAV symbol 6. By actuating the position confirmation symbol, the position of the UAV symbol 6 on the axes 8, 9 is transmitted to the motion generation system, instructing the UAV to fly to the corresponding position in its physical environment.

[0094] exist Figure 2 In the 3D view of FIG. 6 , the first movement mode (which provides the mobility of the UAV symbol 6 limited to the horizontal plane 7) is indicated as not activated. For example, based on the touch input, the UAV symbol 6 can only be dragged to a position located in the horizontal plane 7. Based on this, intuitive controllability of the UAV in the simulated depth plane of the two-dimensional display plane can be provided.

[0095] exist Figure 2 In the 3D view of FIG. 1 , features related to the third movement mode are indicated as not activated. An actuatable ring symbol 12 surrounds another UAV symbol 6 ′, and upon its actuation, the UAV symbol 6 can rotate about its yaw axis 9 in the third movement mode, wherein the UAV symbol 6 rotates about its yaw axis based on the actuated position / location on the ring symbol 12.

[0096] exist Figure 2 In the 3D view of FIG. 6 , a horizontal grid plane 13 at the vertical position of another UAV symbol 6 ′ is shown, and the horizontal grid plane 13 has a horizontal grid 14. Figure 2 In the example shown, the grid plane 13 indicates the vertical position of another UAV symbol 6'. Advantageously, the grid plane 13 adjusts its vertical position to the vertical position of the UAV symbol 6, thereby making it possible to indicate the vertical position of the UAV symbol 6 to the user. In addition, the grid plane 13 gradually disappears after a predetermined distance from the other UAV symbol 6'.

[0097] exist Figure 2In the 3D view of FIG. 6 , a dynamically highlighted point 15 is highlighted at which the horizontal grid 14 meets the object (here the facade of the building). In addition, a through curve 16 is displayed that runs through the horizontal grid plane 13 of the object (here the building). The highlighted point and / or the displayed through curve make the vertical position of another UAV symbol 6 'indicable. Advantageously, the highlighted point and / or the displayed through curve moves along the vertical axis with the UAV symbol 6, thereby making it possible to indicate the vertical position of the UAV symbol 6 to the user when the UAV symbol 6 moves along the vertical axis.

[0098] The circular target indicator symbol 3 is superimposed on Figure 3 During the dragging of the target indication symbol 3 to the desired position, its appearance is dynamically adjusted so that the effect is produced that the orientation of the target indication symbol 3 matches the orientation of the surface (e.g., surface 4) temporarily located below. The target confirmation symbol 5 is superimposed on the 3D view, for example, near the target indication symbol.

Claims

1. A computer-implemented UAV control method, the computer-implemented UAV control method comprising the following steps in a target selection mode: Displaying a 3D view (2) of an environment of a UAV on a touch-sensitive display (1), the UAV having a ranging module implemented as a lidar module and configured to survey a target with the ranging module; superimposing a movable survey target indication symbol (3) onto the 3D view of the environment, wherein the survey target indication symbol can be moved in the 3D view by touch input; While moving the survey target indicator symbol: Continuously determining the position of the survey target indicator symbol in the 3D view; as well as dynamically changing the appearance of the survey target indicator symbol so as to produce the effect that the survey target indicator symbol is displayed in an orientation that matches the orientation of a face (4) located below the survey target indicator symbol, the orientation of the face being derived from the stored 3D data or from the 3D view; as well as The target to be surveyed is selected based on the position of the survey target indicator symbol.

2. The computer-implemented UAV control method of claim 1, wherein: The step of dynamically changing the appearance of the survey target indicator symbol causes the survey target indicator symbol to change its shape and / or color also based on the texture of the surface.

3. The computer-implemented UAV control method of claim 1, wherein the computer-implemented UAV control method comprises the following steps: While the survey target indication symbol is statically located above one side, an actuatable target confirmation symbol (5) is superimposed on the 3D view; After actuating the target confirmation symbol: selecting the target based on the position of the survey target indicator symbol; as well as Instructing the UAV to perform the following operations, wherein the ranging module has a horizontally oriented measurement field of view: Flying to a target position having a predetermined relative position relationship with the target; as well as The UAV is oriented toward the target so that the measurement field of view faces the target.

4. The computer-implemented UAV control method of claim 1 or 2, wherein: The step of producing the effect that the survey target indication symbol is displayed in an orientation matching the orientation of a surface (4) located below the survey target indication symbol is based on projecting the survey target indication symbol onto the surface and displaying the projection of the survey target indication symbol.

5. The computer-implemented UAV control method of claim 1 or 2, wherein: The survey target indication symbol (3) is a ring symbol.

6. The computer-implemented UAV control method of claim 1 or 2, wherein: The 3D data is derived from two-dimensional images provided by the UAV's camera in a real-time view of the environment.

7. The computer-implemented UAV control method of claim 3, wherein: The 3D data are based on a 3D point cloud comprising at least measured 3D coordinates of object points of the environment of the UAV.

8. The computer-implemented UAV control method of claim 7, wherein: The 3D coordinates of the object point are measured by the ranging module of the UAV.

9. The computer-implemented UAV control method of claim 3, wherein: Selecting a target based on the position of the survey target indication symbol (3) means selecting the surface (4) located below the survey target indication symbol as the target, and The control method further comprises the following steps: determining a flight path extending along the selected surface and a predetermined distance from the selected surface; and The UAV is further instructed to fly along the flight path while the measurement field of view faces the selected surface.

10. A computer program product, comprising instructions, which, when the program is executed by a computer, cause the computer to perform the UAV control method according to any one of claims 1 to 9.

Images