Haptic feedback-based drone control methods, terminal devices, and systems

Abstract

This application relates to the field of unmanned aerial vehicle (UAV) flight control technology, and in particular to a UAV control method, terminal device, UAV control system, and storage medium based on haptic feedback. When a user's finger touches the effective touch area, the electronic device can detect the touch point and generate flight control commands for the UAV based on the positional changes of the touch point within the effective touch area. These commands are then sent to the UAV to control its flight. During flight, the UAV reports its flight status data to the terminal device, which can then determine the waveform characteristic parameters corresponding to the flight status data and control the vibration feedback device to drive the touchscreen to vibrate based on these parameters. Therefore, the terminal device can provide flight status data of the UAV through touchscreen vibration feedback during user operation of the virtual remote sensing system, which helps improve the user's control feel and precision.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) flight control technology, and in particular to a UAV control method, terminal device, UAV control system, and storage medium based on haptic feedback. Background Technology

[0002] Unmanned aerial vehicles (UAVs) are unmanned aircraft controlled by radio remote control equipment and their own program control devices. They can be used in military and civilian fields, such as aerial photography, agriculture, and surveying.

[0003] Flight control of drones is typically achieved by a user operating a handheld remote control device. This device can be a traditional physical remote controller, or a terminal device such as a smartphone or tablet. The terminal device can install and run a drone remote control application and display a virtual joystick on a touchscreen. Users can then issue corresponding control commands to the drone by manipulating the virtual joystick.

[0004] When a user operates a physical remote control, they can perceive the damping sensation generated by the joystick's movement and the return force after deviating from the center position in real time through tactile feedback, thus obtaining a clear control feel and positional feedback of the joystick. However, a virtual joystick is merely a dynamic image displayed on a touchscreen, lacking a physical structure and unable to provide any physical feedback to the user, thus affecting the user's control feel and precision. Summary of the Invention

[0005] This application provides a haptic feedback-based drone control method, terminal device, drone control system, and storage medium. It can provide drone flight status data through touchscreen vibration feedback during user operation of virtual remote sensing, which helps improve the user's control feel and control accuracy of virtual remote sensing.

[0006] To address the aforementioned technical problems, the embodiments of this application provide the following technical solutions: In a first aspect of this application, a method for controlling a drone based on haptic feedback is provided. The method includes: displaying a control interface of the drone via a touchscreen, the control interface including a virtual joystick and an effective touch area of ​​the virtual joystick; if a touch point is detected in the effective touch area, adjusting the position of the virtual joystick according to the position change of the touch point within the effective touch area; generating flight control commands for the drone based on the position change of the virtual joystick, and sending the flight control commands to the drone; acquiring flight status data of the drone; determining waveform feature parameters corresponding to the flight status data, and controlling a vibration feedback device to drive the touchscreen to vibrate based on the waveform feature parameters corresponding to the flight status data; wherein the waveform feature parameters include frequency, amplitude, and envelope.

[0007] In the embodiments of this application, the electronic device can display the drone's control interface via a touchscreen. This control interface includes a virtual joystick and an effective touch area for the virtual joystick. When a user's finger touches the effective touch area, the electronic device can detect the touch point and generate flight control commands for the drone based on the positional changes of the touch point within the effective touch area. These commands are then sent to the drone to control its flight. During flight, the drone reports its flight status data to the terminal device. The terminal device can then determine the waveform characteristic parameters corresponding to the flight status data and control the vibration feedback device to drive the touchscreen to vibrate based on these parameters. Therefore, the terminal device can provide vibration feedback of the drone's flight status data through the touchscreen during the user's virtual remote sensing operation, which helps improve the user's control feel and precision.

[0008] In some embodiments, the flight status data includes wind resistance parameters, which include wind speed and wind direction in the flight environment of the UAV; determining the waveform feature parameters corresponding to the flight status data and controlling the vibration feedback device to drive the touch screen to vibrate based on the waveform feature parameters corresponding to the flight status data includes: determining the waveform feature parameters corresponding to the wind resistance parameters, wherein the wind speed and the amplitude corresponding to the wind resistance parameters are positively correlated; determining the vibration direction of the vibration feedback device based on the wind direction, wherein the vibration direction is opposite to the control direction of the virtual remote sensing; and controlling the vibration feedback device to drive the touch screen to vibrate based on the waveform feature parameters corresponding to the wind resistance parameters and the vibration direction.

[0009] In some embodiments, the flight status data further includes the distance between the UAV and the obstacle; determining the waveform feature parameters corresponding to the flight status data and controlling the vibration feedback device to drive the touch screen to vibrate based on the waveform feature parameters corresponding to the flight status data includes: determining the waveform feature parameters corresponding to the distance; controlling the vibration feedback device to drive the target area on the touch screen to vibrate based on the waveform feature parameters corresponding to the distance; wherein, the target area is the area on the touch screen corresponding to the orientation of the obstacle.

[0010] In some embodiments, the flight status data further includes the flight speed of the UAV; determining the waveform feature parameters corresponding to the flight status data and controlling the vibration feedback device to drive the touch screen to vibrate based on the waveform feature parameters corresponding to the flight status data includes: determining the waveform feature parameters corresponding to the flight speed and controlling the vibration feedback device to drive the touch screen to vibrate based on the waveform feature parameters corresponding to the flight speed; wherein, the flight speed is positively correlated with the frequency corresponding to the flight speed.

[0011] In some embodiments, the flight status data further includes the collision time between the UAV and the obstacle; determining the waveform feature parameters corresponding to the flight status data and controlling the vibration feedback device to drive the touch screen to vibrate based on the waveform feature parameters corresponding to the flight status data includes: if it is determined that the collision time is less than a preset safe time threshold, generating waveform feature parameters corresponding to the emergency situation, and controlling the vibration feedback device to vibrate according to the waveform feature parameters corresponding to the emergency situation.

[0012] In some embodiments, the method further includes: if it is determined that the collision time is less than a preset safe time threshold, then triggering an emergency stop mechanism; under the emergency stop mechanism, if a touch point acting on any position of the touch screen is detected, and it is determined that the touch operation corresponding to the touch point is a hard press operation, then issuing an emergency hovering command or a reverse braking command to the drone; wherein, the hard press operation is used to indicate a touch operation in which the pressure applied to the touch screen is greater than a preset pressure threshold; or, the hard press operation is used to indicate a touch operation in which the area of ​​the touch point is greater than a preset area threshold.

[0013] In some embodiments, the flight status data further includes the battery level of the drone; the step of determining the waveform feature parameters corresponding to the flight status data and controlling the vibration feedback device to drive the touch screen to vibrate based on the waveform feature parameters corresponding to the flight status data further includes: determining the waveform feature parameters corresponding to the battery level of the drone and controlling the vibration feedback device to drive the touch screen to vibrate based on the waveform feature parameters corresponding to the battery level.

[0014] In some embodiments, the virtual joystick moves within a range defined by the travel boundary of the virtual joystick; the method further includes: during the process of adjusting the position of the virtual joystick according to the position change of the touch point in the effective touch area, if it is determined that the virtual joystick has reached the travel boundary, then determining the waveform feature parameters corresponding to the travel boundary; and controlling the vibration feedback device to drive the touch screen to vibrate based on the waveform feature parameters corresponding to the travel boundary.

[0015] In some embodiments, the method further includes: during the process of adjusting the position of the virtual joystick according to the position change of the touch point within the effective touch area, determining the offset distance between the current position of the virtual joystick and the initial position of the virtual joystick; determining the waveform feature parameters corresponding to the offset distance; and controlling the vibration feedback device to drive the touch screen to vibrate based on the waveform feature parameters corresponding to the offset distance; wherein the offset distance is positively correlated with the amplitude corresponding to the offset distance.

[0016] In some embodiments, the control interface further includes an ineffective touch area, which is an area in the control interface other than the effective touch area; the method further includes: if it is detected that the touch point has slid from the effective touch area to outside the ineffective touch area, then determining the waveform feature parameters corresponding to the ineffective touch area; and controlling the vibration feedback device to drive the touch screen to vibrate based on the waveform feature parameters corresponding to the ineffective touch area.

[0017] In a second aspect of this application, a terminal device is also provided, the terminal device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0018] In a third aspect of this application, a drone control system is also provided, characterized in that the system comprises: a drone; and the terminal device described in the second aspect, wherein the drone and the terminal device are communicatively connected.

[0019] In a fourth aspect of this application, a non-volatile computer-readable storage medium is also provided, the computer-readable storage medium storing computer-executable instructions that, when executed, enable the execution of the method described in the first aspect.

[0020] It should be understood that the description in the Summary of the Invention section is not intended to limit the key or essential features of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0022] Figure 1 This is a schematic diagram of the structure of an unmanned aerial vehicle (UAV) flight control system provided in some embodiments of this application; Figure 2 These are schematic diagrams of the structure of terminal devices provided in some embodiments of this application; Figure 3 This is a schematic diagram of the control interface of a drone provided in some embodiments of this application; Figure 4This is a flowchart illustrating a haptic feedback-based drone control method provided in some embodiments of this application; Figure 5 These are schematic diagrams of the control interface of a drone provided in other embodiments of this application; Figure 6 This is a schematic diagram of the structure of a controller for performing a haptic feedback-based drone control method, provided in some embodiments of this application. Detailed Implementation

[0023] The principles and spirit of this disclosure will be described below with reference to several exemplary embodiments illustrated in the accompanying drawings. It should be understood that these specific embodiments are described merely to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way. In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art.

[0024] As used herein, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "an embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects and are used only to distinguish the objects referred to, without implying a particular spatial order, temporal order, order of importance, etc., of the objects referred to.

[0025] For example, Figure 1 The present application provides structural schematic diagrams of some embodiments of the unmanned aerial vehicle (UAV) control system, such as... Figure 1 As shown, system 100 includes terminal device 10 and drone 20 that is communicatively connected to terminal device 10.

[0026] Terminal device 10 and drone 20 can be connected directly or indirectly via wired or wireless communication, and this application does not impose any limitations on this connection. Terminal device 10 can be a smartphone, tablet, laptop, or smartwatch, etc., but is not limited to these. Drone 20 can be any suitable type of drone. For example, unmanned fixed-wing aircraft, unmanned vertical take-off and landing aircraft, unmanned airships, unmanned helicopters, unmanned multi-rotor aircraft, or unmanned paragliders, etc.

[0027] For example, Figure 2 The following are schematic diagrams illustrating the hardware structure of a terminal device 10 for implementing a haptic feedback-based drone control method, as provided in some embodiments of this application. Figure 2As shown, the terminal device 10 includes a controller 11, a touch screen 12 and a vibration feedback device 13 that are communicatively connected to the controller 11; wherein, the vibration feedback device 13 may specifically be a wideband linear motor, such as an X-axis linear motor.

[0028] The terminal device 10 has a drone remote control application installed. For example... Figure 2 As shown, when the terminal device 10 is running the drone remote control application, it displays the control interface 13 of the drone 20 via the touch screen 12. The control interface 13 of the drone 20 includes a virtual joystick 131 and an effective touch area 132 of the virtual joystick 131, with the virtual joystick 131 located within the effective touch area 132.

[0029] The virtual joystick 13 is a control used to control the drone 20 to fly in a real environment. Users can control the virtual joystick 13 to move within the joystick's active area 133 by performing touch operations (such as swiping) within the effective touch area 132. The terminal device 10 responds to the user's touch operation within the effective touch area 132, adjusts the position of the virtual joystick 131 according to the change in the position of the touch point within the effective touch area 132, and generates flight control commands for the drone 20 based on the position of the virtual joystick 131. The terminal device 10 then sends the flight control commands to the drone 20 to control its flight.

[0030] Optionally, in this embodiment, the virtual joystick 131 can be circular. In some other embodiments, the virtual joystick 131 can also be triangular, square, hexagonal, octagonal, etc., or other irregular shapes. This embodiment does not limit this. The virtual joystick 131 moves within the joystick activity area 133. Optionally, the shape of the joystick activity area 133 can be the same as or different from the shape of the virtual joystick 131. For example, the virtual joystick 131 is circular, and the joystick activity area 133 is also circular, and the two are concentric circles; or, for example, the virtual joystick 131 is hexagonal, and the joystick activity area 133 is octagonal, and the centers of the two coincide.

[0031] Figure 2 The following description uses a control interface 13 that includes a virtual joystick 131 as an example. In other embodiments, the control interface 13 may also include multiple virtual joysticks 131; and each virtual joystick 131 corresponds to an effective touch area 132 and a joystick activity area 133.

[0032] Those skilled in the art will understand that Figure 2The structure shown does not constitute a limitation on the terminal device 10. The terminal device 10 may include more or fewer components than shown, or combine some components, or split some components, or have different component arrangements.

[0033] During the operation of a physical remote control, the user can perceive the damping sensation generated by the joystick movement and the return force after deviating from the center position in real time through tactile feedback. However, when operating a virtual joystick, the user's finger slides on the touchscreen (glass screen). Because the virtual joystick lacks tactile feedback and physical positioning, the operator needs to continuously look at the screen to confirm the control status, causing a separation between the line of sight and the actual flight environment, significantly increasing the control risk and easily leading to flight accidents. Based on this, this application provides a tactile feedback-based drone control method, terminal device, drone control system, and storage medium. In this method, the terminal device can provide feedback on the drone's flight status data through touchscreen vibration during the user's operation of the virtual remote control, which helps improve the user's control feel and accuracy. To facilitate the reader's understanding of this application, specific embodiments are described below.

[0034] For example, Figure 4 The present application provides flowcharts of a haptic feedback-based drone control method according to some embodiments, which is applied to a terminal device, such as... Figure 2 Terminal device 10, such as Figure 4 As shown, the method includes: Step S41: The controller displays the drone's control interface via a touchscreen. The control interface includes a virtual joystick and the effective touch area of ​​the virtual joystick.

[0035] Step S42: If the controller detects a touch point in the effective touch area, it adjusts the position of the virtual joystick according to the position change of the touch point in the effective touch area.

[0036] Step S43: The controller generates flight control commands for the UAV based on the position changes of the virtual joystick and sends the flight control commands to the UAV.

[0037] Please see Figure 5 In some embodiments, in the initial state, the virtual joystick 10 is located at the center O of the joystick activity area 133. That is, the center O of the joystick activity area 133 is the initial position of the virtual joystick 131. The joystick activity area refers to the range of the virtual joystick that is allowed to move on the interface. If the controller detects that the touch point moves within the joystick control area, it adjusts the position of the virtual joystick within the joystick activity area according to the change in the position of the touch point. When the touch point moves within the joystick control area, the position of the virtual joystick changes synchronously with the position of the touch point in real time, and the effective touch area includes and is larger than the joystick activity area.

[0038] In some embodiments, when the touch point 30 moves within the joystick activity area 133, the position of the touch point 30 is the same as the position of the virtual joystick 10; when the touch point 30 moves outside the effective touch area 132 of the joystick activity area 133, the virtual joystick 10 moves on the edge of the joystick activity area 133.

[0039] In some embodiments, the travel boundary 1331 is used to represent the edge of the joystick activity area 133. The joystick activity area 133 is also the range defined by the travel boundary. The virtual joystick moves within the range defined by the travel boundary; the method further includes: during the process of the controller adjusting the position of the virtual joystick according to the position change of the touch point within the effective touch area, if the controller determines that the virtual joystick has reached the travel boundary, the controller determines the waveform characteristic parameters corresponding to the travel boundary; the controller controls the vibration feedback device to drive the touch screen to vibrate based on the waveform characteristic parameters corresponding to the travel boundary. In this embodiment, if the controller detects that a finger pushes the virtual joystick to the travel boundary, the controller controls the vibration feedback device to output a single 200Hz high-frequency pulse lasting 15ms to simulate the short impact sensation of a real joystick hitting the limit edge.

[0040] In some embodiments, the method further includes: during the process of the controller adjusting the position of the virtual joystick according to the position change of the touch point within the effective touch area, the controller determines the offset distance between the current position of the virtual joystick and the initial position of the virtual joystick; the controller determines the waveform characteristic parameters corresponding to the offset distance; and controls the vibration feedback device to drive the touch screen to vibrate based on the waveform characteristic parameters corresponding to the offset distance; wherein the offset distance is positively correlated with the amplitude corresponding to the offset distance. In this embodiment, the controller can monitor the offset of the current position of the virtual joystick relative to the center point (initial position) of the virtual joystick in real time. After the finger deviates from the center point, the controller gradually increases the continuous micro-vibration of the motor as the offset distance increases, simulating the progressive resistance tactile sensation of a physical joystick spring being stretched.

[0041] In some embodiments, the control interface further includes an inactive touch area, which is an area in the control interface other than the active touch area. The method further includes: if the controller detects that a touch point has slid from the active touch area to outside the inactive touch area, the controller determines the waveform characteristic parameters corresponding to the inactive touch area; and controls a vibration feedback device to drive the touchscreen to vibrate based on the waveform characteristic parameters corresponding to the inactive touch area. For example, if the controller detects that a user's finger has accidentally slid out of the active touch area, it controls the vibration feedback device to trigger two consecutive short vibrations to remind the user to reposition their finger.

[0042] In some embodiments, after the drone remote control application is launched, the virtual joystick enters a standby state. If the controller detects that the user's finger touches the effective touch area, the virtual joystick makes contact with the finger. When the finger is within ±10% of the joystick center (initial position), the virtual joystick maintains its initial position. When the finger slides away from the joystick center, the virtual joystick enters a pushed state. The controller outputs corresponding centering vibrations in segments according to the offset distance (or offset amount) of the virtual joystick relative to the joystick center. For example, when the offset amount is 0%~33%, the controller controls the vibration feedback device to output a 10Hz low-amplitude weak centering vibration; when the offset amount is 33%~66%, the controller controls the vibration feedback device to output a 20Hz medium-amplitude medium centering vibration; when the offset amount is 66%~100%, the controller controls the vibration feedback device to output a 30Hz high-amplitude strong centering vibration. When the virtual joystick is pushed to the travel limit, the controller controls the vibration feedback device to output a single high-frequency pulse of 200Hz for 15ms. If the controller detects that the finger has slipped out of the effective area of ​​the joystick, the controller controls the vibration feedback device to output two sets of short vibrations of 150Hz for 20ms each. If the controller detects that the user's finger has been lifted, the controller controls the virtual joystick to automatically return to the initial position and switch back to standby mode.

[0043] Step S44: The controller acquires the flight status data reported by the UAV. Step S45: The controller determines the waveform characteristic parameters corresponding to the flight status data, and controls the vibration feedback device to drive the touch screen to vibrate based on the waveform characteristic parameters corresponding to the flight status data.

[0044] In some embodiments, flight status data includes wind resistance parameters, which include wind speed and wind direction in the flight environment of the UAV. Step S45 specifically includes: the controller determining the waveform characteristic parameters corresponding to the wind resistance parameters, wherein the amplitude corresponding to the wind speed and wind resistance parameters is positively correlated; the controller determining the vibration direction of the vibration feedback device based on the wind direction, the vibration direction being opposite to the control direction of the virtual remote sensing; the controller controlling the vibration feedback device to drive the touch screen to vibrate based on the waveform characteristic parameters corresponding to the wind resistance parameters and the vibration direction. In this embodiment, the controller can acquire the wind resistance parameters collected by the UAV's wind speed sensor. When the UAV is subjected to crosswinds or headwinds, and the user moves the virtual joystick in the corresponding direction, the controller controls the vibration feedback device to output a reverse low-frequency vibration of 20Hz~50Hz. The controller synchronously increases the vibration amplitude as the wind speed increases, simulating the physical resistance tactile effect of pushing the joystick against the wind.

[0045] In some embodiments, the flight status data also includes the distance between the UAV and the obstacle; step S45 further includes: determining the waveform feature parameters corresponding to the distance; controlling the vibration feedback device to drive the target area on the touch screen to vibrate based on the waveform feature parameters corresponding to the distance; wherein, the target area is the area on the touch screen corresponding to the orientation of the obstacle. The controller receives obstacle distance and orientation data collected by the obstacle avoidance radar, and adjusts the heartbeat pulse vibration frequency in the range of 1Hz~5Hz according to the preset distance mapping rules based on the distance of the obstacle (a vibration frequency of 1Hz corresponds to a distance of 5m between obstacles, and a vibration frequency of 5Hz corresponds to a distance of 1m), and drives the left and right motors to vibrate in zones in combination with obstacle orientation matching, relying on differentiated heartbeat pulse feedback to simulate the distance and orientation information of the obstacle.

[0046] In some embodiments, the flight status data also includes the drone's flight speed; for example, the flight speed may specifically be the drone's flight speed during acceleration. Step S45 further includes: determining the waveform characteristic parameters corresponding to the drone's flight speed, and controlling the vibration feedback device to drive the touchscreen to vibrate based on the waveform characteristic parameters corresponding to the flight speed; wherein, the flight speed is positively correlated with the frequency corresponding to the flight speed. In this embodiment, during the drone's acceleration, the controller synchronously adjusts the vibration motor to output continuous granular pulse tactile signals as the drone's flight speed increases. The higher the flight speed, the denser the pulse arrangement and the higher the vibration frequency. Relying on the tactile feedback with differentiated density, the user can intuitively identify the real-time flight speed level without observing the screen. For example, the controller can receive the flight speed and acceleration data collected by the drone's IMU attitude sensor reported by the drone, and according to the preset speed mapping rules, match the density of the granular vibration according to the flight speed level. During the aircraft's acceleration, the density of the granular vibration is synchronously increased, and the motor is controlled to output high-density granular vibration of 80Hz~150Hz, intuitively simulating the real-time flight speed state through tactile changes.

[0047] In some embodiments, the flight status data also includes the collision time between the UAV and the obstacle; step S45 further includes: if it is determined that the collision time is less than a preset safe time threshold, then generating waveform feature parameters corresponding to the emergency situation, and controlling the vibration feedback device to vibrate according to the waveform feature parameters corresponding to the emergency situation.

[0048] In some embodiments, the method further includes: if the collision time is determined to be less than a preset safe time threshold, then triggering an emergency stop mechanism; under the emergency stop mechanism, if a touch point acting on any position of the touch screen is detected, and it is determined that the touch operation corresponding to the touch point is a hard press operation, then issuing an emergency hovering command or a reverse braking command to the drone; wherein, the hard press operation is used to indicate a touch operation in which the pressure applied to the touch screen is greater than a preset pressure threshold; or, the hard press operation is used to indicate a touch operation in which the area of ​​the touch point is greater than a preset area threshold.

[0049] Specifically, the controller can acquire the drone's current speed and the current distance between the drone and obstacles. Based on the current speed and distance, the controller calculates the current time to collision (TTC). If the current TTC is less than a preset time threshold (e.g., 2 seconds), the controller determines the waveform characteristic parameters corresponding to the emergency situation to generate a preset emergency haptic feedback signal and trigger an emergency stop mechanism. Based on the emergency haptic feedback signal, the controller controls the vibration feedback device to drive the touchscreen to generate sweep frequency vibration. Under the emergency stop mechanism, if a user's heavy press operation on any touch area of ​​the touchscreen is detected, an emergency hover command or a reverse braking command is sent to the drone to instruct the drone to hover or brake in reverse. In some embodiments, the controller divides the flight environment into three safe zones based on the TTC value: a safe zone, a warning zone, and a danger zone. Different safe zones correspond to different waveform characteristic parameters. For example, the safe zone can be TTC > 5s: if the controller determines that the TTC is greater than 5 seconds, the controller determines that the flight environment is in a safe condition and maintains the UAV's normal flight state. The warning zone can be 2s < TTC ≤ 5s: if the controller determines that the TTC is greater than 2 seconds and less than or equal to 5 seconds, the controller enters the collision avoidance warning control stage, drives the vibration motor to output a pulse tactile signal simulating a heartbeat, and the pulse frequency gradually increases from 1Hz to 3Hz, relying on the gradual change in vibration density to achieve a step-by-step warning of danger. The danger zone can be TTC ≤ 2s: if the controller determines that the TTC is less than or equal to 2 seconds, the controller determines that it is a close-range high-risk collision condition, immediately generates a preset emergency tactile feedback signal, and triggers an emergency stop mechanism.

[0050] Specifically, frequency sweep vibration is also known as sinusoidal frequency sweep vibration. Frequency sweep vibration is a sinusoidal periodic vibration whose frequency changes continuously over time within a preset frequency range, and whose amplitude remains constant. For example, the preset frequency range could be 50Hz to 300Hz. The amplitude of the frequency sweep vibration can be the maximum amplitude that the vibration feedback device can output. In touchscreens, frequency sweep vibration is used to create a tingling sensation similar to an "electric shock" or "intense friction" on the finger in contact with the touchscreen, directly stimulating the operator's instinctive reaction.

[0051] In some embodiments, under the emergency stop mechanism, if the controller determines that no re-press operation is detected within a preset emergency time (e.g., 1 second) after the emergency stop mechanism is activated, it issues an emergency hover command to the drone to instruct the drone to hover urgently. After receiving the emergency hover command, the drone executes emergency stop control, completes the flight speed reduction to zero, and locks its body attitude to enter a stationary hovering state. At this time, the controller displays a text prompt message indicating that the emergency stop was successful through the drone's control interface; on the other hand, the controller's vibration feedback device drives the touchscreen to output three gentle pulse vibrations, thereby informing the user that the emergency stop operation has taken effect through tactile feedback.

[0052] In some embodiments, the flight status data also includes the drone's battery level; step S45 further includes: the controller determines the waveform characteristic parameters corresponding to the drone's battery level, and controls the vibration feedback device to drive the touch screen to vibrate based on the waveform characteristic parameters corresponding to the battery level.

[0053] Specifically, if the controller determines that the drone's battery level is below a preset first battery threshold (e.g., 10%), it controls the vibration feedback device to drive the touchscreen to generate a three-hit pulse vibration with a vibration speed of the first speed, based on the waveform characteristic parameters corresponding to the battery level. If the controller determines that the drone's battery level is not lower than the preset first battery threshold (e.g., 10%) and is lower than a preset second battery threshold (e.g., 20%), it controls the vibration feedback device to drive the touchscreen to generate a double-tap pulse vibration with a vibration speed of the second speed, based on the waveform characteristic parameters corresponding to the battery level. The first speed is greater than the second speed.

[0054] This application embodiment uses multi-mode haptic feedback to transmit key flight parameters such as ambient wind speed and obstacle distance to the operator, reducing reliance on screen visual viewing during emergency control and helping to avoid flight hazards caused by the viewpoint being out of the flight environment, thus improving flight safety performance. At the same time, relying on the layered simulation of damped tactile sensation, limit impact tactile sensation, and speed particle tactile sensation generated by the haptic feedback device, the virtual joystick has the control feel of a physical remote control joystick, optimizing the accuracy of touch control to achieve precise blind operation without a screen. In addition, under the emergency stop mechanism, the strong haptic stimulation and full-screen hard press operation to control the drone body shorten the emergency response time from the traditional vision-cognition-operation time (about 1.5 seconds) to the instinctive tactile-muscle reaction time (about 0.3 seconds), which helps to improve the crisis management ability of drone operators.

[0055] In some embodiments, the haptic feedback-based drone control method of this application can be applied to aerial photography control in complex wind field environments. For example, a user controls a drone for aerial photography using the terminal device provided in this application at the beach. The drone encounters strong sea winds (crosswinds), and the wind speed sensor on the drone detects a wind force of 8 m / s on the right. The data is transmitted back to the APP, and the Haptic rendering engine outputs a continuous vibration of 40Hz with medium amplitude when the user pushes the stick to the right (windward side) based on the wind direction and force. The user's fingers can clearly feel the resistance of "not being able to push" the stick. When the user pushes the stick to the left (downwind side), the vibration disappears, and the stick becomes "lighter". The user intuitively perceives the wind field environment through touch, thereby controlling the flight attitude more carefully and avoiding loss of control due to excessive wind force.

[0056] In other embodiments, the haptic feedback-based drone control method of this application can be applied to near-range obstacle avoidance and emergency hovering scenarios for drones. For example, when a drone is moving through a forest, the user may not see a hidden branch in front of them in the image transmission due to foliage obstruction. The drone's forward-looking radar detects the obstacle at a distance of 3 meters, with a TTC calculation of 1.5 seconds. The app instantly triggers an emergency haptic signal, and the phone's linear motor outputs a strong vibration sweeping from 50Hz to 300Hz. The user's hand receives a strong stimulus, instinctively generating a tense reaction, and their fingers subconsciously press the screen harder. The app's multi-touch engine detects a sudden 30% increase in the contact area, determines it as a "press hard to stop" command, and immediately issues an emergency hovering command to the drone. The drone successfully stops 0.5 meters from the branch, avoiding a crash.

[0057] In this embodiment, the terminal device can convert the drone's physical flight status data (such as wind resistance parameters, acceleration, and distance to obstacles) into tactile signals in real time using its linear motor and pressure sensor, allowing the operator to "feel" the drone's flight status. The terminal device can also utilize multi-touch and dynamic tactile feedback to simulate a physical joystick experience with "damping" and "boundary" on a flat screen, enabling blind operation. Furthermore, in emergency situations, the terminal device can directly stimulate the operator's muscle memory through specific high-frequency / high-intensity tactile pulses, shortening emergency response time.

[0058] For example, Figure 6 A schematic diagram of the hardware structure of the controller 11 used to implement the haptic feedback-based drone control method is shown. Figure 6 As shown, the controller 11 includes: One or more processors 710 and memory 720, Figure 6 Take the 710 processor as an example.

[0059] The processor 710 and memory 720 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0060] The memory 720, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of this application. The processor 710 executes various functional applications and data processing of the terminal device by running the non-volatile software programs, instructions, and modules stored in the memory 720, thereby implementing the methods in the above-described method embodiments.

[0061] The memory 720 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 720 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 720 may optionally include memory remotely located relative to the processor 710. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0062] The one or more modules are stored in the memory 720. When executed by the one or more processors 710, they perform the methods in any of the above method embodiments, for example, the methods described above. Figure 4 Method steps S41-S45.

[0063] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0064] This application provides a non-volatile computer-readable storage medium storing computer-executable instructions that are executed by one or more processors, for example... Figure 6 One of the processors 710 can cause the one or more processors to perform the methods in any of the above method embodiments, for example, to perform the methods described above. Figure 4 Method steps S41-S45.

[0065] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0066] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; under the concept of the present invention, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the present invention as described above, which are not provided in detail for the sake of brevity; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for controlling a drone based on haptic feedback, characterized in that, The method includes: The drone's control interface is displayed via a touchscreen, and the control interface includes a virtual joystick and an effective touch area for the virtual joystick. If a touch point is detected in the effective touch area, the position of the virtual joystick is adjusted according to the position change of the touch point within the effective touch area. The flight control commands for the drone are generated based on the position changes of the virtual joystick, and the flight control commands are sent to the drone. Obtain the flight status data reported by the UAV; Determine the waveform characteristic parameters corresponding to the flight status data, and control the vibration feedback device to drive the touch screen to vibrate based on the waveform characteristic parameters corresponding to the flight status data; The waveform characteristic parameters include frequency, amplitude, and envelope.

2. The method according to claim 1, characterized in that, The flight status data includes wind resistance parameters, which include the wind speed and wind direction of the flight environment in which the UAV is located; The step of determining the waveform feature parameters corresponding to the flight status data, and controlling the vibration feedback device to drive the touch screen to vibrate based on the waveform feature parameters corresponding to the flight status data, includes: Determine the waveform characteristic parameters corresponding to the wind resistance parameters, wherein the wind speed and the amplitude corresponding to the wind resistance parameters are positively correlated; The vibration direction of the vibration feedback device is determined based on the wind direction, and the vibration direction is opposite to the control direction of the virtual remote sensing. Based on the waveform characteristic parameters corresponding to the wind resistance parameters and the direction of vibration, the vibration feedback device is controlled to drive the touch screen to vibrate.

3. The method according to claim 1, characterized in that, The flight status data also includes the distance between the drone and the obstacle; The step of determining the waveform feature parameters corresponding to the flight status data, and controlling the vibration feedback device to drive the touch screen to vibrate based on the waveform feature parameters corresponding to the flight status data, includes: Determine the waveform feature parameters corresponding to the distance; Based on the waveform characteristic parameters corresponding to the distance, the vibration feedback device is controlled to drive the target area on the touch screen to vibrate. The target area is the area on the touchscreen that corresponds to the location of the obstacle.

4. The method according to claim 1, characterized in that, The flight status data also includes the flight speed of the drone; The step of determining the waveform feature parameters corresponding to the flight status data, and controlling the vibration feedback device to drive the touch screen to vibrate based on the waveform feature parameters corresponding to the flight status data, includes: Determine the waveform characteristic parameters corresponding to the flight speed, and control the vibration feedback device to drive the touch screen to vibrate based on the waveform characteristic parameters corresponding to the flight speed; The flight speed is positively correlated with the frequency corresponding to the flight speed.

5. The method according to claim 1, characterized in that, The flight status data also includes the collision time between the drone and the obstacle; The step of determining the waveform characteristic parameters corresponding to the flight status data, and controlling the vibration feedback device to drive the touch screen to vibrate based on the waveform characteristic parameters corresponding to the flight status data, includes: If the collision time is determined to be less than a preset safe time threshold, waveform characteristic parameters corresponding to the emergency situation are generated, and the vibration feedback device is controlled to vibrate according to the waveform characteristic parameters corresponding to the emergency situation.

6. The method according to claim 5, characterized in that, The method further includes: If the collision time is determined to be less than a preset safe time threshold, an emergency stop mechanism is triggered. Under the emergency stop mechanism, if a touch point acting on any position of the touch screen is detected, and it is determined that the touch operation corresponding to the touch point is a hard press operation, then an emergency hover command or a reverse braking command is issued to the drone. Wherein, the hard press operation is used to indicate a touch operation where the pressure applied to the touchscreen exceeds a preset pressure threshold; or, The hard press operation is used to indicate a touch operation where the area of ​​the touch point is greater than a preset area threshold.

7. The method according to claim 1, characterized in that, The flight status data also includes the drone's battery level; The step of determining the waveform feature parameters corresponding to the flight status data, and controlling the vibration feedback device to drive the touch screen to vibrate based on the waveform feature parameters corresponding to the flight status data, further includes: The waveform characteristic parameters corresponding to the battery level of the drone are determined, and the vibration feedback device is controlled to drive the touch screen to vibrate based on the waveform characteristic parameters corresponding to the battery level.

8. The method according to any one of claims 1-7, characterized in that, The virtual joystick moves within the range defined by its travel boundaries; The method further includes: During the process of adjusting the position of the virtual joystick according to the position change of the touch point within the effective touch area, if it is determined that the virtual joystick has reached the travel boundary, then the waveform characteristic parameters corresponding to the travel boundary are determined. The vibration feedback device is controlled to drive the touch screen to vibrate based on the waveform characteristic parameters corresponding to the travel boundary.

9. The method according to any one of claims 1-7, characterized in that, The method further includes: During the process of adjusting the position of the virtual joystick according to the position change of the touch point within the effective touch area, the offset distance between the current position of the virtual joystick and the initial position of the virtual joystick is determined; Determine the waveform feature parameters corresponding to the offset distance; The vibration feedback device is controlled to drive the touch screen to vibrate based on the waveform characteristic parameters corresponding to the offset distance. The offset distance is positively correlated with the amplitude corresponding to the offset distance.

10. The method according to any one of claims 1-7, characterized in that, The control interface also includes a non-effective touch area, which is the area in the control interface other than the effective touch area; The method further includes: If the touch point is detected to have slid from the effective touch area to outside the ineffective touch area, then the waveform characteristic parameters corresponding to the ineffective touch area are determined. The vibration feedback device is controlled to drive the touch screen to vibrate based on the waveform characteristic parameters corresponding to the non-effective touch area.

11. A terminal device, characterized in that, The terminal device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-10.

12. A drone control system, characterized in that, The system includes: Drones; and The terminal device of claim 11, wherein the drone and the terminal device are communicatively connected.

13. A non-volatile computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when executed, enable the execution of the method described in any one of claims 1-10.

Images