A rotor unmanned aerial vehicle control system based on multi-modal input and a control method thereof

Abstract

This invention discloses a multimodal input-based control system and method for a rotary-wing unmanned aerial vehicle (UAV). The method comprises the following steps: Step S1: Acquiring the operator's gaze point information; extracting the operator's gaze point information in the environmental coordinate system using a depth sensor; determining the operational intent expressed by the gaze point using a gaze point statistical model and an intention target point estimation method; and combining the operational intent with environmental information to obtain the planned path point. Step S2: Combining the joystick motion input information with the safe path to comprehensively generate navigation and control information for the rotary-wing UAV. Step S3: Using a motion controller to control the rotary-wing UAV to always fly along the planned path; simultaneously, the joystick motion input information is used as a reference input to the motion controller for the rotary-wing UAV's flight speed, achieving path following. This invention solves the problem that traditional mobile UAV teleoperation systems cannot quickly plan safe paths in complex environments.

Description

Technical Field

[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) control technology, specifically relating to a rotary-wing UAV control system and control method based on multimodal input. Background Technology

[0002] With the rapid development of technologies such as electronic information, automatic control, and artificial intelligence, autonomous drones have gradually become a hot area of ​​technological development. However, due to the current limitations in the understanding and autonomous decision-making capabilities of autonomous drones, existing drones are still unable to carry out missions fully autonomously in unknown and complex environments. Therefore, semi-autonomous teleoperation systems that utilize human analysis, understanding, and decision-making capabilities and are integrated with the underlying automatic control of drones have attracted widespread attention in the drone field. Compared with traditional robotic arm teleoperation systems, mobile drone teleoperation systems typically face more complex unstructured environments, placing higher demands on operators' perception and control of the environment. Therefore, a human-machine hybrid teleoperation system that analyzes and understands the operator's operational intentions and then uses the perceived information from the drone to assist the operator in control will become the future development direction.

[0003] In human-machine hybrid teleoperation systems, the key research areas are how to efficiently acquire operator input and how to integrate this input with information from the UAV control system. Traditional teleoperation primarily utilizes position or force signals provided by master controllers such as joysticks as input to the slave UAV. Simultaneously, it uses force / position feedback information generated from the slave UAV or environmental information to achieve operator-teleoperation interaction. Furthermore, electromyography (EMG) and electroencephalography (EEG) signals from the operator's arm surface are also beginning to be used as operator input for controlling the remote slave UAV in teleoperation systems. However, both force / position information and EMG signals are often only directly related to the low-level motion control of the slave UAV and cannot be correlated with more specific and complex task and environmental information. While EEG signal input has a strong correlation with task and environment, its high coupling with other factors makes extracting task- and environment-related information overly complex. Current research has not yet provided an efficient, accurate brain-computer interface that can provide input for complex tasks. Since operators in teleoperation systems typically need to observe the remote environment through images transmitted from the UAV in order to analyze, understand, make task decisions, and perform operations, and since the operator's eye movement information is not only directly related to the environment but may also contain task operation intentions, studying the operator's eye movement information can quickly obtain more and more intuitive operation information, thereby effectively assisting the teleoperation system in carrying out operation tasks. Summary of the Invention

[0004] The purpose of this invention is to provide a rotorcraft unmanned aerial vehicle (UAV) control system and control method based on multimodal input, so as to solve the problem that traditional mobile UAV remote control systems cannot perform rapid safe path planning in complex environments.

[0005] This invention adopts the following technical solution: a control method for a rotary-wing unmanned aerial vehicle (UAV) control system based on multimodal input, comprising the following:

[0006] Step S1: Obtain the operator's gaze point information through an eye tracker, and extract the operator's gaze point information in the environmental coordinate system by combining it with a depth sensor. Then, use the gaze point information statistical model and the intention target point estimation method to determine the operation intention expressed by the gaze point. Finally, combine the operation intention with the environmental information to obtain the planned path point and plan a safe path from the current position to the path point.

[0007] Step S2: Combine the joystick motion input information with the safe path obtained in step S1, comprehensively extract the operation intention, and generate navigation and control information for the rotary-wing UAV;

[0008] Step S3: Use the motion controller to control the rotorcraft to always fly along the path planned by the navigation and control information in step S2, and the heading always points to the direction of the next path point; at the same time, the joystick motion input information is used as the rotorcraft's flight speed reference input to the motion controller to achieve path following.

[0009] Furthermore, in step S1, the intention target point estimation method includes a target point estimation method based on gaze point characteristics in a static environment, specifically:

[0010] First, the remote color images acquired by the RGB-D sensor are provided to the operator for observation of the flight environment;

[0011] Then, an eye tracker is used to obtain the operator's eye movement information, that is, the coordinates of the gaze point in the color image; the coordinates can be used to obtain the depth information in the corresponding coordinates of the RGB-D sensor, thereby obtaining the coordinates of the gaze point in the RGB-D sensor coordinate system;

[0012] Then, by transforming the coordinates between the RGB-D sensor and the rotary-wing drone, and by taking the coordinates of the rotary-wing drone in the world coordinate system, the coordinates of the gaze point in the world coordinate system can be obtained.

[0013] Finally, based on the target characteristics, a preset bias value is added to the mean of the fixation points to generate the target point.

[0014] Furthermore, in step S1, the intention target point estimation method includes a target point estimation method based on gaze point characteristics in a dynamic environment, specifically:

[0015] First, the color images acquired by the RGB-D sensor are provided to the operator for observation of the flight environment;

[0016] Then, an eye tracker is used to obtain the operator's eye movement information, that is, the coordinates of the gaze point in the color image; the coordinates can be used to obtain the depth information in the corresponding coordinates of the RGB-D sensor, thereby obtaining the coordinates of the gaze point in the RGB-D sensor coordinate system;

[0017] Then, by transforming the coordinates between the RGB-D sensor and the rotary-wing drone, and by taking the coordinates of the rotary-wing drone in the world coordinate system, the coordinates of the gaze point in the world coordinate system can be obtained.

[0018] Finally, the obstacle's trajectory is predicted based on its speed, and a preset offset value is added to the endpoint of the predicted trajectory along the direction of the obstacle's movement, ensuring that the target point does not intersect with the predicted trajectory.

[0019] This invention provides a control system for a rotary-wing unmanned aerial vehicle (UAV) based on multimodal input, and a control method for the same system, the structure of which includes:

[0020] A joystick input module, the output of which is connected to a motion controller; an intention target point estimation module, the output of which is connected to the motion controller via a path planner; the output of the motion controller is connected to a rotary-wing UAV; the rotary-wing UAV also provides data feedback to the motion controller, the path planner, and the intention target point estimation module respectively.

[0021] The joystick input module is used to acquire the joystick motion input information of the operator and send the joystick motion input information to the motion controller as a reference speed.

[0022] The intention target point estimation module is used to acquire the operator's visual information and use the intention target point estimation method to generate the intention motion target point of the rotary-wing UAV from the visual information, and input the intention motion target point into the path planner.

[0023] The path planner is used to plan a safe flight path for the rotary-wing UAV based on the intended target point and the current position of the UAV, and input the safe flight path into the motion controller.

[0024] The motion controller is used to simultaneously receive the intended motion target point transmitted from the intended target point estimation module, the safe flight path transmitted from the path planner, and the speed, position, and heading information fed back by the rotary-wing UAV; it is also used to generate flight control requirements and transmit them to the rotary-wing UAV.

[0025] The rotary-wing drone is used to fly according to the control requirements of the motion controller, and also to feed back the current position of the rotary-wing drone to the path planner, as well as to feed back the position and flight path of the rotary-wing drone to the intention target point estimation module.

[0026] Furthermore, the visual information acquired by the intention target point estimation module includes remote color images, depth information, and operator eye movement information. The rotary-wing UAV is equipped with an RGB-D sensor, and the operator carries an eye tracker.

[0027] Among them, the RGB-D sensor is used to acquire remote color images and depth information of the environment, and the eye tracker is used to capture the operator's eye movement information.

[0028] The beneficial effects of this invention are as follows: First, this invention analyzes the characteristics of the operator's gaze point in typical environments and tasks, and establishes an estimation method for estimating the operator's intended target point based on gaze point data; second, by using the operator's intended target point as the endpoint of the local path planner, and using the joystick input as the heuristic condition for heuristic local path planning based on a fast random expansion tree and the speed control command for path following, the operator can achieve rapid motion planning, path following, and speed control of the UAV, thereby ensuring the safe and rapid flight and navigation of the rotary-wing UAV in complex environments. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the structure of a rotary-wing unmanned aerial vehicle control system based on multimodal input according to the present invention;

[0030] Figure 2 This is a schematic diagram of the control method of a rotary-wing unmanned aerial vehicle control system based on multimodal input according to the present invention;

[0031] Figure 3-1 This is a schematic diagram of the flight path of a drone using the control method of the present invention in a static obstacle simulation experiment according to an embodiment of the present invention.

[0032] Figure 3-2 To and Figure 3-1 Time-velocity curves of drones performing the same task;

[0033] Figure 4-1 This is a schematic diagram of the flight path of a UAV using a traditional teleoperation method in a static obstacle simulation experiment according to an embodiment of the present invention.

[0034] Figure 4-2 To and Figure 4-1 Time-velocity curves of drones performing the same task;

[0035] Figure 5-1 This is a schematic diagram of the flight path of a drone using the control method of the present invention in a dynamic obstacle simulation experiment according to an embodiment of the present invention.

[0036] Figure 5-2 To and Figure 5-1 Time-velocity curves of drones performing the same task;

[0037] Figure 5-3 To and Figure 5-1 Distance-time curves of maneuvering obstacles and rotary-wing UAVs in the same task;

[0038] Figure 6-1 This is a schematic diagram of the flight path for a maneuvering obstacle scenario experiment under a traditional remote control system.

[0039] Figure 6-2 The time-velocity curves for a maneuvering obstacle scenario experiment under a traditional remote control system;

[0040] Figure 6-3 This is a distance-time curve of the moving obstacle and the rotary-wing UAV in a scenario experiment under a traditional remote control system.

[0041] The components include: 1. Joystick input module, 2. Intended target point estimation module, 3. Path planner, 4. Motion controller, 5. Rotary-wing UAV, and 6. Operator. Detailed Implementation

[0042] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0043] This invention provides a control method for a rotary-wing unmanned aerial vehicle (UAV) control system based on multimodal input, as follows: Figure 2 As shown, it includes the following steps:

[0044] Step S1: Obtain the operator's gaze point information through an eye tracker, and extract the operator's gaze point information in the environmental coordinate system by combining it with a depth sensor. Then, use the gaze point information statistical model and the intention target point estimation method to determine the operation intention expressed by the gaze point. Finally, combine the operation intention with the environmental information to obtain the path point for path planning, and plan a safe path from the current position to the path point.

[0045] The method for establishing the gaze point information statistical model is as follows: By roughly classifying the UAV's mission environment into four basic types—obstacles in the middle ahead, obstacles on one side ahead, doors or windows ahead, and moving obstacles ahead—modeling of the mission environment can be achieved. Within these four environments, by having multiple operators repeatedly perform basic point-to-point navigation and control tasks and recording the relationship information between the gaze point and obstacles in the environment, a gaze point-obstacle statistical model can be established under different obstacle types. Based on this statistical model, the system can quickly determine the type of obstacle ahead based on the gaze point and environmental perception information, and generate a preliminary intended target point accordingly.

[0046] Step S2: Combine the joystick motion input information with the safe path obtained in step S1, comprehensively extract the operation intention, and generate navigation and control information for the rotary-wing UAV;

[0047] Step S3: Use the motion controller to control the rotorcraft to always fly along the path planned by the navigation and control information in step S2, and the heading always points to the direction of the next path point; at the same time, the joystick motion input information is used as the rotorcraft's flight speed reference input to the motion controller to achieve path following.

[0048] This control method utilizes a relationship model between eye-tracking information and environmental information obtained from prior user research. It can acquire operational intentions based on eye-tracking information and determine specific path planning target points based on environmental information. Simultaneously, by combining joystick operation information with gaze-based inputs, it obtains multi-modal control input information, including path planning target points, movement speed, and direction. This enables comprehensive control of the rotary-wing UAV from multiple system control levels, ensuring the safety of the UAV and the ease of use of the teleoperation system from multiple dimensions, and improving the system's mission performance.

[0049] In some embodiments, the intention target point estimation method includes a target point estimation method based on gaze point characteristics in a static environment, specifically:

[0050] First, the remote color image acquired by the RGB-D sensor is provided to the operator via a display for observation of the flight environment; the rotary-wing UAV is equipped with an RGB-D sensor, which is used to acquire remote color images and depth information of the environment.

[0051] Then, an eye tracker is used to obtain the operator's eye movement information, that is, the coordinates of the gaze point in the color image; the coordinates can be used to obtain the depth information in the corresponding coordinates of the RGB-D sensor, thereby obtaining the coordinates of the gaze point in the RGB-D sensor coordinate system;

[0052] Then, by transforming the coordinates between the RGB-D sensor and the rotary-wing drone, and by taking the coordinates of the rotary-wing drone in the world coordinate system, the coordinates of the gaze point in the world coordinate system can be obtained.

[0053] Finally, based on the target characteristics, a preset bias value is added to the mean of the fixation points to generate the target point.

[0054] In some embodiments, the intention target point estimation method further includes a target point estimation method based on gaze point characteristics in a dynamic environment, specifically:

[0055] First, the color images acquired by the RGB-D sensor are displayed to the operator for observation of the flight environment;

[0056] Then, an eye tracker is used to obtain the operator's eye movement information, that is, the coordinates of the gaze point in the color image; the coordinates can be used to obtain the depth information in the corresponding coordinates of the RGB-D sensor, thereby obtaining the coordinates of the gaze point in the RGB-D sensor coordinate system;

[0057] Then, by transforming the coordinates between the RGB-D sensor and the rotary-wing drone, and by taking the coordinates of the rotary-wing drone in the world coordinate system, the coordinates of the gaze point in the world coordinate system can be obtained.

[0058] Finally, the obstacle's trajectory is predicted based on its speed, and a preset offset value is added to the endpoint of the predicted trajectory along the direction of the obstacle's movement, ensuring that the target point does not intersect with the predicted trajectory.

[0059] In summary, this invention divides the method for estimating the operator's intended target point based on gaze point data into two types: target point estimation based on gaze point characteristics in a static environment and target point estimation based on gaze point characteristics in a dynamic environment.

[0060] In teleoperation scenarios, the operator's gaze has the following characteristics: When observing an environment, the human eye consciously or unconsciously focuses its gaze on a potentially processed part of the external environment, creating a gaze point. Movement between gaze points is accomplished through rapid eye saccades. Human perception and information acquisition of a static target are achieved through a series of gazes and eye saccades. Correspondingly, information acquisition of dynamic targets comes from tracking motion. Eye tracking data can serve as a quantitative indicator of visual attention. When the human eye moves from one position to another through a series of eye saccades, visual attention is formed. Attention is a perceptual selection that reflects a certain human intention.

[0061] In teleoperation systems, vision is a crucial means of acquiring information about the external environment when an operator controls the movement of a drone. The operator's gaze point also reflects the process and result of filtering and processing this environmental information. Therefore, this invention aims to analyze the characteristics of the operator's gaze point and estimate operational intent in typical teleoperation scenarios by capturing the operator's gaze point information and extracting environmental information related to that gaze point.

[0062] For estimating operational intent, the correlation between gaze point information and environmental information is crucial. This invention utilizes a non-invasive Tobii eye tracker to achieve gaze point and eye tracking at a sampling frequency of 60Hz on an Ubuntu system. However, gaze point and eye movement information are based on two-dimensional coordinates and motion in the screen coordinate system. To correlate gaze point and eye movement information with environmental information, and since visual information is typically a line of sight, this invention matches gaze point and eye movement information with depth information acquired by an airborne RGB-D sensor, thereby establishing a correlation between gaze point and the three-dimensional environment. Simultaneously, color image information acquired by the airborne RGB-D sensor is used as image feedback from the UAV to simplify the transformation relationship between the sensor coordinate system, the UAV body coordinate system, and the world coordinate system, achieving the expression of normalized gaze point coordinates in the pixel coordinate system with the top-left corner of the image as the origin in the world coordinate system (East-North-Up, ENU).

[0063]

[0064] The transformation matrices described above include the camera's intrinsic and extrinsic parameter matrices. The intrinsic parameter matrix contains the initial focal length and focus information of the RealSense camera. The extrinsic parameter matrix represents the transformation relationship between the body coordinate system and the world coordinate system, and includes the pose information of the rotary-wing UAV. R is the attitude angle rotation matrix, and T is the position translation matrix.

[0065] This invention also provides a multimodal input-based remote control system for rotary-wing unmanned aerial vehicles (UAVs), employing a control method for such a system, such as... Figure 1 As shown, the system includes a joystick input module 1, an intention target point estimation module 2, a path planner 3, a motion controller 4, and a rotary-wing UAV 5. The output of the joystick input module 1 is connected to the motion controller 4; the output of the intention target point estimation module 2 is connected to the motion controller 4 via the path planner 3; the output of the motion controller 4 is connected to the rotary-wing UAV 5; the rotary-wing UAV 5 also provides data feedback to the motion controller 4, the path planner 3, and the intention target point estimation module 2.

[0066] The joystick input module 1 is used to acquire the joystick motion input information of the operator and send the joystick motion input information to the motion controller 4 as a reference speed.

[0067] The intention target point estimation module 2 is used to acquire the operator's visual information and use the intention target point estimation method to generate the intention motion target point of the rotary-wing UAV from the visual information, and input the intention motion target point into the path planner 3.

[0068] Path planner 3 is used to plan a safe flight path for the rotorcraft UAV based on the intended target point and the current position of the UAV, and inputs the safe flight path to motion controller 4. The path planner needs to be guided by the rotorcraft UAV's current position x and the target point for path planning. Here, x and θ represent the rotorcraft UAV's position and flight path, respectively. The intended target point is first estimated and its coordinates are obtained in the UAV coordinate system, so it needs to be transformed to the world coordinate system, which requires x and θ.

[0069] The motion controller 4 is used to simultaneously receive the intended motion target point transmitted from the intended target point estimation module 2, the safe flight path transmitted from the path planner 3, and the speed, position, and heading information fed back by the rotary-wing UAV 5; it is also used to generate flight control requirements and transmit them to the rotary-wing UAV 5.

[0070] The rotary-wing drone 5 is used to fly according to the control requirements of the motion controller 4, and also to feed back the current position of the rotary-wing drone to the path planner 3, and to feed back the position and flight path of the rotary-wing drone to the intention target point estimation module 2.

[0071] In some embodiments, the visual information acquired by the intention target point estimation module 2 includes remote color images, depth information, and operator eye movement information. The rotary-wing UAV 5 is equipped with an RGB-D sensor, and the operator carries an eye tracker. The RGB-D sensor is used to acquire remote color images and depth information of the environment, while the eye tracker is used to capture the operator's eye movement information. This eye movement information includes various parameters such as fixation point, movement speed, and dwell time. The fixation point combined with the dwell time can effectively express the operator's operational intention.

[0072] Traditional mobile drone telecontrol systems rely on joysticks and other main controllers, which provide limited input information. Inputs such as EEG, EMG, and voice are difficult to understand and complex to match with telecontrol tasks and environments. Traditional joystick-based control systems cannot guarantee drone safety in complex environments, while force feedback and other auxiliary methods increase the operator's workload.

[0073] This invention first analyzes the characteristics of the operator's gaze point in typical environments and tasks, and establishes an estimation method for the operator's intended target point based on gaze point data. It utilizes an eye tracker to acquire information about obstacles in the environment, thereby estimating the intended target point. By using the operator's intended target point as the endpoint of a local path planner and employing a path planning algorithm based on heuristic RRT, it achieves rapid planning of safe paths in complex environments. Furthermore, it uses joystick input as the heuristic condition for heuristic local path planning based on a fast random expansion tree and as the speed control command for path following. This allows the operator to achieve rapid motion planning, path following, and speed control of the UAV, ensuring safe and rapid flight and navigation of the rotary-wing UAV in complex environments. This solves the safety problem of rotary-wing UAVs flying in complex environments, while reducing the operator's workload and improving the control performance of the UAV control system.

[0074] Example

[0075] To avoid interference from other potential factors on the operator's gaze point, this invention establishes a strictly controllable simulation test scenario with a single control variable. Under this simulation test environment, by setting static and dynamic obstacle avoidance navigation tasks for the operator, the gaze point information under the remote control obstacle avoidance navigation task of the rotary-wing UAV is collected.

[0076] The virtual working environment built using the Gazebo simulator measures 6 meters × 4 meters × 2 meters (length × width × height). A three-degree-of-freedom joystick and a Tobii eye tracker were used as the main controller devices. Simultaneously, a rotorcraft drone model based on the 3DRobotics Iris quadcopter and the Realsense D435i depth camera hardware was constructed within the Gazebo simulator. The PX4 open-source flight control firmware was used as the flight control software system for the rotorcraft drone. The world coordinate system in the experimental scene adopted the Northeast-East-Down (ENU) coordinate system, the rotorcraft drone's body coordinate system adopted the North-East-Down (NED) coordinate system, and the joystick coordinate system was consistent with the rotorcraft drone's body coordinate system. The operator could input the x and y axes by pushing the joystick, and the inputs from both axes were processed into linear velocity reference inputs for the rotorcraft drone and sent to the motion controller 4.

[0077] ① Static obstacle simulation experiment

[0078] In this group of experiments, such as Figure 3-1 As shown, a single wall-shaped obstacle and a window-shaped obstacle constitute a static experimental scenario. The operator is required to pilot a rotary-wing UAV from the starting point of star 1 to the target position of star 2. During the experiment, collisions with obstacles should be avoided as much as possible to verify the mission performance of the proposed teleoperation system in this environment.

[0079] In the experiment, the operator first flew a UAV using a multimodal input-based rotary-wing UAV control system according to the present invention. After the experiment, the flight mission was repeated using a traditional teleoperation method. The specific experimental results are analyzed as follows:

[0080] The UAV using the multimodal input-based rotary-wing UAV control system of this invention performed this flight mission. Experimental data are as follows: Figure 3-1 and Figure 3-2 As shown. Figure 3-1 As shown, the rotary-wing UAV quickly departed from position 1 of the star shape along the planned path and arrived at the first intended target point indicated by position 3. It then continued to follow the planned path and finally arrived at the final target position indicated by position 2 of the star shape. During the entire flight, the rotary-wing UAV did not collide with any obstacles. Figure 3-2 The experiment demonstrated the change in the linear velocity of the rotorcraft over time during the mission. It was observed that, except for stopping near the intermediate target point, the rotorcraft maintained a constant speed throughout the mission, with the total mission time being 23 seconds.

[0081] The flight mission was repeated using traditional teleoperation methods, and the relevant experimental data are as follows: Figure 4-1 and Figure 4-2 As shown. Figure 4-1 As shown, since the operator controls the speed and direction of the rotorcraft entirely with the joystick, the rotorcraft adjusted its position forward and backward from the starting position. After the operator adapted to the operation method, the rotorcraft was controlled to reach the target point indicated by the diamond shape No. 3, and finally reached the target position indicated by the star shape No. 2. The rotorcraft did not collide with any obstacles during the mission. Figure 4-2 The paper demonstrates the speed change of a rotary-wing UAV during a mission using traditional remote control methods. The significant speed changes of the rotary-wing UAV are clearly visible, which are caused by turning and obstacle avoidance operations, thus causing the mission time to increase significantly to 35 seconds.

[0082] By observing the comparative experiments in the static obstacle scenario, it can be found that, without collision, the remote operating system proposed in this invention has the characteristics of simple operation, good path smoothness, and high flight speed. This verifies the safety and reliability of the rotorcraft UAV control system and its control method based on multimodal input in rotorcraft UAV navigation missions, thereby effectively improving the system's mission performance.

[0083] ②Simulation experiment of moving obstacles

[0084] In this group of experiments, such as Figure 5-1As shown, a 0.4-meter-sided cubic maneuvering obstacle is placed at (2, 0.8, 1) in the world coordinate system and controlled by a position controller to move towards the target point (2, -2, 1) at a maximum speed of 0.2 m / s. The operator is required to pilot the rotorcraft UAV from point I in the star-shaped configuration and eventually reach the vicinity of point II in the star-shaped configuration. Collisions with the maneuvering obstacle should be avoided as much as possible during the experiment to verify the mission performance of the multimodal input-based rotorcraft UAV control system of this invention in a maneuvering obstacle environment.

[0085] In the experiment, the operator first flew a UAV using a multimodal input-based control system according to the present invention. After completion, the flight mission was repeated using a traditional teleoperation method. The specific experimental results are analyzed as follows:

[0086] The UAV using the multimodal input-based rotary-wing UAV control system of this invention performed this flight mission. Experimental data are as follows: Figures 5-1 to 5-3 As shown. Figure 5-1 As shown in the figure, the vertical dashed line represents the trajectory of the moving target. The dashed rectangle (III) indicates the position of the obstacle when the rotorcraft UAV reaches the horizontal coordinate 2. The solid rectangle represents the position of the moving target when the rotorcraft UAV reaches the target point. This can be observed... Figures 5-1 to 5-3 It can be observed that the rotary-wing drone did not collide with any obstacles during its flight mission, and its overall path and speed were relatively smooth, maintaining a large safe distance from moving obstacles at all times. Figure 5-3 The horizontal dashed line in the figure represents the minimum safe distance between the moving obstacle and the rotorcraft drone to avoid collision. The total flight time of the rotorcraft drone is 9.3 seconds.

[0087] The flight mission was repeated using traditional teleoperation methods, and the relevant experimental data are as follows: Figures 6-1 to 6-3 As shown, observations revealed that the overall flight performance of the rotary-wing UAV was similar to that of an operator using a multimodal input-based rotary-wing UAV control system proposed in this invention. No collisions occurred during the entire flight, and the total flight time was 14 seconds. However, the flight speed fluctuated slightly more compared to the remote control scheme proposed in this invention, and the distance between the UAV and maneuvering obstacles was relatively close in the final stage, posing a potential danger. Furthermore, it was observed that in traditional remote control experiments, the operator tended to move laterally rather than forward, leading to an increase in overall navigation time.

[0088] Based on the above comparison of time delay results in a dynamic obstacle environment, the teleoperation scheme proposed by the multimodal input-based rotary-wing UAV control system of this invention can achieve effective obstacle avoidance. At the same time, compared with the traditional teleoperation system, its flight path is shorter and smoother, saving obstacle avoidance time, and the safe distance between it and the moving obstacle is larger, resulting in better mission safety.

[0089] By constructing typical mission scenarios commonly encountered in teleoperated navigation tasks and conducting comparative experiments with traditional teleoperation systems, the feasibility and effectiveness of the proposed solution were verified. The comparative experimental results also demonstrate that the proposed teleoperation system can effectively shorten flight distance, save flight time, and offer better safety. The simulation experiments in the embodiments, referencing the aforementioned simulation verification, designed two types of experimental scenarios: static obstacles and moving obstacles. These simulate typical scenarios frequently encountered by the teleoperation system in practical applications, thereby verifying the feasibility and performance of the proposed teleoperation method. Simultaneously, traditional teleoperation was used as a comparison in the experimental tasks, further verifying the navigation performance of the proposed solution in practical applications.

[0090] Eye-tracking information, especially gaze point information, can be used to predict the operator's intentions, serving as an important supplement to traditional master controller force-position input methods. Simultaneously, a human-machine hybrid teleoperation system that combines path planning and tracking with the teleoperation system can effectively enhance system task performance and improve operational efficiency. To address the shortcomings of previous studies that only utilize eye-tracking information for environmental recognition or modeling, or use gaze direction as a rotation reference for the UAV in simple environments, and to enhance the task performance of mobile UAV teleoperation systems in complex environments, this invention proposes a rotorcraft UAV control system and its control method based on multimodal input using a shared control framework.

[0091] This invention first analyzes the characteristics of the operator's gaze point in typical environments and tasks, and establishes an estimation method for estimating the operator's intended target point based on gaze point data. Furthermore, by using the operator's intended target point as the endpoint of the local path planner, and using the joystick input as the heuristic condition for heuristic local path planning based on Rapidly-exploring Random Tree (RRT) and the speed control command for path following, the operator can achieve rapid motion planning, path following, and speed control of the UAV, thereby ensuring the safe and rapid flight and navigation of the rotary-wing UAV in complex environments.

[0092] This invention adopts a shared control framework and proposes a rotary-wing UAV teleoperation system that integrates gaze-based intention target point estimation, a path planner based on heuristically improved RRT, and a motion controller. This system improves human-computer interaction efficiency, reduces the operator's workload, and enhances the safety of rotary-wing UAV navigation.

[0093] To address the issues of insufficient and incomplete information acquisition during operator interaction in existing teleoperated rotorcraft UAVs, this invention proposes a rotorcraft UAV teleoperation system that integrates gaze-based intention target point estimation, a path planner based on heuristically improved RRT, and a motion controller. This system improves human-machine interaction efficiency, reduces operator workload, and enhances navigation safety. By constructing typical mission scenarios common in teleoperated navigation tasks and conducting comparative experiments with traditional teleoperation systems, the feasibility and effectiveness of the proposed system are verified. The comparative experimental results also demonstrate that the proposed teleoperation system can effectively shorten flight distance, save flight time, and provide better safety.

[0094] The teleoperation system based on gaze-assisted path planning proposed in this invention is a preliminary attempt to improve the performance of a teleoperation system by integrating various interactive information. In subsequent work, methods such as deep reinforcement learning will be used to further optimize operator intent estimation, so as to better adapt to different application scenarios and task environments. Furthermore, by making full use of fusion frameworks such as shared control, the advantages of humans and automated systems can be complemented, and a more complete human-machine integration can be achieved.

Claims

1. A control method for a rotary-wing unmanned aerial vehicle (UAV) control system based on multimodal input, characterized in that, Includes the following: Step S1: Obtain the operator's gaze point information through an eye tracker, and extract the operator's gaze point information in the environmental coordinate system by combining it with a depth sensor. Then, use the gaze point information statistical model and the intention target point estimation method to determine the operation intention expressed by the gaze point. Finally, combine the operation intention with the environmental information to obtain the planned path point and plan a safe path from the current position to the path point. The method for establishing the gaze point information statistical model is as follows: by roughly classifying the UAV's mission environment into four basic types—obstacles in the middle ahead, obstacles on one side ahead, doors or windows ahead, and moving obstacles ahead—the mission environment can be modeled. In these four types of environments, by having multiple operators repeatedly perform basic point-to-point navigation and control tasks and record the relationship information between the gaze point and obstacles in the environment, a gaze point-obstacle statistical model can be established under different obstacle types. Based on this statistical model, the system can quickly determine the type of obstacle ahead based on the gaze point and environmental perception information, and generate a preliminary intention target point based on this. The intention target point estimation method includes a target point estimation method based on gaze point characteristics in a static environment, specifically: First, the remote color images acquired by the RGB-D sensor are provided to the operator for observation of the flight environment; The eye tracker is then used to obtain the operator's eye movement information, that is, the coordinates of the gaze point in the color image; the coordinates can be used to obtain the depth information in the corresponding coordinates of the RGB-D sensor, thereby obtaining the coordinates of the gaze point in the RGB-D sensor coordinate system; Then, by transforming the coordinates between the RGB-D sensor and the rotary-wing drone, and by taking the coordinates of the rotary-wing drone in the world coordinate system, the coordinates of the gaze point in the world coordinate system can be obtained. Finally, based on the target characteristics, a preset bias value is added to the mean of the fixation points to generate the target points; Step S2: Combine the joystick motion input information with the safe path obtained in step S1, comprehensively extract the operation intention, and generate navigation and control information for the rotary-wing UAV; Step S3: Use the motion controller to control the rotorcraft to always fly along the path planned by the navigation and control information in step S2, and the heading always points to the direction of the next path point; at the same time, the joystick motion input information is used as the rotorcraft's flight speed reference input to the motion controller to achieve path following.

2. The control method for a rotary-wing unmanned aerial vehicle (UAV) control system based on multimodal input as described in claim 1, characterized in that, In step S1, the intention target point estimation method further includes a target point estimation method based on gaze point characteristics in a dynamic environment, specifically: First, the color images acquired by the RGB-D sensor are provided to the operator for observation of the flight environment; The eye tracker is then used to obtain the operator's eye movement information, that is, the coordinates of the gaze point in the color image; the coordinates can be used to obtain the depth information in the corresponding coordinates of the RGB-D sensor, thereby obtaining the coordinates of the gaze point in the RGB-D sensor coordinate system; Then, by transforming the coordinates between the RGB-D sensor and the rotary-wing drone, and by taking the coordinates of the rotary-wing drone in the world coordinate system, the coordinates of the gaze point in the world coordinate system can be obtained. Finally, the obstacle's trajectory is predicted based on its speed, and a preset offset value is added to the endpoint of the predicted trajectory along the direction of the obstacle's movement, ensuring that the target point does not intersect with the predicted trajectory.

3. A control system for a rotary-wing unmanned aerial vehicle based on multimodal input, characterized in that, The control method of a rotorcraft unmanned aerial vehicle control system based on multimodal input as described in claim 1 or 2 has the following structure: A joystick input module (1) has its output data connected to a motion controller (4); an intention target point estimation module (2) has its output data connected to the motion controller (4) through a path planner (3); the output data of the motion controller (4) is connected to a rotary-wing UAV (5); the rotary-wing UAV (5) also has data feedback connected to the motion controller (4), the path planner (3) and the intention target point estimation module (2) respectively. The joystick input module (1) is used to acquire the joystick motion input information of the operator and send the joystick motion input information to the motion controller (4) as a reference speed. The intention target point estimation module (2) is used to acquire the operator's visual information and use the intention target point estimation method to generate the intention motion target point of the rotary-wing UAV from the visual information, and input the intention motion target point into the path planner (3). The path planner (3) is used to plan a safe flight path for the rotorcraft drone based on the intended motion target point and the current position of the rotorcraft drone, and input the safe flight path to the motion controller (4). The motion controller (4) is used to simultaneously receive the intended motion target point transmitted by the intended target point estimation module (2), the safe flight path transmitted by the path planner (3), and the speed, position, and heading information fed back by the rotary-wing UAV (5); it is also used to generate flight control requirements and transmit them to the rotary-wing UAV (5). The rotorcraft (5) is used to fly according to the control requirements of the motion controller (4), and is also used to feed back the current position of the rotorcraft to the path planner (3), and to feed back the position and flight path of the rotorcraft to the intention target point estimation module (2).

4. The rotary-wing unmanned aerial vehicle control system based on multimodal input as described in claim 3, characterized in that, The visual information acquired by the intention target point estimation module (2) includes remote color images, depth information and operator eye movement information. The rotary-wing UAV (5) is equipped with an RGB-D sensor, and the operator carries an eye tracker. The RGB-D sensor is used to acquire remote color images and depth information of the environment, and the eye tracker is used to capture the operator's eye movement information.

Images