Unmanned aerial vehicle control method based on natural language and eye movement data fusion

Abstract

The invention provides an unmanned aerial vehicle control method based on natural language and eye movement data fusion. The method comprises the following steps: acquiring an eye image sequence and voice data of a synchronous user; processing the eye image sequence to obtain an eye movement data sequence, wherein the eye movement data comprises a fixation point coordinate and a sight line vector; processing the voice data by using a large language model to generate a natural language instruction; processing the eye movement data sequence by using an encoder to obtain an eye movement feature sequence; processing the natural language instruction by using a large language model to obtain natural language features; fusing the eye movement feature sequence and the natural language features by using a bidirectional attention mechanism to obtain cross-modal fusion features; processing the cross-modal fusion features by using a self-attention mechanism and a large language model to obtain a target position of the unmanned aerial vehicle; and generating a flight path and a corresponding flight control instruction according to the starting point position and the target position of the unmanned aerial vehicle. The operation flexibility and efficiency of the unmanned aerial vehicle can be improved.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) control technology, and in particular to a UAV control method based on the fusion of natural language and eye-tracking data. Background Technology

[0002] With the continuous development of drone technology, it has been widely applied in various scenarios such as inspection and maintenance, intelligent security, emergency rescue, interactive entertainment, and human-machine collaboration. Currently, drone control methods mainly rely on traditional control terminals, such as pilots, remote control, and graphical user interfaces (GUIs), for task allocation and route management. However, this communication method has gradually revealed some significant problems in practical applications, making it difficult to meet the needs for effective, natural, and intelligent communication in complex environments. These problems are mainly reflected in the following aspects: Low communication efficiency: Traditional methods require users to control buttons, remote controls, etc. step by step. In multitasking or highly dynamic scenarios, it is difficult to quickly convey complex intentions, which limits the responsiveness and flexibility of real-time management, especially in time-sensitive scenarios such as post-disaster search and rescue.

[0003] Insufficient environmental understanding: Traditional control methods only provide operational signals without reflecting the user's attention or semantic intent towards the main target of the scene. They cannot perceive the user's cognitive state (e.g., attention distribution), making it difficult to achieve intelligent assisted control based on user-monitored behavior.

[0004] Communication load: In complex 3D space, users must simultaneously handle target recognition, route planning, and device operation. Traditional interfaces or controllers require a significant amount of work, which is detrimental to continuous management by non-professional users or in high-load scenarios, severely impacting user experience and work efficiency. Summary of the Invention

[0005] In view of this, this application provides a drone control method based on the fusion of natural language and eye-tracking data, which can add autonomous intelligent operation functions to drones and solve the technical problems of low operation flexibility and efficiency of manual remote controllers.

[0006] In a first aspect, embodiments of this application provide a drone control method based on the fusion of natural language and eye-tracking data, including: Acquire synchronized user eye image sequences and voice data; The eye image sequence is processed to obtain an eye movement data sequence, each eye movement data includes: fixation point coordinates and gaze vector; the speech data is processed using a large language model to generate natural language commands; The eye-tracking data sequence is processed using an encoder to obtain an eye-tracking feature sequence; the natural language instructions are processed using a large language model to obtain natural language features. By utilizing a bidirectional attention mechanism, the eye-tracking feature sequence and the natural language features are fused to obtain cross-modal fusion features; The cross-modal fusion features are processed using a self-attention mechanism and a large language model to obtain the target location of the UAV; The flight trajectory and corresponding flight control commands are generated based on the starting point and target position of the UAV.

[0007] In one possible implementation, the eye-tracking data sequence is processed using an encoder to obtain an eye-tracking feature sequence; including: Eye-tracking data sequence Represented as: in, ) is the first Two-dimensional coordinates of the fixation point in an eye-tracking dataset. Let i be the gaze vector in the i-th eye-tracking data; , The number of eye-tracking data; Eye-tracking data sequences are processed by an encoder. Encode the first eye movement feature sequence to obtain the first eye movement feature sequence. : in, For the first First eye movement feature; No. Location encoding of each first eye movement feature: in, It is the first Encoding the position of the first eye movement feature in the j-th dimension; It is the frequency parameter in position coding: in, It is the dimension of the first eye movement feature. It is the first The moment of first eye movement feature; Then the first Second eye movement feature for: Second eye-tracking feature sequence for: .

[0008] In one possible implementation, a large language model is used to process natural language instructions to obtain natural language features, including: The speech recognition engine is used to process the language commands and obtain the text. Text is processed using large language models to obtain natural language features. : in, Let be the semantic embedding vector of the k-th token, with dimension . , , Features of natural language The large language model is based on a multi-layer Transformer architecture and captures the contextual dependencies between language units through a self-attention mechanism.

[0009] In one possible implementation, a bidirectional attention mechanism is used to fuse the eye-tracking feature sequence and the natural language features to obtain cross-modal fusion features; including: Calculate the first attention weight : Calculate the second attention weight : in, For activation functions; Using first attention weights and second attention weights, natural language features are analyzed. With second eye movement feature sequence By performing fusion, cross-modal fusion features are obtained. ,in, For the t-th component, , For cross-modal fusion features Dimensions.

[0010] In one possible implementation, the cross-modal fusion features are processed using a self-attention mechanism and a large language model to obtain the target location of the UAV; including: Utilizing self-attention mechanism for cross-modal feature fusion The data is processed to obtain a globally perceived representation sequence. : Among them, query vector key vector value vector ;in, , and It is a learnable matrix; The dimension of the key vector; For activation functions; Using large language models to analyze representation sequences The process is performed to obtain the target location of the drone.

[0011] In one possible implementation, a flight trajectory and corresponding flight control commands are generated based on the starting point and target position of the UAV; including: Based on the drone's starting position With the target location N trajectory points are generated using linear interpolation or Bézier curves: ,in, = , Each trajectory point contains spatial coordinates, desired velocity, and direction information. A continuous flight path is generated by fitting N trajectory points with a cubic Bézier curve. The nth trajectory point Converted into corresponding flight control commands : in, This represents the control command decoding function; flight control commands include: control commands, position adjustments, attitude angle changes, and speed.

[0012] Secondly, embodiments of this application provide a drone control device based on the fusion of natural language and eye-tracking data, comprising: The acquisition unit is used to acquire synchronized user eye image sequences and voice data; The first processing unit is used to process the eye image sequence to obtain an eye movement data sequence, each eye movement data including: fixation point coordinates and gaze vector; and to process the speech data using a large language model to generate natural language instructions. The second processing unit is used to process the eye-tracking data sequence using an encoder to obtain an eye-tracking feature sequence; and to process the natural language instructions using a large language model to obtain natural language features. The fusion unit is used to fuse the eye-tracking feature sequence and the natural language features using a bidirectional attention mechanism to obtain cross-modal fusion features; The third processing unit is used to process the cross-modal fusion features using a self-attention mechanism and a large language model to obtain the target position of the UAV; The control unit is used to generate flight trajectories and corresponding flight control commands based on the starting point and target position of the UAV.

[0013] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of embodiments of this application. Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the methods of embodiments of this application.

[0014] This application can improve the flexibility and efficiency of drone operation. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 A flowchart of a drone control method based on natural language and eye-tracking data fusion provided in an embodiment of this application; Figure 2 Functional structure diagram of a drone control device based on natural language and eye-tracking data fusion provided in the embodiments of this application; Figure 3 A functional structure diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0018] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0019] The design concept of the embodiments of this application will be briefly introduced below.

[0020] In response to the problems existing in operating drones via remote controllers, there is an urgent need for a more intelligent, natural, and adaptive drone communication method that can overcome the limitations of traditional control paradigms, improve the system's understanding of user intentions, lower the control threshold, and improve human-machine collaboration.

[0021] In recent years, the rapid development of perception and understanding technologies such as eye tracking and natural language processing (NLP) has led to a surge in research on multimodal interaction based on user gaze behavior and verbal expression. Eye tracking data reflects the regional and temporal changes in user visual attention, while verbal expression accurately conveys the user's operational intentions and task objectives. This application integrates and models both to create a unified expression of user intent, making UAV communication more intuitive and intelligent.

[0022] The UAV control method proposed in this application, which integrates natural language and eye-tracking information, has effective intent recognition, multi-modal information alignment, and intelligent navigation control, thereby improving the flexibility and efficiency of UAV operation.

[0023] After introducing the application scenarios and design concepts of the embodiments of this application, the technical solutions provided by the embodiments of this application will be described below.

[0024] like Figure 1 As shown in the figure, this application provides a drone control method based on the fusion of natural language and eye-tracking data, including the following steps: Step 101: Obtain the synchronized user's eye image sequence and voice data; Step 102: Process the eye image sequence to obtain an eye movement data sequence, each eye movement data includes: fixation point coordinates and gaze vector; process the speech data using a large language model to generate natural language instructions; Step 103: Process the eye-tracking data sequence using an encoder to obtain an eye-tracking feature sequence; process the natural language instruction using a large language model to obtain natural language features; Step 104: Using a bidirectional attention mechanism, fuse the eye-tracking feature sequence and the natural language features to obtain cross-modal fusion features; Step 105: Process the cross-modal fusion features using a self-attention mechanism and a large language model to obtain the target location of the UAV; Step 106: Generate the flight trajectory and corresponding flight control commands based on the starting point and target position of the UAV.

[0025] In this embodiment, an infrared emitter emits infrared light to the eye, and a camera continuously captures images of the eye at a high image frequency (1000Hz) to ensure that rapid changes in eye movement can be captured, resulting in a sequence of eye images. The gaze interaction algorithm controls the computer system by analyzing the user's gaze, where gaze tracking technology involves detecting the user's pupil position and gaze direction.

[0026] In some embodiments, the eye-tracking data sequence is processed using an encoder to obtain an eye-tracking feature sequence; including: Eye-tracking data sequence Represented as: in, ) is the first Two-dimensional coordinates of the fixation point in an eye-tracking dataset. Let i be the gaze vector in the i-th eye-tracking data; , The number of eye-tracking data; Eye-tracking data sequences are processed by an encoder. Encode the first eye movement feature sequence to obtain the first eye movement feature sequence. : in, For the first First eye movement feature; No. Location encoding of each first eye movement feature: in, It is the first Encoding the position of the first eye movement feature in the j-th dimension; It is the frequency parameter in position coding: in, It is the dimension of the first eye movement feature. It is the first The moment of first eye movement feature; Then the first Second eye movement feature for: Second eye-tracking feature sequence for: .

[0027] To enhance the ability of gaze features to model temporal and directional information, this embodiment introduces an eye-tracking rotational position encoding mechanism. This mechanism embeds the temporal structure of the gaze sequence into the gaze vector, thereby preserving the sequential and spatial directionality of the gaze trajectory. The rotational position encoding mechanism effectively enhances the temporal sensitivity and spatial directionality of the gaze sequence within the attention mechanism, improving the model's ability to model the dynamic changes in the gaze, and ultimately achieving more accurate semantic-spatial joint modeling.

[0028] In some embodiments, a large language model is used to process natural language instructions to obtain natural language features; including: The speech recognition engine is used to process the language commands and obtain the text. The text is processed using the large language model Qwen to obtain natural language features. : in, Let be the semantic embedding vector of the k-th token, with dimension . , , Features of natural language The large language model is based on a multi-layer Transformer architecture and uses a self-attention mechanism to capture the contextual dependencies between language units in order to maintain temporal consistency and dynamic expressive power.

[0029] This embodiment captures the dependencies between words to obtain a richer semantic context structure.

[0030] The model interprets voice or text input commands into specific control signals. For example, if an operator says, "Let the drone fly 15 meters," the model converts the voice command into text and then uses Natural Language Processing (NLP) to analyze the semantics of the command, extracting key information such as the motion of "fly" and the distance of "15 meters." The model then translates the operator's instructions into control signals for the drone, such as adjusting its flight direction and speed. The system can create natural language feedback based on the operator's instructions and the drone's current state. If the operator's command might lead to a potential collision, the system will generate corresponding natural language feedback: "Obstacle ahead, please adjust flight direction." The model generates natural language feedback based on input environmental information (such as obstacle detection results) and operator instructions to help the operator better understand the current situation and make adjustments.

[0031] This embodiment designs a cross-modal bidirectional attention mechanism to establish an alignment relationship between the semantic space and the physical gaze point, thereby achieving deep fusion of language and eye-tracking features and temporal consistency modeling.

[0032] In some embodiments, a bidirectional attention mechanism is used to fuse the eye-tracking feature sequence and the natural language features to obtain cross-modal fusion features; including: Calculate the first attention weight : Calculate the second attention weight : in, For activation functions; Using first attention weights and second attention weights, natural language features are analyzed. With second eye movement feature sequence By performing fusion, cross-modal fusion features are obtained. ,in, For the t-th component, , For cross-modal fusion features Dimensions.

[0033] To achieve a deep collaborative understanding between natural language expression and user gaze behavior, this embodiment designs a cross-modal alignment mechanism that integrates semantic space and physical space. Based on a bidirectional attention modeling method, it realizes an explicit association between natural language tokens and eye-tracking data, thereby constructing a unified representation of user intent and significantly improving the system's context awareness and intelligent response capabilities in complex task scenarios.

[0034] This embodiment ensures that the language-gaze correspondence is reflected not only in static word meanings but also in the temporal relationship of interaction during dynamic usage. Through bidirectional attention alignment, the model can identify, for example, the dynamic mapping between "left platform" and eye-tracking history location in phrases like "fly to the left platform I just looked at." Then, the fused vector... It will serve as input to the downstream intent parsing module for tasks such as navigation target generation and control command planning, significantly improving the system's ability to understand and respond to multimodal user intents.

[0035] In some embodiments, the cross-modal fusion features are processed using a self-attention mechanism and a large language model to obtain the target location of the UAV; including: Utilizing self-attention mechanism for cross-modal feature fusion The data is processed to obtain a globally perceived representation sequence. : Among them, query vector key vector value vector ;in, , and It is a learnable matrix; The dimension of the key vector; For activation functions; Using large language models to analyze representation sequences The process is performed to obtain the target location of the drone.

[0036] This embodiment uses fused language-eye-tracking cross-modal semantic vectors to parse the user's implicit control intentions during natural interaction and output structured navigation goals and operation strategy instructions. The system core incorporates the Qwen pre-trained large language model with strong semantic understanding and decision-making capabilities to construct a reasoning path from modal fusion representation to spatial behavioral goals.

[0037] In some embodiments, a flight trajectory and corresponding flight control commands are generated based on the starting point and target position of the UAV; including: Based on the drone's starting position With the target location N trajectory points are generated using linear interpolation or Bézier curves: ,in, = , Each trajectory point contains spatial coordinates, desired velocity, and direction information. A continuous flight path is generated by fitting N trajectory points with a cubic Bézier curve. The nth trajectory point Converted into corresponding flight control commands : in, This represents the control command decoding function; flight control commands include: control commands, position adjustments, attitude angle changes, and speed; among them, control commands include: "takeoff", "hover", "fly to target", "obstacle avoidance", etc.

[0038] During the execution of flight control commands, the system can adjust the trajectory in real time based on sensor feedback, supporting dynamic obstacle avoidance and navigation correction, and ensuring stable flight of the UAV in complex environments.

[0039] Based on the above embodiments, this application provides a drone control device based on the fusion of natural language and eye-tracking data. (See attached document.) Figure 2 As shown, the UAV control device 200 based on natural language and eye-tracking data fusion provided in this application embodiment includes at least: The acquisition unit is used to acquire synchronized user eye image sequences and voice data; The first processing unit is used to process the eye image sequence to obtain an eye movement data sequence, each eye movement data including: fixation point coordinates and gaze vector; and to process the speech data using a large language model to generate natural language instructions. The second processing unit is used to process the eye-tracking data sequence using an encoder to obtain an eye-tracking feature sequence; and to process the natural language instructions using a large language model to obtain natural language features. The fusion unit is used to fuse the eye-tracking feature sequence and the natural language features using a cross-modal bidirectional attention mechanism to obtain cross-modal fused features; The third processing unit is used to process the cross-modal fusion features using a self-attention mechanism and a large language model to obtain the target position of the UAV; The control unit is used to generate flight trajectories and corresponding flight control commands based on the starting point and target position of the UAV.

[0040] It should be noted that the principle of the UAV control device 200 based on the fusion of natural language and eye-tracking data provided in this application embodiment to solve the technical problem is similar to the method provided in this application embodiment. Therefore, the implementation of the UAV control device 200 based on the fusion of natural language and eye-tracking data provided in this application embodiment can refer to the implementation of the method provided in this application embodiment, and the repeated parts will not be described again.

[0041] Based on the above embodiments, this application also provides an electronic device, see below. Figure 3 As shown, the electronic device 300 provided in this application embodiment includes at least: a processor 301, a memory 302, and a computer program stored in the memory 302 and capable of running on the processor 301. When the processor 301 executes the computer program, it implements the UAV control method based on natural language and eye-tracking data fusion provided in this application embodiment.

[0042] The electronic device 300 provided in this application embodiment may further include a bus 303 connecting different components (including processor 301 and memory 302). The bus 303 represents one or more types of bus structures, including memory bus, peripheral bus, local area bus, etc.

[0043] The memory 302 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 3021 and / or cache memory 3022, and may further include read-only memory (ROM) 3023.

[0044] The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, including but not limited to: an operating subsystem, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0045] Electronic device 300 can also communicate with one or more external devices 304 (e.g., keyboard, remote control, etc.), and with one or more devices that enable a user to interact with electronic device 300 (e.g., mobile phone, computer, etc.), and / or with any device that enables electronic device 300 to communicate with one or more other electronic devices 300 (e.g., router, modem, etc.). This communication can be performed through input / output (I / O) interface 305. Furthermore, electronic device 300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 306. Figure 3 As shown, network adapter 306 communicates with other modules of electronic device 300 via bus 303. It should be understood that, although... Figure 3 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) subsystems, tape drives, and data backup storage subsystems.

[0046] It should be noted that, Figure 3 The electronic device 300 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0047] This application also provides a computer-readable storage medium storing computer instructions. When executed by a processor, these computer instructions implement the UAV control method based on natural language and eye-tracking data fusion provided in this application. Specifically, the executable program can be built into or installed in an electronic device 300, so that the electronic device 300 can implement the UAV control method based on natural language and eye-tracking data fusion provided in this application by executing the built-in or installed executable program.

[0048] The drone control method based on the fusion of natural language and eye-tracking data provided in this application embodiment can also be implemented as a program product. The program product includes program code. When the program product can run on the electronic device 300, the program code is used to make the electronic device 300 execute the drone control method based on the fusion of natural language and eye-tracking data provided in this application embodiment.

[0049] The program product provided in this application embodiment can be any combination of one or more readable media, wherein the readable media can be a readable signal medium or a readable storage medium, and the readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0050] The program product provided in this application embodiment can be a CD-ROM and include program code, and can also run on a computing device. However, the program product provided in this application embodiment is not limited thereto. In this application embodiment, the readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0051] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0052] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0053] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

[0054] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. Multitasking and parallel processing may be advantageous in certain environments. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

Claims

1. A method for controlling unmanned aerial vehicles (UAVs) based on the fusion of natural language and eye-tracking data, characterized in that, include: Acquire synchronized user eye image sequences and voice data; The eye image sequence is processed to obtain an eye movement data sequence, each eye movement data includes: fixation point coordinates and gaze vector; the speech data is processed using a large language model to generate natural language commands; The eye-tracking data sequence is processed using an encoder to obtain an eye-tracking feature sequence; the natural language instructions are processed using a large language model to obtain natural language features. By utilizing a bidirectional attention mechanism, the eye-tracking feature sequence and the natural language features are fused to obtain cross-modal fusion features; The cross-modal fusion features are processed using a self-attention mechanism and a large language model to obtain the target location of the UAV; The flight trajectory and corresponding flight control commands are generated based on the starting point and target position of the UAV.

2. The method according to claim 1, characterized in that, The eye-tracking data sequence is processed using an encoder to obtain an eye-tracking feature sequence, including: Eye-tracking data sequence Represented as: in, ) is the first Two-dimensional coordinates of the fixation point in an eye-tracking dataset. Let i be the gaze vector in the i-th eye-tracking data; , The number of eye-tracking data; Eye-tracking data sequences are processed by an encoder. Encode the first eye movement feature sequence to obtain the first eye movement feature sequence. : in, For the first First eye movement feature; No. Location encoding of each first eye movement feature: in, It is the first Encoding the position of the first eye movement feature in the j-th dimension; It is the frequency parameter in position coding: in, It is the dimension of the first eye movement feature. It is the first The moment of first eye movement feature; Then the first Second eye movement feature for: Second eye-tracking feature sequence for: 。 3. The method according to claim 2, characterized in that, Natural language instructions are processed using a large language model to obtain natural language features, including: The speech recognition engine is used to process the language commands and obtain the text. Text is processed using large language models to obtain natural language features. : in, Let be the semantic embedding vector of the k-th token, with dimension . , , Features of natural language The large language model is based on a multi-layer Transformer architecture and captures the contextual dependencies between language units through a self-attention mechanism.

4. The method according to claim 3, characterized in that, Using a bidirectional attention mechanism, the eye-tracking feature sequence and the natural language features are fused to obtain cross-modal fusion features; including: Calculate the first attention weight : Calculate the second attention weight : in, For activation functions; Using first attention weights and second attention weights, natural language features are analyzed. With second eye movement feature sequence By performing fusion, cross-modal fusion features are obtained. ,in, For the t-th component, , For cross-modal fusion features Dimensions.

5. The method according to claim 4, characterized in that, The cross-modal fusion features are processed using a self-attention mechanism and a large language model to obtain the target location of the UAV; including: Utilizing self-attention mechanism for cross-modal feature fusion The data is processed to obtain a globally perceived representation sequence. : Among them, query vector key vector value vector ;in, , and It is a learnable matrix; The dimension of the key vector; For activation functions; Using large language models to analyze representation sequences The process is performed to obtain the target location of the drone.

6. The method according to claim 5, characterized in that, Generate a flight trajectory and corresponding flight control commands based on the drone's starting and target positions; including: Based on the drone's starting position With the target location N trajectory points are generated using linear interpolation or Bézier curves: ,in, = , Each trajectory point contains spatial coordinates, desired velocity, and direction information. A continuous flight path is generated by fitting N trajectory points with a cubic Bézier curve. The nth trajectory point Converted into corresponding flight control commands : in, This represents the control command decoding function; flight control commands include: control commands, position adjustments, attitude angle changes, and speed.

7. A drone control device based on the fusion of natural language and eye-tracking data, characterized in that, include: The acquisition unit is used to acquire synchronized user eye image sequences and voice data; The first processing unit is used to process the eye image sequence to obtain an eye movement data sequence, each eye movement data including: fixation point coordinates and gaze vector; and to process the speech data using a large language model to generate natural language instructions. The second processing unit is used to process the eye-tracking data sequence using an encoder to obtain an eye-tracking feature sequence; and to process the natural language instructions using a large language model to obtain natural language features. The fusion unit is used to fuse the eye-tracking feature sequence and the natural language features using a bidirectional attention mechanism to obtain cross-modal fusion features; The third processing unit is used to process the cross-modal fusion features using a self-attention mechanism and a large language model to obtain the target position of the UAV; The control unit is used to generate flight trajectories and corresponding flight control commands based on the starting point and target position of the UAV.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-6.

Images