Unmanned aerial vehicle interactive complex path planning method based on large model

Abstract

The invention discloses an unmanned aerial vehicle interactive complex path planning method based on a large model, and the method comprises the following steps: selecting a dialogue type large model, and calling a cloud voice recognition service to obtain a user voice instruction text; establishing an unmanned aerial vehicle knowledge base for communication; defining a large model as an operation assistant of the unmanned aerial vehicle and a complex path planning rule; based on a preset unmanned aerial vehicle knowledge base, semantic analysis and reasoning are carried out on a natural language instruction through the large model, and a structured flight control instruction sequence is generated; and a multi-thread architecture is used for establishing and keeping communication connection with the flight control of the unmanned aerial vehicle and processing interaction between a user and the large model. According to the invention, interactive complex path planning of the unmanned aerial vehicle is realized, and technical development and application popularization of the unmanned aerial vehicle are promoted.

Description

Technical Field

[0001] This invention provides an interactive complex path planning method for unmanned aerial vehicles (UAVs) based on a large model, belonging to the technical field of UAVs. Background Technology

[0002] Unmanned aerial vehicles (UAVs), also known as drones, are unmanned aircraft controlled by radio remote control devices or built-in programs. In recent years, with the development of sensor technology, flight control algorithms, and communication technology, UAVs have been widely used in military, agriculture, logistics, disaster relief, environmental monitoring, and other fields.

[0003] Traditional drones mainly rely on manual remote control. Operators control the speed of the drone and plan its flight path in real time through a ground control station. This method requires a high level of skill from the operators.

[0004] With the rapid development of artificial intelligence and machine learning technologies, drones are gradually evolving from simple remote control operations to intelligent and autonomous systems. Modern drones can collect environmental data through sensors, cameras, and other devices, and use algorithms for real-time analysis and decision-making.

[0005] To lower the barriers to entry and reduce the difficulty of using drones, and to promote the development of drone technology, this invention proposes an interactive complex path planning method for drones based on a large model. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention aims to provide an interactive complex path planning method for UAVs based on large models. This method addresses complex path planning, which cannot be directly drawn using single waypoint coordinates or simple straight line or arc commands, thereby promoting the development of UAV technology.

[0007] Technical Solution: This invention provides an interactive complex path planning method for unmanned aerial vehicles (UAVs) based on a large model, comprising the following steps: S1. Select a large conversational model and connect to the cloud-based speech recognition service; S2. Establish a drone knowledge base for communication; S3. Configure the large model as an operation assistant for the drone and set rules for complex path planning; S4. Based on the preset UAV knowledge base, the large model is used to perform semantic parsing and reasoning on natural language commands to generate a structured flight control command sequence; S5. Based on a multi-threaded architecture, it establishes and maintains a communication connection with the UAV flight control system and handles user interactions with the large model.

[0008] Furthermore, the speech recognition service mentioned in step S1 is an online speech recognition service.

[0009] Furthermore, the drone knowledge base mentioned in step S2 includes: 1) To address situations where instructions are unclear, rules are established for large-scale models to proactively request flight distance, angle, and altitude parameters from users when controlling drones; 2) A rule stipulates that large models are only allowed to call predefined UAV control functions; 3) Function definitions for routine operation of the UAV, including at least: unlock function, take-off function, positioning function to obtain current position, waypoint flight function, hovering and waiting function, landing function, yaw setting function, yaw angle acquisition function, return-to-home function, flight along preset trajectory function, forward, backward, left and right movement function, flight mode switching function, and flight speed adjustment function; 4) Dialogue templates and examples for complex path planning; 5) Execution logic of complex path planning, including the generation and issuance order of instructions after path planning is completed; 6) The altitude coordinates are defined using the northeast-northeast coordinate system to represent the UAV's position; 7) A set of UAV control functions based on the UAV communication protocol, which are encapsulated into a class for the large model to call.

[0010] Furthermore, in step S3, the large model is configured as a drone operation assistant by: pre-setting prompt information, and when receiving user task instructions, all reasoning behaviors of the large model are based on the drone knowledge base.

[0011] Furthermore, in step S4, the large language model calls a preset function in the UAV knowledge base to generate a structured flight control command sequence based on the natural language instructions input by the user, and interprets the flight control command sequence in natural language for the user to confirm.

[0012] Furthermore, in step S5, the multi-threaded architecture includes: The first thread specifies a fixed port during initialization, establishes a TCP communication connection with the UAV flight controller, and updates the UAV status and handles communication with the UAV periodically in a loop. The second thread receives and recognizes the user's voice input in a loop, sends the recognized text to the large model, and receives the response from the large model.

[0013] Beneficial effects: Compared with the prior art, the significant effect of this invention is that by providing a user-friendly way to operate drones for non-drone fields and people who have never been exposed to drones, through interactive complex path planning based on large models, it promotes the development of drone technology. Attached Figure Description

[0014] Figure 1 This is a general framework diagram of the principle of interactive complex path planning for UAVs based on a large model.

[0015] Figure 2 Pitch angle variation diagram for flight test of UAV interactive complex path planning method based on large model;

[0016] Figure 3 A chart showing the roll angle variation during a flight test of a large-model-based interactive complex path planning method for unmanned aerial vehicles (UAVs).

[0017] Figure 4 A yaw angle variation diagram for a flight test of an interactive complex path planning method for UAVs based on a large model;

[0018] Figure 5 A graph showing the longitudinal velocity variation during a flight test of an interactive complex path planning method for UAVs based on a large model.

[0019] Figure 6 A diagram showing the lateral velocity variation during a flight test of an interactive complex path planning method for UAVs based on a large model.

[0020] Figure 7 Vertical velocity variation diagram for flight test of UAV interactive complex path planning method based on large model;

[0021] Figure 8 A longitudinal position variation diagram for a flight test of an interactive complex path planning method for UAVs based on a large model;

[0022] Figure 9 A lateral position change diagram for a flight test of an interactive complex path planning method for UAVs based on a large model;

[0023] Figure 10 Vertical position variation diagram for flight test of UAV interactive complex path planning method based on large model;

[0024] Figure 11 This is a 3D trajectory variation diagram for a flight test of an interactive complex path planning method for UAVs based on a large model. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] The UAV interactive complex path planning method based on a large model provided by this invention includes the following steps:

[0027] S1. Select a large conversational model and call the cloud-based speech recognition service;

[0028] S2. Establish a drone knowledge base based on drone communication protocols;

[0029] S3. Define the large model as the operation assistant and rules for complex path planning of the drone.

[0030] S4. Based on the preset UAV knowledge base, the large model is used to perform semantic parsing and reasoning on natural language commands to generate a structured flight control command sequence.

[0031] S5. A multi-threaded architecture is used to establish and maintain communication with the UAV flight controller and to handle user interaction with the large model.

[0032] Regarding S1), complete the configuration for dialogue with the large model, register a large model account, obtain the API key, and select the large model's (dialogue completion) API in the program; register a speech recognition service account and complete the configuration of the relevant interfaces in the program.

[0033] The steps for establishing the S2 drone knowledge base are as follows: First, if the instructions are not clear enough, the large model should actively query for details such as flight distance, angle, and altitude. Second, the large model should only call predefined functions. Third, define all functions for routine drone operations, such as unlocking and takeoff, obtaining current position, stationary flight, hovering, landing, setting yaw, obtaining yaw angle, returning to home, flying along a special trajectory, moving forward, backward, left, and right, switching flight modes, changing flight speed, etc. Fourth, provide a dialogue example for complex path planning instructions. Fifth, define the execution logic for complex path planning. Sixth, define altitude coordinates, such as using a northeast-east coordinate system. Seventh, write all functions used by the drone based on the drone communication protocol and package them into a class. Instantiate this class when the program starts executing so that the large model can call the functions.

[0034] The specific definition of S3 is as follows: You are my drone operation assistant. When I instruct you to do something, you should generate control code that uses the drone knowledge base I provide to complete the task, and then explain the purpose of this code. You are only allowed to use the functions I define for you and simple functions from the basic math library.

[0035] Regarding S4), the generation of a structured flight control command sequence through semantic parsing and reasoning of natural language commands using the aforementioned large model is specifically as follows:

[0036] Based on the user's input of natural language commands, the large model uses preset functions in the drone knowledge base to generate structured flight control commands and interprets them in natural language, allowing the user to evaluate their usability.

[0037] Regarding S5, a multi-threading library is introduced to handle multi-threading. During initialization, a thread pool instance is created to store threads for communication with the drone and for interaction between the user and the large model. Thread 1 initializes with a fixed port to establish TCP communication with the drone. This thread contains an infinite loop function used for background periodic updates to the drone's status and handling communication with it. Thread 2 is also a loop function. During initialization, it passes the large model name and the drone knowledge base as message parameters. During the loop, it receives and recognizes the user's voice input, sends it to the large model, and waits for a response. This framework can run in a simulation environment and also allows direct control of actual physical drones.

[0038] Example 1: The interactive complex path planning method for UAVs based on a large model described in this paper includes the following methods:

[0039] S1. Select a large model (GPT-4o-mini), choose chat completions as the API, and select the interaction method based on the Python platform to interact with the large model;

[0040] S1.1 Register an OpenAI account on the OpenAI website and obtain an API key. When creating an OpenAI instance, pass the API key you saved as a parameter;

[0041] S1.2 Register an Azure account on https: / / portal.azure.com / , click on the Azure AI services option and subscribe to the voice service, then save the key;

[0042] S1.3 Set the SPEECH_KEY and SPEECH_REGION environment variables. When creating a SpeechConfig instance in the Python file, pass the two environment variables from the previous step as parameters.

[0043] S1.4 To specify the audio input device, first pass the device parameter when creating the AudioConfig instance, such as use_default_microphone=True. Then, when creating the SpeechRecognizer instance, pass in the AudioConfig instance and the Speech_Config instance set in S1.3;

[0044] Once the S1.5 and SpeechRecognizer instances are running, you can begin voice input.

[0045] S2. Establish a drone knowledge base based on MAVlink for communication. The steps for establishing the drone knowledge base are as follows:

[0046] S2.1 If the instructions are not clear enough, the large model control should actively ask for details, such as flight distance, angle, altitude, etc.

[0047] S2.2 requires that large models only call predefined functions;

[0048] S2.3 defines all functions for routine drone operations, such as unlocking and takeoff, obtaining current position, stationary flight, hovering, landing, setting yaw, obtaining yaw angle, return to home, flying along a special trajectory, moving forward, backward, left, and right, switching flight modes, changing flight speed, etc. A selection of functions is provided as an example, such as:

[0049] 1) aw.arm(): Unlocks the drone.

[0050] 2) aw.takeoff(alt=10): The drone takes off, with a default takeoff altitude of 10 meters. You can change the takeoff altitude by passing the alt parameter.

[0051] 3) aw.get_drone_position(): Gets the current position of the drone. The return value is a three-dimensional vector containing the coordinates of the x, y, and z directions.

[0052] 4) aw.fly_to([x, y, z]): Flies the drone to the specified location, which consists of three parameters corresponding to the x, y, and z coordinates;

[0053] S2.4 Complex Path Planning Dialogue Example, a portion of which is selected as an example, such as:

[0054] 1) Simple instructions:

[0055] Me: Fly forward.

[0056] How far do you need to fly forward?

[0057] Me: Ten meters.

[0058] 2) Complex instructions:

[0059] Me: Please help me fly out a square shape?

[0060] You: What is the side length of the square in meters?

[0061] Me: Five meters;

[0062] S2.5 defines the execution logic for complex path planning, and a portion of it is shown as an example:

[0063] 1) When I request the drone to fly, you need to use the aw.get_drone_position() function to get the current position, calculate the target position according to my instructions, and then use aw.fly_to([x,y,z]) to fly.

[0064] 2) After the drone flies to the target point, you should use aw.wait_until_reached(point) to confirm that the drone has reached the designated location;

[0065] S2.6 Definition of altitude coordinates, such as using the northeast coordinate system;

[0066] S2.7 uses the MAVlink protocol to write all the functions used by the drone and packages them into a class. When the program starts executing, the class is instantiated so that the large model can call the functions.

[0067] S3. Define the large model as a rule for drone operation assistance and complex path planning, specifically as follows:

[0068] You are my drone operation assistant. When I ask you to do something, you should generate Python code that uses the drone knowledge base I provide to complete the task, and then explain what that code does. You are only allowed to use functions I define for you, as well as simple Python functions from libraries such as math and numpy;

[0069] S4. Based on a pre-set UAV knowledge base, the large model performs semantic parsing and reasoning on natural language commands to generate a structured flight control command sequence, specifically:

[0070] Based on the user's input of natural language commands, the large model uses preset functions in the drone knowledge base to generate structured flight control commands and interprets them in natural language, allowing the user to evaluate their usability.

[0071] S5. A multi-threaded architecture is used to establish and maintain communication with the UAV flight control system and to handle user interactions with the large model, specifically:

[0072] The Python `concurrent.futures` library is used to handle multithreading. During initialization, a `ThreadPoolExecutor` instance is created to store threads for communication with the drone and interactions between the user and the large model. Thread 1 initializes with a fixed port to establish TCP communication with the drone. This thread contains an infinite loop function used to periodically update the drone's status and handle communication with it in the background. Thread 2 is also a loop function. During initialization, it passes the large model name and the drone knowledge base as message parameters. During the loop, it receives and recognizes the user's voice input, sends it to the large model, and waits for a response.

[0073] To verify the effectiveness of the interactive complex path planning method for UAVs based on a large model, a real-world flight test was conducted. The following series of commands were input via voice: "Unlock the UAV and take off to an altitude of 15m," "Fly forward 4m and then fly out a square shape with sides of 4m centered on the current position," "Fly forward another 4m and then fly out a pentagram shape with sides of 5m centered on the current position," and "Return to home." The test flight results are as follows: Figures 2-11 As shown.

[0074] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, for the device embodiments, the above descriptions are merely preferred embodiments of the present invention. Since they are fundamentally similar to the method embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions of the method embodiments. The above descriptions are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention, without departing from the principle of the present invention, should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for interactive complex path planning of unmanned aerial vehicles based on a large model, characterized in that, Includes the following steps: S1. Select a large conversational model and connect to the cloud-based speech recognition service; S2. Establish a drone knowledge base for communication; S3. Configure the large model as an operation assistant for the drone and set rules for complex path planning; S4. Based on the preset UAV knowledge base, the large model is used to perform semantic parsing and reasoning on natural language commands to generate a structured flight control command sequence; S5. Based on a multi-threaded architecture, it establishes and maintains a communication connection with the UAV flight control system and handles user interactions with the large model.

2. The interactive complex path planning method for unmanned aerial vehicles based on a large model according to claim 1, characterized in that, The speech recognition service mentioned in step S1 is an online speech recognition service.

3. The interactive complex path planning method for unmanned aerial vehicles based on a large model according to claim 1, characterized in that, The drone knowledge base mentioned in step S2 includes: 1) To address situations where instructions are unclear, rules are established for large-scale models to proactively request flight distance, angle, and altitude parameters from users when controlling drones; 2) A rule stipulates that large models are only allowed to call predefined UAV control functions; 3) Function definitions for routine operation of the UAV, including at least: unlock function, take-off function, positioning function to obtain current position, waypoint flight function, hovering and waiting function, landing function, yaw setting function, yaw angle acquisition function, return-to-home function, flight along preset trajectory function, forward, backward, left and right movement function, flight mode switching function, and flight speed adjustment function; 4) Dialogue templates and examples for complex path planning; 5) Execution logic of complex path planning, including the generation and issuance order of instructions after path planning is completed; 6) The altitude coordinates are defined using the northeast-northeast coordinate system to represent the UAV's position; 7) A set of UAV control functions based on the UAV communication protocol, which are encapsulated into a class for the large model to call.

4. The interactive complex path planning method for UAVs based on a large model according to claim 1, characterized in that: In step S3, the large model is configured as a drone operation assistant by: pre-setting prompt information, and when receiving user task instructions, all reasoning behaviors of the large model are based on the drone knowledge base.

5. The interactive complex path planning method for unmanned aerial vehicles based on a large model according to claim 1, characterized in that: In step S4, the large language model calls a preset function in the UAV knowledge base to generate a structured flight control command sequence based on the natural language instructions input by the user, and interprets the flight control command sequence in natural language for the user to confirm.

6. The interactive complex path planning method for unmanned aerial vehicles based on a large model according to claim 1, characterized in that: In step S5, the multi-threaded architecture includes: The first thread specifies a fixed port during initialization, establishes a TCP communication connection with the UAV flight controller, and updates the UAV status and handles communication with the UAV periodically in a loop. The second thread receives and recognizes the user's voice input in a loop, sends the recognized text to the large model, and receives the response from the large model.

Images