Unmanned aerial vehicle autonomous target searching system and method based on large language model

Abstract

The invention belongs to the technical field of unmanned aerial vehicle autonomous navigation and target searching, and discloses an unmanned aerial vehicle autonomous target searching system based on a large language model, which comprises a natural language processing module, a multi-sensor fusion module, a control and flight module, a target detection module, a path planning module and a task planning and control module, according to the system, task description and state information in a natural language form are processed by using a general large language model, so that autonomous planning of the unmanned aerial vehicle and execution of a complex search task are realized; the system can operate in various complex environments, is suitable for various application scenes, such as disaster rescue, environment monitoring, patrol security and logistics transportation, and has wide application prospect and market value. Through combination of advanced technologies such as natural language processing, multi-sensor fusion and deep learning, and through organic combination of the modules, intelligentization and high efficiency of the unmanned aerial vehicle autonomous target searching system are realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous navigation and target search of unmanned aerial vehicles, and in particular relates to an autonomous target search system and method for unmanned aerial vehicles based on a large language model. Background Art

[0002] With the rapid development of drone technology, drones are increasingly used in military reconnaissance, search and rescue, environmental monitoring and other fields. Traditional drone search systems mainly rely on pre-set flight paths and target locations, lack flexibility and autonomy, and are difficult to cope with complex and changing environments and mission requirements. In recent years, with the development of artificial intelligence and deep learning technology, drones' autonomous navigation and target recognition capabilities have been significantly improved, but there are still some technical bottlenecks.

[0003] Traditional UAV systems usually rely on data from a single sensor for environmental perception and target recognition, which is easily affected by environmental noise and sensor errors, resulting in insufficient perception accuracy and robustness. At the same time, the existing path planning algorithms still need to improve their computational efficiency and real-time performance when dealing with dynamic obstacles in complex environments. In addition, existing UAV mission planning systems usually use traditional methods such as rules and finite state machines, which are difficult to handle complex and unstructured mission descriptions, limiting the autonomous decision-making capabilities and mission execution efficiency of UAVs. Summary of the invention

[0004] The purpose of the present invention is to provide a UAV autonomous target search system and method based on a large language model to solve the above-mentioned technical problems.

[0005] In order to solve the above technical problems, the specific technical solutions of the UAV autonomous target search system and method based on a large language model of the present invention are as follows:

[0006] An autonomous target search system for unmanned aerial vehicles based on a large language model, comprising a natural language processing module, a multi-sensor fusion module, a control and flight module, a target detection module, a path planning module, and a task planning and control module. The system uses a general large language model to process task descriptions and status information in natural language form, thereby enabling the unmanned aerial vehicle to autonomously plan and execute complex search tasks.

[0007] The natural language processing module is mainly composed of a general large language model, and outputs the result of task planning according to the current task goal and state information. The task goal is a text in natural language form, and the state information is usually a text in natural language form describing the current scene and other task-related information. The result of task planning is a task framework generated by the general large language model based on the input information, including task steps and execution order;

[0008] The multi-sensor fusion module is used to receive environmental data from multiple sensors and pre-process the acquired sensor data;

[0009] The control and flight module is used to receive flight instructions and control the movement of the UAV, adjust the flight trajectory of the UAV according to the predetermined path and real-time position, adjust the attitude of the UAV in real time to maintain flight stability, and manage and adjust the power system of the UAV to achieve efficient flight;

[0010] The target detection module obtains environmental data from the multi-sensor fusion module, synchronizes and fuses the multi-sensor data, processes the fused data using a deep learning algorithm to detect and identify the target, and determines the precise coordinates of the target by calculation based on the position of the detected target in the environment;

[0011] The path planning module generates the optimal path for the UAV based on the environmental data and predetermined targets generated by the multi-sensor fusion module, and detects and avoids obstacles in real time during the flight of the UAV; the mission planning and control module calls the corresponding path planning module and the control and flight module to execute the corresponding mission steps according to the mission framework, and controls the state transition according to the target detection results given by the target detection module to ensure that the task is carried out step by step.

[0012] The present invention also discloses a method for autonomous target search of unmanned aerial vehicle (UAV) based on a large language model, comprising the following steps:

[0013] S1, controls the UAV to use the lidar in the multi-sensor fusion module to model the environment map;

[0014] S2, the mission planning and control module enters the initial waiting state and controls the drone to fly to the initial designated location;

[0015] S3, after determining that the UAV has arrived at the initial designated location, the mission planning and control module enters the waiting mission framework state, receives the mission objectives and status information in natural language form by calling the natural language processing module, and generates a mission framework including mission steps and execution order;

[0016] S4, after the mission planning and control module receives the execution plan, it enters the action execution state and calls the path planning module according to the steps in the mission framework to make the drone fly to the designated search location;

[0017] S5, after determining that the UAV has reached the target location, the mission planning and control module enters the target search state, calls the path planning module, and performs a circular search at the target location;

[0018] S6, the mission planning and control module continuously calls the results of the target detection module. If the target is found, it jumps to step 7; if the target is not found after the search at the target location, it jumps to step 9;

[0019] S7, after determining that the UAV has found the target, the mission planning and control module enters the target positioning state, determines the target coordinates by calling the multi-sensor fusion module, and reports the target coordinates; then, the path planning module is called to fly to the target location;

[0020] S8, after determining that the UAV has reached the target location, the mission planning and control module enters a reset state, calls the path planning module, flies to the initial designated location, and returns to step 3 to wait for the input of the next target search mission;

[0021] S9, at this time, after searching the target location, no designated target is found, and the mission planning and control module enters the waiting mission framework state again.

[0022] Furthermore, the state equation corresponding to the environment map modeling in step 1 is as follows:

[0023] x k =f(x k-1 ,u k )+w k

[0024] Among them, x k is the state vector at time k, f(·) is the nonlinear state transfer function, u k is the control input, w k is the process noise, and the observation equation is as follows:

[0025] z k =h(x k )+v k

[0026] Among them, z k is the observed data at time k, h(·) is a nonlinear observation function, which represents the relationship from the state vector to the observed data, v k is the observation noise.

[0027] Furthermore, the UAV flight control in S2 uses PID control to ensure flying from the current position to the target location. The PID control formula is as follows:

[0028]

[0029] Among them, e(t) is the error between the current position and the target position, K p , K i , K d are the proportional, integral, and differential gains of the PID controller.

[0030] Furthermore, S3 calls the natural language processing module M n Receive the task goal g in natural language t and contains historical search information h t and scene map information t Status information t , generates an execution plan containing n task steps and execution order The formula of the natural language processing module is as follows:

[0031] p t =M n (g t ,s t ).

[0032] Furthermore, the S4 task planning and control module receives the execution plan p t After that, it enters the action execution state. According to the steps in the task framework, the path planning module is called to make the drone fly to the specified search location. The A* path planning algorithm formula is as follows:

[0033] f(n)=g(n)+h(n)

[0034] Among them, f(n) is the total estimated cost of the current node, g(n) is the actual cost from the start node to the current node, and h(n) is the estimated cost from the current node to the target node.

[0035] Furthermore, the path generation formula in S5 is as follows:

[0036] x(t)=r·cos(ωt)

[0037] y(t)=r·sin(ωt)

[0038] Where r is the radius, ω is the angular velocity, and t is the time.

[0039] Furthermore, the target detection module M in S6 d Given the target detection probability and the position of the target in the camera coordinate system, the formula is as follows:

[0040] x image ,y image ,P obj =M d (I)

[0041] Among them, (x image ,y image ) represents the coordinates of the target in the image, P obj represents the probability of the target existing, and I is the input image.

[0042] Furthermore, the camera intrinsic parameter matrix in S7 converts the pixel coordinates into normalized coordinates in the camera coordinate system, and the formula is as follows:

[0043]

[0044] Here K -1 It is the inverse matrix of the camera's intrinsic parameter matrix. The camera's extrinsic parameter matrix converts the three-dimensional coordinates in the camera coordinate system into the three-dimensional coordinates in the world coordinate system. The formula is as follows:

[0045]

[0046] Where R is the rotation matrix and t is the translation vector.

[0047] Furthermore, in S9, the current task target and execution action are added to the historical search information h t+1 , and then again receives the task target g in natural language form by calling the natural language processing module t+1 and contains historical search information h t+1 and scene map information t+1 Status information t+1 , generate the next round of execution plan containing n task steps and execution order Skip to step 4.

[0048] The invention discloses an autonomous target search system and method for unmanned aerial vehicle based on a large language model, which has the following advantages: the invention adopts a general large language model to process natural language input, so that the system can quickly and accurately understand the task target. By automatically generating the task framework and steps, human intervention is reduced, the continuous target search task is automatically completed, and the efficiency and accuracy of the target search are improved. The invention integrates multiple sensor data through a multi-sensor fusion module, realizes data denoising, calibration and synchronization, and improves the accuracy and robustness of environmental perception. The complementarity of different sensors enhances the adaptability of the system to complex environments. The invention processes the multi-sensor fusion data through a deep learning algorithm through a target detection module, and has high-precision and high-robust target detection and recognition capabilities. By accurately calculating the target position, the ability of the unmanned aerial vehicle to autonomously search for targets in complex environments is improved. The invention combines real-time environmental data and predetermined targets through a path planning module to generate an optimal flight path, and performs dynamic obstacle avoidance during flight to ensure that the unmanned aerial vehicle can reach the target position safely and quickly. The real-time adjustment capability of the module adapts to the dynamically changing environment and enhances the autonomy and flexibility of the unmanned aerial vehicle. The present invention ensures the stability and efficiency of the UAV during flight by adjusting the flight trajectory and attitude in real time through the control and flight module. By managing the power system, the flight performance is optimized, and the flight time and mission execution capability of the UAV are extended. The system of the present invention can operate in various complex environments and is suitable for a variety of application scenarios, such as disaster relief, environmental monitoring, patrol security and logistics transportation, etc., and has broad application prospects and market value. By combining advanced technologies such as natural language processing, multi-sensor fusion and deep learning, and through the organic combination of various modules, the intelligence and efficiency of the autonomous target search system of the UAV are realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram of a strategy network based on a pre-trained language model in the present invention.

[0050] Figure 2 It is a schematic diagram of the method of the present invention.

[0051] Figure 3 It is a schematic diagram of the memory adapter modification action in the present invention. DETAILED DESCRIPTION

[0052] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of an autonomous target search system and method for a drone based on a large language model in conjunction with the accompanying drawings.

[0053] like Figure 1As shown, an autonomous target search system for unmanned aerial vehicles based on a large language model of the present invention includes a natural language processing module, a multi-sensor fusion module, a control and flight module, a target detection module, a path planning module, and a task planning and control module. The system uses a general large language model to process task descriptions and status information in natural language form to enable the unmanned aerial vehicle to autonomously plan and execute complex search tasks.

[0054] The natural language processing module is mainly composed of a general large language model, and outputs the result of task planning according to the current task goal and state information. The task goal is a text in natural language form, and the state information is usually a text in natural language form describing the current scene and other task-related information. The result of task planning is a task framework generated by the general large language model based on the input information, including task steps and execution order;

[0055] The multi-sensor fusion module can receive environmental data from multiple sensors, including but not limited to cameras, lidars, infrared sensors, and inertial measurement units, and can pre-process the acquired sensor data, including denoising, calibration, and synchronization;

[0056] The control and flight module can receive flight instructions and control the movement of the UAV, can adjust the flight trajectory of the UAV according to the predetermined path and real-time position, can adjust the attitude of the UAV in real time to maintain flight stability, and can manage and adjust the power system of the UAV to achieve efficient flight;

[0057] The target detection module obtains environmental data from the multi-sensor fusion module, and can synchronize and fuse the multi-sensor data to improve the accuracy and robustness of environmental perception, can use a deep learning algorithm to process the fused data to detect and identify targets, and can determine the precise coordinates of the target by calculation based on the position of the detected target in the environment;

[0058] The path planning module is capable of generating an optimal path for the UAV based on the environmental data and predetermined targets generated by the multi-sensor fusion module, and detecting and avoiding obstacles in real time during the flight of the UAV;

[0059] The mission planning and control module can call the corresponding path planning module and the control and flight module to execute the corresponding mission steps according to the mission framework, and can control the state conversion according to the target detection result given by the target detection module to ensure that the mission is carried out step by step.

[0060] This system uses the natural language processing module as the decision-making core. It can plan the UAV's mission in real time through mission objectives and status information in natural language form, and use multi-sensor data and target detection modules to achieve environmental perception and target detection, thereby achieving efficient and accurate target search mission execution.

[0061] like Figure 2 As shown, a method for autonomous target search of a UAV based on a large language model of the present invention comprises the following steps:

[0062] S1, controls the drone to use the laser radar in the multi-sensor fusion module to model the environment map. The corresponding state equation is as follows:

[0063] x k =f(x k-1 ,u k )+w k

[0064] Among them, x k is the state vector at time k, f(·) is the nonlinear state transfer function, u k is the control input, w k is the process noise. The observation equation is as follows:

[0065] z k =h(x k )+v k

[0066] Among them, z k is the observed data at time k, h(·) is a nonlinear observation function, which represents the relationship from the state vector to the observed data, v k is the observation noise.

[0067] S2, the mission planning and control module enters the initial waiting state and controls the drone to fly to the initial designated location. UAV flight control usually uses PID control to ensure that it flies from the current location to the target location. The PID control formula is as follows:

[0068]

[0069] Among them, e(t) is the error between the current position and the target position, K p , K i , K d are the proportional, integral, and differential gains of the PID controller.

[0070] S3, after determining that the UAV has arrived at the initial designated location, the mission planning and control module enters the waiting mission framework state and calls the natural language processing module M n Receive the task goal g in natural language tand contains historical search information h t and scene map information t Status information t , generates an execution plan containing n task steps and execution order The formula of the natural language processing module is as follows:

[0071] p t =M n (g t ,s t )

[0072] S4, the mission planning and control module receives the execution plan p t After that, it enters the action execution state and calls the path planning module according to the steps in the task framework to make the drone fly to the specified search location. The A* path planning algorithm formula is as follows:

[0073] f(n)=g(n)+h(n)

[0074] Among them, f(n) is the total estimated cost of the current node, g(n) is the actual cost from the start node to the current node, and h(n) is the estimated cost from the current node to the target node.

[0075] S5, after determining that the UAV has reached the target location, the mission planning and control module enters the target search state and calls the path planning module to perform a circular search at the target location. The path generation formula is as follows:

[0076] x(t)=r·cos(ωt)

[0077] y(t)=r·sin(ωt)

[0078] Where r is the radius, ω is the angular velocity, and t is the time.

[0079] S6, the mission planning and control module continuously calls the results of the target detection module. If the target is found, it jumps to step 7; if the target is not found after the search at the target location, it jumps to step 9. d Given the target detection probability and the position of the target in the camera coordinate system, the formula is as follows:

[0080] x image ,y image ,P obj =M d (I)

[0081] Among them, (x image ,y image ) represents the coordinates of the target in the image, P obj represents the probability of the target existing, and I is the input image.

[0082] S7, after determining that the drone has found the target, the mission planning and control module enters the target positioning state, determines the target coordinates by calling the multi-sensor fusion module, and reports the target coordinates; then, it calls the path planning module to fly to the target location. The camera intrinsic parameter matrix converts the pixel coordinates into normalized coordinates in the camera coordinate system, and the formula is as follows:

[0083]

[0084] Here K -1 It is the inverse matrix of the camera's intrinsic parameter matrix. The camera's extrinsic parameter matrix is ​​used to convert the three-dimensional coordinates in the camera coordinate system into the three-dimensional coordinates in the world coordinate system. The formula is as follows:

[0085]

[0086] Where R is the rotation matrix and t is the translation vector.

[0087] S8, after determining that the UAV has reached the target location, the mission planning and control module enters the reset state, calls the path planning module, flies to the initial designated location, and returns to step 3 to wait for the input of the next target search mission.

[0088] S9: At this time, after searching the target location, the specified target is not found, and the task planning and control module enters the waiting task framework state again, adding the current task target and execution action to the historical search information h t+1 , and then again receives the task target g in natural language form by calling the natural language processing module t+1 and contains historical search information h t+1 and scene map information t+1 Status information t+1 , generate the next round of execution plan containing n task steps and execution order Skip to step 4.

[0089] like Figure 2As shown in the figure, the overall framework of the scheme is built in this form. Channel 1, the mission planning and control module transmits instructions such as building a map or obtaining environmental data to the multi-sensor fusion module, and then the multi-sensor fusion module returns a map or environmental photo of the environment; Channel 2, the mission planning and control module transmits the current mission goal and status information to the natural language processing module, and then the natural language processing module returns a task framework containing mission steps and execution order; Channel 3, the mission planning and control module transmits the destination coordinates to the path planning module, and then the path planning module returns the planning results; Channel 4, the path planning module transmits the path planning to the control and flight module, and then the control and flight module returns the flight status to the path planning module; Channel 5, the mission planning and control module transmits the search target information and environmental image data to the target detection module, and then the target detection module transmits the detection results to the mission planning and control module.

[0090] like Figure 3 As shown, one application scenario of the invention is that a drone carrying goods goes to a designated delivery point to search for a designated recipient. After the system receives a customer order, the logistics system generates a task instruction, including cargo information, delivery address, and designated recipient. The natural language processing module parses the instruction and generates a task framework: the drone takes off from warehouse A with the goods and flies to delivery point B. At delivery point B, it searches for a recipient with specified characteristics. After the search target is successfully found, it flies to the target recipient, and then the recipient unloads the goods from the drone and signs for them. Finally, the drone completes the task and returns to warehouse A.

[0091] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. An autonomous target search system for unmanned aerial vehicles based on a large language model, characterized in that: The system includes a natural language processing module, a multi-sensor fusion module, a control and flight module, a target detection module, a path planning module, and a mission planning and control module. The system uses a general large language model to process mission descriptions and status information in natural language form, thereby enabling the UAV to autonomously plan and execute complex search missions. The natural language processing module is mainly composed of a general large language model, and outputs the result of task planning according to the current task goal and state information. The task goal is a text in natural language form, and the state information is usually a text in natural language form describing the current scene and other task-related information. The result of task planning is a task framework generated by the general large language model based on the input information, including task steps and execution order; The multi-sensor fusion module is used to receive environmental data from multiple sensors and pre-process the acquired sensor data; The control and flight module is used to receive flight instructions and control the movement of the UAV, adjust the flight trajectory of the UAV according to the predetermined path and real-time position, adjust the attitude of the UAV in real time to maintain flight stability, and manage and adjust the power system of the UAV to achieve efficient flight; The target detection module obtains environmental data from the multi-sensor fusion module, synchronizes and fuses the multi-sensor data, processes the fused data using a deep learning algorithm to detect and identify the target, and determines the precise coordinates of the target by calculation based on the position of the detected target in the environment; The path planning module generates the optimal path for the UAV based on the environmental data and predetermined targets generated by the multi-sensor fusion module, and detects and avoids obstacles in real time during the flight of the UAV; the mission planning and control module calls the corresponding path planning module and the control and flight module to execute the corresponding mission steps according to the mission framework, and controls the state transition according to the target detection results given by the target detection module to ensure that the task is carried out step by step.

2. A method for autonomous target search of unmanned aerial vehicle based on large language model of unmanned aerial vehicle autonomous target search system as claimed in claim 1, characterized in that: The steps include: S1, controls the UAV to use the lidar in the multi-sensor fusion module to model the environment map; S2, the mission planning and control module enters the initial waiting state and controls the drone to fly to the initial designated location; S3, after determining that the UAV has arrived at the initial designated location, the mission planning and control module enters the waiting mission framework state, receives the mission objectives and status information in natural language form by calling the natural language processing module, and generates a mission framework including mission steps and execution order; S4, after the mission planning and control module receives the execution plan, it enters the action execution state and calls the path planning module according to the steps in the mission framework to make the drone fly to the designated search location; S5, after determining that the UAV has reached the target location, the mission planning and control module enters the target search state, calls the path planning module, and performs a circular search at the target location; S6, the mission planning and control module continuously calls the results of the target detection module. If the target is found, it jumps to step 7; if the target is not found after the search at the target location, it jumps to step 9; S7, after determining that the UAV has found the target, the mission planning and control module enters the target positioning state, determines the target coordinates by calling the multi-sensor fusion module, and reports the target coordinates; then, the path planning module is called to fly to the target location; S8, after determining that the UAV has reached the target location, the mission planning and control module enters a reset state, calls the path planning module, flies to the initial designated location, and returns to step 3 to wait for the input of the next target search mission; S9, at this time, after searching the target location, no designated target is found, and the mission planning and control module enters the waiting mission framework state again.

3. The autonomous target search method of a UAV according to claim 2, characterized in that: The state equation corresponding to the environment map modeling in step 1 is as follows: x k =f(x k-1 ,u k )+w k Among them, x k is the state vector at time k, f(·) is the nonlinear state transfer function, u k is the control input, w k is the process noise, and the observation equation is as follows: z k =h(x k )+v k Among them, z k is the observed data at time k, h(·) is a nonlinear observation function, which represents the relationship from the state vector to the observed data, v k is the observation noise.

4. The autonomous target search method of a UAV according to claim 2, characterized in that: The UAV flight control in S2 uses PID control to ensure that it flies from the current position to the target location. The PID control formula is as follows: Among them, e(t) is the error between the current position and the target position, K p , K i , K d are the proportional, integral, and differential gains of the PID controller.

5. The autonomous target search method of a UAV according to claim 2, characterized in that: S3 calls the natural language processing module M n Receive the task goal g in natural language t and contains historical search information h t and scene map information t Status information t , generates an execution plan containing n task steps and execution order The formula of the natural language processing module is as follows: p t =M n (g t ,s t )。 6. The autonomous target search method of a UAV according to claim 2, characterized in that: The S4 mission planning and control module receives the execution plan p t After that, it enters the action execution state. According to the steps in the task framework, the path planning module is called to make the drone fly to the specified search location. The A* path planning algorithm formula is as follows: f(n)=g(n)+h(n) Among them, f(n) is the total estimated cost of the current node, g(n) is the actual cost from the start node to the current node, and h(n) is the estimated cost from the current node to the target node.

7. The autonomous target search method of a UAV according to claim 2, characterized in that: The path generation formula in S5 is as follows: x(t)=r·coS(ωt) y(t)=r·sin(ωt) Where r is the radius, ω is the angular velocity, and t is the time.

8. The autonomous target search method of a UAV according to claim 2, characterized in that: The target detection module M in S6 d Given the target detection probability and the position of the target in the camera coordinate system, the formula is as follows: x image ,y image ,P obj =M d (I) Among them, (x image ,y image ) represents the coordinates of the target in the image, P obj represents the probability of the target existing, and I is the input image.

9. The autonomous target search method of a UAV according to claim 2, characterized in that: The camera intrinsic parameter matrix in S7 converts the pixel coordinates into normalized coordinates in the camera coordinate system, and the formula is as follows: Here K -1 It is the inverse matrix of the camera's intrinsic parameter matrix. The camera's extrinsic parameter matrix converts the three-dimensional coordinates in the camera coordinate system into the three-dimensional coordinates in the world coordinate system. The formula is as follows: Where R is the rotation matrix and t is the translation vector.

10. The autonomous target search method of a UAV according to claim 2, characterized in that: In S9, the current task target and execution action are added to the historical search information h t+1 , and then again receives the task target g in natural language form by calling the natural language processing module t+1 and contains historical search information h t+1 and scene map information t+1 Status information t+1 , generate the next round of execution plan containing n task steps and execution order Skip to step 4.

Images