Unmanned aerial vehicle body intelligent operation method based on visual feedback and behavior tree control, readable storage medium and program product

Abstract

The invention belongs to the technical field of unmanned aerial vehicle control, and particularly relates to an unmanned aerial vehicle body intelligent operation method based on visual feedback and behavior tree control, a readable storage medium and a program product. The method comprises the following steps: firstly, optimizing a large-scale language model through prompt word engineering and instruction fine tuning, and converting a user password into an unmanned aerial vehicle instruction; then obtaining a potential risk target in the route based on a visual target detection algorithm of ensemble learning; then, behavior tree nodes are designed, and a behavior tree system architecture is designed from top to bottom; and finally, information interaction among the modules is carried out based on a robot operation system, and stable operation of the unmanned aerial vehicle body intelligent algorithm system is realized. According to the method, the unmanned aerial vehicle can effectively understand user instructions, intelligently detect risk points in the cruising process and autonomously execute unmanned aerial vehicle tasks such as fly-around under the condition that no waypoint is preset, dynamic environments and complex tasks are effectively dealt with, and intelligent application of the unmanned aerial vehicle is promoted.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicle control, and more specifically, relates to an unmanned aerial vehicle embodied intelligent computing method based on visual feedback and behavior tree control, a readable storage medium and a program product. Background Art

[0002] With the rapid development of drone technology, drones are increasingly used in military reconnaissance, civil aerial photography, logistics distribution, environmental monitoring and other fields. However, drones often face complex and changing environments and uncertain factors when performing tasks, such as obstacle avoidance, target tracking, flight route planning, etc., which puts higher requirements on the intelligence and autonomy of drones.

[0003] Traditional UAV control systems mainly rely on pre-programmed flight instructions and simple sensor data, such as flight modes based on pilot pre-flight and preset GPS route control, which makes it difficult to achieve adaptive and intelligent decision-making in complex environments. In recent years, vision-based UAV navigation and target recognition technology has received widespread attention, but most methods still remain at the level of simple image processing and target tracking, lacking advanced intelligent decision-making and behavior planning capabilities. In addition, existing UAV control systems usually adopt a single control strategy, which is difficult to cope with a variety of different tasks and scenarios. Behavior tree control theory provides a structured and hierarchical control framework that can dynamically adjust the control strategy according to task requirements and environmental changes, and realize intelligent decision-making and flexible control of UAVs.

[0004] Chinese patent document CN115525061A discloses a multi-UAV cooperative control method based on graph theory, comprising the following steps: step S1, determining the communication network diagram of the multi-UAV through the communication relationship between each UAV of the multi-UAV; step S2, generating a multi-UAV control relationship tree through the communication network diagram; step S3, determining the reference point of the relative motion control of each UAV through the control relationship tree generated in step S2; step S4, determining the control relationship between each UAV to achieve multi-UAV cooperative control. Step S5, when an abnormality occurs in a UAV participating in a cooperative task, the control relationship tree changes, resulting in some UAVs having no reference point for relative motion control. At this time, cooperative control is achieved by reconstructing the control relationship tree. This method still remains at the level of simple image processing and target tracking, lacking advanced intelligent decision-making and behavior planning capabilities.

[0005] Chinese patent document CN117369795A discloses a behavior tree model reachability analysis method and device for UAV decision-making, wherein the method comprises: obtaining a behavior tree constructed based on the decision-making behavior of the UAV, wherein each node in the behavior tree corresponds to the decision-making behavior of the UAV for various factors of the external situation; converting the behavior tree into a control flow graph; based on the control flow graph, performing a node reachability analysis of the behavior tree to obtain a reachability result of the node of the behavior tree; the reachability result is used to reflect whether the decision of the UAV represented by the node of the behavior tree can be executed, but the method described in the patent is only a UAV reachability analysis, which can only provide the UAV with judgment information on whether it can reach the specified area, but cannot realize the embodied intelligent control of the UAV, that is, it cannot realize autonomous flight based on the behavior tree.

[0006] In view of this, the present invention designs an embodied intelligent computing method for a UAV based on visual feedback and behavior tree control, a computer-readable storage medium, and a computer program product to enhance the autonomy and intelligence of the UAV in a complex environment. Summary of the invention

[0007] In view of the shortcomings of the prior art, the present invention provides a UAV embodied intelligent computing method based on visual feedback and behavior tree control, which uses visual sensors to obtain environmental information in real time, and implements intelligent decision-making and behavior planning through behavior tree control theory, so that the UAV can complete tasks adaptively and improve the efficiency and safety of task execution.

[0008] The present invention also discloses a computer-readable storage medium for implementing the above method;

[0009] The invention also discloses a program product for realizing the method.

[0010] To achieve the above purpose, the technical solution adopted by the present invention is:

[0011] A method for embodied intelligent computing of a drone based on visual feedback and behavior tree control, the method comprising the following steps:

[0012] S1: Optimize the large-scale language model through prompt word engineering and command fine-tuning, decompose user commands into machine commands that the drone can understand and execute, and provide good information support for the subsequent autonomous flight and mission execution of the drone;

[0013] S2: Utilizes the target recognition algorithm based on ensemble learning, fully integrates the advantages of different visual models, and identifies potential risk targets, such as dangerous fishing, flames, and radiation. This can provide timely and accurate risk information for subsequent visual information-based modules, and assist the behavior tree in making more informed decisions.

[0014] S3: In order to build a flexible and adaptable drone behavior system, a series of behavior tree nodes with visual feedback information as the core are designed. The behavior tree nodes will make conditional judgments based on different risk targets and trigger corresponding behavior tasks. These function-rich nodes will become the basic units for building a complete behavior tree, ensuring that the drone can flexibly respond and complete tasks in various complex scenarios.

[0015] S4: Build a complete behavior tree from top to bottom by using behavior tree nodes. The behavior tree nodes include sequence nodes, selection nodes, condition nodes, and action nodes, providing a top-level logic control framework for the drone, so that the drone can follow a reasonable logic flow when performing tasks, and providing a flexible expansion mechanism, so that the behavior tree can be customized and expanded according to actual needs.

[0016] S5: Based on the robot operating system ROS, information communication between modules is realized, including real-time communication and data sharing between the drone and the ground master station, computing power module and sensors, and interconnection between the dual versions ROS1 and ROS2 is realized based on ROS Bridge, further improving the intelligence level of the drone and the efficiency of task execution.

[0017] Preferably, according to the present invention, step S3 includes the following detailed steps:

[0018] S31: establishing a patrol action node action_patrol, comprehensively analyzing the coupling characteristics of the patrol action node action_patrol and the sequence node squen in information release, and setting the action instructions of the patrol action node action_patrol to node state writing and patrol state display;

[0019] The patrol action node action_patrol is a child node of the selection node select, and the sequence node squen is a child node of select_transition;

[0020] S32: establishing a condition node condition_is_have_aim for monitoring the target state, the node is configured with a receiver for real-time capturing BOX information output by the target detection algorithm based on ensemble learning, the BOX information including the target identification and pixel position, and another receiver is set for receiving an indication of task completion;

[0021] S33: Create the centripetal motion action node action_goto_center, obtain the target BOX coordinate center and image center, and calculate the drone linear velocity x vector Vel from the image center to the BOX center x and the y vector Vel yThe sum of the two vectors is the combined velocity direction vector of the drone, which needs to be multiplied by the velocity coefficient to adjust it to the appropriate velocity value. The calculation formula is as follows:

[0022]

[0023] In formulas (1) and (2), ψ represents the velocity coefficient; (x image ,y image ) represents the center of the image; (x box ,y box ) represents the coordinate center of the target BOX; the x direction of the drone is in front of the nose, and the y direction is to the right of the nose, and there is a pose transformation between the pixel coordinates of the image taken by the camera;

[0024] S34: Establish the center judgment condition node condition_is_arrive_center, and calculate the pixel distance D between the center position of the BOX and the center position of the image according to the following formula center :

[0025]

[0026] In formula (3), (x image ,y image ) represents the center of the image; (x box ,y box ) represents the target BOX coordinate center;

[0027] When the distance D center When the distance D is less than the preset threshold, the condition node returns success, that is, the drone has reached the top of the target; center If it is greater than the preset threshold, the condition node returns failure, that is, the drone has not reached directly above the target;

[0028] S35: Establish the height adjustment action node action_goto_height. This node first calculates the pixel area of ​​the BOX through the upper left coordinate and the lower right coordinate of the BOX, and calculates the ratio of the pixel area of ​​the BOX to the pixel area of ​​the image. Then, the basic value of the UAV's z-direction speed is calculated by the difference between the BOX ratio and the optimal ratio. Finally, it is multiplied by the speed coefficient to obtain the final speed to be released. Through the above process, the height adjustment speed instruction Vel is obtained. z The calculation formula is as follows:

[0029]

[0030] In formula (4), ψ represents the speed coefficient, (x 1,box ,y 1,box ) represents the upper left coordinate of BOX, (x 2,box ,y 2,box) represents the lower right coordinate of BOX, W represents the length of BOX pixel area in image pixels, H represents the width of BOX pixel area in image pixels, r best Indicates the optimal ratio;

[0031] S36: Establish the height judgment condition node condition_is_ratio_right, calculate the ratio of the pixel area of ​​BOX to the pixel area of ​​the image, and compare it with the optimal ratio r best Make a difference to determine whether the difference with the optimal ratio is too large. If the difference exceeds the preset threshold, the condition node status is set to failure; otherwise, it is set to success.

[0032] S37: Create a detour action node action_goto_detour, which is used to issue detour speed instructions for the drone. Since the drone needs to rotate its fuselage to observe the target during detour, the radius setting is required when setting the detour angular velocity and linear velocity, as shown in the formula:

[0033] Vel=ω*Rad(5)

[0034] In formula (5), ω represents the angular velocity of the flight, Vel represents the linear velocity of the flight, and Rad represents the radius;

[0035] S38: Establish the detour completion judgment condition node condition_is_detour_finish, calculate the difference between the GPS position before the detour starts and the real-time GPS position, and calculate the actual position meter distance d according to the following steps gps :

[0036] First, convert the longitude and latitude into radians using the following formula:

[0037]

[0038] Then calculate the radian difference:

[0039]

[0040] Finally, the distance between two points is calculated based on the Haversine formula:

[0041]

[0042] In formulas (6) to (13), La real and Lo real Indicates real-time GPS longitude and latitude, La start and Lo start Indicates the GPS longitude and latitude at the beginning of the flight;

[0043] Metric distance d calculated from GPS difference gps, and considering the circling time constraint, the circling start time is not 0 and the circling flag is 1, and a joint judgment is made. If the circling start time is not 0 and the circling flag is 1, the circling has been completed and the return status value is success; if the circling has not been completed, the return status value is failure;

[0044] S39: Establish a detour completion publishing action node action_pub_detour_finish, set the detour completion flag topic, write a value into the message and then publish the topic, so as to realize the effective publication of the detour completion flag.

[0045] Preferably, according to the present invention, step S4 includes the following detailed steps:

[0046] S41: Design the first layer of the behavior tree, add two child nodes, select_transition and action_patrol, to the select node select; add multiple sequence nodes as child nodes to the select node select_transition, so as to make different response actions for different target IDs; at the same time, adjust the order of multiple sequence nodes of select_transition and arrange the execution priority of tasks with different target IDs; add the target feedback condition node condition_is_have_aim and the select node select1 as child nodes to one of the sequence nodes squen;

[0047] S42: Design the second layer of the behavior tree, add the sequence node squen1 and the centripetal motion action node action_goto_center to the selection node select1; add the center judgment condition node condition_is_arrive_center and the selection node select2 to the sequence node squen1;

[0048] S43: Design the third layer of the behavior tree, add the sequence node squen2 and the height adjustment action node action_goto_height to the selection node select2; add the height judgment condition node condition_is_ratio_right and the selection node select3 to the sequence node squen2;

[0049] S44: Design the fourth layer of the behavior tree, add the sequence node squen3 and the detour action node action_goto_detour to the selection node select3; add the detour completion judgment condition node condition_is_detour_finish and the detour completion publishing action node action_pub_detour_finish to the sequence node squen3;

[0050] S45: Set the target feedback condition node condition_is_have_aim, add a state setting condition in the node, and use the orbit completion flag value as a necessary condition for the node state to succeed, so as to avoid the drone under the behavior tree control falling into an endless loop of repeatedly executing tasks at the same target position;

[0051] S46: Use the "circling" instruction to lock the states of the centripetal motion and altitude adjustment action nodes to prevent the circling state from being interrupted due to the target not being in the center of the field of view or the altitude changing during the circling process;

[0052] S47: The detour flag released by the detour action node action_goto_detour, the detour completion judgment condition node condition_is_detour_finish uses a receiver to subscribe to the detour flag released by the detour action node action_goto_detour, and the metric distance d obtained by combining the GPS difference calculation gps The status of whether the circling is completed is set according to the circling time constraint, and the circling completion flag instruction is issued at the same time;

[0053] S48: Set the external node update_label. When receiving the circling completion flag instruction, immediately refresh the circling flag, and refresh the circling completion flag after 3 seconds. This can not only ensure the refresh of the flag to cope with the next round of task execution, but also fly away from the completed task area within 3 seconds to prevent repeated execution of tasks at the same target.

[0054] Preferably, according to the present invention, the robot operating system ROS includes two versions, namely ROS1 and ROS2.

[0055] Preferably, according to the present invention, step S5 includes the following detailed steps:

[0056] S51: Encapsulate the ensemble learning-based target detection algorithm with ROS1, obtain the drone camera image frame stream and publish the detection result Topic;

[0057] S52: Write behavior tree nodes based on ROS1 to implement node communication, receive GPS information from the drone and target detection information from the target detection algorithm, publish speed command Topic to control the movement of the drone, and publish route control command Topic to control the pause or continue of route movement;

[0058] S53: Obtain the drone GPS information Topic based on ROS2, and receive the drone center of mass motion speed instruction issued by the behavior tree;

[0059] S54: Based on ROS Bridge, information bridging between the dual versions of ROS1 and ROS2 is performed to achieve information interaction of the complete system.

[0060] A computer-readable storage medium stores computer instructions, which can implement the steps of the above method when executed by a processor.

[0061] A computer program product loaded with the above method comprises computer instructions, which can implement the steps of the above method when executed by a processor.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] The present invention cleverly combines the visual feedback mechanism with the control strategy driven by the behavior tree, and incorporates a large-scale language model to achieve in-depth analysis and precise decomposition of user commands. The present invention uses advanced visual feedback technology to efficiently identify and locate the category and spatial position of the target object, while relying on the behavior tree to build a high-level logical architecture to ensure that the drone exhibits excellent adaptability when facing changing environments and complex task configurations. This design fundamentally breaks through the limitations of traditional drones that rely on preset flight routes, and introduces a control framework that is both structured and clearly hierarchical. The framework can flexibly optimize and adjust the control strategy according to real-time task requirements and dynamic changes in the external environment, thereby enabling drones to have intelligent decision-making capabilities and a high degree of control flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 is a flow chart of the drone embodied intelligent computing method of the present invention;

[0065] Figure 2 It is the design diagram of the behavior tree constructed by the present invention;

[0066] Figure 3 It is the actual operation diagram of the behavior tree constructed by the present invention;

[0067] Figure 4 It is the ROS communication diagram of the actual operation of the algorithm system of the present invention;

[0068] Figure 5 It is a simulated flight diagram of the unmanned aerial vehicle of the present invention;

[0069] Figure 6 It is the actual flight diagram of the UAV of the present invention;

[0070] Figure 7 It is the autonomous flight route record of the unmanned aerial vehicle of the present invention. DETAILED DESCRIPTION

[0071] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0072] Embodiment 1,

[0073] like Figure 1 As shown, a method for embodied intelligent computing of a UAV based on visual feedback and behavior tree control comprises the following steps:

[0074] S1: Optimize the large-scale language model through prompt word engineering and command fine-tuning, decompose user commands into machine commands that the drone can understand and execute, and provide good information support for the subsequent autonomous flight and mission execution of the drone;

[0075] S2: Utilizes the target recognition algorithm based on ensemble learning, fully integrates the advantages of different visual models, and accurately identifies potential risk targets such as dangerous fishing, flames, and radiation. It can provide timely and accurate risk information for subsequent modules based on visual information, and assist the behavior tree in making more intelligent decisions.

[0076] S3: In order to build a flexible and adaptable drone behavior system, a series of behavior tree nodes with visual feedback information as the core are designed. The behavior tree nodes will make conditional judgments based on different risk targets and trigger corresponding behavior tasks. Specifically, these nodes will cover a series of drone functions, including but not limited to: centripetal movement based on visual recognition technology to approach the target, dynamic altitude adjustment to adapt to different flight environments, circling flight to obtain all-round photo data, remote communication through a shouting system, and the use of flashing lights for warnings or information transmission. These feature-rich nodes will become the basic units for building a complete behavior tree, ensuring that the drone can flexibly respond and complete tasks in various complex scenarios;

[0077] S4: Build a complete behavior tree from top to bottom using behavior tree nodes. The behavior tree nodes include sequence nodes, selection nodes, condition nodes, and action nodes. This provides a top-level logic control framework for the drone, so that the drone can follow a reasonable logic flow when performing tasks. It also provides a flexible expansion mechanism, so that the behavior tree can be customized and expanded according to actual needs.

[0078] S5: Based on the Robot Operating System (ROS), information communication between modules is realized, including real-time communication and data sharing between the UAV and the ground master station, computing power module or sensor, and interconnection between the two versions of ROS is realized based on ROS Bridge, further improving the intelligence level of the UAV and the efficiency of task execution.

[0079] S3 includes the following detailed steps:

[0080] S31: a cruise action node action_patrol is established, the coupling characteristics of the child node action_patrol of the selection node select and the child node select_transition, i.e., the sequence node squen, in information release are comprehensively analyzed, and the action instructions of the cruise action node action_patrol are set to node state writing and cruise state display;

[0081] S32: a target feedback condition node condition_is_have_aim is established, a receiver is created in the node for receiving BOX information returned by the visual target detection algorithm based on ensemble learning in real time, including target ID, pixel coordinates, etc., and another receiver is created to wait for the task completion instruction;

[0082] S33: The centripetal motion action node action_goto_center is established to obtain the target BOX coordinate center and the image center, and the linear velocity x vector and y vector of the drone pointing from the image center to the BOX center are calculated. The sum of the two vectors is the combined velocity direction vector of the drone, which needs to be multiplied by the velocity coefficient ψ to adjust it to a suitable velocity value;

[0083] S34: A center judgment condition node condition_is_arrive_center is established to calculate the pixel distance between the center position of the BOX and the center position of the image. When the distance is less than the threshold value 20, the condition node returns success, that is, the drone has arrived directly above the target; when the distance is greater than the threshold value 20, the condition node returns failure, that is, the drone has not arrived directly above the target;

[0084] S35: The height adjustment action node action_goto_height is established. This node first calculates the pixel area of ​​the BOX through the upper left coordinate and the lower right coordinate of the BOX, and calculates the ratio of the pixel area of ​​the BOX to the pixel area of ​​the image. Then, the basic value of the UAV's z-direction speed is calculated by the difference between the BOX ratio and the optimal ratio, and finally multiplied by the speed coefficient to obtain the final speed to be published.

[0085] S36: Establish a height judgment condition node condition_is_ratio_right, calculate the ratio of the pixel area of ​​the BOX to the pixel area of ​​the image, and make a difference with the optimal ratio, and judge whether the difference with the optimal ratio is too large. If the difference exceeds the threshold, the state of the condition node is set to failure; otherwise, it is set to success;

[0086] S37: The detour action node action_goto_detour is established. This node is used to issue detour speed instructions for the drone. Since the drone needs to rotate its fuselage to observe the target during detour, the radius setting is required when setting the detour angular velocity and linear velocity.

[0087] S38: Establish the detour completion judgment condition node condition_is_detour_finish, calculate the difference between the GPS position before the detour and the real-time GPS position, and calculate the metric distance of the actual position. gps , and considering the circling time constraint, the circling start time is not 0 and the circling flag is 1, and a joint judgment is made. If the circling start time is not 0 and the circling flag is 1, the circling has been completed and the return status value is success; if the circling has not been completed, the return status value is failure;

[0088] S39: Establish a detour completion publishing action node action_pub_detour_finish, set the detour completion flag topic, write a value into the message and then publish the topic, so as to realize the effective publication of the detour completion flag.

[0089] The above shows the complete steps of how the drone flies around a designated target based on visual feedback and behavior tree control after receiving the disassembly command issued by the large-scale language model LLM. The same is true for other functions such as taking pictures, shouting, and flashing lights, which will not be described here.

[0090] like Figure 2 As shown, S4 includes the following detailed steps:

[0091] S41: Designed the first layer of the behavior tree, adding two child nodes, select_transition and action_patrol, to the select node select; adding n sequence nodes squen, suaen00, etc. as child nodes to the select node select_transition, so as to respond to different target IDs with different actions; at the same time, by adjusting the order of the n sequence nodes of select_transition, the priority of task execution for different target IDs can be arranged; adding the target feedback condition node condition_is_have_aim and the select node select1 as child nodes to the sequence node squen;

[0092] S42: Designed the second layer of the behavior tree, added the sequence node squen1 and the centripetal motion action node action_goto_center to the selection node select1; added the center judgment condition node condition_is_arrive_center and the selection node select2 to the sequence node squen1;

[0093] S43: Designed the third layer of the behavior tree, added the sequence node squen2 and the height adjustment action node action_goto_height to the selection node select2; added the height judgment condition node condition_is_ratio_right and the selection node select3 to the sequence node squen2;

[0094] S44: Designed the fourth layer of the behavior tree, added the sequence node squen3 and the detour action node action_goto_detour to the selection node select3; added the detour completion judgment condition node condition_is_detour_finish and the detour completion publishing node action_pub_detour_finish to the sequence node squen3;

[0095] S45: Add a state setting condition in the target feedback condition node condition_is_have_aim, that is, take the value of the circling completion flag as a necessary condition for the success of the node state, so as to avoid the drone under the behavior tree control falling into an endless loop of repeatedly executing tasks at the same target position;

[0096] S46: Use the "circling" instruction to lock the states of the centripetal motion and altitude adjustment action nodes to prevent the circling state from being interrupted due to the target not being in the center of the field of view or the altitude changing during the circling process;

[0097] S47: The detour completion judgment condition node condition_is_detour_finish needs to use a receiver to subscribe to the detour flag released by the detour action node action_goto_detour, and set the state of whether the detour is completed by combining the metric distance calculated by the GPS difference value and the constraints between detours. If the detour start time is not 0 and the detour flag is 1, the detour is completed, and the detour completion flag instruction is released at the same time;

[0098] S48: Set the external node update_label. When the circling completion flag is received, the circling flag is refreshed immediately, and the circling completion flag is refreshed after 3 seconds. This can ensure the refresh of the flag to cope with the next round of task execution, and fly away from the completed task area within 3 seconds to prevent repeated execution of tasks at the same target.

[0099] like Figure 3 and Figure 4 As shown, it is the actual operation diagram of the behavior tree constructed by the present invention and the actual operation ROS communication diagram of the algorithm system of the present invention, including the following detailed steps:

[0100] S51: Encapsulate the ensemble learning-based target detection algorithm with ROS1, obtain the drone camera image frame stream and publish the detection result Topic;

[0101] S52: Write internal nodes of the behavior tree based on ROS1 to realize node communication, receive GPS information from the drone and target detection information from the target detection algorithm, publish speed command Topic to control the center of mass movement of the drone, and publish route control command Topic to control the pause or continue of route movement;

[0102] S53: Obtain the drone GPS information Topic based on ROS2, and receive the drone center of mass motion speed instruction issued by the behavior tree;

[0103] S54: Based on ROS Bridge, information bridging between two versions of ROS is performed to achieve information interaction of the complete system;

[0104] S55: By setting it in the target feedback condition node condition_is_have_aim, it is made to issue a route control flag instruction before the target is found and the task completion instruction is not received. At the same time, considering the default value of the message data in the ROS topic, it is determined that the flag has a value of 1 to indicate route interruption and a value of 0 to indicate route continuation.

[0105] Figure 5 and Figure 6They are respectively the simulated flight graph and the actual flight graph of the UAV of the present invention, which shows that the UAV embodied intelligent control algorithm system based on behavior tree can support the UAV to realize autonomous flight in simulated environment and actual scene.

[0106] Figure 7 It is a record of the autonomous flight route of the UAV, which shows that the present invention can realize task decomposition and high-level understanding based on the large model, discover specific targets in combination with target recognition, and realize intelligent control of the UAV based on the designed flexible and adaptable UAV behavior tree, and can realize the autonomous circumnavigation function after discovering the specific target, that is, realize the autonomous flight of the UAV based on the behavior tree. The method of the present invention can realize autonomous circumnavigation (red route) after discovering the specific target, while other methods can only realize straight-line flight (yellow route) based on waypoints (i.e., numbers 0 and 1 in the figure), and cannot make maneuverable responses to specific targets.

[0107] Embodiment 2,

[0108] This embodiment also provides a machine-readable storage medium storing executable instructions, which, when executed, enable the machine to execute the above-mentioned drone embodied intelligent computing method based on visual feedback and behavior tree control.

[0109] Specifically, a system or device equipped with a readable storage medium can be provided, on which software program codes that implement the functions of any of the above-mentioned embodiments are stored, and a computer or processor of the system or device can read and execute instructions stored in the readable storage medium.

Claims

1. A method for embodied intelligent computing of unmanned aerial vehicles based on visual feedback and behavior tree control, characterized in that: The method comprises the following steps: S1: Optimize large-scale language models through prompt word engineering and command fine-tuning to break down user commands into machine commands that the drone can understand and execute; S2: Use the target recognition algorithm based on ensemble learning to fully integrate the advantages of different visual models and identify potential risk targets; S3: Design a behavior tree node with visual feedback information as the core as the basic unit of the complete behavior tree. The behavior tree node will make conditional judgments based on different risk targets and trigger corresponding behavior tasks; S4: constructing a complete behavior tree from top to bottom using the behavior tree nodes, wherein the behavior tree nodes include sequence nodes, selection nodes, condition nodes, and action nodes; S5: Based on the robot operating system ROS, information communication between modules is realized, including real-time communication and data sharing between the drone and the ground master station, computing power module and sensors, and interconnection between the two versions ROS1 and ROS2 is realized based on ROS Bridge.

2. The method for embodied intelligent computing of a UAV based on visual feedback and behavior tree control according to claim 1, characterized in that: The step S3 includes the following detailed steps: S31: establishing a patrol action node action_patrol, comprehensively analyzing the coupling characteristics of the patrol action node action_patrol and the sequence node squen in information release, and setting the action instructions of the patrol action node action_patrol to node state writing and patrol state display; The patrol action node action_patrol is a child node of the selection node select, and the sequence node squen is a child node of select_transition; S32: establishing a condition node condition_is_have_aim for monitoring the target state, the node is configured with a receiver for real-time capturing BOX information output by the target detection algorithm based on ensemble learning, the BOX information including the target identification and pixel position, and another receiver is set for receiving an indication of task completion; S33: Create the centripetal motion action node action_goto_center, obtain the target BOX coordinate center and image center, and calculate the drone linear velocity x vector Vel from the image center to the BOX center x and the y vector Vel y The sum of the two vectors is the combined velocity direction vector of the drone, which needs to be multiplied by the velocity coefficient to adjust it to the appropriate velocity value. The calculation formula is as follows: In formulas (1) and (2), ψ represents the velocity coefficient; (x image ,y image ) represents the center of the image; (x box ,y box ) represents the coordinate center of the target BOX; the x direction of the drone is in front of the nose, and the y direction is to the right of the nose, and there is a pose transformation between the pixel coordinates of the image taken by the camera; S34: Establish the center judgment condition node condition_is_arrive_center, and calculate the pixel distance D between the center position of the BOX and the center position of the image according to the following formula center : In formula (3), (x image ,y image ) represents the center of the image; (x box ,y box ) represents the target BOX coordinate center; When the distance D center When the distance D is less than the preset threshold, the condition node returns success, that is, the drone has reached the top of the target; center If it is greater than the preset threshold, the condition node returns failure, that is, the drone has not reached directly above the target; S35: Establish the height adjustment action node action_goto_height. This node first calculates the pixel area of ​​the BOX through the upper left coordinate and the lower right coordinate of the BOX, and calculates the ratio of the pixel area of ​​the BOX to the pixel area of ​​the image. Then, the basic value of the UAV's z-direction speed is calculated by the difference between the BOX ratio and the optimal ratio. Finally, it is multiplied by the speed coefficient to obtain the final speed to be released. Through the above process, the height adjustment speed instruction Vel is obtained. z The calculation formula is as follows: In formula (4), ψ represents the speed coefficient, (x 1,b0x ,y 1,box ) represents the upper left coordinate of BOX, (x 2,box ,y 2,box ) represents the lower right coordinate of BOX, W represents the length of BOX pixel area in image pixels, H represents the width of BOX pixel area in image pixels, r best Indicates the optimal ratio; S36: Establish the height judgment condition node condition_is_ratio_right, calculate the ratio of the pixel area of ​​BOX to the pixel area of ​​the image, and compare it with the optimal ratio r best Take the difference. If the difference exceeds the preset threshold, the condition node status is set to failure; otherwise, it is set to success. S37: Create a detour action node action_goto_detour, which is used to issue detour speed instructions for the drone. When setting the detour angular velocity and linear velocity, the radius setting needs to be constrained, as shown in the formula: Vel=ω*Rad(5) In formula (5), ω represents the angular velocity of the flight, Vel represents the linear velocity of the flight, and Rad represents the radius; S38: Establish the detour completion judgment condition node condition_is_detour_finish, calculate the difference between the GPS position before the detour starts and the real-time GPS position, and calculate the actual position distance d in meters gps ; The actual position distance in meters calculated based on the GPS difference gps and the circling time constraint, and make a joint judgment based on the circling start time not being 0 and the circling flag being 1. If the circling start time is not 0 and the circling flag is 1, the circling has been completed and the return status value is success; if the circling has not been completed, the return status value is failure; S39: Establish a detour completion publishing action node action_pub_detour_finish, set the detour completion flag topic, write a value into the message and then publish the topic, so as to realize the effective publication of the detour completion flag.

3. The method for embodied intelligent computing of a UAV based on visual feedback and behavior tree control according to claim 2, characterized in that: The calculated actual position distance d in meters gps , including the following steps: First, convert the longitude and latitude into radians using the following formula: Then calculate the radian difference: Finally, the distance between two points is calculated based on the Haversine formula: In formulas (6) to (13), La real and Lo real Indicates real-time GPS longitude and latitude, La start and Lo start Indicates the GPS longitude and latitude at the beginning of the orbit.

4. The method for embodied intelligent computing of a UAV based on visual feedback and behavior tree control according to claim 1, characterized in that: The step S4 includes the following detailed steps: S41: Design the first layer of the behavior tree, add two child nodes, select_transition and action_patrol, to the select node select; add multiple sequence nodes as child nodes to the select node select_transition, so as to make different response actions for different target IDs; at the same time, adjust the order of multiple sequence nodes of select_transition; add the target feedback condition node condition_is_have_aim and the select node select1 as child nodes to one of the sequence nodes squen; S42: Design the second layer of the behavior tree, add the sequence node squen1 and the centripetal motion action node action_goto_center to the selection node select1; add the center judgment condition node condition_is_arrive_center and the selection node select2 to the sequence node squen1; S43: Design the third layer of the behavior tree, add the sequence node squen2 and the height adjustment action node action_goto_height to the selection node select2; add the height judgment condition node condition_is_ratio_right and the selection node select3 to the sequence node squen2; S44: Design the fourth layer of the behavior tree, add the sequence node squen3 and the detour action node action_goto_detour to the selection node select3; add the detour completion judgment condition node condition_is_detour_finish and the detour completion publishing action node action_pub_detour_finish to the sequence node squen3; S45: setting the target feedback condition node condition_is_have_aim, adding a state setting condition in the node, and taking the value of the flag bit of the fly-around completion as a necessary condition for the success of the node state; S46: Use the "circling" instruction to lock the states of the centripetal motion and altitude adjustment action nodes; S47: The detour flag released by the detour action node action_goto_detour, the detour completion judgment condition node condition_is_detour_finish uses a receiver to subscribe to the detour flag released by the detour action node action_goto_detour, and the metric distance d obtained by combining the GPS difference calculation gps The status of whether the circling is completed is set according to the circling time constraint, and the circling completion flag instruction is issued at the same time; S48: Set the external node update_label. When receiving the circling completion flag instruction, immediately refresh the circling flag, and refresh the circling completion flag after 3 seconds.

5. The method for embodied intelligent computing of a UAV based on visual feedback and behavior tree control according to claim 1, characterized in that: The robot operating system ROS includes two versions, namely ROS1 and ROS2.

6. The method for embodied intelligent computing of a drone based on visual feedback and behavior tree control according to claim 1, characterized in that: The step S5 comprises the following detailed steps: S51: Encapsulate the ensemble learning-based target detection algorithm with ROS1, obtain the drone camera image frame stream and publish the detection result Topic; S52: Write behavior tree nodes based on ROS1, receive GPS information from the drone and target detection information from the target detection algorithm, publish speed command Topic to control the movement of the drone, and publish route control command Topic to control the pause or continue of route movement; S53: Obtain the drone GPS information Topic based on ROS2, and receive the drone center of mass motion speed instruction issued by the behavior tree; S54: Based on ROS Bridge, information bridging between the dual versions of ROS1 and ROS2 is performed to achieve information interaction of the complete system.

7. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, they are used to implement the steps of the method as described in any one of claims 1 to 6.

8. A computer program product comprising computer instructions, characterized in that: When the computer instruction is executed by a processor, it is used to implement the steps of the method according to any one of claims 1 to 6.

Images