Three-dimensional AI direction sensing method and system based on unmanned aerial vehicle

Abstract

The invention discloses a three-dimensional AI direction sensing method and system based on an unmanned aerial vehicle, and the method comprises the steps: sensing the visible light, thermal radiation, spatial distance and motion state of the surrounding environment of the unmanned aerial vehicle in real time through integrating a plurality of sensors, constructing a comprehensive and accurate three-dimensional environment map, carrying out the semantic segmentation through deep learning, recognizing an object, endowing a semantic tag, and carrying out the recognition of a target object. The environment understanding capability of the unmanned aerial vehicle is improved, dynamic object behaviors are predicted based on a machine learning prediction model, and the adaptability and safety of the unmanned aerial vehicle in a complex environment are enhanced. The self-adaptive decision tree is dynamically constructed, the path planning and flight instruction is generated according to the task demand and the prediction result, the environment change is flexibly coped, the task execution efficiency and success rate are improved, the unmanned aerial vehicle autonomously adjusts the flight, the task is completed according to the planning path, the efficient autonomous flight is realized, and the manual operation burden is reduced.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional AI perception direction technology, and in particular to a three-dimensional AI perception direction method and system based on an unmanned aerial vehicle. Background Art

[0002] With the rapid development of drone technology, drones are increasingly used in military reconnaissance, environmental monitoring, logistics distribution, agricultural plant protection and other fields. However, the autonomous navigation and obstacle avoidance capabilities of drones are one of the key technologies to realize these applications. Traditional drone navigation systems mainly rely on GPS signals and preset flight paths, but in complex and changeable environments, such as urban canyons, forested areas or severe weather conditions, GPS signals may be interfered with or blocked, resulting in the inability of drones to navigate accurately.

[0003] In addition, traditional drone obstacle avoidance methods are mostly based on simple sensor data (such as ultrasonic or infrared obstacle avoidance sensors) to detect and avoid obstacles. This method is prone to collision accidents due to misjudgment or missed judgment in complex environments. Therefore, it is particularly important to develop a drone system that can perceive the three-dimensional environment in real time, accurately identify obstacles and predict their dynamic changes, and then achieve autonomous navigation and obstacle avoidance. Summary of the invention

[0004] In view of this, the present invention proposes a three-dimensional AI perception direction method and system based on drones, which can significantly improve the autonomous navigation and obstacle avoidance capabilities of drones, enabling them to complete tasks safely and efficiently in various complex environments.

[0005] The technical solution of the present invention is achieved in this way:

[0006] A three-dimensional AI perception direction method based on drone, specifically including:

[0007] The drone is integrated with cameras, infrared sensors, laser radars and inertial measurement units, which generate three-dimensional environmental maps;

[0008] Use a deep learning network to perform semantic segmentation on the constructed 3D environment map, identify the locations of different objects in the 3D environment map, and assign corresponding semantic labels to each object;

[0009] Predict the future behavior of dynamic objects in a 3D environment map based on a machine learning prediction model;

[0010] Dynamically build an adaptive decision tree based on current task requirements and prediction results;

[0011] Generate path planning instructions and flight instructions for the drone based on the adaptive decision tree;

[0012] The generated path planning instructions and flight instructions are sent to the UAV's flight control system. The UAV autonomously adjusts its flight attitude and speed based on the received path planning instructions and flight instructions to complete the task according to the planned path.

[0013] As a further optional solution of the three-dimensional AI perception direction method based on drones, the three-dimensional environment map is generated based on cameras, infrared sensors, laser radars and inertial measurement units, specifically including:

[0014] Preprocess visible light images, infrared images, point cloud data and IMU data to obtain preprocessed data of each modality;

[0015] Extract features from each modality data after preprocessing;

[0016] Spatial registration of features of each modality data;

[0017] Perform decision-level fusion on the registered features to obtain fused multimodal data;

[0018] Extract feature points from the fused multimodal data, match them, and establish corresponding relationships under different perspectives;

[0019] Construct an initial three-dimensional map based on the corresponding relationships under different viewing angles;

[0020] The initial three-dimensional map is optimized according to the bundle adjustment method to obtain an optimized initial three-dimensional map;

[0021] Use the geometric information in the point cloud data to reconstruct the surface and generate a three-dimensional model containing the geometric shape of the object;

[0022] Combining the semantic information in visible light images and infrared images, the 3D model is semantically annotated to generate a 3D environment map.

[0023] As a further optional solution of the drone-based 3D AI perception direction method, the 3D environment map constructed by using a deep learning network is subjected to semantic segmentation processing, the positions of different objects in the 3D environment map are identified, and corresponding semantic labels are assigned to each object, specifically including:

[0024] Build deep learning models;

[0025] The 3D environment map is input into a deep learning model, which assigns a predicted probability distribution to each point or voxel in the 3D environment map.

[0026] According to the probability distribution output by the deep learning model, a semantic label is assigned to each point or voxel to obtain the semantic segmentation result;

[0027] Identify and process the connected regions in the semantic segmentation results to identify the locations of different objects in the 3D environment map.

[0028] As a further optional solution of the drone-based 3D AI perception direction method, the prediction of the future behavior of dynamic objects in the 3D environment map based on the machine learning prediction model specifically includes:

[0029] Extract real-time data of dynamic objects from 3D environment maps;

[0030] Convert the extracted real-time data and static information of the three-dimensional environment map into feature subsets;

[0031] The feature subset is input into the trained machine learning prediction model, and the machine learning prediction model outputs the prediction result of the future behavior of the dynamic object;

[0032] Output the prediction results in a visual form.

[0033] As a further optional solution of the three-dimensional AI perception direction method based on drones, the adaptive decision tree is dynamically constructed according to the current task requirements and prediction results, specifically including:

[0034] Based on the prediction results and the current mission requirements of the drone, select the feature subset that has the greatest impact on the decision;

[0035] Build an initial decision tree using a decision tree building algorithm and the selected feature subset;

[0036] An adaptive adjustment strategy is designed to dynamically adjust the splitting point of the decision tree according to the actual flight conditions of the UAV and the real-time behavior prediction results of dynamic objects.

[0037] As a further optional solution of the three-dimensional AI perception direction method based on the drone, the generation of the path planning instructions and flight instructions of the drone based on the adaptive decision tree specifically includes:

[0038] Obtain the current state of the drone, the behavior prediction results of dynamic objects, and environmental constraints;

[0039] The current state of the drone, the behavior prediction results of the dynamic objects, and the environmental constraints are input into the adaptive decision tree;

[0040] The adaptive decision tree traverses its internal structure, and by comparing the features and thresholds on the nodes, it finally reaches a leaf node, which contains the decision on the next action of the drone based on the current input data, and outputs the decision;

[0041] The path planning algorithm generates a flight path based on the output of the adaptive decision tree;

[0042] Convert the generated flight path into flight instructions that the drone can execute.

[0043] A three-dimensional AI perception direction system based on drone, comprising:

[0044] Integrated module, which is used for drones with cameras, infrared sensors, lidar and inertial measurement units, to generate three-dimensional environmental maps based on cameras, infrared sensors, lidar and inertial measurement units;

[0045] The semantic segmentation module is used to perform semantic segmentation processing on the constructed 3D environment map using a deep learning network, identify the locations of different objects in the 3D environment map, and assign corresponding semantic labels to each object;

[0046] Dynamic behavior prediction module, used to predict the future behavior of dynamic objects in the three-dimensional environment map based on machine learning prediction models;

[0047] Adaptive decision generation module, used to dynamically build an adaptive decision tree based on current task requirements and prediction results;

[0048] The path planning and flight instruction generation module is used to generate the path planning instructions and flight instructions of the UAV based on the adaptive decision tree;

[0049] The flight control module is used to send the generated path planning instructions and flight instructions to the UAV's flight control system. The UAV autonomously adjusts its flight attitude and speed based on the received path planning instructions and flight instructions to complete the mission according to the planned path.

[0050] A computing device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned three-dimensional AI perception direction method based on a drone are implemented.

[0051] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-mentioned three-dimensional AI perception direction method based on a drone.

[0052] The beneficial effects of the present invention are as follows: by integrating cameras, infrared sensors, lidar and inertial measurement units, the system can perceive the surrounding environment of the drone in real time and in all directions, including visible light images, thermal radiation information, spatial distance and the drone's own motion state. The fusion of these multi-source data makes the generated three-dimensional environmental map more comprehensive, detailed and accurate, providing a solid foundation for subsequent processing and analysis. The deep learning network is used to perform semantic segmentation on the three-dimensional environmental map, which can not only effectively identify the positions of different objects in the map, but also assign corresponding semantic labels to each object, so that the drone can understand and interpret objects in the surrounding environment, thereby making more intelligent and accurate decisions. Based on the machine learning prediction model, dynamic objects (such as pedestrians, vehicles) in the three-dimensional environmental map can be segmented and processed. The future behavior of the drone (such as the real-time environment) can be predicted, so that the drone can perceive and respond to potential risks or changes in advance, thereby enhancing its adaptability and safety in complex dynamic environments. According to the current mission requirements and prediction results, an adaptive decision tree can be dynamically constructed, and the path planning instructions and flight instructions of the drone are generated accordingly. This adaptive decision-making mechanism enables the drone to flexibly adjust the flight strategy and path planning according to different mission scenarios and real-time environmental changes, thereby improving the execution efficiency and success rate of the mission. By sending the generated path planning instructions and flight instructions to the flight control system of the drone, the drone can autonomously adjust the flight attitude and speed and complete the mission according to the planned path. This autonomous flight capability not only reduces the burden of manual operation, but also improves the response speed and mission execution efficiency of the drone. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 A schematic diagram of a flow chart of a three-dimensional AI direction perception method based on a drone according to the present invention;

[0055] Figure 2 The present invention is a schematic diagram of the composition of a three-dimensional AI perception direction system based on a drone. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] refer to Figure 1 to Figure 2 , a three-dimensional AI perception direction method based on drone, specifically including:

[0058] The drone is integrated with cameras, infrared sensors, laser radars and inertial measurement units, which generate three-dimensional environmental maps;

[0059] Use a deep learning network to perform semantic segmentation on the constructed 3D environment map, identify the locations of different objects in the 3D environment map, and assign corresponding semantic labels to each object;

[0060] Predict the future behavior of dynamic objects in a 3D environment map based on a machine learning prediction model;

[0061] Dynamically build an adaptive decision tree based on current task requirements and prediction results;

[0062] Generate path planning instructions and flight instructions for the drone based on the adaptive decision tree;

[0063] The generated path planning instructions and flight instructions are sent to the UAV's flight control system. The UAV autonomously adjusts its flight attitude and speed based on the received path planning instructions and flight instructions to complete the task according to the planned path.

[0064] In this embodiment, by integrating cameras, infrared sensors, lidar and inertial measurement units, the system can perceive the surrounding environment of the drone in real time and in all directions, including visible light images, thermal radiation information, spatial distance and the drone's own motion state. The fusion of these multi-source data makes the generated three-dimensional environment map more comprehensive, detailed and accurate, providing a solid foundation for subsequent processing and analysis. The deep learning network is used to perform semantic segmentation on the three-dimensional environment map, which can not only effectively identify the positions of different objects in the map, but also assign corresponding semantic labels to each object, so that the drone can understand and interpret objects in the surrounding environment, thereby making more intelligent and accurate decisions. Based on the machine learning prediction model, dynamic objects (such as pedestrians, vehicles, etc.) in the three-dimensional environment map can be segmented. )’s future behavior can be predicted, so that the UAV can perceive and respond to potential risks or changes in advance, thereby enhancing its adaptability and safety in complex dynamic environments. According to the current mission requirements and prediction results, an adaptive decision tree can be dynamically constructed, and the path planning instructions and flight instructions of the UAV can be generated accordingly. This adaptive decision-making mechanism enables the UAV to flexibly adjust the flight strategy and path planning according to different mission scenarios and real-time environmental changes, thereby improving the execution efficiency and success rate of the mission. By sending the generated path planning instructions and flight instructions to the flight control system of the UAV, the UAV can autonomously adjust the flight attitude and speed and complete the mission according to the planned path. This autonomous flight capability not only reduces the burden of manual operation, but also improves the response speed and mission execution efficiency of the UAV.

[0065] Preferably, the three-dimensional environment map is generated based on the camera, infrared sensor, laser radar and inertial measurement unit, specifically including:

[0066] Preprocess visible light images, infrared images, point cloud data and IMU data to obtain preprocessed data of each modality;

[0067] Extract features from each modality data after preprocessing;

[0068] Spatial registration of features of each modality data;

[0069] Perform decision-level fusion on the registered features to obtain fused multimodal data;

[0070] Extract feature points from the fused multimodal data, match them, and establish corresponding relationships under different perspectives;

[0071] Construct an initial three-dimensional map based on the corresponding relationships under different viewing angles;

[0072] The initial three-dimensional map is optimized according to the bundle adjustment method to obtain an optimized initial three-dimensional map;

[0073] Use the geometric information in the point cloud data to reconstruct the surface and generate a three-dimensional model containing the geometric shape of the object;

[0074] Combining the semantic information in visible light images and infrared images, the 3D model is semantically annotated to generate a 3D environment map.

[0075] In this embodiment, the visible light image, infrared image, point cloud data and IMU data are preprocessed to effectively remove noise and redundant information, improve the efficiency and accuracy of subsequent processing, and make the preprocessed data clearer and more stable, laying a good foundation for subsequent feature extraction and fusion; features are extracted from each modal data after preprocessing, and these features can represent important information of the data, providing an effective basis for subsequent spatial registration and fusion. The feature extraction process enhances the representation ability of the data, so that the correlation between different modal data can be highlighted; the features of each modal data are spatially registered to achieve spatial alignment of multi-source data. This step is the key to subsequent data fusion and ensures the consistency and comparability of different modal data in the same spatial reference system; the registered features are fused at the decision level to obtain fused multimodal data. The decision level fusion fully considers the characteristics and advantages of different modal data, and improves the accuracy and reliability of the overall information by comprehensively evaluating the decision results of each modal data; feature points are extracted from the fused multimodal data, and Matching, establish the correspondence under different perspectives, this step is an important prerequisite for achieving 3D reconstruction, and provides accurate corresponding point information for the subsequent 3D map construction; construct the initial 3D map based on the correspondence under different perspectives, this process preliminarily restores the 3D structure of the scene, and provides a basic framework for subsequent optimization and reconstruction; optimize the initial 3D map using the bundle adjustment method to obtain the optimized initial 3D map, the bundle adjustment method optimizes the camera pose and 3D point position by minimizing the reprojection error, and significantly improves the accuracy and consistency of the 3D map; use the geometric information in the point cloud data to reconstruct the surface, and generate a 3D model containing the geometric shape of the object. This process makes the 3D map more realistic and complete, and provides richer information for subsequent semantic annotation and application; combine the semantic information in the visible light image and the infrared image, and semantically annotate the 3D model to generate a 3D environment map. Semantic annotation not only gives the 3D map a richer semantic connotation, but also enables the drone to understand and interpret objects in the surrounding environment, so as to make more intelligent and accurate decisions.

[0076] It should be noted that the registered features are fused at the decision level to obtain fused multimodal data. The specific steps include:

[0077] First, within each individual modality, use the respective classifier or regression model to make decisions or predictions on the features. For example, for high-definition images, an image classifier can be used to identify objects in the image; for infrared images, a temperature analysis model can be used to detect hot spots; for point cloud data, a shape recognition algorithm can be used to identify the geometry of the object; for IMU data, a motion analysis model can be used to estimate the motion state of the device;

[0078] Collect the decision results of each modality, which include classification labels, regression values, or probability distributions;

[0079] According to specific application scenarios and requirements, formulate appropriate decision fusion rules, which can be based on algebraic combinations (such as maximum, minimum, average, weighted sum, etc.), voting mechanisms (such as majority voting, weighted voting, etc.), Bayesian reasoning, dynamic Bayesian networks, adaptive enhancement algorithms, etc.;

[0080] The collected decision results of each modality are used as input, and the previously formulated fusion rules are applied to generate the final fusion decision. For example, if the weighted sum method is adopted, different weights can be assigned to each modality according to their reliability and importance, and then the weighted decision results are added together to obtain the final decision.

[0081] Evaluate the fused decision results to check their accuracy, consistency and robustness. If the results do not meet the requirements, return to the previous steps for optimization, such as adjusting the fusion rules, improving the unimodal decision model, etc.

[0082] Preferably, the method of using a deep learning network to perform semantic segmentation processing on the constructed three-dimensional environment map, identifying the positions of different objects in the three-dimensional environment map, and assigning a corresponding semantic label to each object specifically includes:

[0083] Build deep learning models;

[0084] The 3D environment map is input into a deep learning model, which assigns a predicted probability distribution to each point or voxel in the 3D environment map.

[0085] According to the probability distribution output by the deep learning model, a semantic label is assigned to each point or voxel to obtain the semantic segmentation result;

[0086] Identify and process the connected regions in the semantic segmentation results to identify the locations of different objects in the 3D environment map.

[0087] In this embodiment, by constructing a special deep learning model, the scheme can accurately semantically classify each point or voxel in the three-dimensional environment map, and assign a predicted probability distribution to each point or voxel. This classification method based on probability distribution makes the classification result more refined and accurate, and can effectively distinguish objects of different categories; the introduction of the deep learning model realizes the automation and intelligence of semantic segmentation. Compared with the traditional manual annotation or rule-based segmentation method, the deep learning model can automatically learn and extract features in the map, and complete complex semantic segmentation tasks without manual intervention. This not only improves processing efficiency, but also reduces labor costs; after obtaining the semantic segmentation results, the scheme further identifies and processes the connected areas in the segmentation results, thereby identifying the positions of different objects in the three-dimensional environment map. This process not only helps autonomous systems such as drones to accurately locate objects in the surrounding environment, but also provides an important basis for subsequent path planning, obstacle avoidance and other tasks; by assigning corresponding semantic labels to each object in the three-dimensional environment map, the scheme significantly enhances the understanding ability of autonomous systems such as drones to the surrounding environment. This understanding ability enables autonomous systems to respond to complex and changing environments more intelligently and improve the accuracy and efficiency of task execution.

[0088] It should be noted that the specific steps of building a deep learning model include:

[0089] Collect 3D environment data and generate 3D point cloud or voxelized 3D grid map;

[0090] Finely annotate the collected 3D data and assign semantic labels to each object in the map (such as roads, buildings, trees, vehicles, etc.);

[0091] Preprocess the annotated 3D data, including denoising, filtering, downsampling, etc., to reduce the amount of calculation and improve the efficiency of model training;

[0092] Use deep learning models suitable for 3D data processing, such as PointNet, PointNet++, VoteNet, etc. These models can directly process point cloud data or voxelized data;

[0093] Design a customized deep learning network according to specific needs, which may include feature extraction module, context information fusion module, multi-scale feature fusion module, etc.

[0094] Design a reasonable loss function to optimize the semantic segmentation performance of the model. Commonly used loss functions include cross entropy loss, D ice loss, etc. You can also combine multiple loss functions for training.

[0095] Preferably, the predicting of the future behavior of dynamic objects in the three-dimensional environment map based on the machine learning prediction model specifically includes:

[0096] Extract real-time data of dynamic objects from 3D environment maps;

[0097] Convert the extracted real-time data and static information of the three-dimensional environment map into feature subsets;

[0098] The feature subset is input into the trained machine learning prediction model, and the machine learning prediction model outputs the prediction result of the future behavior of the dynamic object;

[0099] Output the prediction results in a visual form.

[0100] In this embodiment, by extracting the data of dynamic objects from the three-dimensional environment map in real time, it is ensured that the input of the prediction model is the latest and most accurate information. The real-time data is combined with the static environment information to form a comprehensive feature subset, which provides rich contextual information for the machine learning model and helps to improve the accuracy of the prediction. The complex three-dimensional data and static environment information are converted into a feature subset that can be understood by the machine learning model. This step not only simplifies the data processing process, but also enables the model to flexibly adapt to different types of data input, thereby enhancing the versatility and scalability of the model. This solution does not rely on a specific machine learning algorithm. As long as the model can process the input features and output the prediction results, it can be applied, such as deep learning, random forest, support vector machine, etc., which provides more possibilities for technical implementation. The prediction results are presented in a visual form, so that non-technical personnel can also intuitively understand the future behavior of dynamic objects, providing an intuitive reference for decision makers, which helps to make judgments and decisions quickly. By predicting the future behavior of dynamic objects, potential risk areas or conflict points can be identified in advance, and sufficient response time can be reserved for the system, thereby effectively reducing the possibility of accidents. The entire prediction process is highly automated, reducing the need for manual intervention and improving work efficiency.

[0101] It should be noted that the construction of the machine learning prediction model specifically includes:

[0102] In addition to the static data of the 3D environment map, it is also necessary to collect dynamic data such as the real-time position, speed, acceleration, and direction of dynamic objects;

[0103] Annotate the collected dynamic data to mark the behavior patterns of the objects (such as movement trajectory, speed change, turning, etc.), and may include future behavior tags (such as predicted future location);

[0104] The collected data is cleaned, denoised, and normalized to ensure data quality. At the same time, the 3D environment map is fused with the data of dynamic objects to form a comprehensive data set that can be used for model training.

[0105] Extract features related to the future behavior of dynamic objects from the preprocessed data. These features may include the object's historical position, speed, acceleration, direction change, environmental obstacle distribution, road structure, etc.

[0106] Use feature selection algorithms (such as principal component analysis, mutual information method, etc.) to select the feature subset that has the greatest impact on the performance of the prediction model;

[0107] Split the synthetic dataset into training, validation, and test sets to ensure the generalization ability of the model;

[0108] Use the training set data to train the model and adjust the model parameters through the back propagation algorithm to minimize the prediction error;

[0109] Tune the model's hyperparameters through cross-validation, grid search and other methods to improve the model's predictive performance;

[0110] Use the validation set and test set to evaluate the trained model, and calculate indicators such as prediction accuracy, recall rate, and F1 score to evaluate the performance of the model.

[0111] Preferably, dynamically constructing an adaptive decision tree according to current task requirements and prediction results specifically includes:

[0112] Based on the prediction results and the current mission requirements of the drone, select the feature subset that has the greatest impact on the decision;

[0113] Build an initial decision tree using a decision tree building algorithm and the selected feature subset;

[0114] An adaptive adjustment strategy is designed to dynamically adjust the splitting point of the decision tree according to the actual flight conditions of the UAV and the real-time behavior prediction results of dynamic objects.

[0115] In this embodiment, by combining the current mission requirements and prediction results of the UAV, the feature subset with the greatest impact on the decision is selected, ensuring that the construction of the decision tree directly serves the current mission objectives. This task-oriented method helps to reduce the interference of irrelevant information and improve the pertinence and effectiveness of decision-making; the initial decision tree is constructed based on the selected feature subset, providing a basic decision framework for the UAV. However, the environment is dynamically changing, especially when it comes to real-time behavior prediction of dynamic objects. By designing an adaptive adjustment strategy, the decision tree can dynamically adjust the split point according to the actual flight situation and the update of the prediction results, thereby maintaining sensitivity and adaptability to environmental changes; the adaptive decision tree can be dynamically adjusted according to real-time data, avoiding the traditional Static decision-making methods may cause decision lags or errors due to environmental changes. This dynamic adjustment mechanism enables drones to respond to environmental changes more quickly and make more accurate decisions, thereby improving the overall execution efficiency and success rate of the task. By dynamically building and adjusting the decision tree, the system can better handle uncertainty and complexity. Even in the face of emergencies or prediction errors, the system can maintain stable decision-making capabilities through adaptive adjustments, thereby enhancing the robustness and reliability of the entire system. In a complex three-dimensional environment, drones need to face a variety of dynamic objects and complex scenes. The adaptive decision tree provides an effective autonomous decision-making mechanism, allowing drones to make reasonable decisions based on real-time data and mission requirements without human intervention. This is of great significance for realizing autonomous navigation, obstacle avoidance and mission execution of drones.

[0116] Preferably, the generating of the path planning instructions and flight instructions of the drone based on the adaptive decision tree specifically includes:

[0117] Obtain the current state of the drone, the behavior prediction results of dynamic objects, and environmental constraints;

[0118] The current state of the drone, the behavior prediction results of the dynamic objects, and the environmental constraints are input into the adaptive decision tree;

[0119] The adaptive decision tree traverses its internal structure, and by comparing the features and thresholds on the nodes, it finally reaches a leaf node, which contains the decision on the next action of the drone based on the current input data, and outputs the decision;

[0120] The path planning algorithm generates a flight path based on the output of the adaptive decision tree;

[0121] Convert the generated flight path into flight instructions that the drone can execute.

[0122] In this embodiment, by acquiring the current state of the drone, the behavior prediction results of dynamic objects, and environmental constraints in real time, and inputting this information into the adaptive decision tree, a decision can be made quickly on the next action of the drone. This real-time performance ensures that the drone can respond quickly when facing a complex environment to avoid potential dangers or conflicts. The adaptive decision tree can make the most practical decision based on the current input data by traversing and comparing its internal structure. This decision-making process not only takes into account the current state of the drone, but also incorporates the behavior prediction of dynamic objects and environmental constraints, thereby ensuring the accuracy and rationality of the decision. The path planning algorithm generates a flight path based on the output of the adaptive decision tree. This process makes full use of the decision information provided by the decision tree, so that the generated path meets the mission requirements and can avoid potential dangerous areas. Efficient path planning not only improves the flight efficiency of the drone, but also reduces energy consumption and flight risks. Converting the generated flight path into flight instructions that can be executed by the drone is a key step to ensure that the drone flies according to the planned path. This process requires the accurate conversion of path information into control instructions for the drone, including parameters such as speed, altitude, and direction. Accurate conversion ensures that the drone can accurately execute the planned path and achieve the predetermined task. The entire decision-making and control process is highly dependent on the intelligent decision-making ability of the adaptive decision tree, which enables the drone to autonomously plan paths and control flights without human intervention. This autonomy and intelligence not only improves the operating efficiency and safety of the drone, but also reduces dependence on and cost of manual operation. The adaptive decision tree can dynamically adjust its decision-making process according to changes in actual input data, so that the drone can maintain stable decision-making capabilities when facing complex and changing environments. This dynamic adaptability and robustness ensures that the drone can make reasonable decisions in various situations and complete tasks safely.

[0123] A three-dimensional AI perception direction system based on drone, comprising:

[0124] Integrated module, which is used for drones with cameras, infrared sensors, lidar and inertial measurement units, to generate three-dimensional environmental maps based on cameras, infrared sensors, lidar and inertial measurement units;

[0125] The semantic segmentation module is used to perform semantic segmentation processing on the constructed 3D environment map using a deep learning network, identify the locations of different objects in the 3D environment map, and assign corresponding semantic labels to each object;

[0126] Dynamic behavior prediction module, used to predict the future behavior of dynamic objects in the three-dimensional environment map based on machine learning prediction models;

[0127] Adaptive decision generation module, used to dynamically build an adaptive decision tree based on current task requirements and prediction results;

[0128] The path planning and flight instruction generation module is used to generate the path planning instructions and flight instructions of the UAV based on the adaptive decision tree;

[0129] The flight control module is used to send the generated path planning instructions and flight instructions to the UAV's flight control system. The UAV autonomously adjusts its flight attitude and speed based on the received path planning instructions and flight instructions to complete the mission according to the planned path.

[0130] A computing device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned three-dimensional AI perception direction method based on a drone are implemented.

[0131] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-mentioned three-dimensional AI perception direction method based on a drone.

[0132] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A three-dimensional AI perception direction method based on drone, characterized in that: Specifically include: The drone is integrated with cameras, infrared sensors, laser radars and inertial measurement units, which generate three-dimensional environmental maps; Use a deep learning network to perform semantic segmentation on the constructed 3D environment map, identify the locations of different objects in the 3D environment map, and assign corresponding semantic labels to each object; Predict the future behavior of dynamic objects in a 3D environment map based on a machine learning prediction model; Dynamically build an adaptive decision tree based on current task requirements and prediction results; Generate path planning instructions and flight instructions for the drone based on the adaptive decision tree; The generated path planning instructions and flight instructions are sent to the UAV's flight control system. The UAV autonomously adjusts its flight attitude and speed based on the received path planning instructions and flight instructions to complete the task according to the planned path.

2. The three-dimensional AI direction perception method based on drone according to claim 1 is characterized in that: The three-dimensional environment map is generated based on the camera, infrared sensor, laser radar and inertial measurement unit, specifically including: Preprocess visible light images, infrared images, point cloud data and IMU data to obtain preprocessed data of each modality; Extract features from each modality data after preprocessing; Spatial registration of features of each modality data; Perform decision-level fusion on the registered features to obtain fused multimodal data; Extract feature points from the fused multimodal data, match them, and establish corresponding relationships under different perspectives; Construct an initial three-dimensional map based on the corresponding relationships under different viewing angles; The initial three-dimensional map is optimized according to the bundle adjustment method to obtain an optimized initial three-dimensional map; Use the geometric information in the point cloud data to reconstruct the surface and generate a three-dimensional model containing the geometric shape of the object; Combining the semantic information in visible light images and infrared images, the 3D model is semantically annotated to generate a 3D environment map.

3. The three-dimensional AI direction perception method based on drone according to claim 2 is characterized in that: The method of using a deep learning network to perform semantic segmentation on the constructed three-dimensional environment map, identifying the locations of different objects in the three-dimensional environment map, and assigning corresponding semantic labels to each object specifically includes: Build deep learning models; The 3D environment map is input into a deep learning model, which assigns a predicted probability distribution to each point or voxel in the 3D environment map. According to the probability distribution output by the deep learning model, a semantic label is assigned to each point or voxel to obtain the semantic segmentation result; Identify and process the connected regions in the semantic segmentation results to identify the locations of different objects in the 3D environment map.

4. The three-dimensional AI direction perception method based on drone according to claim 3 is characterized in that: The prediction of the future behavior of dynamic objects in the three-dimensional environment map based on the machine learning prediction model specifically includes: Extract real-time data of dynamic objects from 3D environment maps; Convert the extracted real-time data and static information of the three-dimensional environment map into feature subsets; The feature subset is input into the trained machine learning prediction model, and the machine learning prediction model outputs the prediction result of the future behavior of the dynamic object; Output the prediction results in a visual form.

5. The three-dimensional AI direction perception method based on drone according to claim 4 is characterized in that: The method of dynamically building an adaptive decision tree based on current task requirements and prediction results specifically includes: Based on the prediction results and the current mission requirements of the drone, select the feature subset that has the greatest impact on the decision; Build an initial decision tree using a decision tree building algorithm and the selected feature subset; An adaptive adjustment strategy is designed to dynamically adjust the splitting point of the decision tree according to the actual flight conditions of the UAV and the real-time behavior prediction results of dynamic objects.

6. The three-dimensional AI direction perception method based on drone according to claim 5 is characterized in that: The generating of the path planning instructions and flight instructions of the UAV according to the adaptive decision tree specifically includes: Obtain the current state of the drone, the behavior prediction results of dynamic objects, and environmental constraints; The current state of the drone, the behavior prediction results of the dynamic objects, and the environmental constraints are input into the adaptive decision tree; The adaptive decision tree traverses its internal structure, and by comparing the features and thresholds on the nodes, it finally reaches a leaf node, which contains the decision on the next action of the drone based on the current input data, and outputs the decision; The path planning algorithm generates a flight path based on the output of the adaptive decision tree; Convert the generated flight path into flight instructions that the drone can execute.

7. A three-dimensional AI perception direction system based on drones, characterized in that: include: Integrated module, which is used for drones with cameras, infrared sensors, lidar and inertial measurement units, to generate three-dimensional environmental maps based on cameras, infrared sensors, lidar and inertial measurement units; The semantic segmentation module is used to perform semantic segmentation processing on the constructed 3D environment map using a deep learning network, identify the locations of different objects in the 3D environment map, and assign corresponding semantic labels to each object; Dynamic behavior prediction module, used to predict the future behavior of dynamic objects in the three-dimensional environment map based on machine learning prediction models; Adaptive decision generation module, used to dynamically build an adaptive decision tree based on current task requirements and prediction results; The path planning and flight instruction generation module is used to generate the path planning instructions and flight instructions of the UAV based on the adaptive decision tree; The flight control module is used to send the generated path planning instructions and flight instructions to the UAV's flight control system. The UAV autonomously adjusts its flight attitude and speed based on the received path planning instructions and flight instructions to complete the mission according to the planned path.

8. A computing device, characterized in that The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the three-dimensional AI direction perception method based on a drone as described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the steps of the three-dimensional AI direction perception method based on a drone as described in any one of claims 1 to 6.

Images