A real-scene modeling unmanned aerial vehicle system and control method thereof

Abstract

The present invention provides a real-scene modeling UAV system and a control method thereof, which relates to the technical field of UAV modeling, including using a multimodal sensor carried by a UAV to perform a preliminary scan of a real scene to form a first environmental perception map; using an improved ant colony algorithm to perform task area division and UAV task assignment to generate an initial flight path; using a reinforcement learning algorithm to pre-train the acquired historical flight data to obtain a real-time environmental feedback report; using an edge computing module to analyze the data in the real-time environmental feedback report to generate a second environmental perception map; using a data fusion algorithm to fuse data from different sensors and generate a high-precision three-dimensional environmental model. The beneficial effect of the present invention is that through algorithms and processing methods, the UAV can perceive and adapt to different environmental conditions in real time and accurately, ensure the safety of flight and the efficient completion of tasks, and has broad application prospects and significant technical advantages.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle modeling, and in particular to a real-scene modeling unmanned aerial vehicle system and a control method thereof. Background Art

[0002] In rescue scenarios after natural disasters such as earthquakes, real-time, high-precision terrain modeling is crucial for search and rescue work. In the existing technology, the traditional single control method of UAVs is difficult to cope with complex and changing environments and large amounts of data processing requirements. Among them, the real-scene modeling method of UAVs has the following main defects: Most of the existing UAV real-scene modeling technologies rely on the data of a single sensor. Especially in a changing environment, the data of a single sensor is difficult to fully and accurately reflect the actual situation; traditional modeling methods often need to collect a large amount of data first, and then centrally process it at the ground station. This batch processing method leads to poor real-time performance. The UAV cannot adjust the path and obstacle avoidance strategy in real time during flight, thereby increasing the risk and failure rate of the mission; temperature distribution and obstacle location are important factors affecting the flight safety of UAVs, but the existing methods do not take this into consideration enough, resulting in UAVs being prone to collisions and failures in complex environments; and there is also unoptimized path planning and lack of intelligent control. Summary of the invention

[0003] The purpose of the present invention is to provide a real-scene modeling drone system and a control method thereof to improve the above problems. In order to achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0004] In a first aspect, the present application provides a real-scene modeling drone system, comprising:

[0005] Acquisition module: used to use the multimodal sensor carried by the drone to perform a preliminary scan of the real scene, obtain the terrain, temperature distribution and obstacle information in the real scene, and form a first environment perception map, wherein the first environment perception map includes a preliminary terrain map, a temperature distribution map and an obstacle distribution map, and the multimodal sensor includes a laser radar, an infrared sensor and a high-definition camera;

[0006] Allocation module: used to use the improved ant colony algorithm to divide the task area and allocate the UAV tasks, and dynamically adjust the allocation strategy according to the first environmental perception map to generate the initial flight path;

[0007] Training optimization module: used to pre-train the acquired historical flight data according to the reinforcement learning algorithm, optimize and adjust the flight path and posture of the drone, and obtain real-time environmental feedback reports;

[0008] Analysis module: used to analyze the data in the real-time environment feedback report using the edge computing module to generate a second environment perception map, wherein the second environment perception map includes a real-time terrain map, a real-time temperature distribution map, and a real-time obstacle distribution map;

[0009] Fusion module: used to obtain other data information collected by the drone, input other data information and the second environment perception map into the data fusion system, pre-process the data in the data fusion system, and use the data fusion algorithm to fuse the data from different sensors and generate a high-precision three-dimensional environment model, where other data information includes high-definition images and video streams.

[0010] Preferably, the acquisition module includes:

[0011] Fitting unit: used to pre-process the point cloud data in the real scene collected by the laser radar carried by the drone, and randomly select the plane model fitted by the first point, the second point and the third point from the pre-processed point cloud data, evaluate the fitness of the plane model, determine the plane model containing the most internal points, and record it as the optimal plane model, where the point cloud data includes ground points, building points and other obstacle points;

[0012] Segmentation unit: used to extract all points in the point cloud data that meet the internal point conditions in the optimal plane model, record them as real ground points, use clustering algorithm to segment the real ground points, identify the ground in different areas, and obtain a preliminary topographic map. The segmentation process is to select a real ground point as a core point, determine whether the number of points in the neighborhood of the core point reaches a preset threshold, and if the threshold is reached, classify the core point and its neighborhood points into a cluster, and continue to expand the cluster until no new core points can be added. Repeat the above process until all real ground points are classified or marked as noise points;

[0013] The first processing unit is used to process the thermal imaging data collected by the infrared sensor carried by the UAV through the Gaussian filter algorithm, and obtain the temperature distribution map through thermal imaging data smoothing and temperature value conversion;

[0014] The second processing unit is used to obtain an obstacle distribution map through real-time target detection and obstacle position recognition based on the image data collected by the high-definition camera on the drone after being processed by the YOLO deep learning algorithm;

[0015] Summarizing unit: used to summarize the preliminary terrain map, temperature distribution map and obstacle distribution map to obtain the first environment perception map.

[0016] Preferably, the allocation module includes:

[0017] Analysis unit: used for analyzing the terrain complexity, temperature anomaly and obstacle density in the mission area through an image processing algorithm based on the first environment perception map, and obtaining a mission area complexity map, which includes a comprehensive evaluation of terrain complexity, temperature anomaly and obstacle density;

[0018] Allocation unit: used to allocate weights to the task areas according to the task area complexity map, using the hierarchical analysis method and the multi-criteria decision-making method, divide the task areas into priority areas, obtain the priority area map, and mark the priority of each area on the priority area map, where the priorities include high-risk areas, medium-priority areas and low-priority areas. High-risk areas include areas with complex terrain, abnormal temperature and dense obstacles;

[0019] Iteration unit: It is used to simulate the movement of ants in each sub-area according to the priority area map, find the optimal task allocation plan through multiple iterations, and generate a task allocation map.

[0020] Preferably, the training optimization module includes:

[0021] Extraction unit: used to extract terrain and obstacle features based on historical flight data through reinforcement learning algorithm pre-training model, input the extracted terrain and obstacle features into the reinforcement learning model, and train the model's flight strategy in different environments;

[0022] Computing unit: used to construct a flight topology map based on the flight strategy, identify key nodes using graph theory algorithms, and calculate key nodes and connection strengths between nodes using the shortest path algorithm;

[0023] Path adjustment unit: It is used to take key nodes and connection strengths as input values ​​of the deep reinforcement learning model, train the deep reinforcement learning model using historical data and real-time environmental data, dynamically adjust the flight path according to the real-time status, and generate a real-time flight path and attitude adjustment strategy.

[0024] Preferably, the computing unit includes:

[0025] Define unit: It is used to divide the flight area into several polygons based on terrain and obstacle features using the Thiessen polygon partitioning method. Each polygon represents a terrain unit or obstacle unit. The center point of each polygon is defined as a node, and edges are established between adjacent nodes. The weight of the edge is based on the distance between the two nodes, the flight cost and the risk level.

[0026] Marking node unit: used to calculate the importance of each node using graph theory algorithms, and mark nodes with high values ​​as key nodes, where key nodes include high-altitude terrain, narrow passages, and major obstacles;

[0027] Calculation path unit: used to adjust the edge weights in the flight topology graph according to the importance of key nodes and the distance of edges, and use the shortest path algorithm to calculate the shortest path from the starting point to the end point, while recording the connection strength between each node, where the connection strength can be comprehensively calculated based on the path length, the number of key nodes passed through, and the risk level index of the path;

[0028] Update flight unit: used to mark flight paths and dynamically update flight topology maps based on real-time environmental data and flight mission requirements. The marking process marks paths with high connection strength as priority flight paths and excludes paths with low connection strength or high risk.

[0029] In a second aspect, the present application also provides a real-scene modeling drone control method, comprising:

[0030] Using the multimodal sensor carried by the drone to perform a preliminary scan of the real scene, obtain the terrain, temperature distribution and obstacle information in the real scene, and form a first environmental perception map, wherein the first environmental perception map includes a preliminary terrain map, a temperature distribution map and an obstacle distribution map, and the multimodal sensor includes a laser radar, an infrared sensor and a high-definition camera;

[0031] The improved ant colony algorithm is used to divide the task area and allocate the UAV tasks, and the allocation strategy is dynamically adjusted according to the first environmental perception map to generate the initial flight path;

[0032] Use reinforcement learning algorithms to pre-train the acquired historical flight data, optimize and adjust the UAV flight path and attitude, and obtain real-time environmental feedback reports;

[0033] The edge computing module is used to analyze the data in the real-time environment feedback report to generate a second environment perception map, wherein the second environment perception map includes a real-time terrain map, a real-time temperature distribution map, and a real-time obstacle distribution map;

[0034] Acquire other data information collected by the drone, input the other data information and the second environment perception map into the data fusion system, pre-process the data in the data fusion system, and use the data fusion algorithm to fuse the data from different sensors to generate a high-precision three-dimensional environmental model, where the other data information includes high-definition images and video streams.

[0035] The beneficial effects of the present invention are:

[0036] The present invention utilizes point cloud data preprocessing and plane model fitting to improve the accuracy and efficiency of ground point extraction, laying a solid foundation for the subsequent generation of high-precision preliminary topographic maps, realizing refined division of terrain, generating accurate preliminary topographic maps, improving the resolution and accuracy of environmental perception, accurately reflecting temperature changes in the environment, and providing reliable data support for the flight decision-making of UAVs in different temperature environments. Reinforcement learning model pre-training and flight strategy optimization ensure the safety and effectiveness of path planning, improve the reliability and efficiency of task execution, and the edge computing module generates a second environmental perception map to provide reliable support for the real-time flight decision-making of UAVs, further improve flight navigation and obstacle avoidance capabilities, and effectively enhance the autonomous flight capability and task execution efficiency of UAVs in complex environments. Through algorithms and processing methods, UAVs can perceive and adapt to different environmental conditions in real time and accurately, ensuring the safety of flight and the efficient completion of tasks. It has broad application prospects and significant technical advantages.

[0037] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or be understood by implementing the embodiments of the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 Schematic diagram of the structure of the real scene modeling UAV system described in an embodiment of the present invention;

[0040] Figure 2 The figure is a flow chart of the real scene modeling UAV method described in an embodiment of the present invention.

[0041] In the figure: 701, acquisition module; 7011, fitting unit; 7012, segmentation unit; 7013, first processing unit; 7014, second processing unit; 7015, summary unit; 702, allocation module; 7021, analysis unit; 7022, allocation unit; 7023, iteration unit; 703, training optimization module; 7031, extraction unit; 7032, calculation unit; 70321, definition unit; 70322, marking node unit; 70323, calculation path unit; 70324, update flight unit; 7033, adjustment path unit; 704, analysis module; 705, fusion module. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0043] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0044] Embodiment 1:

[0045] like Figure 1 As shown, this embodiment provides a real scene modeling drone system, see Figure 1 The system comprises:

[0046] Acquisition module 701: used to use the multimodal sensor carried by the drone to perform a preliminary scan of the real scene, obtain the terrain, temperature distribution and obstacle information in the real scene, and form a first environment perception map, wherein the first environment perception map includes a preliminary terrain map, a temperature distribution map and an obstacle distribution map, and the multimodal sensor includes a laser radar, an infrared sensor and a high-definition camera;

[0047] Allocation module 702: used to use the improved ant colony algorithm to perform task area division and UAV task allocation, and dynamically adjust the allocation strategy according to the first environmental perception map to generate an initial flight path;

[0048] Training optimization module 703: used to pre-train the acquired historical flight data according to the reinforcement learning algorithm, optimize and adjust the flight path and posture of the UAV, and obtain a real-time environmental feedback report;

[0049] Analysis module 704: used to analyze the data in the real-time environment feedback report using the edge computing module to generate a second environment perception map, wherein the second environment perception map includes a real-time terrain map, a real-time temperature distribution map, and a real-time obstacle distribution map;

[0050] Fusion module 705: used to obtain other data information collected by the drone, input the other data information and the second environment perception map into the data fusion system, pre-process the data in the data fusion system, and use the data fusion algorithm to fuse the data from different sensors and generate a high-precision three-dimensional environment model, where the other data information includes high-definition images and video streams.

[0051] Specifically, the acquisition module 701 includes:

[0052] The fitting unit 7011 is used to pre-process the point cloud data in the real scene collected by the laser radar carried by the UAV, and randomly select the plane model fitted by the first point, the second point and the third point from the pre-processed point cloud data, evaluate the fitness of the plane model, determine the plane model containing the most internal points, and record it as the optimal plane model, wherein the point cloud data includes ground points, building points and other obstacle points;

[0053] Segmentation unit 7012: used to extract all points in the point cloud data that meet the internal point conditions in the optimal plane model, record them as real-scene ground points, use clustering algorithm to segment the real-scene ground points, identify the ground in different areas, and obtain a preliminary topographic map, wherein the segmentation process is to select a real-scene ground point as a core point, determine whether the number of points in the neighborhood of the core point reaches a preset threshold, if the threshold is reached, classify the core point and its neighborhood points into a cluster, continue to expand the cluster until no new core points can be added, and repeat the above process until all real-scene ground points are classified or marked as noise points;

[0054] The first processing unit 7013 is used to obtain a temperature distribution map by processing the thermal imaging data collected by the infrared sensor carried by the drone through a Gaussian filter algorithm, smoothing the thermal imaging data and converting the temperature value;

[0055] The second processing unit 7014 is used to obtain an obstacle distribution map through real-time target detection and obstacle position recognition based on the image data collected by the high-definition camera carried by the drone and processed by the YOLO deep learning algorithm;

[0056] Summarizing unit 7015: used to summarize the preliminary terrain map, temperature distribution map and obstacle distribution map to obtain a first environment perception map.

[0057] Specifically, the allocation module 702 includes:

[0058] The analysis unit 7021 is used to analyze the terrain complexity, temperature anomaly and obstacle density in the task area through an image processing algorithm based on the first environment perception map, and obtain a task area complexity map, which includes a comprehensive evaluation of terrain complexity, temperature anomaly and obstacle density;

[0059] Allocation unit 7022: used to allocate weights to the task areas according to the task area complexity map by using the hierarchical analysis method and the multi-criteria decision-making method, divide the task areas into priority areas, obtain a priority area map, and mark the priority of each area on the priority area map, wherein the priorities include high-risk areas, medium-priority areas and low-priority areas, and the high-risk areas include areas with complex terrain, abnormal temperature and dense obstacles;

[0060] Iteration unit 7023: used to simulate the movement of ants in each sub-area according to the priority area map, find the optimal task allocation solution through multiple iterations, and generate a task allocation map.

[0061] Specifically, the training optimization module 703 includes:

[0062] Extraction unit 7031: used to extract terrain and obstacle features based on historical flight data by pre-training the model through reinforcement learning algorithm, input the extracted terrain and obstacle features into the reinforcement learning model, and train the model's flight strategies in different environments;

[0063] Calculation unit 7032: used to construct a flight topology map according to the flight strategy, identify key nodes using graph theory algorithms, and calculate key nodes and connection strengths between nodes using the shortest path algorithm;

[0064] Path adjustment unit 7033: used to use key nodes and connection strengths as input values ​​of the deep reinforcement learning model, use historical data and real-time environmental data to train the deep reinforcement learning model, dynamically adjust the flight path according to the real-time status, and generate a real-time flight path and attitude adjustment strategy.

[0065] Specifically, the computing unit 7032 includes:

[0066] Definition unit 70321: used to divide the flight area into several polygons based on terrain and obstacle features using the Thiessen polygon partitioning method, each polygon represents a terrain unit or an obstacle unit, the center point of each polygon is defined as a node, and edges are established between adjacent nodes, where the weight of the edge is based on the distance between the two nodes, the flight cost and the risk level;

[0067] Marking node unit 70322: used to calculate the importance of each node using graph theory algorithms, and mark nodes with high values ​​as key nodes, where key nodes include high-altitude terrain, narrow passages, and major obstacles;

[0068] Calculation path unit 70323: used to adjust the edge weights in the flight topology graph according to the importance of key nodes and the distance of edges, calculate the shortest path from the starting point to the end point using the shortest path algorithm, and record the connection strength between each node, wherein the connection strength can be comprehensively calculated based on the path length, the number of key nodes passed through, and the risk level index of the path;

[0069] Update flight unit 70324: used to mark flight paths and dynamically update the flight topology map based on real-time environmental data and flight mission requirements. The marking process is to mark paths with high connection strength as priority flight paths and exclude paths with low connection strength or high risk.

[0070] It should be noted that, regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0071] Embodiment 2:

[0072] This embodiment provides a method for real-scene modeling of a drone.

[0073] See also Figure 2 , the figure shows that the method includes step S100, step S200, step S300, step S400 and step S500.

[0074] S100. Use the multimodal sensor carried by the drone to perform a preliminary scan of the real scene, obtain the terrain, temperature distribution and obstacle information in the real scene, and form a first environment perception map, wherein the first environment perception map includes a preliminary terrain map, a temperature distribution map and an obstacle distribution map, and the multimodal sensor includes a lidar, an infrared sensor and a high-definition camera.

[0075] It can be understood that step S100 includes S101, S102, S103, S104 and S105, wherein:

[0076] S101, preprocessing the point cloud data in the real scene collected by the laser radar carried by the drone, and randomly selecting the plane model fitted by the first point, the second point and the third point from the preprocessed point cloud data, evaluating the fitness of the plane model, determining the plane model containing the most internal points, and recording it as the optimal plane model, wherein the point cloud data includes ground points, building points and other obstacle points;

[0077] It should be noted that these three points are used to fit a plane model. The mathematical expression of the plane model can be expressed as ax+by+cz+d=0, where (a, b, c) is the normal vector, d is the distance from the plane to the origin, and the fitness of the plane model is evaluated. The fitness is calculated by evaluating the distance from each point in the point cloud data to the plane, and defining the points whose distance is less than a certain threshold as internal points. The fitness formula is:

[0078]

[0079] In the formula, is the indicator function, ∈ is the distance threshold, and N is the total number of points in the point cloud data; repeat the above steps multiple times and select the plane model containing the most internal points as the optimal plane model.

[0080] S102, extracting all points in the point cloud data that meet the internal point conditions in the optimal plane model and recording them as real-scene ground points, segmenting the real-scene ground points using a clustering algorithm, identifying the ground in different areas, and obtaining a preliminary topographic map, wherein the segmentation process is to select a real-scene ground point as a core point, determine whether the number of points in the neighborhood of the core point reaches a preset threshold, and if the threshold is reached, classify the core point and its neighborhood points into a cluster, and continue to expand the cluster until no new core points can be added, and repeat the above process until all real-scene ground points are classified or marked as noise points;

[0081] It should be noted that a point is selected from the real ground points as a core point, recorded as the core point, and the neighborhood of the core point is calculated, that is, the number of points within a certain radius, recorded as the number of neighborhood points; if the number of neighborhood points reaches a preset threshold (for example, 50 points), the core point and its neighborhood points are classified into a cluster and marked; a new core point is selected, and the above process is repeated to continue expanding the current cluster until no new core points can be added; the whole process is repeated until all real ground points are classified into a cluster or marked as noise points.

[0082] In summary, the real ground points in different clusters are marked and mapped to generate a preliminary topographic map, which shows the ground distribution in different areas, which is helpful for the subsequent task area division and UAV task allocation. This method combines the RANSAC and DBSCAN algorithms to ensure that ground points can be accurately extracted and segmented in complex terrain environments, providing a reliable data basis for UAV task allocation and flight path planning.

[0083] S103, according to the thermal imaging data collected by the infrared sensor carried by the UAV, the thermal imaging data is processed by a Gaussian filter algorithm, and the temperature distribution map is obtained by smoothing the thermal imaging data and converting the temperature value;

[0084] It should be noted that during the flight of the drone, the infrared sensor continuously obtains thermal imaging data of the area. These data contain the infrared radiation intensity of each pixel, mapped as temperature information; first, a two-dimensional Gaussian filter needs to be created, and then the Gaussian filter is convolved with the thermal imaging data according to the convolution operation to obtain the original imaging data, and then the temperature value is converted, that is, the smoothed data is converted into the actual temperature value through the temperature calibration model. The processed data is displayed in the form of an image to generate a temperature distribution map. This figure shows the temperature distribution in different areas, and the temperature gradient is represented by color coding for intuitive analysis.

[0085] S104, according to the image data collected by the high-definition camera carried by the drone, after being processed by the YOLO deep learning algorithm, through real-time target detection and obstacle position recognition, an obstacle distribution map is obtained;

[0086] S105: Summarize the preliminary topographic map, the temperature distribution map, and the obstacle distribution map to obtain a first environment perception map.

[0087] It should be noted that during the flight of the drone, the high-definition camera captures image data on the ground and in the air. These data include potential obstacles such as roads, buildings, trees, and vehicles. The YOLO algorithm regards the target detection problem as a regression problem, directly predicts the bounding box and category probability on the image, scales the collected image data to a uniform size (such as 416x416), and normalizes it. The processed image is input into the pre-trained YOLO model, and the model outputs a set of bounding boxes, the confidence of each box, and the category probability; it filters out overlapping bounding boxes and only retains the box with the highest confidence. The specific algorithm is:

[0088]

[0089] In the formula, IOU is the intersection-over-union ratio. If it exceeds the set threshold, the box with low confidence is removed. The final output obstacle distribution map marks the location and category of each obstacle, using bounding boxes and labels. This map helps the navigation system avoid potential dangers and optimize the flight path.

[0090] In summary, when flying over the city, the infrared sensor captures thermal imaging data of buildings, roads, and green spaces. A Gaussian filter is applied to each frame to eliminate noise and smooth the temperature distribution. The smoothed infrared radiation intensity is then converted into actual temperature values ​​to generate a temperature distribution map. These temperature data help identify the heat island effect and abnormally high temperature areas. At the same time, the high-definition camera captures real-time images of city streets, and the YOLO model detects vehicles, pedestrians, and other obstacles in real time. Through non-maximum suppression technology, overlapping bounding boxes are filtered out to ensure that the location and category of each obstacle are accurately marked. The obstacle distribution map finally generated shows the location and category information of all detected obstacles, which facilitates the drone to plan a safe path. This method not only improves the navigation capability of drones in complex environments, but also enhances the real-time and accuracy of data processing.

[0091] S200, using an improved ant colony algorithm to divide the task area and allocate UAV tasks, and dynamically adjust the allocation strategy according to the first environmental perception map to generate an initial flight path.

[0092] It can be understood that step S200 includes S201, S202 and S203, wherein:

[0093] S201, based on the first environment perception map, analyzing the terrain complexity, temperature anomaly and obstacle density in the mission area through an image processing algorithm to obtain a mission area complexity map, which includes a comprehensive evaluation of terrain complexity, temperature anomaly and obstacle density;

[0094] It should be noted that the Sobel operator is used to calculate the gradient of the preliminary topographic map, detect terrain changes and slope information, and evaluate the complexity of the terrain; the Gaussian filter is used to smooth the temperature distribution map, and the temperature standard deviation of each area is calculated to detect temperature anomalies; the obstacle distribution map is analyzed by counting the obstacle density (the number of obstacles per unit area).

[0095] S202. According to the task area complexity map, the task area is weighted and divided into priority areas using the analytic hierarchy process and the multi-criteria decision-making method, and a priority area map is obtained. The priority of each area is marked on the priority area map, where the priorities include high-risk areas, medium-priority areas, and low-priority areas. The high-risk areas include areas with complex terrain, abnormal temperature, and dense obstacles.

[0096] It should be noted that the analytic hierarchy process (AHP) was applied to determine the weights of terrain complexity, temperature anomaly, and obstacle density through expert scoring, and the multi-criteria decision-making method (MCDM) was applied to calculate the comprehensive priority score of each area. The weighted sum was combined with each weight to obtain a priority area map, and the priority of each area was marked, such as high-risk areas (complex terrain, temperature anomaly, and dense obstacles), medium-priority areas, and low-priority areas (flat areas).

[0097] S203, based on the priority area map, simulate the movement of ants in each sub-area, find the optimal task allocation solution through multiple iterations, and generate a task allocation map.

[0098] It should be noted that when the ant colony is initialized, each ant represents a task allocation scheme. The individual ants select paths through pheromone trajectories and priority scores, update pheromones, introduce priority weights, and leave more pheromones in high-priority areas. Then, in each iteration, the pheromone concentration in low-priority areas is gradually reduced, and the pheromone concentration in high-priority areas is increased. Through multiple iterations, the optimal task allocation scheme is found. The task allocation graph can be obtained, the task area is divided into several sub-areas, and preliminary tasks are assigned to each sub-area.

[0099] It should be noted that this step also includes the allocation of UAV tasks, and dynamically adjusts the allocation strategy according to the first environmental perception map to generate an initial flight path.

[0100] Specifically, the regional division is dynamically adjusted according to the real-time status of the UAV (battery level, flight capability) and mission requirements; edge computing technology is used to analyze temperature data in real time to identify high-temperature areas with temperatures above the set threshold, and to identify high-priority areas (high-temperature areas and complex terrain areas) based on the temperature distribution map and obstacle distribution map; high-temperature areas are preferentially assigned to UAVs with sufficient battery power and strong flight capabilities, and complex terrain areas are assigned to UAVs with good flight stability. Task assignment information is sent to the UAV through the communication module. After receiving the task assignment information, the UAV updates the flight path and prepares to perform the task. It obtains environmental feedback data in real time (new temperature distribution and obstacle locations), and dynamically adjusts the UAV's tasks and paths based on the real-time environmental feedback report processed by the edge computing module).

[0101] S300, use reinforcement learning algorithm to pre-train the acquired historical flight data, optimize and adjust the UAV flight path and posture, and obtain real-time environmental feedback report.

[0102] It can be understood that step S300 includes S301, S302 and S303, wherein:

[0103] S301. Based on historical flight data, a reinforcement learning algorithm is used to pre-train a model to extract terrain and obstacle features, and the extracted terrain and obstacle features are input into the reinforcement learning model to train the model's flight strategies in different environments.

[0104] S302, constructing a flight topology map according to the flight strategy, identifying key nodes using a graph theory algorithm, and calculating key nodes and connection strengths between nodes using a shortest path algorithm;

[0105] It can be understood that historical flight data includes records of UAV flights in different terrains and environments, including sensor data (such as lidar point cloud data, infrared imaging data, camera image data) and flight status (such as speed, attitude, path); extracting terrain features such as altitude changes, slope, ground type, etc. from lidar data; extracting the position, shape and dynamic properties of obstacles (such as moving speed) from images and lidar data, and then performing a pre-training process, which includes defining the state space, including the position, speed, sensor data, etc. of the UAV; defining the action space, including the flight instructions of the UAV (such as direction, speed adjustment); designing a reward function, taking into account factors such as the safety and efficiency of the flight path; and using historical data for reinforcement learning training in a simulated environment, so that the model learns the best flight strategy under different terrain and obstacle conditions.

[0106] It should be noted that step S302 includes S3021, S3022, S3023 and S3024, wherein:

[0107] S3021. Based on the terrain and obstacle features, the flight area is divided into several polygons using the Thiessen polygon partitioning method. Each polygon represents a terrain unit or an obstacle unit. The center point of each polygon is defined as a node. Edges are established between adjacent nodes, where the weight of the edge is based on the distance between the two nodes, the flight cost, and the risk level.

[0108] It should be noted that the entire flight area is divided into several polygonal units using the Thiessen polygon partitioning method. Each polygonal unit is generated by the nearest sampling point, and the center point of each polygon is defined as a node. The center point can usually be determined by calculating the centroid of the polygon. In the Voronoi diagram, the center points of adjacent polygons are connected into edges. The flight cost is calculated based on the terrain and obstacle characteristics on the flight path. The flight cost in complex terrain and obstacle-dense areas is higher. The risk level of the flight path is evaluated by combining historical data and real-time perception data. The path with high risk has a greater weight.

[0109] S3022. Calculate the importance of each node using graph theory algorithms, and mark nodes with high values ​​as key nodes, where key nodes include high-altitude terrain, narrow passages, and major obstacles;

[0110] It should be noted that nodes with higher graph theory algorithm values ​​are marked as key nodes, which are usually important turning points in the flight path or areas with higher risks, including high-altitude terrain, narrow passages, and major obstacles.

[0111] S3023. According to the importance of key nodes and the distance of edges, the edge weights in the flight topology graph are adjusted, and the shortest path from the starting point to the end point is calculated using the shortest path algorithm, and the connection strength between each node is recorded, wherein the connection strength can be comprehensively calculated based on the path length, the number of key nodes passed through, and the risk level index of the path;

[0112] It should be noted that the distance of the starting node is set to 0, and the distances of other nodes are infinite; the node with the shortest current distance is gradually selected, and the distances of its adjacent nodes are updated; for each adjacent node, if the path through the current node is shorter, its distance is updated; the shortest path from the starting point to the end point is calculated, and the connection strength between each node is recorded; based on the comprehensive calculation of the path length, the number of key nodes passed through and the risk level of the path, the path with high connection strength is marked as the priority flight path.

[0113] S3024. Mark the flight path and dynamically update the flight topology map based on real-time environmental data and flight mission requirements. The marking process is to mark the paths with high connection strength as priority flight paths and exclude the paths with low connection strength or high risk.

[0114] It should be noted that these paths are usually optimized, with few key nodes and low risk levels, and are suitable as priority flight paths for drones. Based on the connection strength and real-time data, the priority of the path is dynamically evaluated to ensure that the drone chooses the safest and most efficient path. Usually, the paths are longer, more risky, or pass through many key nodes, and these paths will be excluded. Based on real-time data, such as sudden obstacles or harsh. The flight topology map is dynamically adjusted according to the latest environmental data and mission requirements. For example, if a sudden obstacle appears ahead, the topology map needs to be updated in time, the optimal path needs to be recalculated, and the flight path needs to be continuously optimized to ensure that the drone can still complete the task efficiently and safely in a changing environment.

[0115] In summary, by dynamically updating the flight topology map, the drone can avoid sudden obstacles in real time and ensure flight safety. For example, after detecting sudden moving obstacles (such as vehicles and pedestrians) in real time, the system can quickly eliminate high-risk paths and avoid collision accidents; by updating the flight path in real time, the drone can choose a priority path with high connection strength and low flight cost to improve the efficiency of task completion. For example, in an emergency mission, the system can dynamically adjust the path so that the drone can reach the target location at the fastest speed; the human-machine flight environment is complex and changeable, and the real-time update of the topology map enables it to flexibly respond to various changes and improve its adaptability. For example, in an urban environment, drones need to frequently avoid obstacles such as tall buildings and bridges, and dynamically updating the path can ensure that they can successfully complete the task; using reinforcement learning and graph theory algorithms, the system can intelligently select the optimal path and improve the drone's autonomous decision-making ability. For example, by analyzing historical flight data and real-time environmental data, the system can intelligently predict potential risks and adjust flight strategies. Therefore, dynamically updating the flight topology map plays a key role in the UAV control system. It not only ensures flight safety and mission efficiency, but also improves the adaptability and intelligent decision-making level of the UAV in complex environments. Through comprehensive processing and analysis of real-time data, the system can dynamically adjust the flight path and provide the optimal flight strategy for the UAV.

[0116] S303. Use key nodes and connection strengths as input values ​​of the deep reinforcement learning model, use historical data and real-time environmental data to train the deep reinforcement learning model, dynamically adjust the flight path according to the real-time status, and generate a real-time flight path and attitude adjustment strategy.

[0117] S400. Analyze the data in the real-time environment feedback report using the edge computing module to generate a second environment perception map, wherein the second environment perception map includes a real-time terrain map, a real-time temperature distribution map, and a real-time obstacle distribution map.

[0118] It can be understood that in this step, the real-time terrain map, real-time temperature distribution map and real-time obstacle distribution map are fused to generate a second environment perception map, and data fusion algorithms (such as Kalman filtering and Bayesian networks) are used to comprehensively analyze different types of data to ensure the consistency and accuracy of each data layer; the final second environment perception map provides comprehensive real-time environmental information, including terrain features, temperature distribution and obstacle locations, to provide support for the flight decision-making and path optimization of the UAV.

[0119] In summary, edge computing can process and analyze the data collected by drones in real time, reduce data transmission delays, and improve response speed. Data fusion and comprehensive analysis ensure the accuracy and comprehensiveness of the environmental perception map, providing reliable data support for the safe flight of drones. The second environmental perception map provides key environmental information for the drone's path planning and flight strategy, helping drones complete tasks safely and efficiently in complex environments.

[0120] S500, obtaining other data information collected by the drone, inputting the other data information and the second environment perception map into the data fusion system, preprocessing the data in the data fusion system, and using the data fusion algorithm to fuse the data from different sensors, and generate a high-precision three-dimensional environment model, wherein the other data information includes high-definition images and video streams.

[0121] It can be understood that in this step, the real-time terrain map is fused with the UAV's LiDAR data and high-definition image data in the data fusion system to generate terrain data; the real-time temperature distribution map is fused with the UAV's infrared sensor data in the data fusion system to generate temperature data; the real-time obstacle distribution map is fused with the UAV's high-definition image and video data in the data fusion system to generate obstacle data.

[0122] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0123] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A real scene modeling UAV system, characterized in that: include: Acquisition module: used to use the multimodal sensor carried by the drone to perform a preliminary scan of the real scene, obtain the terrain, temperature distribution and obstacle information in the real scene, and form a first environment perception map, wherein the first environment perception map includes a preliminary terrain map, a temperature distribution map and an obstacle distribution map, and the multimodal sensor includes a laser radar, an infrared sensor and a high-definition camera; Allocation module: used to use the improved ant colony algorithm to divide the task area and allocate the UAV tasks, and dynamically adjust the allocation strategy according to the first environmental perception map to generate the initial flight path; Training optimization module: used to pre-train the acquired historical flight data according to the reinforcement learning algorithm, optimize and adjust the flight path and posture of the drone, and obtain real-time environmental feedback reports; Analysis module: used to analyze the data in the real-time environment feedback report using the edge computing module to generate a second environment perception map, wherein the second environment perception map includes a real-time terrain map, a real-time temperature distribution map, and a real-time obstacle distribution map; Fusion module: used to obtain other data information collected by the drone, input other data information and the second environment perception map into the data fusion system, pre-process the data in the data fusion system, and use the data fusion algorithm to fuse the data from different sensors and generate a high-precision three-dimensional environment model, where other data information includes high-definition images and video streams; The acquisition module includes: Fitting unit: used to pre-process the point cloud data in the real scene collected by the laser radar carried by the drone, and randomly select the plane model fitted by the first point, the second point and the third point from the pre-processed point cloud data, evaluate the fitness of the plane model, determine the plane model containing the most internal points, and record it as the optimal plane model, where the point cloud data includes ground points, building points and other obstacle points; Segmentation unit: used to extract all points in the point cloud data that meet the internal point conditions in the optimal plane model, record them as real ground points, use clustering algorithm to segment the real ground points, identify the ground in different areas, and obtain a preliminary topographic map. The segmentation process is to select a real ground point as a core point, determine whether the number of points in the neighborhood of the core point reaches a preset threshold, and if the threshold is reached, classify the core point and its neighborhood points into a cluster, and continue to expand the cluster until no new core points can be added. Repeat the above process until all real ground points are classified or marked as noise points; The first processing unit is used to process the thermal imaging data collected by the infrared sensor carried by the UAV through the Gaussian filter algorithm, and obtain the temperature distribution map through thermal imaging data smoothing and temperature value conversion; The second processing unit is used to obtain an obstacle distribution map through real-time target detection and obstacle position recognition based on the image data collected by the high-definition camera on the drone after being processed by the YOLO deep learning algorithm; Summarizing unit: used to summarize the preliminary terrain map, temperature distribution map and obstacle distribution map to obtain a first environment perception map; The allocation module includes: Analysis unit: used for analyzing the terrain complexity, temperature anomaly and obstacle density in the mission area through an image processing algorithm based on the first environment perception map, and obtaining a mission area complexity map, which includes a comprehensive evaluation of terrain complexity, temperature anomaly and obstacle density; Allocation unit: used to allocate weights to the task areas according to the task area complexity map, using the hierarchical analysis method and the multi-criteria decision-making method, divide the task areas into priority areas, obtain the priority area map, and mark the priority of each area on the priority area map, where the priorities include high-risk areas, medium-priority areas and low-priority areas. High-risk areas include areas with complex terrain, abnormal temperature and dense obstacles; Iteration unit: used to simulate the movement of ants in each sub-area according to the priority area map, find the optimal task allocation scheme through multiple iterations, and generate a task allocation map; the training optimization module includes: Extraction unit: used to extract terrain and obstacle features based on historical flight data through reinforcement learning algorithm pre-training model, input the extracted terrain and obstacle features into the reinforcement learning model, and train the model's flight strategy in different environments; Computing unit: used to construct a flight topology map based on the flight strategy, identify key nodes using graph theory algorithms, and calculate key nodes and connection strengths between nodes using the shortest path algorithm; Path adjustment unit: It is used to take key nodes and connection strengths as input values ​​of the deep reinforcement learning model, train the deep reinforcement learning model using historical data and real-time environmental data, dynamically adjust the flight path according to the real-time status, and generate a real-time flight path and attitude adjustment strategy.

2. The real scene modeling UAV system according to claim 1, characterized in that: The computing unit includes: Define unit: It is used to divide the flight area into several polygons based on terrain and obstacle features using the Thiessen polygon partitioning method. Each polygon represents a terrain unit or obstacle unit. The center point of each polygon is defined as a node, and edges are established between adjacent nodes. The weight of the edge is based on the distance between the two nodes, the flight cost and the risk level. Marking node unit: used to calculate the importance of each node using graph theory algorithms, and mark nodes with high values ​​as key nodes, where key nodes include high-altitude terrain, narrow passages, and major obstacles; Calculation path unit: used to adjust the edge weights in the flight topology graph according to the importance of key nodes and the distance of edges, and use the shortest path algorithm to calculate the shortest path from the starting point to the end point, while recording the connection strength between each node, where the connection strength can be comprehensively calculated based on the path length, the number of key nodes passed through, and the risk level index of the path; Update flight unit: used to mark flight paths and dynamically update flight topology maps based on real-time environmental data and flight mission requirements. The marking process marks paths with high connection strength as priority flight paths and excludes paths with low connection strength or high risk.

3. A method for controlling a real-scene modeling UAV, based on the real-scene modeling UAV system according to claim 1, characterized in that: include: Using the multimodal sensor carried by the drone to perform a preliminary scan of the real scene, obtain the terrain, temperature distribution and obstacle information in the real scene, and form a first environmental perception map, wherein the first environmental perception map includes a preliminary terrain map, a temperature distribution map and an obstacle distribution map, and the multimodal sensor includes a laser radar, an infrared sensor and a high-definition camera; The improved ant colony algorithm is used to divide the task area and allocate the UAV tasks, and the allocation strategy is dynamically adjusted according to the first environmental perception map to generate the initial flight path; Use reinforcement learning algorithms to pre-train the acquired historical flight data, optimize and adjust the UAV flight path and attitude, and obtain real-time environmental feedback reports; The edge computing module is used to analyze the data in the real-time environment feedback report to generate a second environment perception map, wherein the second environment perception map includes a real-time terrain map, a real-time temperature distribution map, and a real-time obstacle distribution map; Acquire other data information collected by the drone, input the other data information and the second environment perception map into the data fusion system, pre-process the data in the data fusion system, and use the data fusion algorithm to fuse the data from different sensors to generate a high-precision three-dimensional environmental model, where the other data information includes high-definition images and video streams.

4. The real scene modeling UAV control method according to claim 3 is characterized in that: The multimodal sensor carried by the drone is used to perform a preliminary scan of the real scene to obtain terrain, temperature distribution and obstacle information in the real scene, and form a first environmental perception map, which includes: According to the point cloud data in the real scene collected by the laser radar carried by the drone, the point cloud data is preprocessed, and the plane model fitted by the first point, the second point and the third point is randomly selected from the preprocessed point cloud data, and the fitness of the plane model is evaluated to determine the plane model containing the most internal points and record it as the optimal plane model, wherein the point cloud data includes ground points, building points and other obstacle points; All points in the point cloud data that meet the internal point conditions in the optimal plane model are extracted and recorded as real ground points. The real ground points are segmented using a clustering algorithm to identify the ground in different areas and obtain a preliminary topographic map. The segmentation process is to select a real ground point as a core point and determine whether the number of points in the neighborhood of the core point reaches a preset threshold. If the threshold is reached, the core point and its neighborhood points are classified into a cluster, and the cluster is continued to expand until no new core points can be added. The above process is repeated until all real ground points are classified or marked as noise points. The thermal imaging data collected by the infrared sensor carried by the drone is processed by the Gaussian filter algorithm, and the temperature distribution map is obtained by smoothing the thermal imaging data and converting the temperature value; Based on the image data collected by the high-definition camera on the drone, the obstacle distribution map is obtained through real-time target detection and obstacle position recognition after being processed by the YOLO deep learning algorithm; The preliminary terrain map, temperature distribution map and obstacle distribution map are summarized to obtain the first environment perception map.

5. The real scene modeling UAV control method according to claim 3 is characterized in that: The improved ant colony algorithm is used to divide the task area, including: Based on the first environmental perception map, the terrain complexity, temperature anomaly and obstacle density in the mission area are analyzed by image processing algorithm to obtain a mission area complexity map, which includes a comprehensive evaluation of terrain complexity, temperature anomaly and obstacle density; According to the task area complexity map, the task area is weighted and divided into priority areas using the analytic hierarchy process and multi-criteria decision-making method to obtain a priority area map, and the priority of each area is marked on the priority area map. The priorities include high-risk areas, medium-priority areas and low-priority areas. High-risk areas include areas with complex terrain, abnormal temperature and dense obstacles. According to the priority area map, the movement of ants in each sub-area is simulated. Through multiple iterations, the optimal task allocation scheme is found and the task allocation map is generated.

6. The real scene modeling UAV control method according to claim 3, characterized in that: The obtained historical flight data is pre-trained according to the reinforcement learning algorithm to optimize and adjust the flight path and posture of the drone to obtain a real-time environmental feedback report, including: Based on historical flight data, the model is pre-trained through reinforcement learning algorithm to extract terrain and obstacle features, and the extracted terrain and obstacle features are input into the reinforcement learning model to train the model's flight strategies in different environments; According to the flight strategy, a flight topology map is constructed, key nodes are identified using graph theory algorithms, and key nodes and connection strengths between nodes are calculated using the shortest path algorithm; Key nodes and connection strengths are used as input values ​​of the deep reinforcement learning model. Historical data and real-time environmental data are used to train the deep reinforcement learning model. The flight path is dynamically adjusted according to the real-time status to generate a real-time flight path and attitude adjustment strategy.

7. The real scene modeling UAV control method according to claim 6, characterized in that: According to the flight strategy, a flight topology graph is constructed, key nodes are identified using graph theory algorithms, and key nodes and connection strengths between nodes are calculated using the shortest path algorithm, including: Based on the terrain and obstacle characteristics, the flight area is divided into several polygons using the Thiessen polygon partitioning method. Each polygon represents a terrain unit or an obstacle unit. The center point of each polygon is defined as a node, and edges are established between adjacent nodes. The weight of the edge is based on the distance between the two nodes, the flight cost, and the risk level. Using graph theory algorithms, the importance of each node is calculated and nodes with high values ​​are marked as key nodes, where key nodes include high-altitude terrain, narrow passages, and major obstacles; According to the importance of key nodes and the distance of edges, the edge weights in the flight topology graph are adjusted, and the shortest path algorithm is used to calculate the shortest path from the starting point to the end point, while recording the connection strength between each node. The connection strength can be comprehensively calculated based on the path length, the number of key nodes passed through, and the risk level index of the path; Mark the flight path and dynamically update the flight topology map based on real-time environmental data and flight mission requirements. The marking process marks the paths with high connection strength as priority flight paths and excludes the paths with low connection strength or high risk.

Images