Dynamic path planning method and system of unmanned aerial vehicle based on multi-modal perception

Abstract

The invention discloses a dynamic path planning method and system for an unmanned aerial vehicle based on multi-modal sensing, and relates to the technical field of dynamic path planning methods, and the method comprises the steps: constructing multi-modal sensing data of each environment region; the area form of each environment area is determined based on the identification of the multi-mode sensing data of each environment area, the flight stereo scene graph of the unmanned aerial vehicle is determined according to the area form and the area position of each environment area and the current position of the unmanned aerial vehicle, and the accuracy of the flight stereo scene graph of the unmanned aerial vehicle is improved. Determining an obstacle in front of the unmanned aerial vehicle according to a preset inspection path of the unmanned aerial vehicle, the current position of the unmanned aerial vehicle and the flight three-dimensional scene graph, and determining a dynamic avoidance path according to the obstacle range of the obstacle in front and the form of the unmanned aerial vehicle; and determining the dynamic path of the unmanned aerial vehicle according to the remaining inspection paths, each dynamic avoidance path and the electric quantity of the unmanned aerial vehicle, thereby improving the planning accuracy of the dynamic path of the unmanned aerial vehicle.

Description

Technical Field

[0001] The present invention relates to the technical field of dynamic path planning methods, and in particular to a dynamic path planning method and system for an unmanned aerial vehicle (UAV) based on multimodal perception. Background Art

[0002] With the development of science and technology, drones are gradually used in various inspection tasks. Drones fly in the air and conduct circumferential inspections of the areas to be inspected. In the existing technology, drones use visual detectors to shoot the surrounding environment to collect corresponding environmental image data, determine the corresponding environmental features through the recognition of environmental image data, and determine the drone's flight stereo scene map based on the synthesis of multiple environmental features. The environmental image data is used for single-dimensional control and cannot form corresponding multimodal perception data, which affects the accuracy of the drone's flight stereo scene map and makes the drone's dynamic path planning accuracy low. Summary of the Invention

[0003] The purpose of the present invention is to overcome the deficiencies of the prior art and to provide a method and system for dynamic path planning of a drone based on multimodal perception.

[0004] An embodiment of the present invention provides a dynamic path planning method for an unmanned aerial vehicle based on multimodal perception, comprising: When the UAV is in the inspection state, the UAV's integrated module collects the UAV's environmental data set, and the integrated module integrates radar, visual detector and thermal imaging module; Determine a point cloud data combination, an image data combination, and a thermal imaging data combination based on the environmental data set to construct multimodal perception data of each environmental area; Determine the regional morphology of each environmental area based on the recognition of multimodal perception data of each environmental area, and determine the flight stereoscopic scene map of the drone based on the regional morphology, regional location and current location of each environmental area; Determine the obstacle in front of the drone based on the drone's preset inspection path, the drone's current position, and the flight 3D scene map, and determine the dynamic avoidance path based on the obstacle range and the drone's shape; The remaining inspection path of the drone is determined based on the preset inspection path and current position of the drone, and the dynamic path of the drone is determined according to the remaining inspection path, various dynamic avoidance paths and the power of the drone.

[0005] An embodiment of the present invention provides a dynamic path planning system for an unmanned aerial vehicle based on multimodal perception. The dynamic path planning system for an unmanned aerial vehicle based on multimodal perception is applied to the above-mentioned dynamic path planning method for an unmanned aerial vehicle based on multimodal perception. The dynamic path planning system for an unmanned aerial vehicle based on multimodal perception includes: An environmental data module is used to collect environmental data from the drone when the drone is in an inspection state based on the drone's integrated module, which integrates radar, visual detectors, and thermal imaging modules; A multimodal perception data module is used to determine a point cloud data combination, an image data combination, and a thermal imaging data combination based on an environmental data set to construct multimodal perception data of each environmental area; A flight stereo scene graph module is used to determine the regional morphology of each environmental area based on the recognition of multimodal perception data of each environmental area, and to determine the flight stereo scene graph of the drone based on the regional morphology, regional location and current location of each environmental area; The dynamic avoidance path module is used to determine the obstacle in front of the drone based on the preset inspection path of the drone, the current position of the drone and the flight three-dimensional scene map, and determine the dynamic avoidance path according to the obstacle range of the obstacle in front and the shape of the drone; The dynamic path module is used to determine the remaining inspection path of the drone based on the preset inspection path and current position of the drone, and to determine the dynamic path of the drone based on the remaining inspection path, various dynamic avoidance paths and the power of the drone.

[0006] Compared with the prior art, the present invention has the following beneficial effects: In an embodiment of the present invention, through the method in the embodiment of the present invention, when the UAV is in an inspection state, an environmental data set of the UAV is collected based on the integrated module of the UAV, and the integrated module integrates a radar, a visual detector and a thermal imaging module; based on the environmental data set, a point cloud data combination, an image data combination and a thermal imaging data combination are determined to construct multimodal perception data of each environmental area; based on the identification of the multimodal perception data of each environmental area, the regional morphology of each environmental area is determined, and the flight stereoscopic scene map of the UAV is determined according to the regional morphology, regional position and current position of each environmental area, the multimodal perception data of each environmental area is introduced, and the overall consideration of the regional morphology, regional position and current position of each environmental area is compatible, thereby improving the accuracy of the flight stereoscopic scene map of the UAV.

[0007] Therefore, the obstacle in front of the drone is determined based on the drone's preset inspection path, the drone's current position and the flight stereo scene map, and the dynamic avoidance path is determined based on the obstacle range of the obstacle in front and the shape of the drone; the remaining inspection path of the drone is determined based on the drone's preset inspection path and current position, and the dynamic path of the drone is determined based on the remaining inspection path, each dynamic avoidance path and the drone's power level, thereby ensuring the accuracy of the remaining inspection path, and further dynamically controlling the remaining inspection path, thereby improving the planning accuracy of the drone's dynamic path and improving the drone's inspection compatibility in various environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 1 is a flow chart of a dynamic path planning method for a UAV based on multimodal perception in an embodiment of the present invention; Figure 2 1 is a flow chart of step S11 in the dynamic path planning method for a UAV based on multimodal perception in an embodiment of the present invention; Figure 3 1 is a flow chart of step S12 in the dynamic path planning method for a UAV based on multimodal perception in an embodiment of the present invention; Figure 4 1 is a flow chart of step S13 in the dynamic path planning method for a UAV based on multimodal perception in an embodiment of the present invention; Figure 5 1 is a flow chart of step S14 in the dynamic path planning method for a UAV based on multimodal perception in an embodiment of the present invention; Figure 6 1 is a flow chart of step S15 in the dynamic path planning method for a UAV based on multimodal perception in an embodiment of the present invention; Figure 7 Schematic diagram of the structure of a dynamic path planning system for a UAV based on multimodal perception in an embodiment of the present invention. DETAILED DESCRIPTION

[0009] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0010] See also Figures 1 to 7 A dynamic path planning method for a UAV based on multimodal perception is applied to a dynamic path planning scenario for a UAV. The dynamic path planning method for a UAV based on multimodal perception includes: Step S11: When the UAV is in an inspection state, an integrated module of the UAV is used to collect an environmental data set of the UAV, where the integrated module is integrated with a radar, a visual detector, and a thermal imaging module; Step S12: determining a point cloud data combination, an image data combination, and a thermal imaging data combination based on the environmental data set to construct multimodal perception data of each environmental area; Step S13: determining the regional form of each environmental area based on the recognition of the multimodal perception data of each environmental area, and determining the flight stereoscopic scene map of the UAV according to the regional form and regional position of each environmental area and the current position of the UAV; Step S14: Determine the obstacle in front of the drone based on the preset inspection path of the drone, the current position of the drone, and the flight stereoscopic scene image, and determine a dynamic avoidance path based on the obstacle range of the obstacle and the shape of the drone; Step S15: determining the remaining inspection path of the drone based on the preset inspection path and the current position of the drone, and determining the dynamic path of the drone based on the remaining inspection path, each dynamic avoidance path, and the power of the drone; refer to Figure 2 In step S11, when the UAV is in an inspection state, an integrated module of the UAV is used to collect an environmental data set of the UAV, where the integrated module integrates a radar, a visual detector, and a thermal imaging module; In the specific implementation process of the present invention, the specific steps are: S111: collecting the current flight data and the current inspection data of the UAV, and determining the inspection status of the UAV according to the current flight data and the current inspection data of the UAV; S112: Real-time monitoring of the drone's flight process during inspection. The integrated module is configured at the front end of the drone and collects environmental data as the drone performs dynamic inspections. The integrated module integrates radar, visual detectors, and thermal imaging modules. S113: In the environmental data space, the collection time of each environmental data is marked, and dynamic sorting of each environmental data is triggered according to the sorting of the collection time to form an environmental data set of the UAV.

[0011] In an embodiment of the present application, the current flight data of the drone is collected. At this time, the current flight speed of the drone is usually measured in meters per second (m / s), which reflects the speed of the drone's movement; the height of the drone relative to the ground is usually measured in meters (m), which ensures that the drone operates within a safe flight altitude; the current flight direction of the drone is usually expressed in degrees (°), which is the angle relative to the north direction, which helps to determine the flight trajectory of the drone; the pitch, yaw and roll angles of the drone, these parameters describe the spatial direction of the drone; the latitude and longitude coordinates of the drone or the relative position relative to a certain point, which provides the exact position of the drone in geographic space.

[0012] Collect the drone's current inspection data, including the number of inspection tasks the drone has completed, which helps evaluate the drone's inspection progress; the number of inspection tasks that the drone still needs to complete, which helps plan subsequent task allocation; specific inspection areas have specific requirements, such as specific image resolution, inspection speed, flight altitude, etc. These requirements vary depending on the inspection target; the drone's current battery level, which is crucial to ensure that the drone has enough power to complete the remaining tasks.

[0013] Combining current flight data with inspection data, the algorithm assesses the drone's inspection status. For example, if the drone's speed is too slow or its altitude is too low, it may have encountered an obstacle or adverse weather conditions. If the number of completed tasks approaches the number of remaining tasks, the inspection is nearing completion. Based on the inspection status assessment, the algorithm automatically adjusts the drone's inspection strategy. For example, if the battery is low, the drone will prioritize completing the most important remaining tasks and search for the nearest charging station. Specifically, suppose a drone is performing a power line inspection mission. During the inspection, the drone collects the following current flight data and inspection data: Flight data: speed = 15m / s, altitude = 50m, heading = 90° (due east), stable attitude, and the position is the latitude and longitude coordinates of a point in the city center. Inspection data: number of completed tasks = 10 (a total of 20 power lines), number of remaining tasks = 10, the inspection area requires an image resolution of no less than 1080p, the flight altitude is maintained at around 50m, and the battery status is 70% (a full charge is 100%). Based on this data, the algorithm evaluates The drone's inspection status is normal, but the battery level needs to be noted. To ensure sufficient battery power to complete the remaining tasks, the algorithm prioritizes the inspection of the most important power lines and searches for the nearest charging station for charging when the battery level drops below 50%. At the same time, since the inspection area requires an image resolution of no less than 1080p, the algorithm also ensures that the drone's camera settings meet this requirement. This specific example shows the importance of step S111 in practical applications, as it ensures that the drone can flexibly adjust its inspection strategy based on actual conditions, improving inspection efficiency and safety.

[0014] Furthermore, the drone is monitored in real time through its built-in flight control system, which can continuously track the drone's key flight parameters such as position, speed, altitude, and heading; various sensors on the drone (such as gyroscopes, accelerometers, GPS, etc.) provide data to the flight control system in real time to ensure that the drone can fly stably and follow the predetermined route; the flight control system also has an anomaly detection function, which can promptly detect and respond to any abnormal conditions during the drone's flight, such as sudden wind changes, mechanical failures, etc.

[0015] To maximize the efficiency and accuracy of data collection, radar, visual detectors, and thermal imaging modules are carefully configured at the front of the drone. This layout ensures that the drone can directly face and capture environmental information in front of it during flight. These modules are connected to the flight control system through internal interfaces within the drone, enabling real-time data transmission and processing. At the same time, the radar module detects obstacles and terrain changes ahead by emitting and receiving electromagnetic waves, generating point cloud data that provides three-dimensional structural information about the environment. Visual detectors (such as high-definition cameras) capture visible light images ahead, providing detailed visual information about the environment that helps identify specific objects and scenes. The thermal imaging module generates thermal images by capturing infrared radiation ahead, revealing the thermal signature of objects, which is very useful for detecting hidden heat sources or areas of abnormal temperature. At this point, the drone is equipped with a dedicated environmental data storage space for collected radar data, image data, and thermal imaging data. To ensure data storage efficiency and readability, the collected data is converted to a specific format and compressed as necessary. To prevent data loss or tampering, the environmental data space also has data encryption and backup functions.

[0016] Therefore, in the environmental data space, the collection time of each environmental data is marked, and the dynamic sorting of each environmental data is triggered according to the sorting of the collection time to form the environmental data set of the drone, which introduces the formation of the environmental data set of the drone.

[0017] At this time, when the drone collects each piece of environmental data (such as radar data, image data, and thermal imaging data), the system automatically adds a timestamp to each piece of data. This timestamp records the exact time when the data was collected. To ensure the timeliness of the data, the timestamp usually has high precision, such as milliseconds or higher, which helps to accurately track the changing trends of the data in subsequent data processing and analysis.

[0018] After collecting a certain amount of environmental data, the system will sort the data according to the timestamp; the sorting algorithm is usually based on the comparison of timestamps to ensure that the data is arranged in the order of collection time; because the drone will continuously collect data during the inspection process, the sorting process needs to be dynamic; that is, every time new data is collected and added to the environmental data space, the system will automatically update the sorting results.

[0019] Sorting is triggered based on a variety of conditions, such as the collection of new data, user requests, preset time intervals, etc. Once the triggering conditions are met, the system will perform the sorting operation. After the sorting is completed, all environmental data arranged in chronological order will form a complete data set. This set contains all relevant environmental information collected by the drone during the inspection process, providing a basis for subsequent data processing and analysis.

[0020] Specifically, suppose a drone is conducting a power line inspection. During the inspection, the drone flies along a predetermined route and collects environmental data in real time. During flight, the radar module, visual detector, and thermal imaging module collect terrain data, visible light images, and infrared images of the coastline, respectively. Each data point is automatically timestamped when collected, such as "2023-05-15 10:00:01.234" (year-month-day hour:minute:second.millisecond).

[0021] As the inspection progresses, the drone continuously collects new data; the system sorts these data according to their timestamps, ensuring that the earliest collected data is in front and the latest collected data is in the back. For example, if radar data is collected first, then image data, and finally thermal imaging data, then the sorted data sets will be arranged in this order. When the drone completes the inspection task or reaches the preset data collection volume, the system integrates all the sorted environmental data into a complete data set. This set contains all the environmental information related to coastline erosion collected by the drone during the inspection, such as terrain changes, vegetation cover, sea water temperature, etc. This data will be used in subsequent coastline erosion analysis and early warning systems.

[0022] refer to Figure 3 , in step S12, determining a point cloud data combination, an image data combination, and a thermal imaging data combination based on the environmental data set to construct multimodal perception data of each environmental area; In the specific implementation process of the present invention, the specific steps are: S121: Collecting an environmental data set, traversing the environmental data set, and marking corresponding data types and corresponding environmental area labels for each environmental data, and determining a plurality of point cloud data, a plurality of image data, and a plurality of thermal imaging data based on matching of the environmental data set and the data types; S122: determining a point cloud data combination based on a combination of a plurality of point cloud data; determining an image data combination based on a combination of a plurality of image data; and determining a thermal imaging data combination based on a combination of a plurality of thermal imaging data. S123: In the environmental data space of the UAV, the environmental area markers are aligned for the point cloud data combination, the image data combination, and the thermal imaging data combination. At this time, the corresponding point cloud data, image data, and thermal imaging data are presented in the same environmental area marker, and multimodal perception data of each environmental area is constructed based on the point cloud data, image data, and thermal imaging data.

[0023] In an embodiment of the present application, a set of collected and stored environmental data is obtained from the storage system or real-time data stream of the drone. These data are usually collected in real time during the inspection process of the drone and have been sorted and integrated according to chronological order or other standards; a loop structure in a programming language (such as a for loop, a while loop) is used to access each data item in the environmental data set one by one; the purpose of the traversal is to process and mark each data item.

[0024] During the traversal process, the content or format of each data item is checked to determine its data type; data types include point cloud data, image data, thermal imaging data, etc.; according to the different data types, corresponding tags or labels are added to each data item; at the same time, during the traversal process, it is also necessary to determine the environmental area to which each data item belongs. This is achieved by analyzing the geographic location information in the data content, preset map data or area tags entered by the user; and a tag indicating the environmental area to which each data item belongs is added to each data item.

[0025] After traversal and labeling are completed, the environmental data set is split into multiple subsets according to the data type and the labeling of the environmental area; each subset contains all data items belonging to the same data type and the same environmental area; for example, multiple point cloud data subsets, multiple image data subsets, and multiple thermal imaging data subsets are obtained.

[0026] Specifically, assume that a drone collects the following environmental data during a power line inspection mission and stores this data in an environmental data set: Data Item 1: Point cloud data representing the topographical features of a certain area along the coastline, collected at 8:00 AM and labeled "Area A"; Data Item 2: Image data showing a visible light image of a certain area along the coastline, collected at 8:10 AM and also labeled "Area A"; Data Item 3: Thermal imaging data showing the seawater temperature distribution in the same area, collected at 8:15 AM and also labeled "Area A"; Data Item 4: Point cloud data representing the topographical features of another area along the coastline, collected at 9:00 AM and labeled "Area B"; Data Item 5: Image data showing a visible light image of "Area B" collected at 9:10 AM; ... (other data items omitted). The above environmental data set is obtained from the storage system of the UAV; a for loop is used to iterate over each data item in the environmental data set; data item 1 is marked as point cloud data; data item 2 is marked as image data; data item 3 is marked as thermal imaging data; ... (the data type markings of other data items are omitted); data items 1, 2, and 3 are all marked as belonging to "area A"; data items 4 and 5 are both marked as belonging to "area B"; ... (the environmental area markings of other data items are omitted).

[0027] Based on the data type and environmental area labeling, the environmental data set is split into: point cloud data subset 1: containing data item 1 (point cloud data belonging to "area A"); image data subset 1: containing data item 2 (image data belonging to "area A"); thermal imaging data subset 1: containing data item 3 (thermal imaging data belonging to "area A"); point cloud data subset 2: containing data item 4 (point cloud data belonging to "area B"); image data subset 2: containing data item 5 (image data belonging to "area B"); ... (other data subsets are omitted). In this way, the traversal and labeling of the environmental data set are completed, and multiple data subsets split by data type and environmental area are obtained. These subsets are used for subsequent tasks such as environmental monitoring, target recognition, or anomaly detection.

[0028] Furthermore, multiple point cloud data belonging to the same environmental area or having some correlation are combined together to form a point cloud data set or dataset; at this time, the data items to be combined are filtered out from all point cloud data according to the environmental area tag or other related attributes (such as timestamp, acquisition height, etc.); these filtered point cloud data are integrated into a data structure, such as a list, array or a special point cloud data format.

[0029] Combine multiple image data belonging to the same environmental area or with time series correlation to form an image data set; at the same time, filter out the image data that need to be combined according to attributes such as environmental area tags or timestamps; integrate the filtered image data into a data structure such as a list or image sequence in a certain order (such as time series).

[0030] Combine multiple thermal imaging data belonging to the same environmental area or having some correlation to form a thermal imaging data set; at this time, filter out the thermal imaging data to be combined according to attributes such as environmental area tag, timestamp or temperature range; integrate the filtered thermal imaging data into a data structure, such as a list or a special thermal imaging data format.

[0031] Specifically, assume that in a power line inspection mission, the drone has traversed and marked the environmental data according to step S121 and obtained the following data subsets: point cloud data subset: contains all point cloud data belonging to "area A"; image data subset: contains all image data belonging to "area A", and these image data are arranged in time series; thermal imaging data subset: contains all thermal imaging data belonging to "area A", and these data also correspond to the image data in time series; All point cloud data belonging to "Region A" are filtered out from the point cloud data subset; these data are integrated into a point cloud data set to form a complete point cloud data set containing the terrain features of "Region A".

[0032] All image data belonging to "region A" and arranged in time series are filtered out from the image data subset; these data are integrated into an image dataset to form a set containing visible light image sequences of "region A", which is used for tasks such as time series analysis, video generation or dynamic monitoring.

[0033] All thermal imaging data belonging to "Region A" and corresponding to the image data in time series are filtered out from the thermal imaging data subset; these data are integrated into a thermal imaging dataset to form a set containing the seawater temperature distribution sequence of "Region A". This set is used in combination with the image data set for multimodal analysis or anomaly detection.

[0034] Therefore, in the environmental data space of the UAV, the point cloud data combination, image data combination and thermal imaging data combination are aligned for environmental area markings. At this time, the corresponding point cloud data, image data and thermal imaging data are presented in the same environmental area mark, and multimodal perception data of each environmental area is constructed based on the point cloud data, image data and thermal imaging data. Point cloud data, image data and thermal imaging data are introduced to construct multimodal perception data of each environmental area.

[0035] At this point, ensure that the point cloud data, image data, and thermal imaging data under the same environmental area mark are matched in time and space, that is, they describe the environmental information at the same time and place; at this point, check the timestamp information of each data item to ensure that the data belonging to the same time point are combined together; if there are slight differences in the timestamps (such as due to delays in data acquisition and processing), interpolation or averaging is required based on the proximity of the timestamps; secondly, use geographic coordinate systems (such as GPS coordinates) or relative position information (such as the correspondence between pixel positions in the image and three-dimensional coordinates in the point cloud) to ensure that the data are spatially matched, which requires the use of a registration algorithm to align the spatial positions in different data sets; finally, verify whether the environmental area mark of each data item is consistent to ensure that no data is incorrectly assigned to other areas.

[0036] Combine the aligned point cloud data, image data, and thermal imaging data to form multimodal perception data of the environmental area containing multiple data types; at the same time, design a data structure to store the multimodal perception data. This structure should be able to accommodate different types of data and allow easy access and processing of these data; for example, use a class or structure containing multiple fields, each field corresponding to a data type; integrate the aligned point cloud data, image data, and thermal imaging data into the designed data structure; ensure that the data of each data type is correctly organized according to the environmental area tag and timestamp; optionally, according to application requirements, preprocess the integrated multimodal perception data, such as normalization, dimensionality reduction, feature extraction, etc., to improve the accuracy and efficiency of subsequent analysis.

[0037] Specifically, assuming that in a power line inspection mission, the drone has traversed, marked, and combined the environmental data according to steps S121 and S122, and obtained the following data sets: point cloud data set: containing all point cloud data of "area A", which describes the three-dimensional terrain of a certain area of ​​the coastline; image data set: containing all image data of "area A", which shows the visible light image of the coastline in the form of a time series; thermal imaging data set: containing all thermal imaging data of "area A" corresponding to the image data time series, which shows the sea water temperature distribution in the same area.

[0038] Check the timestamp of each data item to ensure that the point cloud data, image data, and thermal imaging data are all collected at the same time; in this example, assume that all data were collected between 8 and 9 am, and the data items in each data set are arranged in chronological order; use a registration algorithm to spatially align the point cloud data with the image data and thermal imaging data, which requires using the feature points in the image and the corresponding points in the point cloud to establish a transformation matrix to achieve accurate alignment of the data; in this example, assume that an automatic registration algorithm has been used to align all data; verify the environmental region labeling of each data item to ensure that all data belongs to "Region A"; in this example, assume that the environmental region labeling of all data is correct.

[0039] Design a class containing point cloud data fields, image data fields, and thermal imaging data fields to store multimodal perception data. This class should also contain environmental area tags and timestamp fields for subsequent data access and processing; integrate the aligned point cloud data, image data, and thermal imaging data into the designed data structure; in this example, a multimodal perception data object is created, and the data items in each data set are filled into the corresponding fields according to the timestamp and environmental area tag; optionally, the integrated multimodal perception data is preprocessed according to application requirements; in this example, it is assumed that the subsequent analysis task requires the extraction of edge features in the image and height features in the point cloud, so image processing algorithms and point cloud processing algorithms are used to extract these features and add them as additional fields to the multimodal perception data object; through step S123, a multimodal perception data set of "region A" containing point cloud data, image data, and thermal imaging data is obtained. This set is used for subsequent environmental monitoring, target recognition, anomaly detection, or data analysis tasks, providing comprehensive data support for coastline erosion inspections.

[0040] refer to Figure 4 In step S13, the regional form of each environmental area is determined based on the recognition of the multimodal perception data of each environmental area, and the flight stereoscopic scene map of the UAV is determined according to the regional form, regional position and current position of each environmental area; In the specific implementation process of the present invention, the specific steps are: S131: Determine, in the multimodal perception data of each environmental area, a first sub-area morphology based on a synthesis of point cloud data and image data, and determine a second sub-area morphology based on a synthesis of point cloud data and thermal imaging data; S132: Determine the regional morphology of the environmental region based on a synthesis of the regional position of the environmental region, the morphology of the first sub-region, and the morphology of the second sub-region, so as to collect the regional morphology of each environmental region; S133: Mark the corresponding regional form and regional position in each environmental area, determine the first scene according to the current position of the drone and the regional form of each environmental area, determine the second scene according to the current position of the drone and the regional position of each environmental area, based on the first scene, the second scene and the flight stereoscopic scene map.

[0041] In an embodiment of the present application, in the multimodal perception data of each environmental area, the first sub-area morphology is determined based on the synthesis of point cloud data and image data, and the second sub-area morphology is determined based on the synthesis of point cloud data and thermal imaging data, which is compatible with the overall consideration of the synthesis of point cloud data and thermal imaging data and ensures the accuracy of the second sub-area morphology.

[0042] At this point, the point cloud data and image data are preprocessed, including denoising, filtering, and registration, to ensure the quality and consistency of the data; registration is a key step, which ensures that the three-dimensional coordinates in the point cloud correctly correspond to the pixel positions in the image, which is usually achieved through feature matching and transformation matrix calculation; the preprocessed point cloud data and image data are fused to generate a synthetic view containing three-dimensional geometric information and two-dimensional texture information, which is achieved by applying the image data as a texture map to the point cloud data, or using a projection method to map the image information into three-dimensional space.

[0043] Morphological analysis is performed on the synthetic view to extract key features, such as terrain undulations, building outlines, vegetation distribution, etc. This is achieved through image processing technologies such as edge detection, region segmentation, and feature extraction. Based on the results of the morphological analysis, the morphology of the first sub-region is determined, including the three-dimensional structure of the terrain, texture characteristics, main objects and obstacles, etc.

[0044] The point cloud data and thermal imaging data are preprocessed and aligned to ensure the spatial correspondence between them; the temperature information in the thermal imaging data is mapped to the point cloud data to generate a synthetic view containing 3D geometric information and temperature distribution information. This is achieved by converting the pixel values ​​in the thermal imaging data into temperature values ​​and assigning these values ​​to the corresponding point cloud data points.

[0045] The temperature information in the synthetic view is analyzed to detect temperature anomaly areas, such as hot spots, cold spots, or areas with significant temperature gradient changes. This is achieved by setting temperature thresholds, calculating temperature gradients, or using clustering algorithms. Based on the results of temperature anomaly detection, the morphology of the second sub-area is determined, including the location, range, intensity, and relationship of the temperature anomaly with the terrain structure.

[0046] Specifically, suppose a drone is performing a power line inspection mission, and its flight path covers multiple environmental areas, including forests, farmlands, and urban edges. The drone first collects point cloud data and image data of the forest area. Through preprocessing and alignment steps, the precise correspondence between the point cloud and the image is ensured. The image data is applied to the point cloud data as a texture map to generate a synthetic view containing the three-dimensional structure and texture information of the forest. Through morphological analysis, key features such as tree outlines, terrain undulations, and vegetation distribution in the forest are extracted. Finally, the morphology of the first sub-area, namely the three-dimensional structure and texture characteristics of the forest, as well as obstacles (such as large trees, rocks, etc.), are determined.

[0047] The drone also collected thermal imaging data of the forest area; the temperature information in the thermal imaging data was mapped onto point cloud data to generate a composite view containing the forest's three-dimensional structure and temperature distribution information; through temperature anomaly detection, an area with significantly increased temperature was discovered, namely a potential fire area; the location, scope and intensity of the fire area were analyzed, as well as its relationship with the terrain structure (such as whether it is close to flammable materials and whether there are wind ducts, etc.); and finally the morphology of the second sub-area was determined, namely the location, scope and intensity of the forest fire, and its interaction with the surrounding environment.

[0048] Furthermore, the regional morphology of the environmental area is determined based on the synthesis of the regional position of the environmental area, the first sub-regional morphology and the second sub-regional morphology, so as to collect the regional morphology of each environmental area, which is compatible with the overall consideration of the synthesis of the regional position of the environmental area, the first sub-regional morphology and the second sub-regional morphology, and ensures the accuracy of the regional morphology of the environmental area.

[0049] At this time, the regional location information (such as latitude and longitude, altitude, terrain type, etc.) of each environmental area, the first sub-area morphology (based on the synthesis result of point cloud and image data), and the second sub-area morphology (based on the synthesis result of point cloud and thermal imaging data) are summarized; ensure that all collected information is consistent in space and time, that is, they correspond to the same environmental area and the same time point.

[0050] The morphology of the first and second sub-regions is fused to generate a comprehensive view that includes 3D geometry, texture information, temperature distribution, and anomalies or feature points. Spatial analysis is performed on this comprehensive view to understand the spatial relationships between different morphological features; for example, analyzing the relationship between terrain relief and temperature distribution, or identifying the association between specific textures and potential obstacles.

[0051] Extract key features from the comprehensive view, which can comprehensively describe the morphology of the environmental area. These features include the three-dimensional structure of the terrain, the location and shape of major objects, the distribution and intensity of temperature anomalies, etc.; construct a morphological description of the environmental area based on the extracted features; the morphological description should be detailed enough so that the drone can accurately identify and understand the structure and characteristics of the environmental area; store the morphological description of each environmental area in a database for subsequent query and analysis.

[0052] Specifically, suppose a drone is performing a power line inspection mission and needs to conduct a detailed morphological analysis of the disaster area in order to plan rescue routes and allocate resources; the drone first collects the regional location information of the disaster area, including latitude and longitude, altitude and terrain type; then, the drone collects point cloud data, image data and thermal imaging data of the disaster area, and generates the first sub-area morphology (based on the synthesis result of point cloud and image data) and the second sub-area morphology (based on the synthesis result of point cloud and thermal imaging data) respectively.

[0053] The drone fuses the morphology of the first and second sub-regions to generate a comprehensive view of the disaster area that includes three-dimensional geometric information, texture information, temperature distribution, and potential anomalies. In this comprehensive view, the drone conducts spatial analysis and finds a clear correlation between the topographic undulations and temperature distribution in the disaster area. For example, low-lying areas tend to have higher temperatures and are potential areas for water accumulation or fire. From this comprehensive view, the drone extracts key features, including the three-dimensional structure of the terrain, the location and shape of major obstacles (such as collapsed buildings and trees), and the distribution and intensity of temperature anomalies. Based on these features, the drone constructs a morphological description of the disaster area. For example, it describes the topographic undulations of the disaster area, the location and extent of water accumulation areas, and the location and intensity of potential fire points. Finally, the drone stores the morphological description of the disaster area in a database for subsequent query and analysis. This information is crucial for planning rescue routes, allocating resources, and formulating disaster response strategies.

[0054] Therefore, the corresponding regional morphology and regional position are marked in each environmental area, and the first scene is determined according to the current position of the drone and the regional morphology of each environmental area. The second scene is determined according to the current position of the drone and the regional position of each environmental area. Based on the first scene, the second scene and the flight stereoscopic scene graph, at the same time, the multimodal perception data of each environmental area is introduced, which is compatible with the overall consideration of the regional morphology, regional position and current position of each environmental area, thereby improving the accuracy of the drone's flight stereoscopic scene graph.

[0055] At this time, in the environmental region database, the corresponding regional morphology and regional location information are marked for each environmental region; the regional morphology includes the three-dimensional structure, texture characteristics, temperature distribution, etc. of the terrain, while the regional location includes geographical coordinates such as longitude and latitude and altitude; ensure that the regional morphology and regional location information are associated with the unique identifier of the environmental region so that they can be quickly retrieved and used in subsequent steps.

[0056] The drone uses its built-in GPS system, inertial navigation system or visual positioning system to perceive its current position in real time; the drone's current position is matched with the regional morphological information in the environmental area database to determine the environmental area and its morphology in which the drone is currently located; based on the matching results, the first scene is constructed, that is, a detailed morphological description of the environmental area in which the drone is currently located; the first scene includes information such as terrain undulations, obstacle distribution, and temperature anomalies, which are crucial to the drone's flight safety and mission execution.

[0057] Analyze the relative positional relationship between the drone's current position and various environmental areas, including information such as the distance and direction between the drone and the nearest environmental area, target point or danger zone. Based on the results of the positional relationship analysis, construct a second scene, which is a macro-environmental view of the drone's current location. The second scene includes information such as multiple environmental areas around the drone, potential obstacles on the flight path, and the location of the target point.

[0058] The first scene and the second scene are matched with a predefined flight stereo scene graph; the flight stereo scene graph is a database containing different environmental areas, flight paths and corresponding flight strategies; based on the matching results, the flight strategy that best suits the current scene is selected from the flight stereo scene graph; the flight strategy includes flight altitude, speed, heading adjustment, obstacle avoidance, etc.; the selected flight strategy is executed, and environmental changes during the flight are monitored in real time through the drone's sensor system; the flight strategy is adjusted in a timely manner according to environmental changes to ensure the drone's safe flight and successful mission execution.

[0059] Specifically, suppose a drone is performing a power line inspection mission and passes through a forest area. It needs to quickly locate the fire point in the forest area and plan a safe flight path. Before the mission begins, the drone has collected environmental data of the forest area through previous flights and built a database containing the morphology and location of each environmental area. Each environmental area is marked with a unique identifier and associated with its corresponding regional morphology (such as terrain, vegetation type, temperature distribution, etc.) and regional location (such as latitude and longitude, altitude, etc.) information.

[0060] Determine the first scenario: When the drone enters a forest area, it uses the GPS system to sense its current location in real time. This location is matched with the regional morphological information in the database to determine the specific environmental area (such as a valley or hillside) where the drone is currently located. This constructs the first scenario, including information such as the terrain undulations, vegetation distribution, and temperature anomalies in the area.

[0061] Determine the second scenario: Analyze the relative position relationship between the drone's current position and the surrounding environment area; determine the distance and direction between the drone and the nearest fire point, safe landing point or potential obstacle; construct a second scenario, including information such as multiple environmental areas around the drone, potential obstacles on the flight path, and the location of the target point.

[0062] Match the first and second scenes with the pre-defined flight stereo scene graph; select the flight strategy that best suits the current scenario from the flight stereo scene graph, such as flying low along the valley to avoid high-temperature areas, flying quickly in open areas to shorten the time to reach the fire point, etc.; execute the selected flight strategy and monitor environmental changes in real time during the flight through the drone's thermal imaging and visual sensors; when new temperature anomalies or potential obstacles are discovered, adjust the flight strategy in a timely manner to ensure the drone's safe flight and successful mission execution; through this example, see the importance of step S133 in drone environmental perception, scene construction, and flight strategy planning; by marking the regional morphology and location, determining the first and second scenes, and combining the flight stereo scene graph for flight strategy planning, the drone can more accurately understand its current environment and make more informed decisions.

[0063] In some embodiments of the present application, the following weights and scoring criteria are assumed: terrain relief: weight 0.3, score (based on relief): 0-10; vegetation density: weight 0.2, score (based on density): 0-10; longitude and latitude range: weight 0.2, score (based on proximity to the mission target): 0-10; altitude: weight 0.1, score (based on safe flight altitude): 0-10 (points will be deducted for being below or above the safe range); flight strategy A (low-fly through): score = 0.3 terrain relief score + 0.2 vegetation density score - 0.1*(altitude - absolute value of safe altitude); flight strategy B (high-fly around): score = 0.2 terrain relief score + 0.1 vegetation density score + 0.3 longitude and latitude range score + 0.2 altitude score (points will be added for being above the safe altitude); Specifically, assuming the drone's current conditions are: terrain relief score 8, vegetation density score 6, latitude and longitude range score 9 (close to the mission target), and altitude 800 meters (within the safe altitude range), then: Flight Strategy A score: 0.38 + 0.26 -0.1*(800 - the absolute value of the safe altitude, assuming it is 500, 300) = 2.4 - 30 = -27.6 (a negative score indicates that the strategy is not suitable under the current conditions); Flight Strategy B score: 0.28 + 0.16 + 0.39 + 0.2 (assuming the altitude is within the safe range, the full score is 10) = 1.6 + 0.6 + 2.7 + 2 = 7.9 (a high score indicates that the strategy is more suitable under the current conditions). Therefore, the drone should choose Flight Strategy B for flight.

[0064] refer to Figure 5 In step S14, the obstacle in front of the drone is determined based on the preset inspection path of the drone, the current position of the drone, and the flight stereoscopic scene map, and a dynamic avoidance path is determined based on the obstacle range of the obstacle in front and the shape of the drone; In the specific implementation process of the present invention, the specific steps are: S141: Acquire a flight stereoscopic scene image, determine a corresponding flight area based on the flight stereoscopic scene image and the shape of the UAV, and determine a corresponding flight deviation range based on the flight area and the current position of the UAV; S142: Obtaining the inspection mission of the drone, determining a preset inspection path for the drone based on the analysis of the inspection mission, marking the current position of the drone on the preset inspection path, and determining obstacles ahead of the drone based on the preset inspection path and the flight deviation range of the drone; S143: Collect the outer contour of the obstacle in front of the drone, determine the obstacle range of the obstacle in front according to the outer contour of the obstacle in front and the corresponding activity event, and determine the corresponding dynamic avoidance path according to the obstacle range of the obstacle in front, the shape of the drone, and the spatial distance of the drone relative to the obstacle in front.

[0065] In an embodiment of the present application, a flight stereo scene graph is collected, and a corresponding flight area is determined based on the flight stereo scene graph and the shape of the drone. The corresponding flight deviation range is determined according to the flight area and the current position of the drone. This takes into account the overall consideration of the flight area and the current position of the drone, and ensures the accuracy of the corresponding flight deviation range.

[0066] At this time, detailed images or data of the drone's current flight environment are obtained to construct a flight stereo scene map. At the same time, the drone uses its onboard cameras, radars, laser radar (LiDAR), infrared sensors, etc. to collect high-definition images, three-dimensional point cloud data, temperature distribution maps and other information of the surrounding environment in real time. The collected data undergoes preprocessing, such as denoising, enhancement, and registration, to generate a high-quality flight stereo scene map.

[0067] Determine a safe and suitable area for drone flight based on the drone's morphology (such as size, weight, flight speed, maneuverability, etc.) and the flight stereo scene map. The drone's morphology affects its flight capabilities in different environments. For example, large drones are limited in narrow spaces, while high-speed drones require a larger safety margin in complex terrain. Combining factors such as terrain, buildings, vegetation, weather conditions, and the drone's morphological limitations in the flight stereo scene map, the drone determines a safe flight area through algorithms or manually preset rules.

[0068] Ensure that the drone remains within a safe flight area during flight, while allowing a certain flight deviation to adapt to minor changes in actual flight; the drone calculates the allowable flight deviation range based on its current relative position in the flight area and a preset safety margin (such as the minimum safe distance from obstacles); the flight deviation range is dynamically adjusted as the drone flies to adapt to environmental changes and flight mission requirements.

[0069] Specifically, suppose a drone is conducting a power line inspection mission; the drone uses its high-definition camera and lidar to collect high-definition images and three-dimensional point cloud data of urban areas in real time. These data include the outlines of buildings, the layout of roads, the location of power lines, and the distribution of trees; the drone analyzes the collected flight stereo scene images and identifies key elements such as high-rise buildings, narrow streets, and power lines; taking into account the drone's shape (such as moderate size, stable flight speed, and good maneuverability), the drone determines a flight area that avoids the tops of high-rise buildings and stays close to power lines but maintains a safe distance. This area ensures that the drone will not collide with buildings or power lines during flight.

[0070] The drone is currently located on a wide street in the city, a certain distance away from the nearest building. Combining the flight area and the drone's current position, the drone calculates the allowable flight deviation range as a circular area centered on the current position with a radius of 20 meters. This range ensures that the drone can remain within the safe flight area even if it encounters slight changes in wind direction or operational errors during flight. In actual flight, the drone will dynamically adjust the flight deviation range based on changes in its position, speed, and flight area to ensure the safe completion of the inspection mission.

[0071] Furthermore, the inspection mission of the drone is obtained, and the preset inspection path of the drone is determined based on the analysis of the drone's inspection mission. The current position of the drone is marked on the preset inspection path of the drone. The preset inspection path of the drone and the flight deviation range of the drone determine the obstacles in front of the drone, which is compatible with the overall consideration of the analysis of the drone's inspection mission and ensures the accuracy of the preset inspection path of the drone.

[0072] At this point, the specific inspection tasks that the drone needs to perform are understood, including the inspection targets, scope, time requirements, etc. At this point, the drone receives inspection task instructions from the ground control station or cloud platform through wireless communication. These instructions usually contain information such as the coordinates of the inspection target, description of the inspection route, inspection priority and special precautions. After receiving the task instructions, the drone first parses them to understand the specific requirements of the inspection task.

[0073] Based on the requirements of the inspection task, an efficient and safe flight path is planned from the starting point to the end point; the drone uses built-in algorithms (such as the Dijkstra algorithm, the RRT algorithm, etc.) or combines external data (such as meteorological data, terrain data, etc.) to plan a preset inspection path based on the distribution and priority of the inspection targets; the preset inspection path needs to be optimized to reduce flight time, improve inspection efficiency, or adapt to special environmental conditions.

[0074] Track the drone's position in real time to determine its deviation from the pre-set inspection path; the drone obtains its current position in real time through GPS, INS (Inertial Navigation System), or other positioning technology; and marks the drone's current position on the pre-set inspection path for subsequent analysis and comparison.

[0075] Identify obstacles encountered by the drone on the preset inspection path to ensure flight safety; the drone uses sensors such as cameras, radars, and lidar to detect obstacles in the surrounding environment in real time; combine the preset inspection path and the drone's flight deviation range to analyze whether the obstacle poses a threat to the drone's flight; if the obstacle is on the preset inspection path or close to the drone's flight deviation range, it will be regarded as a front obstacle.

[0076] Specifically, suppose a drone is performing a power line inspection mission. The drone receives an inspection mission instruction from the ground control station, requiring it to inspect multiple power lines within the urban area, starting from a substation on the east side of the city and ending at another substation on the west side of the city. The inspection mission also includes information such as the coordinates of the power lines, the inspection priority, and safety issues that require special attention.

[0077] After parsing the inspection mission instructions, the drone uses a built-in algorithm to combine the map data of the urban area and the location information of the power lines to plan a preset inspection route from the starting substation to the ending substation. This route is distributed along the power lines while avoiding high-rise buildings and densely populated residential areas in the city to ensure safe and efficient flight. During the flight, the drone obtains its current position in real time through GPS and INS technology and marks the location on the preset inspection route. In this way, the ground control station can monitor the drone's flight progress and position in real time.

[0078] During flight, the drone uses its camera and lidar sensor to detect obstacles in the surrounding environment in real time. When the drone approaches a high-rise building, the sensor detects the building's outline and height information. Combining the preset inspection path and the drone's flight deviation range (assuming it is 20 meters), the drone analyzes that the building poses a threat to its flight and therefore regards it as an obstacle ahead. To ensure flight safety, the drone will adjust its flight altitude and speed to bypass the high-rise building and continue to fly along the preset inspection path.

[0079] Therefore, the outer contour of the obstacle in front of the drone is collected, and the obstacle range of the obstacle in front is determined according to the outer contour of the obstacle in front and the corresponding activity event. The corresponding dynamic avoidance path is determined according to the obstacle range of the obstacle in front, the shape of the drone, and the spatial distance of the drone relative to the obstacle in front. This is compatible with the overall consideration of the obstacle range of the obstacle in front, the shape of the drone, and the spatial distance of the drone relative to the obstacle in front, ensuring the accuracy of the corresponding dynamic avoidance path.

[0080] At this time, the precise shape and size of the obstacle ahead are obtained for subsequent analysis. At this time, the drone uses its onboard camera, laser radar (LiDAR) or other sensors to scan and image the obstacle ahead. Through image processing algorithms or point cloud processing algorithms, the outer contour information of the obstacle, including edges, vertices, curvature, etc., is extracted. The impact range of the obstacle on the drone's flight is evaluated, considering the static characteristics (such as size and shape) and dynamic characteristics (such as moving speed and direction) of the obstacle. Based on the outer contour of the obstacle, the spatial range occupied by it is determined, that is, the static obstacle range. If the obstacle is moving (such as vehicles, pedestrians, flying birds, etc.), the drone needs to predict its future motion trajectory and expand the static obstacle range based on the prediction results to form a dynamic obstacle range. Other events caused by the obstacle, such as wind noise, airflow changes, etc., are considered, which also affect the flight of the drone.

[0081] Plan a safe path that avoids obstacles and is consistent with the drone's flight capabilities. At this point, the drone uses a path planning algorithm (such as the Dijkstra algorithm, the RRT algorithm, etc.) to generate multiple avoidance paths based on the obstacle range, the drone's shape (such as size, speed, maneuverability, etc.), and spatial distance constraints. The generated avoidance paths are evaluated, and the optimal path is selected as the dynamic avoidance path, taking into account factors such as the path's length, safety, and smoothness. The drone flies along the selected dynamic avoidance path and makes fine adjustments based on actual conditions to adapt to environmental changes and uncertainties during flight.

[0082] Specifically, suppose a drone is performing a power line inspection mission in a forest area; during flight, the drone suddenly encounters a fallen tree as an obstacle in front of it; the drone uses its lidar to scan the tree and obtain its precise outer contour information, including the diameter and length of the trunk and the distribution of branches; the drone analyzes the static characteristics of the tree and determines the spatial range it occupies as the static obstacle range; since the trees are stationary, there are no dynamic characteristics to consider; in addition, the drone also takes into account the airflow changes caused by the trees, which affect the flight stability of the drone, and therefore appropriately expands the obstacle range based on the static obstacle range.

[0083] Using the RRT* algorithm, the drone generated multiple avoidance paths, taking into account obstacle range, its own shape (such as moderate size, stable speed, and good maneuverability), and spatial distance constraints. After evaluating these paths, the drone selected a path that bypassed the tree while maintaining a safe distance as a dynamic avoidance path. This path not only avoided collisions with the tree but also ensured a smooth and safe flight. During the execution of the avoidance path, the drone made fine-tuning adjustments based on actual conditions to adapt to airflow changes and other uncertainties in flight. Ultimately, the drone successfully avoided the fallen tree and continued to fly along the preset inspection path.

[0084] refer to Figure 6 In step S15, the remaining inspection path of the drone is determined based on the preset inspection path and the current position of the drone, and the dynamic path of the drone is determined according to the remaining inspection path, each dynamic avoidance path and the power of the drone; In the specific implementation process of the present invention, the specific steps are: S151: collecting the preset inspection path and current position of the drone, and determining the remaining inspection path based on the comparison between the preset inspection path and the current position of the drone; S152: Determine a first sub-dynamic path based on a combination of the remaining inspection path and each dynamic avoidance path, determine a second sub-dynamic path based on the remaining inspection path and the drone's battery level, and determine the drone's dynamic path based on the first sub-dynamic path, the second sub-dynamic path, and the path mapping relationship; S153: Marking the emergency avoidance nodes of the UAV during flight according to the dynamic path of the UAV and the flight speed of the UAV, and triggering the emergency avoidance logic of the UAV based on the emergency avoidance nodes.

[0085] In an embodiment of the present application, the preset inspection path and current position of the drone are collected, and the remaining inspection path is determined based on the comparison between the preset inspection path and the current position of the drone, which is compatible with the overall consideration of the comparison between the preset inspection path and the current position of the drone, and ensures the accuracy of the remaining inspection path.

[0086] At this time, the complete inspection path planned by the UAV before starting the inspection mission is obtained; at this time, the preset inspection path is usually stored in the UAV's flight management system in the form of a series of coordinate points or path segments. These path points or path segments define the inspection target locations that the UAV needs to visit and the connection paths between them; the preset inspection path is planned in advance by the ground control station based on mission requirements, terrain, obstacle distribution and other factors, and sent to the UAV via wireless communication.

[0087] The precise position of the drone during flight is obtained in real time. At the same time, the drone calculates and updates its current position information in real time through its onboard GPS, INS (Inertial Navigation System) or other positioning technologies. To ensure the accuracy of positioning, the drone will integrate data from multiple positioning technologies, such as GPS and INS, to improve the accuracy and reliability of position information.

[0088] By comparing the drone's current position with the preset inspection path, the drone calculates the inspection targets that need to be visited and the optimal connection path between them. At this time, the drone compares its current position with each path point or path segment in the preset inspection path one by one to determine which path points have been visited and which path points have not been visited. For path points that have not been visited, the drone needs to calculate the optimal connection path from the current position to these path points. This usually involves path planning algorithms such as Dijkstra or RRT to ensure that the drone can reach the remaining inspection targets efficiently and safely. As the drone flies, its current position will continue to change, and the remaining inspection path will also be updated accordingly to reflect the drone's latest flight progress and remaining tasks.

[0089] Specifically, suppose a drone is performing a power line inspection mission. The preset inspection route is to start from substation A, inspect lines B, C, and D in sequence, and finally return to substation A. Before the drone starts flying, it has received complete preset inspection route information from the ground control station, including the coordinates of substation A, the coordinates of key points on lines B, C, and D, and the connection paths between them. During the flight, the drone obtains its current position information in real time through GPS and INS technology. For example, when the drone flies to the midpoint of line B, its current position information has been updated to the coordinates of the midpoint of line B.

[0090] The drone compares its current location with the preset inspection route and finds that substation A and the first half of line B have been visited, while the second half of line B, lines C and D, and the path back to substation A have not yet been visited. Therefore, the drone calculates the remaining inspection path as follows: from the current location (the midpoint of line B), continue along line B to the end point, then inspect lines C and D in sequence, and finally return to substation A. As the drone flies, the remaining inspection path will be continuously updated until all inspection targets have been visited.

[0091] Furthermore, the first sub-dynamic path is determined based on the synthesis of the remaining inspection paths and each dynamic avoidance path, the second sub-dynamic path is determined based on the remaining inspection paths and the power of the UAV, and the dynamic path of the UAV is determined based on the first sub-dynamic path, the second sub-dynamic path and the path mapping relationship. The overall consideration of the first sub-dynamic path, the second sub-dynamic path and the path mapping relationship is compatible, ensuring the accuracy of the dynamic path of the UAV. At the same time, the accuracy of the remaining inspection paths is guaranteed, and the remaining inspection paths are further dynamically controlled, thereby improving the planning accuracy of the UAV's dynamic path and improving the inspection compatibility of the UAV in various environments.

[0092] At this time, the remaining inspection path is combined with the various dynamic avoidance paths planned when encountering obstacles during flight to form a preliminary dynamic path that meets the inspection needs and avoids obstacles; at this time, the UAV first plans a basic flight trajectory based on the remaining inspection path; then, when encountering an obstacle, the UAV uses the obstacle detection and avoidance system to generate a dynamic avoidance path. These avoidance paths need to be seamlessly connected with the remaining inspection path to ensure that the UAV can smoothly return to the preset inspection path after avoiding the obstacle; the UAV uses the path planning algorithm to synthesize the avoidance path with the remaining inspection path, adjust the flight trajectory to avoid obstacles, and at the same time maintain consistency with the preset inspection path.

[0093] Taking into account the current battery level of the drone and the power required for the remaining inspection tasks, an alternative path is planned that can safely return or land in the event of insufficient battery. At this time, the drone calculates the minimum battery threshold required to safely complete the remaining tasks based on the length of the remaining inspection path, flight speed, expected power consumption, and the current remaining battery level. If the current battery level is lower than this threshold, the drone needs to plan a path back to the charging station or a safe landing point, which is the second sub-dynamic path. When planning the second sub-dynamic path, the drone needs to consider factors such as flight distance, terrain conditions, and obstacle distribution to ensure the safety and feasibility of the path.

[0094] The first sub-dynamic path and the second sub-dynamic path are integrated, and the final dynamic flight instructions are generated according to the UAV's flight control system and the path mapping relationship. The path mapping relationship refers to a set of rules or algorithms stored in the UAV's flight control system, which is used to convert the planned path into flight instructions for the UAV to execute, which includes the setting of parameters such as flight altitude, speed, direction, and attitude. The UAV generates the final dynamic flight instructions based on the first sub-dynamic path and the second sub-dynamic path (if applicable), combined with the path mapping relationship. These instructions will guide the UAV on how to avoid obstacles during flight, how to adjust the flight trajectory to adapt to environmental changes, and how to return or land safely when the battery is low.

[0095] Specifically, suppose a drone is performing a power line inspection mission, and the remaining inspection path includes several main streets and several key power towers; the drone first plans a basic flight trajectory based on the remaining inspection path; during the flight, the drone detects a street under construction ahead, so it generates a dynamic avoidance path that bypasses the construction area. This avoidance path is combined with the remaining inspection path to form the first sub-dynamic path; during the flight, the drone also continuously monitors its current power level and the power required for the remaining inspection mission; suppose that halfway through the flight, the drone finds that its remaining power level is insufficient to safely complete the remaining mission; therefore, the drone plans a path back to the nearest charging station based on its current position and remaining power level, which is the second sub-dynamic path.

[0096] The drone integrates the first sub-dynamic path and the second sub-dynamic path (applicable when the battery is low) and generates the final dynamic flight instructions based on the path mapping relationship within its flight control system. These instructions guide the drone to continue flying along the preset inspection path after bypassing the construction area until the battery is low, at which point it automatically turns back to the path of the charging station. Through this process, the drone can both efficiently complete its inspection mission and ensure flight safety.

[0097] Therefore, the emergency avoidance nodes of the UAV during flight are marked according to the dynamic path of the UAV and the flight speed of the UAV. Based on the emergency avoidance nodes triggering the emergency avoidance logic of the UAV, the emergency avoidance logic of the UAV is introduced.

[0098] At this time, in the dynamic path of the drone, key points where the drone needs to make emergency avoidance due to obstacles, terrain changes, sudden changes in flight conditions, etc. are identified and marked; at this time, the drone uses its built-in obstacle detection and avoidance system, combined with dynamic path information and real-time flight data (such as speed, altitude, heading, etc.) to analyze potential dangers encountered during the flight. These dangers include sudden obstacles, terrain undulations, severe weather conditions, etc.; the drone marks emergency avoidance nodes on the dynamic path according to the location, size and impact of these dangerous factors; emergency avoidance nodes usually exist in the form of coordinate points, distance marks or time marks. These marked points will be recorded by the drone flight management system and updated and monitored in real time during the flight.

[0099] When a drone approaches an emergency avoidance node, the emergency avoidance logic is automatically triggered to ensure that the drone can quickly and safely avoid potential dangers. The emergency avoidance logic is a set of preset algorithms and rules used to guide the drone on how to make avoidance actions in emergency situations. These logics include changing the flight altitude, speed, direction, or performing specific maneuvers. The design of the emergency avoidance logic needs to take into account factors such as the drone's performance limitations, safety margins, and flight environment. At the same time, the drone monitors the distance or time relationship between its current position and the emergency avoidance node in real time. When it approaches the preset trigger threshold, the emergency avoidance logic is automatically triggered. The trigger threshold is usually dynamically adjusted based on factors such as the drone's flight speed, the size of the obstacle, and the movement speed.

[0100] Specifically, consider a drone conducting a power line inspection, using a dynamic path along a winding mountain road. During flight, the drone uses its built-in sensors, such as lidar and cameras, to monitor the terrain and obstacles ahead in real time. When the drone approaches a steep cliff, the system identifies the risk of rockfall or landslides at the edge and marks an emergency avoidance node on the dynamic path. This node is a specific coordinate point approximately 500 meters from the current location. As the drone flies, when it approaches the emergency avoidance node within approximately 200 meters (a preset trigger threshold), the drone automatically triggers the emergency avoidance logic. This emergency avoidance logic instructs the drone to immediately climb to a certain altitude to avoid the potential landslide and adjust its flight direction to continue the inspection along a safer path. During the emergency avoidance process, the drone also monitors its flight parameters and environmental changes in real time to ensure the safety and effectiveness of the avoidance maneuver. Through this process, the drone can automatically identify and respond to potential hazards during flight, ensuring the safe and efficient completion of the inspection mission.

[0101] In some embodiments of the present application, an emergency avoidance node matching table is collected, and the emergency avoidance node matching table is shown in Table 1: Table 1 Emergency avoidance node matching table

[0102] During flight, the drone compares its dynamic path and flight speed with the conditions in the emergency avoidance node matching table in real time; when the emergency avoidance conditions of a certain type of danger are met, the drone marks an emergency avoidance node at that location.

[0103] See also Figure 7 , Figure 7 : is a schematic diagram of the structural composition of a dynamic path planning system for a UAV based on multimodal perception in an embodiment of the present invention; the dynamic path planning system for a UAV based on multimodal perception includes: An environmental data module 21 is used to collect environmental data of the drone when the drone is in an inspection state based on the drone's integrated module, which integrates a radar, a visual detector, and a thermal imaging module; A multimodal perception data module 22 is configured to determine a point cloud data combination, an image data combination, and a thermal imaging data combination based on the environmental data set to construct multimodal perception data of each environmental area; A flight stereo scene graph module 23 is configured to determine the regional morphology of each environmental region based on the recognition of the multimodal perception data of each environmental region, and to determine the flight stereo scene graph of the UAV based on the regional morphology, regional position, and current position of each environmental region; The dynamic avoidance path module 24 is used to determine the obstacle in front of the drone based on the preset inspection path of the drone, the current position of the drone and the flight three-dimensional scene image, and determine the dynamic avoidance path based on the obstacle range of the obstacle and the shape of the drone; The dynamic path module 25 is used to determine the remaining inspection path of the drone based on the preset inspection path and the current position of the drone, and to determine the dynamic path of the drone according to the remaining inspection path, each dynamic avoidance path and the power of the drone.

[0104] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A dynamic path planning method for an unmanned aerial vehicle based on multimodal perception, characterized in that: include: When the UAV is in the inspection state, the UAV's integrated module collects the UAV's environmental data set, and the integrated module integrates radar, visual detector and thermal imaging module; Determine a point cloud data combination, an image data combination, and a thermal imaging data combination based on the environmental data set to construct multimodal perception data of each environmental area; Determine the regional morphology of each environmental area based on the recognition of multimodal perception data of each environmental area, and determine the flight stereoscopic scene map of the drone based on the regional morphology, regional location and current location of each environmental area; Determine the obstacle in front of the drone based on the drone's preset inspection path, the drone's current position, and the flight 3D scene map, and determine the dynamic avoidance path based on the obstacle range and the drone's shape; The remaining inspection path of the drone is determined based on the preset inspection path and current position of the drone, and the dynamic path of the drone is determined according to the remaining inspection path, various dynamic avoidance paths and the power of the drone.

2. The dynamic path planning method for a UAV based on multimodal perception according to claim 1 is characterized in that: When the UAV is in the inspection state, the UAV's integrated module collects the UAV's environmental data set, and the integrated module integrates a radar, a visual detector, and a thermal imaging module, including: Collect the current flight data and current inspection data of the drone, and determine the inspection status of the drone based on the current flight data and current inspection data of the drone; Real-time monitoring of the drone's flight process during inspection. The integrated module is configured at the front end of the drone and collects environmental data as the drone performs dynamic inspections. The data is stored in the drone's environmental data space. In this case, the integrated module integrates radar, visual detectors, and thermal imaging modules. In the environmental data space, the collection time of each environmental data is marked, and the dynamic sorting of each environmental data is triggered according to the sorting of the collection time to form the environmental data set of the drone.

3. The dynamic path planning method for a UAV based on multimodal perception according to claim 1, characterized in that: The step of determining a point cloud data combination, an image data combination, and a thermal imaging data combination based on the environmental data set to construct multimodal perception data for each environmental area includes: Collecting an environmental data set, traversing the environmental data set, and marking corresponding data types and corresponding environmental area labels for each environmental data, and determining a plurality of point cloud data, a plurality of image data, and a plurality of thermal imaging data based on matching between the environmental data set and the data types; Determining a point cloud data combination based on a combination of a plurality of point cloud data; determining an image data combination based on a combination of a plurality of image data; determining a thermal imaging data combination based on a combination of a plurality of thermal imaging data; In the environmental data space of the drone, the environmental area markers are aligned for the point cloud data combination, image data combination, and thermal imaging data combination. At this time, the corresponding point cloud data, image data, and thermal imaging data are presented in the same environmental area marker, and multimodal perception data of each environmental area is constructed based on the point cloud data, image data, and thermal imaging data.

4. The dynamic path planning method for a UAV based on multimodal perception according to claim 1, characterized in that: The method of determining the regional form of each environmental area based on the recognition of the multimodal perception data of each environmental area, and determining the flight stereoscopic scene graph of the drone according to the regional form, regional position and current position of each environmental area, includes: In the multimodal perception data of each environmental area, a first sub-area morphology is determined based on a synthesis of point cloud data and image data, and a second sub-area morphology is determined based on a synthesis of point cloud data and thermal imaging data; The regional morphology of the environmental area is determined based on a synthesis of the regional position of the environmental area, the first sub-region morphology, and the second sub-region morphology, so as to collect the regional morphology of each environmental area.

5. The dynamic path planning method for a UAV based on multimodal perception according to claim 4 is characterized in that: The method further includes: determining the regional form of each environmental area based on the recognition of the multimodal perception data of each environmental area, and determining the flight stereoscopic scene graph of the drone according to the regional form, regional position and current position of each environmental area. Mark the corresponding regional morphology and regional position in each environmental area, determine the first scene according to the current position of the drone and the regional morphology of each environmental area, determine the second scene according to the current position of the drone and the regional position of each environmental area, based on the first scene, the second scene and the flight stereo scene map.

6. The dynamic path planning method for a UAV based on multimodal perception according to claim 1, characterized in that: The method of determining an obstacle ahead of the drone based on the preset inspection path of the drone, the current position of the drone, and the flight stereoscopic scene image, and determining a dynamic avoidance path based on the obstacle range of the obstacle ahead and the shape of the drone, includes: Collect a 3D flight scene image, determine a corresponding flight area based on the 3D flight scene image and the UAV's morphology, and determine a corresponding flight deviation range based on the flight area and the UAV's current position; Obtain the inspection mission of the drone, determine the preset inspection path of the drone based on the analysis of the inspection mission of the drone, mark the current position of the drone on the preset inspection path of the drone, and determine the obstacles in front of the drone based on the preset inspection path of the drone and the flight deviation range of the drone.

7. The dynamic path planning method for a UAV based on multimodal perception according to claim 6, characterized in that: The method further includes determining an obstacle ahead of the drone based on the preset inspection path of the drone, the current position of the drone, and the flight stereoscopic scene image, and determining a dynamic avoidance path based on the obstacle range of the obstacle ahead and the shape of the drone: The outer contour of the obstacle in front of the drone is collected, and the obstacle range of the obstacle in front is determined based on the outer contour of the obstacle in front and the corresponding activity event. The corresponding dynamic avoidance path is determined based on the obstacle range of the obstacle in front, the shape of the drone, and the spatial distance of the drone relative to the obstacle in front.

8. The dynamic path planning method for a UAV based on multimodal perception according to claim 1, characterized in that: The method of determining the remaining inspection path of the drone based on the preset inspection path and the current position of the drone, and determining the dynamic path of the drone based on the remaining inspection path, each dynamic avoidance path, and the power of the drone, includes: The preset inspection path and current position of the drone are collected, and the remaining inspection path is determined based on the comparison between the preset inspection path and the current position of the drone.

9. The dynamic path planning method for a UAV based on multimodal perception according to claim 8, characterized in that: The method further includes determining the remaining inspection path of the drone based on the preset inspection path and the current position of the drone, and determining the dynamic path of the drone based on the remaining inspection path, each dynamic avoidance path, and the power level of the drone. A first sub-dynamic path is determined based on the synthesis of the remaining inspection path and each dynamic avoidance path. A second sub-dynamic path is determined based on the remaining inspection path and the power level of the drone. The dynamic path of the drone is determined based on the first sub-dynamic path, the second sub-dynamic path, and the path mapping relationship. According to the dynamic path and flight speed of the UAV, the emergency avoidance nodes of the UAV during flight are marked, and the emergency avoidance logic of the UAV is triggered based on the emergency avoidance nodes.

10. A dynamic path planning system for UAV based on multimodal perception, characterized in that: The dynamic path planning system for a UAV based on multimodal perception is applied to the dynamic path planning method for a UAV based on multimodal perception as described in any one of claims 1 to 9. The dynamic path planning system for a UAV based on multimodal perception includes: An environmental data module is used to collect environmental data from the drone when the drone is in an inspection state based on the drone's integrated module, which integrates radar, visual detectors, and thermal imaging modules; A multimodal perception data module is used to determine a point cloud data combination, an image data combination, and a thermal imaging data combination based on an environmental data set to construct multimodal perception data of each environmental area; A flight stereo scene graph module is used to determine the regional morphology of each environmental area based on the recognition of multimodal perception data of each environmental area, and to determine the flight stereo scene graph of the drone based on the regional morphology, regional location and current location of each environmental area; The dynamic avoidance path module is used to determine the obstacle in front of the drone based on the preset inspection path of the drone, the current position of the drone and the flight three-dimensional scene map, and determine the dynamic avoidance path according to the obstacle range of the obstacle in front and the shape of the drone; The dynamic path module is used to determine the remaining inspection path of the drone based on the preset inspection path and current position of the drone, and to determine the dynamic path of the drone based on the remaining inspection path, various dynamic avoidance paths and the power of the drone.

Images