A method and system for unmanned aerial vehicle inspection planning based on multi-source data

Abstract

The application relates to the technical field of unmanned planes, and relates to an unmanned plane inspection planning method and system based on multi-source data, which comprises the following steps: confirming a region to be inspected and an unmanned plane to be planned, collecting images of the region to be inspected to obtain region original image data, extracting key targets from the region original image data to obtain a plurality of key targets, confirming an inspection priority score based on target state scoring and target multi-source data, summarizing the inspection priority scores to obtain a plurality of inspection priority scores, confirming an initial inspection route based on the plurality of inspection priority scores and the plurality of key targets, collecting parameters of the unmanned plane to be planned to obtain inspection parameters, confirming a final planning route based on the inspection parameters and the initial inspection route, issuing the final planning route to the unmanned plane to be planned to obtain a target unmanned plane, and completing unmanned plane inspection planning. The application can improve the efficiency of unmanned plane inspection planning, and in turn improve the inspection accuracy.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a UAV inspection planning method and system based on multi-source data. Background Technology

[0002] With increasingly stringent requirements for infrastructure safety inspections in sectors such as power, transportation, and energy, the limitations of traditional manual inspection methods in terms of efficiency, coverage, and safety risks are becoming increasingly apparent. Drones, with their advantages of maneuverability and the ability to carry multiple sensors, are gradually becoming the core platform for inspection operations, with inspection solutions integrating multi-source data such as visible light, infrared, and lidar emerging as an important development direction.

[0003] Currently, existing drone inspection plans mainly rely on statically preset waypoints or local obstacle avoidance methods based on a single sensor.

[0004] While the above methods can achieve UAV inspection planning, they are difficult to respond to changes in the environment or target status in real time. Static routes cannot cope with dynamic obstructions or temporary obstacles, leading to mission omissions or flight risks. They also have excessively high requirements for the spatiotemporal synchronization accuracy of multi-source data, which can easily cause fusion deviations due to sampling delays, affecting inspection efficiency and detection accuracy. Therefore, how to improve the efficiency of UAV inspection planning and thus improve inspection accuracy has become an urgent problem to be solved. Summary of the Invention

[0005] This invention provides a UAV inspection planning method based on multi-source data and a computer-readable storage medium. Its main purpose is to improve the efficiency of UAV inspection planning and thus improve the inspection accuracy.

[0006] To achieve the above objectives, the present invention provides a UAV inspection planning method based on multi-source data, comprising: Identify the areas to be inspected and the drones to be planned; Images are acquired from the area to be inspected to obtain the raw image data of the area; Key targets were extracted from the original image data of the region, resulting in multiple key targets. For each of the multiple key objectives, perform the following actions: Target status scores were determined based on key objectives; Multi-source data collection is performed on key targets to obtain multi-source target data; Inspection priority scores are determined based on target status scores and multi-source target data. Summarize the inspection priority scores to obtain multiple inspection priority scores; The initial inspection route was determined based on multiple inspection priority scores and multiple key objectives. Parameters are collected from the drone to be planned to obtain inspection parameters; The final planned route is determined based on the inspection parameters and the initial inspection route. The final planned route is sent to the drones to be planned, the target drones are obtained, and the drone inspection plan is completed.

[0007] Optionally, the step of acquiring images of the area to be inspected to obtain raw image data of the area includes: The area to be inspected is divided into multiple unit areas; Multiple unit regions are numbered to obtain multiple numbered regions; Extracting the number from multiple numbered regions There are numbered areas, among which... The initial value is 1; For the Visible light images were acquired from the numbered regions to obtain the first... Visible light images of the region; For the Thermal infrared images were acquired from the numbered regions to obtain the first... Thermal infrared images of the area; Based on the The numbered area was confirmed to be the first Area number; The first Visible light image of the region, the first Thermal infrared images of the region and the first Bind the area number to get the first one. Image data for each region; The first Image data from each region is fed into a pre-constructed initial image dataset to obtain an updated image dataset; make ,Will As The updated image dataset is used as the initial image dataset, and the result is returned from the extraction of the first image from multiple numbered regions. The steps for each numbered area, until... The updated image dataset will be used as the original image data for the region, where, This represents the number of numbered regions among multiple numbered regions.

[0008] Optionally, the key target extraction from the original image data of the region yields multiple key targets, including: Based on the original image data of the region, the visible stitched image and the infrared stitched image of the region were identified. Band registration is performed on the visible mosaic image and the infrared mosaic image of the region to obtain the registered region image; Semantic segmentation is performed on the registered region image to obtain multiple device instances; Perform the following operation on each of the multiple device instances: Visible texture features and infrared temperature distribution features were identified based on device examples; The credibility score was determined based on visible texture features and infrared temperature distribution features. The credibility scores are aggregated to obtain multiple credibility scores; Multiple target scores were identified based on multiple credibility scores and preset scoring thresholds; Several key objectives were identified based on multiple target scores and multiple device instances.

[0009] Optionally, the step of identifying the visible stitched image and the infrared stitched image of the region based on the original image data of the region includes: Based on the original image data of the region, multiple visible light images and multiple thermal infrared images of the region were identified. Perform the following operation on each of the multiple visible light images: The coordinates of the visible light image of the region are corrected to obtain the corrected visible light image; By summarizing the corrected visible light images, multiple corrected visible light images are obtained; Multiple corrected visible light images are stitched together to obtain a regional visible stitched image; The regional infrared mosaic image was confirmed based on multiple regional thermal infrared images.

[0010] Optionally, the confidence score determined based on visible texture features and infrared temperature distribution features includes: Texture sharpness analysis is performed on visible texture features to obtain texture sharpness indices; Thermal anomaly analysis was performed on the infrared temperature distribution characteristics to obtain thermal anomaly indices; The target credibility score is calculated based on the texture sharpness index and the thermal anomaly index, using the following formula: in, Indicates the target credibility score. Indicates texture sharpness index, Indicates thermal anomaly index, Represents the natural logarithm. Indicates taking the absolute value; Compare the target credibility score with a preset credibility threshold. If the target credibility score is greater than the credibility threshold, then the target credibility score is used as the credibility score. If the target credibility score is less than or equal to the credibility threshold, then the credibility threshold will be used as the credibility score.

[0011] Optionally, the step of determining the target status score based on key objectives includes: Feature extraction is performed on key targets to obtain target appearance features, target temperature features, and target location features; Defect identification and analysis are performed on the target's appearance features to obtain appearance anomaly indicators; Temperature rise shift analysis is performed on the target temperature characteristics to obtain temperature anomaly indicators; Environmental correlation analysis is performed on the target location characteristics to obtain location risk indicators; The target status score is calculated based on appearance anomaly indicators, temperature anomaly indicators, and location risk indicators.

[0012] Optionally, the step of determining the inspection priority score based on the target status score and target multi-source data includes: Multi-source data analysis is performed on the target multi-source data to obtain historical inspection data, real-time sensor data and operation and maintenance record data; Historical anomaly frequency indicators were identified based on historical inspection data. Real-time volatility risk indicators are identified based on real-time sensor data. Maintenance urgency indicators were identified based on maintenance record data; The target risk coefficient is calculated based on historical anomaly frequency indicators, real-time fluctuation risk indicators, and maintenance urgency indicators. The inspection priority score is calculated based on the target risk coefficient and target status score, using the following formula: in, Indicates the inspection priority score. Represents the natural constant. Indicates the target state score. This represents the target risk coefficient.

[0013] Optionally, determining the initial inspection route based on multiple inspection priority scores and multiple key targets includes: Based on multiple key objectives, the coordinates of multiple target locations were confirmed. By associating multiple target location coordinates with multiple inspection priority scores, a set of target route plans is obtained; The target route planning set is sorted according to the inspection priority score to obtain the target access sequence; Multiple target distance values ​​were identified based on the target access sequence; The initial inspection route is obtained by performing path optimization processing based on multiple target distance values ​​and target access sequences.

[0014] Optionally, determining the final planned route based on inspection parameters and the initial inspection route includes: Based on the inspection parameters, the remaining battery power, maximum range, payload capacity, and flight speed parameters are confirmed. The drone's operational range was determined based on the remaining battery power and maximum flight range parameters. The mission execution capability value of the UAV is determined based on the payload capacity parameters and flight speed parameters; The UAV's executable range, UAV mission execution capability value, and initial inspection route are matched and analyzed to obtain the route matching result, which is either a match or a non-match. If the route matching result is a match, the initial inspection route will be used as the final planned route. If the route matching result is a mismatch, the initial inspection route is corrected to obtain the corrected inspection route. The revised inspection route will be used as the final planned route.

[0015] To achieve the above objectives, the present invention also provides a UAV inspection planning system based on multi-source data, comprising: The key target acquisition module is used to identify the area to be inspected and the drone to be planned, to collect images of the area to be inspected, to obtain the original image data of the area, and to extract key targets from the original image data of the area to obtain multiple key targets. The priority score calculation module is used to perform the following operations for each of the multiple key objectives: determine the target status score based on the key objective, collect multi-source data on the key objective to obtain target multi-source data, and determine the inspection priority score based on the target status score and the target multi-source data. The multi-source data acquisition module is used to summarize the inspection priority scores, obtain multiple inspection priority scores, determine the initial inspection route based on multiple inspection priority scores and multiple key targets, and collect parameters of the UAV to be planned to obtain inspection parameters. The path planning and distribution module is used to determine the final planned route based on the inspection parameters and the initial inspection route, and then distribute the final planned route to the drone to be planned, thereby obtaining the target drone and completing the drone inspection planning.

[0016] To address the above problems, the present invention also provides an electronic device, the electronic device comprising: Memory, storing at least one instruction; and The processor executes the instructions stored in the memory to implement the UAV inspection planning method based on multi-source data described above.

[0017] To address the aforementioned issues, the present invention also provides a computer-readable storage medium storing at least one instruction, which is executed by a processor in an electronic device to implement the aforementioned UAV inspection planning method based on multi-source data.

[0018] To address the problems described in the background art, this invention identifies the area to be inspected and the drone to be planned. This clearly defines the scope of the inspection task and the execution equipment, establishing a basic framework for subsequent path planning. Then, image acquisition is performed on the area to be inspected to obtain raw image data. This invention provides raw data support for automatic identification of inspection targets by acquiring visual information of the area. Key targets are extracted from the raw image data to obtain multiple key targets. This invention intelligently filters out objects requiring priority inspection from the images, achieving target positioning and classification, improving the efficiency of drone inspection planning, and thus improving inspection accuracy. For each key target, the following operations are performed: A target status score is determined based on the key target. This invention quantitatively assesses the current operating status or degree of abnormality of each target to form an urgency indicator. Multi-source data is collected from the key targets to obtain target multi-source data. This invention comprehensively reflects the actual state of the target by fusing information from multiple sensors such as temperature and vibration. An inspection priority score is determined based on the target status score and the target multi-source data. This invention calculates the inspection priority score by comprehensively considering the status and multi-source data. The invention prioritizes the inspection of targets, summarizing inspection priority scores to obtain multiple inspection priority scores. This demonstrates that by aggregating the priority scores of all targets, a globally ordered inspection list is formed. Based on multiple inspection priority scores and multiple key targets, an initial inspection route is determined. This invention improves the efficiency of UAV inspection planning and thus improves inspection accuracy by planning preliminary flight paths covering all key targets according to priority. Parameters are collected from the UAV to be planned to obtain inspection parameters. This invention provides constraints for route optimization by acquiring the UAV's endurance, speed, obstacle avoidance, and other performance constraints. Based on the inspection parameters and the initial inspection route, the final planned route is determined. This invention generates a safe and executable inspection track by adjusting the initial route according to actual flight capabilities. The final planned route is then sent to the UAV to be planned, obtaining the target UAV and completing the UAV inspection planning. This invention improves the intelligence level and operational efficiency of inspection tasks by realizing an automated closed loop from planning to execution, thus improving the efficiency of UAV inspection planning and consequently improving inspection accuracy. Therefore, this invention can improve the efficiency of UAV inspection planning and thus improve inspection accuracy. Attached Figure Description

[0019] Figure 1This is a flowchart illustrating a UAV inspection planning method based on multi-source data, provided in an embodiment of the present invention. Figure 2 This is a functional block diagram of a UAV inspection planning system based on multi-source data provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device for implementing the UAV inspection planning method based on multi-source data, according to an embodiment of the present invention.

[0020] Explanation of reference numerals in the attached figures: 10. Electronic device; 11. Processor; 12. Memory; 13. Bus.

[0021] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0023] This application provides a drone inspection planning method based on multi-source data. The executing entity of the multi-source data-based drone inspection planning method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application embodiment: a server, a terminal, etc. In other words, the multi-source data-based drone inspection planning method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.

[0024] Reference Figure 1 The diagram shown is a flowchart illustrating a UAV inspection planning method based on multi-source data according to an embodiment of the present invention. In this embodiment, the UAV inspection planning method based on multi-source data includes: S1. Identify the areas to be inspected and the drones to be planned.

[0025] For example, Xiao Zhang is a staff member of a new energy power plant and needs to use drones to inspect multiple photovoltaic panels in the plant. So Xiao Zhang identifies the area to be inspected and the drones to be planned, so that the drones to be planned can be used to inspect the area to be inspected later.

[0026] It should be explained that the area to be inspected refers to the area that needs to be inspected, and the inspection refers to the process of using a drone to monitor the operational status of equipment within the area. The drone to be planned is a type of drone, optionally using a DJI-Mini5-Pro.

[0027] S2. Collect images of the area to be inspected to obtain the original image data of the area. Extract key targets from the original image data of the area to obtain multiple key targets.

[0028] Specifically, the process of acquiring images of the area to be inspected to obtain raw image data of the area includes: The area to be inspected is divided into multiple unit areas; Multiple unit regions are numbered to obtain multiple numbered regions; Extracting the number from multiple numbered regions There are numbered areas, among which... The initial value is 1; For the Visible light images were acquired from the numbered regions to obtain the first... Visible light images of the region; For the Thermal infrared images were acquired from the numbered regions to obtain the first... Thermal infrared images of the area; Based on the The numbered area was confirmed to be the first Area number; The first Visible light image of the region, the first Thermal infrared images of the region and the first Bind the area number to get the first one. Image data for each region; The first Image data from each region is fed into a pre-constructed initial image dataset to obtain an updated image dataset; make ,Will As The updated image dataset is used as the initial image dataset, and the result is returned from the extraction of the first image from multiple numbered regions. The steps for each numbered area, until... The updated image dataset will be used as the original image data for the region, where, This represents the number of numbered regions among multiple numbered regions.

[0029] It should be explained that the process of dividing the area to be inspected into units refers to dividing the area into multiple smaller areas based on unit length (e.g., 1 meter). These smaller areas are called multiple unit areas. For example, if the area to be inspected is a rectangle 20 meters long and 10 meters wide, then after dividing the area into units, 200 rectangles 1 meter long and 1 meter wide will be obtained. These 1-meter long and 1-meter wide rectangles are called unit areas. The process of numbering these multiple unit areas refers to assigning different numbers to each unit area. The numbered area refers to the unit area after being numbered.

[0030] For example, if the multiple numbered regions are: region 1, region 2, region 3, and region 4, then the first numbered region extracted from the multiple numbered regions is: region 1.

[0031] Understandably, the above refers to the first Visible light image acquisition of the numbered area refers to: using image acquisition equipment (e.g., remote sensing satellite) to capture images of the numbered area. The visible light image of the surface of the numbered region, the first The visible light image of the surface of the numbered region is the first... The visible light image of the region, the one for the first Thermal infrared image acquisition of the numbered area refers to: using image acquisition equipment (e.g., remote sensing satellite) to capture images of the numbered area. Thermal infrared image of the surface of the numbered region, the first The thermal infrared image of the surface of the numbered region is the first... Thermal infrared images of the region. The area number refers to the first... The number corresponding to the numbered area, the first The image data of the first region refers to the data of the first region. Visible light image of the region, the first Thermal infrared images of the region and the first The data obtained after binding each region number.

[0032] It should be understood that updating the image dataset refers to putting in the first... The initial image dataset is the result of processing image data from multiple regions. The initial image dataset is an empty set. The original image data of a region refers to the data obtained when... The image dataset is updated in real time.

[0033] In detail, the key target extraction of the original image data of the region yields multiple key targets, including: Based on the original image data of the region, the visible stitched image and the infrared stitched image of the region were identified. Band registration is performed on the visible mosaic image and the infrared mosaic image of the region to obtain the registered region image; Semantic segmentation is performed on the registered region image to obtain multiple device instances; Perform the following operation on each of the multiple device instances: Visible texture features and infrared temperature distribution features were identified based on device examples; The credibility score was determined based on visible texture features and infrared temperature distribution features. The credibility scores are aggregated to obtain multiple credibility scores; Multiple target scores were identified based on multiple credibility scores and preset scoring thresholds; Several key objectives were identified based on multiple target scores and multiple device instances.

[0034] It should be explained that the band registration of the visible and infrared mosaic images refers to the process of aligning the visible and infrared mosaic images in terms of spatial geometry. The method for aligning the visible and infrared mosaic images in terms of spatial geometry is existing technology and will not be elaborated here. The registered region image refers to the visible mosaic image after band registration. The semantic segmentation of the registered region image refers to predicting each pixel in the registered region image and dividing spatially connected pixel regions with the same device category label into independent regions. These independent regions are device instances. It should be noted that the device category label refers to the device type identifier assigned to each pixel in the registered region image during the semantic segmentation process. For example: Label 1: Transformer, Label 2: Circuit Breaker, Label 3: Disconnect Switch, Label 4: Insulator. When the device type identifier of two pixels is both Label 1, then the two pixels have the same device category label.

[0035] Understandably, visible texture features refer to the gray-level co-occurrence matrix extracted from the device instance, infrared temperature distribution features refer to the mean temperature extracted from the device instance, and the target score refers to the confidence score that is greater than or equal to the scoring threshold. The scoring threshold is set manually by the staff of the new energy power plant and is optional; the scoring threshold is 60. The key target refers to the device instance corresponding to the target score.

[0036] Specifically, the process of identifying the visible stitched image and the infrared stitched image of the region based on the original image data of the region includes: Based on the original image data of the region, multiple visible light images and multiple thermal infrared images of the region were identified. Perform the following operation on each of the multiple visible light images: The coordinates of the visible light image of the region are corrected to obtain the corrected visible light image; By summarizing the corrected visible light images, multiple corrected visible light images are obtained; Multiple corrected visible light images are stitched together to obtain a regional visible stitched image; A consistency check is performed on the visible stitched images to obtain the image check results; Based on the image inspection results and the visible stitched image of the region, the visible stitched image of the region was confirmed; The regional infrared mosaic image was confirmed based on multiple regional thermal infrared images.

[0037] It should be explained that the regional visible light image refers to the visible light image in the original regional image data, and the regional thermal infrared image refers to the thermal infrared image in the original regional image data. The coordinate correction of the regional visible light image refers to the geometric correction of the regional visible light image, and the method for performing this geometric correction is existing technology and will not be elaborated here.

[0038] It should be understood that a corrected visible light image refers to a regional visible light image after coordinate correction. The process of stitching together multiple corrected visible light images refers to combining them into a single, unified image, which is the regional visible stitched image. The method for identifying a regional infrared stitched image based on multiple regional thermal infrared images is the same as the method for identifying a regional visible stitched image based on multiple regional visible light images, and will not be repeated here.

[0039] In detail, the confidence score determined based on visible texture features and infrared temperature distribution features includes: Texture sharpness analysis is performed on visible texture features to obtain texture sharpness indices; Thermal anomaly analysis was performed on the infrared temperature distribution characteristics to obtain thermal anomaly indices; The target credibility score is calculated based on the texture sharpness index and the thermal anomaly index, using the following formula: in, Indicates the target credibility score. Indicates texture sharpness index, Indicates thermal anomaly index, Represents the natural logarithm. Indicates taking the absolute value; Compare the target credibility score with a preset credibility threshold. If the target credibility score is greater than the credibility threshold, then the target credibility score is used as the credibility score. If the target credibility score is less than or equal to the credibility threshold, then the credibility threshold will be used as the credibility score.

[0040] It should be explained that the texture sharpness analysis of visible texture features refers to measuring the sharpness of visible texture features using a sharpness measurement method (e.g., joint histogram). The sharpness of the visible texture features is the texture sharpness index. The method used to measure the sharpness of visible texture features using a sharpness measurement method (e.g., joint histogram) is existing technology and will not be elaborated here. The thermal anomaly analysis of infrared temperature distribution features refers to calculating the difference between the infrared temperature distribution features and the ambient temperature (room temperature). The difference between the infrared temperature distribution features and the ambient temperature (room temperature) is the thermal anomaly index. The target credibility score reflects the severity of the problem with the critical target. The higher the target credibility score, the greater the severity of the problem with the critical target. The credibility score refers to the target credibility score that is greater than the credibility threshold. The credibility threshold is set manually by the staff of the new energy power plant and is optional; the credibility threshold is 80.

[0041] S3. For each of the multiple key objectives, perform the following operations: Based on the key objective, determine the target status score, collect multi-source data for the key objective, and obtain target multi-source data.

[0042] In detail, the process of determining the target status score based on key objectives includes: Feature extraction is performed on key targets to obtain target appearance features, target temperature features, and target location features; Defect identification and analysis are performed on the target's appearance features to obtain appearance anomaly indicators; Temperature rise shift analysis is performed on the target temperature characteristics to obtain temperature anomaly indicators; Environmental correlation analysis is performed on the target location characteristics to obtain location risk indicators; The target status score is calculated based on appearance anomaly indicators, temperature anomaly indicators, and location risk indicators, using the following formula: in, Indicates the target state score. Indicates an abnormal appearance indicator. Indicates an abnormal temperature index. This indicates a location risk indicator.

[0043] It should be explained that the feature extraction of the key target refers to the process of extracting the operational parameters of the key target. Target appearance features refer to the image of the key target's appearance, target temperature features refer to the temperature of the key target, and target location features refer to the location of the key target.

[0044] It is understood that the defect identification analysis of the target appearance features refers to: using a target detection algorithm (e.g., YOLOv8) to detect the number of defects in the target appearance features. The number of defects in the target appearance features is the appearance anomaly index. The method of using a target detection algorithm (e.g., YOLOv8) to detect the number of defects in the target appearance features is existing technology and will not be elaborated here. The temperature rise offset analysis of the target temperature features refers to calculating the difference between the target temperature feature and the standard temperature (ambient temperature). The difference between the target temperature feature and the standard temperature (ambient temperature) is the temperature anomaly index. The environmental correlation analysis of the target location features refers to: determining whether the target location features are key locations. If the target location features are key locations, the location risk index is 1; otherwise, it is 0. Key locations refer to the locations of critical equipment (e.g., transformers, photovoltaic panels) in a substation. The target status score reflects the health level of the critical target. The higher the target status score, the more dangerous the critical target is, and the more priority it needs to be inspected.

[0045] S4. Based on the target status score and target multi-source data, the inspection priority score is determined, and the inspection priority scores are summarized to obtain multiple inspection priority scores.

[0046] In detail, the process of determining the inspection priority score based on the target status score and target multi-source data includes: Multi-source data analysis is performed on the target multi-source data to obtain historical inspection data, real-time sensor data and operation and maintenance record data; Historical anomaly frequency indicators were identified based on historical inspection data. Real-time volatility risk indicators are identified based on real-time sensor data. Maintenance urgency indicators were identified based on maintenance record data; The target risk coefficient is calculated based on historical anomaly frequency indicators, real-time fluctuation risk indicators, and maintenance urgency indicators. The calculation formula is as follows: in, Indicates the target risk coefficient. Indicator representing the frequency of historical anomalies This indicates a real-time volatility risk indicator. Indicates the urgency of maintenance; The inspection priority score is calculated based on the target risk coefficient and target status score, using the following formula: in, Indicates the inspection priority score. Represents the natural constant. Indicates the target state score. This represents the target risk coefficient.

[0047] It should be explained that the multi-source data parsing of the target multi-source data refers to the process of parsing historical inspection data, real-time sensor data and operation and maintenance record data from the target multi-source data. Historical inspection data refers to the data recorded in the past for inspecting key targets, real-time sensor data refers to the real-time sensor data of key targets, and operation and maintenance record data refers to the data recorded in the past for maintaining key targets.

[0048] Understandably, the historical anomaly frequency index refers to the number of times critical targets have experienced anomalies recorded in historical inspection data; the real-time fluctuation risk index refers to the number of data anomalies recorded in real-time sensor data; and the maintenance urgency index refers to the number of historical maintenance records. The target risk coefficient reflects the urgency of critical targets requiring maintenance; the higher the target risk coefficient, the greater the urgency of critical targets requiring maintenance. The inspection priority score reflects the priority of critical targets requiring inspection; the higher the inspection priority score, the higher the priority of critical targets requiring inspection.

[0049] S5. Based on multiple inspection priority scores and multiple key targets, the initial inspection route is determined, and the parameters of the planned UAV are collected to obtain the inspection parameters.

[0050] It should be noted that the inspection parameters refer to the relevant parameters of the UAV to be planned during operation, and a detailed explanation of the inspection parameters can be found in the following embodiments.

[0051] In detail, the process of determining the initial inspection route based on multiple inspection priority scores and multiple key targets includes: Based on multiple key objectives, the coordinates of multiple target locations were confirmed. By associating multiple target location coordinates with multiple inspection priority scores, a set of target route plans is obtained; The target route planning set is sorted according to the inspection priority score to obtain the target access sequence; Multiple target distance values ​​were identified based on the target access sequence; The initial inspection route is obtained by performing path optimization processing based on multiple target distance values ​​and target access sequences.

[0052] It should be explained that target location coordinates refer to the coordinates corresponding to key targets. Associating multiple target location coordinates with multiple inspection priority scores means associating each target location coordinate with the corresponding inspection priority score, thus obtaining multiple target location coordinates containing inspection priority scores. The target route planning set is a set of multiple target location coordinates containing inspection priority scores. Sorting the target route planning set according to inspection priority scores means sorting the multiple target coordinate positions containing inspection priority scores in the target route planning set in descending order of inspection priority scores. The target access sequence refers to the sequence obtained after sorting the multiple target coordinate positions containing inspection priority scores in the target route planning set in descending order of inspection priority scores.

[0053] Understandably, the target distance value refers to the distance between any two target coordinate positions in the target access sequence. Optionally, the distance formula between two points can be used to calculate the target distance value. The path optimization process based on multiple target distance values ​​and the target access sequence refers to: using multiple target coordinate positions in the target access sequence as multiple nodes, and using multiple target distance values ​​as the cost of connecting the nodes, a swarm intelligence algorithm is used to iteratively search for the shortest or second shortest closed loop in the multiple nodes, thereby obtaining an access path that takes into account both inspection priority and total flight mileage. This access path is the initial inspection route.

[0054] S6. Based on the inspection parameters and the initial inspection route, the final planned route is determined.

[0055] Specifically, the process of determining the final planned route based on inspection parameters and the initial inspection route includes: Based on the inspection parameters, the remaining battery power, maximum range, payload capacity, and flight speed parameters are confirmed. The drone's operational range was determined based on the remaining battery power and maximum flight range parameters. The mission execution capability value of the UAV is determined based on the payload capacity parameters and flight speed parameters; The UAV's executable range, UAV mission execution capability value, and initial inspection route are matched and analyzed to obtain the route matching result, which is either a match or a non-match. If the route matching result is a match, the initial inspection route will be used as the final planned route. If the route matching result is a mismatch, the initial inspection route is corrected to obtain the corrected inspection route. The revised inspection route will be used as the final planned route.

[0056] It should be explained that the remaining power parameter refers to the remaining power of the drone to be inspected in the inspection parameters; the maximum flight time parameter refers to the maximum flight time of the drone to be inspected in the inspection parameters; the payload capacity parameter refers to the weight of the payload that the drone to be inspected can carry in the inspection parameters; and the flight speed parameter refers to the maximum flight speed of the drone to be inspected in the inspection parameters.

[0057] It is understandable that determining the executable range of the UAV based on the remaining battery power parameter and the maximum endurance parameter means: using the remaining battery power parameter of the UAV to be planned as the energy constraint boundary, and combining it with the maximum endurance parameter, calculating the maximum round-trip distance that the UAV can fly under the current safe battery margin, and defining the geographical space covered by this distance as the executable range of the UAV with the takeoff point as the center. Determining the UAV's mission execution capability value based on the payload capacity parameter and the flight speed parameter means: using the payload capacity parameter as the upper limit of the weight of the mission payload carried by the UAV when performing inspection tasks, and combining it with the flight speed parameter to comprehensively evaluate the efficiency of the tasks that the UAV to be planned can complete when performing inspection tasks. This efficiency is the mission execution capability value. It should be noted that the higher the flight speed, the larger the coverage area per unit time, but excessively high speeds will lead to a decrease in detection accuracy due to motion blur, while also increasing energy consumption. Therefore, the mission execution capability value can be quantified as the product of flight speed and effective detection probability, divided by the energy consumption per unit distance. A higher mission execution capability value indicates a stronger overall capability of the UAV to perform inspection tasks under the current payload capacity parameters and flight speed parameters. Both the effective detection probability and energy consumption per unit distance can be obtained from the historical operational data of the UAV to be inspected. The matching analysis of the UAV's executable range, mission execution capability value, and initial inspection route refers to: verifying whether the total flight mileage of the initial inspection route is entirely within the executable range of the UAV to be inspected; and verifying whether the number and distribution of key targets in the initial inspection route require a mission execution capability value within the limits of the UAV's current payload and speed. If both verifications are satisfied, the route matching result is determined to be a match; otherwise, it is determined to be a mismatch.

[0058] It should be understood that the final planned route refers to the initial inspection route when the route matching result is a match. The modification of the initial inspection route refers to: when the route matching result is a mismatch, adjusting the initial inspection route based on the constraints of the UAV's executable range and mission execution capability; the adjustment operation includes: removing the lowest inspection priority score from the target access sequence; and re-executing the matching analysis until the adjusted inspection route meets the dual constraints of the planned UAV's executable range and mission execution capability. The modified inspection route refers to the modified initial inspection route.

[0059] S7. Send the final planned route to the drone to be planned, obtain the target drone, and complete the drone inspection planning.

[0060] It should be explained that the process of sending the final planned route to the drone to be planned refers to the process of transmitting the final planned route to the drone to be planned, and the target drone refers to the drone to be planned that receives the final planned route.

[0061] For example, once the target drone is acquired, Xiao Zhang can start the target drone and use it to inspect the area to be inspected.

[0062] To address the problems described in the background art, this invention identifies the area to be inspected and the drone to be planned. This clearly defines the scope of the inspection task and the execution equipment, establishing a basic framework for subsequent path planning. Then, image acquisition is performed on the area to be inspected to obtain raw image data. This invention provides raw data support for automatic identification of inspection targets by acquiring visual information of the area. Key targets are extracted from the raw image data to obtain multiple key targets. This invention intelligently filters out objects requiring priority inspection from the images, achieving target positioning and classification, improving the efficiency of drone inspection planning, and thus improving inspection accuracy. For each key target, the following operations are performed: A target status score is determined based on the key target. This invention quantitatively assesses the current operating status or degree of abnormality of each target to form an urgency indicator. Multi-source data is collected from the key targets to obtain target multi-source data. This invention comprehensively reflects the actual state of the target by fusing information from multiple sensors such as temperature and vibration. An inspection priority score is determined based on the target status score and the target multi-source data. This invention calculates the inspection priority score by comprehensively considering the status and multi-source data. The invention prioritizes the inspection of targets, summarizing inspection priority scores to obtain multiple inspection priority scores. This demonstrates that by aggregating the priority scores of all targets, a globally ordered inspection list is formed. Based on multiple inspection priority scores and multiple key targets, an initial inspection route is determined. This invention improves the efficiency of UAV inspection planning and thus improves inspection accuracy by planning preliminary flight paths covering all key targets according to priority. Parameters are collected from the UAV to be planned to obtain inspection parameters. This invention provides constraints for route optimization by acquiring the UAV's endurance, speed, obstacle avoidance, and other performance constraints. Based on the inspection parameters and the initial inspection route, the final planned route is determined. This invention generates a safe and executable inspection track by adjusting the initial route according to actual flight capabilities. The final planned route is then sent to the UAV to be planned, obtaining the target UAV and completing the UAV inspection planning. This invention improves the intelligence level and operational efficiency of inspection tasks by realizing an automated closed loop from planning to execution, thus improving the efficiency of UAV inspection planning and consequently improving inspection accuracy. Therefore, this invention can improve the efficiency of UAV inspection planning and thus improve inspection accuracy.

[0063] like Figure 2 The diagram shown is a functional block diagram of a UAV inspection planning system based on multi-source data provided in an embodiment of the present invention.

[0064] The UAV inspection and planning system 100 based on multi-source data described in this invention can be installed in an electronic device. Depending on the functions implemented, the UAV inspection and planning system 100 may include a key target acquisition module 101, a priority scoring calculation module 102, a multi-source data acquisition module 103, and a path planning and distribution module 104. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.

[0065] The key target acquisition module 101 is used to identify the area to be inspected and the drone to be planned, to collect images of the area to be inspected, to obtain the original image data of the area, and to extract key targets from the original image data of the area to obtain multiple key targets. The priority scoring calculation module 102 is used to perform the following operations on each of the multiple key targets: confirm the target status score based on the key target, collect multi-source data on the key target to obtain target multi-source data, and confirm the inspection priority score based on the target status score and the target multi-source data. The multi-source data acquisition module 103 is used to summarize the inspection priority scores to obtain multiple inspection priority scores, determine the initial inspection route based on the multiple inspection priority scores and multiple key targets, and collect parameters of the UAV to be planned to obtain inspection parameters. The path planning and distribution module 104 is used to determine the final planned route based on the inspection parameters and the initial inspection route, and distribute the final planned route to the drone to be planned, thereby obtaining the target drone and completing the drone inspection planning.

[0066] In detail, the modules in the UAV inspection planning system 100 based on multi-source data described in this embodiment of the invention employ the same methods as described above. Figure 1 The method used is the same as the UAV inspection planning method based on multi-source data described above, and can produce the same technical effect, so it will not be repeated here.

[0067] like Figure 3 The diagram shown is a structural schematic of an electronic device for implementing a UAV inspection planning method based on multi-source data, according to an embodiment of the present invention.

[0068] The electronic device 1 may include a processor 10, a memory 11 and a bus 12, and may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as a UAV inspection planning method program based on multi-source data.

[0069] The memory 11 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as the portable hard drive of the electronic device 1. In other embodiments, the memory 11 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 1. Furthermore, the memory 11 includes both internal storage units and external storage devices of the electronic device 1. The memory 11 can be used not only to store application software and various types of data installed on the electronic device 1, such as the code of a UAV inspection planning method program based on multi-source data, but also to temporarily store data that has been output or will be output.

[0070] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., a UAV inspection planning method program based on multi-source data) and calls data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

[0071] The bus 12 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 12 can be divided into an address bus, a data bus, a control bus, etc. The bus 12 is configured to realize the connection and communication between the memory 11 and at least one processor 10, etc.

[0072] Figure 3 Only electronic devices with components are shown; those skilled in the art will understand that... Figure 3The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0073] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0074] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device 1 and other electronic devices.

[0075] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual user interface.

[0076] The UAV inspection planning method program based on multi-source data stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When run in the processor 10, it can achieve the following: Identify the areas to be inspected and the drones to be planned; Images are acquired from the area to be inspected to obtain the raw image data of the area; Key targets were extracted from the original image data of the region, resulting in multiple key targets. For each of the multiple key objectives, perform the following actions: Target status scores were determined based on key objectives; Multi-source data collection is performed on key targets to obtain multi-source target data; Inspection priority scores are determined based on target status scores and multi-source target data. Summarize the inspection priority scores to obtain multiple inspection priority scores; The initial inspection route was determined based on multiple inspection priority scores and multiple key objectives. Parameters are collected from the drone to be planned to obtain inspection parameters; The final planned route is determined based on the inspection parameters and the initial inspection route. The final planned route is sent to the drones to be planned, the target drones are obtained, and the drone inspection plan is completed.

[0077] Specifically, the processor 10's implementation method for the above instructions can be found in [reference needed]. Figures 1 to 3 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0078] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0079] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following: Identify the areas to be inspected and the drones to be planned; Images are acquired from the area to be inspected to obtain the raw image data of the area; Key targets were extracted from the original image data of the region, resulting in multiple key targets. For each of the multiple key objectives, perform the following actions: Target status scores were determined based on key objectives; Multi-source data collection is performed on key targets to obtain multi-source target data; Inspection priority scores are determined based on target status scores and multi-source target data. Summarize the inspection priority scores to obtain multiple inspection priority scores; The initial inspection route was determined based on multiple inspection priority scores and multiple key objectives. Parameters are collected from the drone to be planned to obtain inspection parameters; The final planned route is determined based on the inspection parameters and the initial inspection route. The final planned route is sent to the drones to be planned, the target drones are obtained, and the drone inspection plan is completed.

[0080] In the embodiments provided by this invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and actual implementations may have other classification methods.

[0081] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0082] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0083] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for planning UAV inspections based on multi-source data, characterized in that, The method includes: Identify the areas to be inspected and the drones to be planned; Images are acquired from the area to be inspected to obtain the raw image data of the area; Key targets were extracted from the original image data of the region, resulting in multiple key targets. For each of the multiple key objectives, perform the following actions: Target status scores were determined based on key objectives; Multi-source data collection is performed on key targets to obtain multi-source target data; Inspection priority scores are determined based on target status scores and multi-source target data. Summarize the inspection priority scores to obtain multiple inspection priority scores; The initial inspection route was determined based on multiple inspection priority scores and multiple key objectives. Parameters are collected from the drone to be planned to obtain inspection parameters; The final planned route is determined based on the inspection parameters and the initial inspection route. The final planned route is sent to the drones to be planned, the target drones are obtained, and the drone inspection plan is completed.

2. The UAV inspection planning method based on multi-source data as described in claim 1, characterized in that, The process of acquiring images of the area to be inspected to obtain raw image data of the area includes: The area to be inspected is divided into multiple unit areas; Multiple unit regions are numbered to obtain multiple numbered regions; Extracting the number from multiple numbered regions There are numbered regions, among which... The initial value is 1; For the Visible light images were acquired from the numbered regions to obtain the first... Visible light images of the region; For the Thermal infrared images were acquired from the numbered regions to obtain the first... Thermal infrared images of the area; Based on the The numbered area was confirmed to be the first Area number; The first Visible light image of the region, the first Thermal infrared images of the region and the first Bind the area number to get the first one. Image data for each region; The first Image data from each region is fed into a pre-constructed initial image dataset to obtain an updated image dataset; make ,Will As The updated image dataset is used as the initial image dataset, and the result is returned from the extraction of the first image from multiple numbered regions. The steps for each numbered area, until... The updated image dataset will be used as the original image data for the region, where, This represents the number of numbered regions among multiple numbered regions.

3. The UAV inspection planning method based on multi-source data as described in claim 2, characterized in that, The process of extracting key targets from the original image data of the region yields multiple key targets, including: Based on the original image data of the region, the visible stitched image and the infrared stitched image of the region were identified. Band registration is performed on the visible mosaic image and the infrared mosaic image of the region to obtain the registered region image; Semantic segmentation is performed on the registered region image to obtain multiple device instances; Perform the following operation on each of the multiple device instances: Visible texture features and infrared temperature distribution features were identified based on device examples; The credibility score was determined based on visible texture features and infrared temperature distribution features. The credibility scores are aggregated to obtain multiple credibility scores; Multiple target scores were identified based on multiple credibility scores and preset scoring thresholds; Several key objectives were identified based on multiple target scores and multiple device instances.

4. The UAV inspection planning method based on multi-source data as described in claim 3, characterized in that, The process of identifying the visible stitched image and the infrared stitched image of the region based on the original image data of the region includes: Based on the original image data of the region, multiple visible light images and multiple thermal infrared images of the region were identified. Perform the following operation on each of the multiple visible light images: The coordinates of the visible light image of the region are corrected to obtain the corrected visible light image; By summarizing the corrected visible light images, multiple corrected visible light images are obtained; Multiple corrected visible light images are stitched together to obtain a regional visible stitched image; The regional infrared mosaic image was confirmed based on multiple regional thermal infrared images.

5. The UAV inspection planning method based on multi-source data as described in claim 4, characterized in that, The confidence score determined based on visible texture features and infrared temperature distribution features includes: Texture sharpness analysis is performed on visible texture features to obtain texture sharpness indices; Thermal anomaly analysis was performed on the infrared temperature distribution characteristics to obtain thermal anomaly indices; The target credibility score is calculated based on the texture sharpness index and the thermal anomaly index, using the following formula: in, Indicates the target credibility score. Indicates texture sharpness index, Indicates thermal anomaly index, Represents the natural logarithm. Indicates taking the absolute value; Compare the target credibility score with a preset credibility threshold. If the target credibility score is greater than the credibility threshold, then the target credibility score is used as the credibility score. If the target credibility score is less than or equal to the credibility threshold, then the credibility threshold will be used as the credibility score.

6. The UAV inspection planning method based on multi-source data as described in claim 5, characterized in that, The process of determining the target status score based on key objectives includes: Feature extraction is performed on key targets to obtain target appearance features, target temperature features, and target location features; Defect identification and analysis are performed on the target's appearance features to obtain appearance anomaly indicators; Temperature rise shift analysis is performed on the target temperature characteristics to obtain temperature anomaly indicators; Environmental correlation analysis is performed on the characteristics of the target location to obtain location risk indicators; The target status score is calculated based on appearance anomaly indicators, temperature anomaly indicators, and location risk indicators.

7. The UAV inspection planning method based on multi-source data as described in claim 6, characterized in that, The inspection priority score determined based on the target status score and multi-source target data includes: Multi-source data analysis is performed on the target multi-source data to obtain historical inspection data, real-time sensor data and operation and maintenance record data; Historical anomaly frequency indicators were identified based on historical inspection data. Real-time volatility risk indicators are identified based on real-time sensor data. Maintenance urgency indicators were identified based on maintenance record data; The target risk coefficient is calculated based on historical anomaly frequency indicators, real-time fluctuation risk indicators, and maintenance urgency indicators. The inspection priority score is calculated based on the target risk coefficient and target status score, using the following formula: in, Indicates the inspection priority score. Represents the natural constant. Indicates the target state score. This represents the target risk coefficient.

8. The UAV inspection planning method based on multi-source data as described in claim 7, characterized in that, The initial inspection route is determined based on multiple inspection priority scores and multiple key targets, including: Based on multiple key objectives, the coordinates of multiple target locations were confirmed. By associating multiple target location coordinates with multiple inspection priority scores, a set of target route plans is obtained; The target route planning set is sorted according to the inspection priority score to obtain the target access sequence; Multiple target distance values ​​were identified based on the target access sequence; The initial inspection route is obtained by performing path optimization processing based on multiple target distance values ​​and target access sequences.

9. The UAV inspection planning method based on multi-source data as described in claim 8, characterized in that, The process of determining the final planned route based on inspection parameters and the initial inspection route includes: Based on the inspection parameters, the remaining battery power, maximum range, payload capacity, and flight speed parameters are confirmed. The drone's operational range was determined based on the remaining battery power and maximum flight range parameters. The mission execution capability value of the UAV is determined based on the payload capacity parameters and flight speed parameters; The UAV's executable range, UAV mission execution capability value, and initial inspection route are matched and analyzed to obtain the route matching result, which is either a match or a non-match. If the route matching result is a match, the initial inspection route will be used as the final planned route. If the route matching result is a mismatch, the initial inspection route is corrected to obtain the corrected inspection route. The revised inspection route will be used as the final planned route.

10. A drone inspection planning system based on multi-source data, characterized in that, The system includes: The key target acquisition module is used to identify the area to be inspected and the drone to be planned, to collect images of the area to be inspected, to obtain the original image data of the area, and to extract key targets from the original image data of the area to obtain multiple key targets. The priority score calculation module is used to perform the following operations for each of the multiple key objectives: determine the target status score based on the key objective, collect multi-source data on the key objective to obtain target multi-source data, and determine the inspection priority score based on the target status score and the target multi-source data. The multi-source data acquisition module is used to summarize the inspection priority scores, obtain multiple inspection priority scores, determine the initial inspection route based on multiple inspection priority scores and multiple key targets, and collect parameters of the UAV to be planned to obtain inspection parameters. The path planning and distribution module is used to determine the final planned route based on the inspection parameters and the initial inspection route, and then distribute the final planned route to the drone to be planned, thereby obtaining the target drone and completing the drone inspection planning.

Images