An unmanned aerial vehicle inspection overhead line hidden danger online detection method and system

Abstract

This invention discloses an online detection method and system for overhead power line hazards using unmanned aerial vehicle (UAV) inspection. The method includes: identifying the component type of the component to be inspected to obtain its modal sensitivity; combining inspection distance normalization factors and inspection speed normalization factors; registering visible light and infrared images and calculating real-time alignment confidence; dynamically adjusting the dual-modal fusion weights based on the above parameters to generate a fusion feature map and obtaining an overlap threshold; detecting temperature anomaly regions and visual defect regions based on the fusion feature map, calculating the overlap degree, and determining the existence of a composite hazard when it exceeds the threshold. This invention solves the problem of existing technologies' difficulty in adaptively optimizing fusion strategies and quantitatively assessing defect correlation through dynamic adjustment of fusion weights and quantitative analysis of defect spatial correlation, thereby improving the accuracy of online detection of composite hazards in overhead power lines.

Description

Technical Field

[0001] This invention relates to the field of power line inspection technology, specifically to an online detection method and system for potential hazards in overhead power lines using unmanned aerial vehicles (UAVs). Background Technology

[0002] Overhead lines are a crucial component of the power system, and their operational status directly impacts power supply reliability. Due to their long-term exposure to the open outdoor environment, components such as conductors, insulators, and fittings gradually deteriorate under the combined effects of continuous mechanical tension, electrical load, and natural weather conditions. Traditionally, power departments have relied on manual inspections for regular line checks to promptly identify potential hazards. However, manual inspections are inefficient, labor-intensive, and severely limited by terrain and weather conditions. In recent years, drone inspections have been widely adopted in overhead line inspections due to their flexibility, efficiency, and wide coverage.

[0003] Unmanned aerial vehicle (UAV) inspection systems are typically equipped with both visible light cameras and infrared thermal imagers. The former is used to acquire information about the texture, color, and shape of the component surface, while the latter is used to acquire information about the component's temperature distribution. By analyzing visible light images, visual defects such as broken strands, corrosion, and damage can be detected, while by analyzing infrared images, temperature anomalies caused by poor contact, overload, and other factors can be detected.

[0004] Existing UAV inspection methods, when simultaneously utilizing visible light and infrared images for hazard detection, suffer from the following shortcomings: the fusion strategy is insufficiently adaptable to changes in inspection conditions and component characteristics; it lacks effective analysis of the spatial correlation between visual defects and temperature anomalies, easily leading to missed or misjudged complex hazards. Therefore, how to dynamically adjust the fusion strategy of visible light and infrared features and quantitatively assess the degree of spatial correlation between visual defects and temperature anomalies to improve the accuracy of online detection of complex hazards on overhead lines is a technical problem that needs to be solved in this field. Summary of the Invention

[0005] This invention provides an online detection method for potential hazards on overhead power lines using unmanned aerial vehicle (UAV) inspections. It addresses the technical problems of existing technologies, such as the difficulty in dynamically adjusting the fusion strategy of visible light and infrared features and quantitatively assessing the spatial correlation between visual defects and temperature anomalies, leading to inaccurate online detection of complex hazards on overhead power lines. In view of the above problems, this invention provides an online detection method and system for potential hazards on overhead power lines using unmanned aerial vehicle (UAV) inspections.

[0006] In a first aspect, the present invention provides an online detection method for potential hazards in overhead power lines using unmanned aerial vehicle (UAV) inspections, the method comprising: The modal sensitivity and standard detection distance of the component to be tested are obtained, and the measured distance and inspection speed between the UAV and the component to be tested are obtained. The inspection distance normalization factor and the inspection speed normalization factor are calculated and obtained. The visible light image and infrared image of the component to be detected are registered, and the real-time alignment confidence is calculated. Based on the modal sensitivity of the component, the normalization factor of the inspection distance and the normalization factor of the inspection speed, the fusion weight of the visible light feature and the infrared feature is dynamically adjusted to generate a fused feature map. Based on the inspection distance normalization factor, the inspection speed normalization factor, and the real-time alignment confidence, the overlap threshold for hazard judgment is obtained. Based on the fused feature map, temperature anomaly areas and visual defect areas are detected, and the degree of overlap is calculated. When the degree of overlap is greater than the overlap threshold, it is determined that there is a composite hidden danger.

[0007] Secondly, the present invention also provides an online detection system for potential hazards in overhead power lines inspected by unmanned aerial vehicles (UAVs), the system comprising: The inspection parameter acquisition module is used to acquire the modal sensitivity and standard detection distance of the component to be inspected, and to acquire the measured distance and inspection speed between the UAV and the component to be inspected, and to calculate and acquire the inspection distance normalization factor and the inspection speed normalization factor. The image registration module is used to register the visible light image and the infrared image of the component to be detected, and to calculate the real-time alignment confidence. The feature fusion module is used to dynamically adjust the fusion weights of visible light features and infrared features based on the modal sensitivity of the component, the normalization factor of the inspection distance, and the normalization factor of the inspection speed, and to generate a fused feature map. The threshold acquisition module is used to acquire the overlap threshold of the hidden danger judgment based on the inspection distance normalization factor, the inspection speed normalization factor and the real-time alignment confidence. The hazard determination module is used to detect temperature abnormality areas and visual defect areas based on the fused feature map, calculate the degree of overlap, and determine that there is a composite hazard when the degree of overlap is greater than the overlap threshold.

[0008] One or more technical solutions provided in this invention have at least the following technical effects or advantages: First, this invention dynamically adjusts the fusion weights of visible light and infrared features by using component modal sensitivity, inspection distance normalization factor, and inspection speed normalization factor. When the inspection distance increases or the speed accelerates, the fusion weights tilt towards modes with stronger noise resistance; when a component exhibits more significant hazard characteristics in a particular mode, the fusion weights tilt towards that mode. Compared to the fixed-weight fusion method of existing technologies, this invention can adaptively optimize the fusion strategy based on different component types and real-time inspection status, ensuring that the fused feature map always retains the most valuable feature information for hazard identification.

[0009] Secondly, this invention establishes a spatial correlation quantification mechanism for dual-modal defects by calculating the degree of overlap between temperature anomaly regions and visual defect regions and comparing it with a dynamically adjusted overlap threshold. The overlap threshold is dynamically adjusted based on inspection distance normalization factors, inspection speed normalization factors, and real-time alignment confidence. The threshold is widened accordingly when image registration quality deteriorates or inspection conditions worsen, avoiding missed detections due to data quality fluctuations. Compared to existing technologies that analyze visible light defects and infrared temperature anomalies independently, this invention effectively distinguishes between complex defects and single-modal anomalies, solving the problem of missed or misjudged complex defects.

[0010] In summary, this invention solves the technical problem of inaccurate online detection of complex hazards in overhead lines by dynamically adjusting the fusion strategy of visible light and infrared features and quantitatively assessing the spatial correlation between visual defects and temperature anomalies, through dynamic adjustment of fusion weights and spatial correlation quantification of dual-modal defects. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating an online detection method for potential hazards in overhead power lines using unmanned aerial vehicle (UAV) inspection, as provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of an online detection system for potential hazards in overhead power lines provided by an embodiment of the present invention. The components represented by each number in the attached diagram are described as follows: Inspection parameter acquisition module 11, image registration module 12, feature fusion module 13, threshold acquisition module 14, and hidden danger judgment module 15. Detailed Implementation

[0013] This invention provides an online detection method and system for overhead power line hazards using unmanned aerial vehicle (UAV) inspections. It addresses the technical problem of inaccurate online detection of complex hazards in overhead power lines, which is difficult to address in existing technologies due to the difficulty in dynamically adjusting the fusion strategy of visible light and infrared features and quantitatively assessing the spatial correlation between visual defects and temperature anomalies.

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0015] It should be noted that the terms "comprising" and "having" are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or modules that are not explicitly listed or that are inherent to these processes, methods, products, or devices.

[0016] Example 1

[0017] like Figure 1 As shown, this invention provides an online detection method for potential hazards in overhead power lines using unmanned aerial vehicle (UAV) inspections. The method includes: S100: Obtain the modal sensitivity and standard detection distance of the component to be tested, and obtain the measured distance and inspection speed between the UAV and the component to be tested, and calculate and obtain the inspection distance normalization factor and the inspection speed normalization factor. When drones inspect overhead power lines, different components exhibit varying capabilities in representing potential hazards under visible light and infrared imaging modalities. Furthermore, the distance between the drone and the component under inspection, as well as its flight speed, dynamically change during the inspection, directly impacting image acquisition quality. If a fixed fusion weight is applied to the visible light and infrared images for subsequent analysis, excessive ambiguity may be introduced into the fused feature map when the inspection distance is too great or the flight speed is too high, leading to a decrease in visible light image quality. Conversely, when a component's hazard is more pronounced under a particular modality, the fixed weight cannot favor that modality to highlight key features.

[0018] Step S100 in the method provided in this embodiment of the invention includes: Identify the component to be detected from the visible light image to obtain the component type; From the pre-established component type and attribute mapping table, query and obtain the component modal sensitivity and standard detection distance corresponding to the component type, wherein the component modal sensitivity is the ratio of the distinguishability of the hidden danger features of the component type in the visible light mode to the distinguishability of the hidden danger features in the infrared mode; Obtain the measured distance between the drone and the component to be inspected, divide the measured distance by the standard detection distance, and obtain the inspection distance normalization factor; The inspection speed of the UAV within the time window during which it identifies the component to be inspected is obtained. The inspection speed is then divided by the standard inspection speed to obtain the inspection speed normalization factor. The inspection speed is obtained based on the average inspection speed during the identification of the component to be inspected.

[0019] The specific implementation method is as follows: First, the component type of the target component is identified from the visible light images captured by the visible light camera mounted on the UAV. Component identification can be achieved using a target detection algorithm based on a convolutional neural network, such as the YOLO algorithm. This algorithm divides the input image into several grid cells, each responsible for predicting the position, size, and class probability of the bounding box falling within it. The network structure consists of three parts: a feature extraction backbone network, a feature pyramid fusion layer, and a detection head. The backbone network extracts multi-scale features of the image through layer-by-layer convolution and downsampling. The feature pyramid fusion layer fuses feature maps from different levels to take into account the detection capabilities of both large and small targets. The detection head outputs the coordinate offset, confidence score, and class score of each bounding box. Images of common line component types and bounding box positions, such as insulators, clamps, splicing pipes, conductors, and vibration dampers, collected during historical inspections and manually labeled, are used as training samples. A total loss function is constructed using the positional deviation loss between the labeled box and the predicted box and the class prediction loss. The network weight parameters are iteratively updated through the backpropagation algorithm until the loss function converges, resulting in the trained component identification model. Input the currently acquired visible light image into the model, and the component recognition model outputs the bounding box location of the component to be detected and its corresponding component category label to obtain the component type.

[0020] Furthermore, the modal sensitivity and standard detection distance corresponding to the component type are retrieved from a pre-established component type and attribute mapping table. This mapping table is pre-built and stored based on historical detection data of different circuit components in both visible and infrared modes. Component modal sensitivity is defined as the ratio of the distinguishability of hazard features for that component type in the visible light mode to that in the infrared mode. Hazard feature distinguishability can be quantified by the classification accuracy of normal and abnormal samples of that component in the corresponding mode in historical data. Specifically, a binary classifier is trained in that mode to distinguish between normal and abnormal samples, with classification accuracy used as the measure of distinguishability. When the component modal sensitivity is greater than 1, it indicates that the hazard features of that component type are more easily distinguishable in the visible light mode; when the component modal sensitivity is less than 1, it indicates that the features are more easily distinguishable in the infrared mode.

[0021] For example, cracks and damage on the surface of an insulator have obvious texture features in visible light images, and its component modal sensitivity can be set to 1.5; poor contact of a splice or clamp mainly reflects temperature rise, and its component modal sensitivity can be set to 0.6. The standard detection distance refers to the reference distance for image acquisition of this component type under ideal inspection conditions. It can be set according to the size of different components and the imaging clarity requirements. For example, the standard detection distance for an insulator can be set to 5 meters, and the standard detection distance for a conductor can be set to 8 meters.

[0022] Further, the measured distance between the drone and the component under inspection is obtained, and the inspection distance normalization factor is calculated. The measured distance can be obtained in real time by a laser rangefinder sensor mounted on the drone. The laser rangefinder sensor emits a laser pulse towards the component under inspection, receives the reflected pulse, and measures the round-trip time. This time is multiplied by the speed of light and divided by two to obtain the measured distance value. The measured distance is then divided by the standard detection distance corresponding to this component type to obtain the inspection distance normalization factor. When the inspection distance normalization factor is greater than 1, it indicates that the drone is currently farther from the component than the standard distance. This results in a decrease in the spatial resolution of the visible light image, blurred texture details, and a reduction in the registration accuracy between the subsequent visible light and infrared images due to the greater distance. When the inspection distance normalization factor is less than 1, it indicates a closer distance and richer image details. For example, if the measured distance between the drone and the insulator in a certain inspection is 10 meters, and the standard detection distance of the insulator is 5 meters, then the inspection distance normalization factor is 10 ÷ 5 = 2.0.

[0023] Finally, the inspection speed of the UAV within the time window for identifying the component under inspection is obtained, and the inspection speed normalization factor is calculated. The inspection speed can be obtained by measuring the position changes of the inertial measurement unit or satellite positioning module mounted on the UAV at continuous time points. The position change is divided by the time interval to obtain the instantaneous speed, and the average of the instantaneous speeds at each moment within the time window is taken as the average inspection speed. The time window refers to the time period from when the component under inspection enters the field of view of the visible light camera to when image acquisition is completed. The average inspection speed is divided by the standard inspection speed to obtain the inspection speed normalization factor. The standard inspection speed refers to the preset cruising speed of the UAV in this inspection task, which can be set according to the inspection task specifications. For example, the standard inspection speed can be set to 5 meters per second. When the inspection speed normalization factor is greater than 1, it indicates that the current flight speed is faster than the standard speed, and the visible light image may have motion blur. Moreover, the rapid flight leads to an increase in the change of perspective between adjacent frames, increasing the difficulty of image registration. When the inspection speed normalization factor is less than 1, it indicates that the flight speed is slower, and the image quality is relatively stable. For example, if the average inspection speed within the time window for identifying the component to be inspected during a certain inspection is 7 meters per second, and the standard inspection speed is 5 meters per second, then the normalization factor for the inspection speed is 7 ÷ 5 = 1.4.

[0024] The following technical effects were achieved through this step: First, we introduce component modal sensitivity as an indicator to quantify the differences in the distinguishability of hazard features between different component types in visible and infrared modes. When the component modal sensitivity is greater than 1, it indicates that the hazard features of the component are easier to distinguish in the visible light mode, and the base adjustment amount of the subsequent visible light fusion weight is increased accordingly, causing the fused feature map to tend to retain more visible light details. When the component modal sensitivity is less than 1, it indicates that the infrared mode is more advantageous, and the fusion result is tilted towards infrared features. This parameter, which is set individually based on component type, provides a basis for the differentiated fusion of different components.

[0025] Second, by using inspection distance and inspection speed normalization factors, the real-time inspection conditions of the UAV are quantified into standardized dimensionless parameters. A larger inspection distance normalization factor indicates a greater distance between the UAV and the component, resulting in a decrease in visible light image quality; a larger inspection speed normalization factor indicates a faster flight speed, increasing motion blur in the visible light image. These two normalization factors provide a unified quantitative input for the subsequent dynamic adjustment of fusion weights and the calculation of the overlap threshold, enabling the fusion strategy to adapt to different inspection conditions.

[0026] S200: Register the visible light image and infrared image of the component to be detected, and calculate the real-time alignment confidence. Visible light cameras and infrared thermal imagers are typically used on drones at different installation angles and optical axis positions, resulting in spatial discrepancies between the two types of images. Furthermore, due to the different operating principles of the two cameras—visible light images record reflected light from an object's surface, while infrared images record the object's own thermal radiation—the texture and edge features of the same scene appear significantly different in the two images. Therefore, image registration is necessary to ensure that the same physical location corresponds to the same coordinate position in both images. However, due to the aforementioned differences in imaging principles, drone flight vibrations, and variations in inspection distance and speed, the registration quality between the two images is not constant. If the registration quality is poor but subsequent analysis is performed using default standards, significant spatial discrepancies may occur between temperature anomaly areas and visual defect areas. This can easily lead to misjudging two anomalies belonging to the same physical location as originating from different locations, or vice versa, resulting in the missed or misjudged detection of complex hazards.

[0027] Step S200 in the method provided in this embodiment of the invention includes: Feature points are extracted from the visible light image and the infrared image respectively, and the two sets of feature points are matched to obtain a set of matching point pairs; Count the number of internal point pairs in the set of matching point pairs that satisfy the preset geometric consistency constraint, and obtain the total number of matching point pairs in the set of matching point pairs; Divide the number of internal point pairs by the total number of matched point pairs to obtain the real-time alignment confidence.

[0028] The specific implementation method is as follows: First, feature points are extracted from both visible light and infrared images. The Scale Invariant Feature Transform (SIFT) algorithm is used for feature point extraction. SIFT detects local extrema in different scale spaces of the image as candidate feature points. Specifically, the input image is subjected to multiple Gaussian blurs to generate image pyramids at different scales. Differences are performed between adjacent scales to obtain a Gaussian difference pyramid. In this pyramid, each pixel is compared with its 8 neighbors at its current scale and 18 neighbors at the adjacent scales (a total of 26 neighbors). If a pixel is a maximum or minimum value, it is considered a candidate keypoint. The candidate keypoints are then precisely located, and low-contrast keypoints and unstable keypoints located at edges are removed. Next, based on the gradient direction distribution within the keypoint's neighborhood, one or more principal directions are assigned to each keypoint to ensure rotation invariance. A 16×16 neighborhood window centered on the keypoint is divided into 4×4 sub-regions. Gradient histograms in eight directions are calculated within each sub-region, generating a 4×4×8 feature description vector of 128 dimensions. Perform the above steps on the visible light image and the infrared image respectively to obtain two sets of feature description vectors.

[0029] Furthermore, the two sets of feature description vectors are matched to obtain a set of matching point pairs. Matching can be performed using a brute-force matching algorithm combined with a nearest neighbor / second nearest neighbor distance ratio check. The core idea of ​​the brute-force matching algorithm is exhaustive search: for each feature description vector to be matched, calculate the distance between it and all vectors in the other set of feature description vectors, and find the closest matching object. In feature point matching, the two sets of feature description vectors have the same dimension. The similarity between two feature points is measured by calculating the distance between the vectors; the smaller the distance, the more similar the two feature description vectors are, and the more likely the two feature points correspond to the same physical location in space.

[0030] Furthermore, Euclidean distance can be used as the distance metric. Euclidean distance comprehensively considers the differences between feature description vectors across all dimensions and is a standard method for measuring the straight-line distance between two points in a multidimensional space. The smaller the Euclidean distance, the closer the two feature points are in the 128-dimensional feature space, and the higher the similarity of their local image textures. For each feature description vector in the visible light image, its Euclidean distance to all feature description vectors in the infrared image is calculated. Then, the nearest neighbor to second nearest neighbor distance ratio test is used to filter the matches.

[0031] Specifically, for a given feature description vector in a visible light image, find the nearest neighbor with the smallest Euclidean distance and the second nearest neighbor with the second smallest Euclidean distance among all feature description vectors in the infrared image. If the ratio of the nearest neighbor distance to the second nearest neighbor distance is less than a preset ratio threshold (e.g., 0.8), it indicates that the feature point is much closer to its nearest neighbor in the feature space than to other points, and the match is unique and discriminative, so the match is considered reliable, and this pair of feature points is added to the matching point pair set. If the ratio is greater than or equal to the preset ratio threshold, it indicates that the nearest and second nearest neighbors are close to the feature point, and the match is ambiguous, possibly due to feature texture duplication or noise interference, so the matching pair is discarded. This process is repeated for all feature points in the visible light image to obtain the initial matching point pair set.

[0032] Furthermore, the number of inlier pairs satisfying the preset geometric consistency constraints in the matching point pair set is counted. The Random Sample Consensus (RANSAC) algorithm can be used to estimate the geometric transformation relationship between the two images, and matching point pairs conforming to this transformation relationship are designated as inliers. The RANSAC algorithm is executed as follows: several point pairs are randomly selected from the matching point pair set, and the homography matrix is ​​calculated as a candidate transformation model; the remaining point pairs in the matching point pair set are substituted into the candidate model, and the projection error is calculated; if the projection error is less than a preset error threshold (e.g., 3 pixels), the point pair is included in the inlier set; this random sampling and inlier counting process is repeated several times (e.g., 500 times), and the inlier set with the largest number of inliers is selected as the final inlier set. The number of point pairs in the inlier set is counted and recorded as the inlier pair count, while the total number of matching point pairs in the matching point pair set is obtained.

[0033] Finally, the real-time alignment confidence score is obtained by dividing the number of inlier pairs by the total number of matched pairs. The real-time alignment confidence score ranges from 0 to 1. The closer the value is to 1, the higher the proportion of matched pairs satisfying the same geometric transformation relationship between the two images, and the better the registration quality. The closer the value is to 0, the lower the proportion of inlier pairs satisfying geometric consistency constraints, and the poorer the registration quality. This may be due to sparse image features caused by excessive inspection distance, severe motion blur caused by excessive inspection speed, or large differences in surface texture of the component in the two modalities. For example, if the total number of matched pairs in a certain registration is 120, and the number of inlier pairs after RANSAC filtering is 96, then the real-time alignment confidence score is 96 ÷ 120 = 0.8.

[0034] The following technical effects were achieved through this step: First, based on the registration of visible light and infrared images, a real-time alignment confidence score is introduced as a quantitative indicator to measure the registration quality. Compared with existing technologies that only focus on whether registration is completed without evaluating the registration quality, the confidence score value output in this step can dynamically reflect the reliability of spatial alignment between the two modal images under the current working conditions, providing a quantitative basis for the adaptive adjustment of thresholds in subsequent judgment of complex hidden dangers.

[0035] Second, the real-time alignment confidence score is calculated based on the ratio of inlier pairs that satisfy geometric consistency constraints in feature point matching. This metric comprehensively reflects the quality of visible and infrared images in the three stages of feature extraction, feature matching, and geometric transformation estimation. When the inspection distance is long or the speed is fast, leading to a decrease in image quality, the ratio of inlier pairs decreases, and the confidence score decreases accordingly. This automatically relaxes the judgment conditions or issues an early warning in subsequent steps, improving the adaptability of the entire detection process to changes in inspection conditions.

[0036] S300: Based on the modal sensitivity of the component, the normalization factor of the inspection distance, and the normalization factor of the inspection speed, dynamically adjust the fusion weight of visible light features and infrared features to generate a fused feature map; When simultaneously using visible light and infrared images for overhead line hazard detection, the fusion of features from both modes directly impacts the accuracy of subsequent hazard assessment. If a fixed weight is applied to feature fusion for all component types and all inspection conditions, when the inspection distance is too far or the flight speed is too high, causing a decrease in resolution or motion blur in the visible light image, the fused feature map may introduce poor-quality visible light features, weakening the overall discrimination capability. Conversely, when a component exhibits significant hazard features in the visible light mode while infrared information contributes little, if the fusion weights are not tilted towards visible light, the high-discrimination information in the visible light image may be wasted.

[0037] Step S300 in the method provided in this embodiment of the invention includes: Obtain the basic visible light fusion weight and the basic infrared fusion weight, wherein the sum of the basic visible light fusion weight and the basic infrared fusion weight is 1; The product of the basic visible light fusion weights and the modal sensitivity of the component is recorded as the sensitivity adjustment value. Divide the sensitivity adjustment value by the inspection distance normalization factor to obtain the first adjustment value; Divide the first adjustment value by the inspection speed normalization factor to obtain the visible light fusion weight adjustment value; Divide the visible light fusion weight adjustment value by the sum of the visible light fusion weight adjustment value and 1 to obtain the visible light fusion weight; Calculate 1 and subtract the visible light fusion weight to obtain the infrared fusion weight; A visible light feature map is extracted from the visible light image, and an infrared feature map is extracted from the infrared image, wherein the visible light feature map and the infrared feature map have the same feature dimension; Each channel of the visible light feature map extracted from the visible light image is multiplied by the visible light fusion weight to obtain a weighted visible light feature map; Each channel of the infrared feature map extracted from the infrared image is multiplied by the infrared fusion weight to obtain a weighted infrared feature map; The weighted visible light feature map and the weighted infrared feature map are concatenated along the channel dimension to generate a fused feature map.

[0038] The specific implementation method is as follows: First, obtain the basic visible light fusion weight and the basic infrared fusion weight. The basic visible light fusion weight refers to the default weight of visible light characteristics when component characteristics and inspection conditions are not considered, and the basic infrared fusion weight is similarly determined. The sum of the two is 1. For example, the basic visible light fusion weight can be set to 0.5, and the basic infrared fusion weight can be set to 0.5 accordingly.

[0039] Furthermore, based on the component modal sensitivity, inspection distance normalization factor, and inspection speed normalization factor obtained in step S100, the basic visible light fusion weight is dynamically adjusted. The basic visible light fusion weight is multiplied by the component modal sensitivity, and the product is recorded as the sensitivity adjustment value. For example, if the basic visible light fusion weight is 0.5 and the insulator's component modal sensitivity is 1.5, then the sensitivity adjustment value is 0.5 × 1.5 = 0.75.

[0040] Further, the sensitivity adjustment value is divided by the inspection distance normalization factor obtained in step S100 to obtain the first adjustment value. When the inspection distance normalization factor is greater than 1, it indicates that the UAV is currently far from the component, and the spatial resolution of the visible light image decreases. Dividing by this factor reduces the visible light fusion weight. For example, if the inspection distance normalization factor is 2.0, then the first adjustment value is 0.75 ÷ 2.0 = 0.375. The first adjustment value is divided by the inspection speed normalization factor obtained in step S100 to obtain the visible light fusion weight adjustment value. When the inspection speed normalization factor is greater than 1, it indicates that the flight speed is fast, and the visible light image may have motion blur. Dividing by this factor further reduces the visible light fusion weight. For example, if the inspection speed normalization factor is 1.4, then the visible light fusion weight adjustment value is 0.375 ÷ 1.4, which is approximately equal to 0.268.

[0041] Further, the visible light fusion weight adjustment value is divided by the sum of the visible light fusion weight adjustment value and 1 to obtain the visible light fusion weight. This step uses normalization to limit the weight to between 0 and 1. For example, the visible light fusion weight is approximately 0.211, which is 0.268 ÷ (0.268 + 1). The infrared fusion weight is obtained by subtracting the visible light fusion weight from 1. The infrared fusion weight is 1 - 0.211 = 0.789. The sum of the two weights is always 1.

[0042] Component modal sensitivity reflects the component's static properties, while the inspection distance normalization factor and inspection speed normalization factor reflect real-time dynamic conditions. When the inspection distance is long or the flight speed is high, the visible light image quality deteriorates, and the visible light fusion weight is gradually reduced, while the infrared fusion weight increases accordingly. When the component modal sensitivity is greater than 1, the visible light is positively modulated. Through serial calculation, the fusion weight comprehensively reflects the combined influence of these three factors on the image quality of the two modalities.

[0043] Finally, visible light feature maps are extracted from the visible light image, and infrared feature maps are extracted from the infrared image. Both sets of feature maps have the same feature dimension. In this embodiment, feature map extraction is achieved through the feature extraction branch of a pre-trained multi-task analysis model. The weighted visible light feature map and the weighted infrared feature map are concatenated along the channel dimension to generate a fused feature map. Channel concatenation refers to stacking the two feature maps end-to-end along the channel dimension. The fused feature map simultaneously contains feature information from both the visible light and infrared modes, and the contribution ratio of the two modes is dynamically adjusted through the fusion weights.

[0044] Step S300 in the method provided in this embodiment of the invention further includes: Extracting visible light feature maps from the visible light image and extracting infrared feature maps from the infrared image includes: Collect visible light and infrared sample images of the components to be inspected along the same route from multiple historical inspections, and label each set of sample images with temperature anomaly area mask, visual defect area mask, and the hazard type label corresponding to the sample component to be inspected, to obtain a sample training set. The hazard type label includes composite hazard type and single hazard type. Construct a multi-task analysis model; The total loss function is constructed by combining the deviation between the temperature anomaly prediction probability map and the temperature anomaly area mask, the deviation between the visual defect prediction probability map and the visual defect area mask, and the deviation between the hazard type prediction result and the hazard type label. By minimizing the total loss function, the feature extraction branch, feature fusion branch, and multi-task analysis branch of the multi-task analysis model are jointly trained using the sample training set to obtain the trained multi-task analysis model.

[0045] First, we collected visible light and infrared sample images of the components to be inspected from multiple historical inspections along the same route. Each set of sample images corresponds to the two modal imaging results of the same component, simultaneously acquired by a visible light camera and an infrared thermal imager mounted on a UAV at the same time. Each set of sample images was manually annotated, including the following three items: Temperature anomaly area mask: This is a binary image of the same size as the infrared sample image. Continuous pixel areas in the infrared image where the temperature exceeds the upper limit of the component's normal operating temperature are marked as 1, indicating an abnormal temperature area; other pixel areas are marked as 0, indicating a normal temperature area. Visual defect area mask: This is a binary image of the same size as the visible light sample image. Continuous pixel areas in the visible light image with physical surface defects such as broken strands, corrosion, cracks, and damage are marked as 1, indicating a visual defect area; other pixel areas are marked as 0, indicating a normal area. Hazard type label: This label indicates the hazard category corresponding to the sample. Hazard type labels include compound hazard types and single hazard types. Composite hazard types refer to hazard categories where the component exhibits both temperature anomalies and visual defects; single hazard types refer to hazard categories where the component only exhibits temperature anomalies or only exhibits visual defects. Multiple sets of labeled sample images are used as the training set.

[0046] Furthermore, a multi-task analysis model is constructed. This model consists of three parts: a feature extraction branch, a feature fusion branch, and a multi-task analysis branch. It can extract the original visible light feature map and the original infrared feature map from the input visible light sample image and infrared sample image, respectively. It fuses the two sets of feature maps according to the fusion weight and simultaneously outputs the temperature anomaly prediction probability map, the visual defect prediction probability map, and the hazard type prediction result.

[0047] Furthermore, a total loss function is constructed by combining the deviation between the temperature anomaly prediction probability map and the temperature anomaly region mask, the deviation between the visual defect prediction probability map and the visual defect region mask, and the deviation between the hazard type prediction result and the hazard type label.

[0048] The temperature anomaly prediction bias is calculated using pixel-wise binary cross-entropy loss. For each pixel in the temperature anomaly prediction probability map, the predicted probability value at that location is compared with the true label value at the corresponding location in the temperature anomaly region mask. When the true label is 1, the closer the predicted probability is to 1, the smaller the loss; the closer it is to 0, the larger the loss. Conversely, when the true label is 0, the closer the predicted probability is to 0, the smaller the loss; the closer it is to 1, the larger the loss. The average loss value for all pixel locations is then used to obtain the temperature anomaly prediction bias. The visual defect prediction bias also uses pixel-wise binary cross-entropy loss, calculated in the same way as the temperature anomaly prediction bias. The hazard type prediction bias is calculated using multi-class cross-entropy loss. The predicted probability for each hazard category is output, with the category corresponding to the true hazard type label taken as the positive class. The negative logarithm of the predicted probability for this positive class is taken as the loss value; the closer the predicted probability for the positive class is to 1, the smaller the loss; the closer it is to 0, the larger the loss. The three loss terms are then weighted and summed according to preset weighting coefficients to obtain the total loss function. The weighting coefficients can be set according to the needs of the actual detection task. For example, the weighting for the loss of temperature anomaly prediction can be set to 0.3, the weighting for the loss of visual defect prediction can be set to 0.3, and the weighting for the loss of hazard type prediction can be set to 0.4.

[0049] Finally, by minimizing the total loss function, the feature extraction branch, feature fusion branch, and multi-task analysis branch of the multi-task analysis model are jointly trained using the sample training set.

[0050] The training process employs the backpropagation algorithm and a gradient descent optimizer. The backpropagation algorithm works as follows: In the forward propagation phase, the input sample sequentially passes through the feature extraction branch, the feature fusion branch, and the multi-task analysis branch, calculating the prediction results of the three output heads layer by layer. In the backpropagation phase, starting from the total loss function, using the chain rule, the gradient of the loss function with respect to the weight parameters of each layer in each branch is calculated layer by layer, beginning from the output of the multi-task analysis branch, until the propagation reaches the input of the feature extraction branch. The gradient indicates the steepest direction for adjusting the parameter to reduce the loss function value.

[0051] During backpropagation, all learnable parameters of the feature extraction branch, feature fusion branch, and multi-task analysis branch—including convolutional kernel weights and biases in convolutional layers, scaling factors and offsets in batch normalization layers, and connection weights and biases in fully connected layers—simultaneously receive gradients and are updated in each backpropagation. This synchronous update mechanism enables the three branches to coordinate optimization under a unified loss function constraint: the feature extraction branch learns to extract feature representations that are discriminative for both temperature anomaly and visual defect discrimination; the feature fusion branch learns to effectively integrate bimodal information given fusion weights; and the multi-task analysis branch learns to accurately predict the three types of outputs from the fused features.

[0052] The gradient descent optimizer updates the parameter values ​​in the reverse direction of the gradient based on the gradients of each branch and layer and the preset learning rate. During training, the sample training set is divided into a training set and a validation set. All learnable parameters of the model are updated in batches on the training set. After each iteration through all training samples, the total loss function is calculated on the validation set. When the total loss function value on the validation set no longer decreases for several consecutive training iterations, training stops, and the model parameters corresponding to the iteration with the smallest loss function value on the validation set are saved, resulting in the trained multi-task analysis model.

[0053] Step S300 in the method provided in this embodiment of the invention further includes: The multi-task analysis model includes a feature extraction branch, a feature fusion branch, and a multi-task analysis branch. The feature extraction branch includes a visible light feature extraction branch and an infrared feature extraction branch, which take the visible light sample image and the infrared sample image as inputs and the original visible light feature map and the original infrared feature map as outputs, respectively. The feature fusion branch takes the original visible light feature map and the original infrared feature map as inputs and the fused feature map as output. The multi-task analysis branch takes the fused feature map as input and outputs a temperature anomaly prediction probability map, a visual defect prediction probability map, and a hazard type prediction result.

[0054] The multi-task analysis model consists of three parts: feature extraction branch, feature fusion branch, and multi-task analysis branch.

[0055] The feature extraction branch includes visible light feature extraction and infrared feature extraction branches, both employing convolutional neural networks with identical structures but independent parameters. Each branch consists of multiple convolutional layers, batch normalization layers, activation function layers, and pooling layers stacked alternately. The convolutional layers perform convolution operations by sliding learnable convolutional kernels across the input image. The kernel weights are multiplied element-wise with the corresponding pixel values ​​in the input image, and the sum is obtained to obtain the output value. The kernels slide with a preset stride to generate feature maps, and multiple kernels generate multi-channel feature maps in parallel. The batch normalization layer standardizes each channel's feature map, making its mean 0 and variance 1, accelerating training convergence. The activation function layer uses a linear rectified function, setting values ​​less than 0 to 0, introducing non-linear expressive power into the network. The pooling layer downsamples the feature maps using max pooling or average pooling, compressing the feature map size while preserving the main features.

[0056] The visible light feature extraction branch takes a visible light sample image as input and outputs a raw visible light feature map; the infrared feature extraction branch takes an infrared sample image as input and outputs a raw infrared feature map. Both sets of raw feature maps have the same number of channels and spatial dimensions.

[0057] The feature fusion branch takes the original visible light feature map and the original infrared feature map as input. This branch receives the visible light fusion weights and infrared fusion weights output from the fusion weight calculation step. It multiplies each channel of the original visible light feature map by the visible light fusion weight to obtain a weighted visible light feature map; and multiplies each channel of the original infrared feature map by the infrared fusion weight to obtain a weighted infrared feature map. Then, the two weighted feature maps are concatenated along the channel dimension to generate the fused feature map.

[0058] The multi-task analysis branch takes the fused feature map as input and includes a shared convolutional layer and three parallel output heads. The shared convolutional layer performs further deep feature extraction on the fused feature map. The three output heads correspond to the three tasks of temperature anomaly prediction, visual defect prediction, and hazard type prediction, respectively.

[0059] The temperature anomaly output head uses a series of convolution and upsampling operations to gradually restore the abstract features output by the shared convolutional layer to the same spatial size as the input infrared image, outputting a temperature anomaly prediction probability map. Each pixel in the map takes a value between 0 and 1, representing the probability of a temperature anomaly at that pixel location. The visual defect output head uses a similar structure to the temperature anomaly output head, outputting a visual defect prediction probability map.

[0060] The hazard type output header aggregates the spatial information of each channel of the fused feature map into a scalar value through global average pooling. All scalar values ​​from all channels form a fixed-length feature vector. Global average pooling works by averaging all pixel values ​​in the feature map of each channel. This feature vector is then mapped to a score for each hazard category via a fully connected layer. The Softmax function then converts the score into a predicted probability distribution for each hazard category, with the sum of all category probabilities equal to 1. The Softmax function works by exponentially operating on each element of the input vector, using the sum of all exponents as the denominator and the exponent of each element as the numerator to obtain the probability value for that category.

[0061] The three branches are connected in a series, with the output of the previous branch serving as the input to the next branch. During training, the parameters of the three branches are updated synchronously under the constraint of the total loss function, enabling the model to simultaneously complete the three tasks of locating temperature anomaly areas, locating visual defect areas, and identifying hazard types.

[0062] The following technical effects were achieved through this step: First, a quantitative calculation relationship was established between the fusion weight and the component modal sensitivity, inspection distance normalization factor, and inspection speed normalization factor. The greater the component modal sensitivity, the larger the base adjustment amount of the visible light fusion weight, causing the fused feature map to retain more visible light details. The larger the inspection distance normalization factor and inspection speed normalization factor, the lower the visible light fusion weight is gradually achieved, while the infrared fusion weight is correspondingly increased. This mechanism enables the fusion strategy to adaptively adjust according to component type and real-time inspection conditions, solving the problem of insufficient adaptability in existing fixed-weight fusion methods.

[0063] Second, through joint training of the multi-task analysis model, the feature extraction branch, feature fusion branch, and multi-task analysis branch are simultaneously optimized under a unified loss function constraint. The feature extraction branch learns to extract feature representations that are discriminative for both temperature anomaly and visual defect discrimination. The multi-task analysis branch learns to simultaneously predict temperature anomaly regions, visual defect regions, and hazard types from the fused features. The gradients of the three tasks mutually promote each other during training, improving the model's comprehensive detection capability for various types of hazards.

[0064] S400: Based on the inspection distance normalization factor, the inspection speed normalization factor, and the real-time alignment confidence, obtain the overlap threshold for hazard judgment; When determining whether a composite hazard exists in an overhead power line, it is necessary to compare the spatial overlap between the temperature anomaly area and the visual defect area with a preset overlap threshold. If a fixed threshold is used, when the registration quality of visible light and infrared images deteriorates due to changes in inspection conditions, positional shifts may occur between the two types of areas due to registration errors. In this case, if a strict fixed threshold is still applied to ensure high spatial overlap, temperature anomalies and visual defects caused by the same component and the same hazard may be judged as spatially non-overlapping due to the shift, resulting in the missed detection of a composite hazard. Conversely, if the threshold is set too loosely, two anomalies that are not spatially related may be misjudged as a composite hazard.

[0065] Step S400 in the method provided in this embodiment of the invention includes: Obtain the preset basic overlap threshold; Calculate 1 and subtract the real-time alignment confidence to obtain the alignment deviation. The mean of the inspection distance normalization factor and the inspection speed normalization factor is calculated and multiplied by the alignment deviation to obtain the threshold relaxation amount. Calculate the basic overlap threshold multiplied by the sum of the threshold relaxation amount to obtain the dynamically adjusted overlap threshold. When the overlap threshold is greater than 1, the dynamically adjusted overlap threshold is corrected to 1.

[0066] The specific implementation method is as follows: First, obtain the preset basic overlap threshold. The basic overlap threshold is the cross-union ratio threshold value used to determine complex hidden dangers under standard inspection conditions. Standard inspection conditions refer to ideal working conditions where the drone collects images at a standard inspection speed and a standard detection distance, and the visible light and infrared images are well registered. The basic overlap threshold can be set based on experience or historical inspection data, for example, set to 0.5.

[0067] Further, based on the real-time alignment confidence obtained in step S200, the alignment deviation is calculated. The alignment deviation is equal to 1 minus the real-time alignment confidence, reflecting the difference between the current registration result and the ideal perfect alignment. The closer the real-time alignment confidence is to 1, the closer the alignment deviation is to 0; the lower the real-time alignment confidence, the larger the alignment deviation. For example, if the real-time alignment confidence is 0.80, then the alignment deviation is 0.20.

[0068] Further, the threshold relaxation amount is calculated. The arithmetic mean of the inspection distance normalization factor and the inspection speed normalization factor obtained in step S100 is calculated, and then multiplied by the alignment deviation. The resulting product is recorded as the threshold relaxation amount. The inspection distance normalization factor and the inspection speed normalization factor reflect the degree of deviation of the current inspection distance and inspection speed from the standard working conditions, respectively. The larger the values ​​of the two, the more unfavorable the current acquisition conditions are for image registration. The mean value comprehensively reflects the influence of the working conditions on the registration quality. For example, if the inspection distance normalization factor is 2.0, the inspection speed normalization factor is 1.4, the mean value is 1.7, and the alignment deviation is 0.20, then the threshold relaxation amount is 1.7 × 0.20 = 0.34.

[0069] Finally, calculate the dynamically adjusted overlap threshold. Multiply the base overlap threshold by 1 and add the threshold relaxation amount. If the result is greater than 1, adjust it to 1. Continuing with the previous example, if the base overlap threshold is 0.5 and the threshold relaxation amount is 0.34, then the dynamically adjusted overlap threshold is 0.5 × (1 + 0.34) = 0.67.

[0070] The adjusted overlap threshold of 0.67 is higher than the base overlap threshold of 0.5, indicating that the current inspection distance is relatively far and the flight speed is relatively fast, resulting in a decrease in registration quality. Therefore, the threshold is appropriately increased to impose stricter requirements on spatial overlap and avoid misjudgments due to registration errors. Conversely, if all normalization factors are close to 1 and the alignment confidence is close to 1, the threshold relaxation is close to zero, the overlap threshold is close to the base value, and a high detection sensitivity is maintained.

[0071] The following technical effects were achieved through this step: A quantitative adjustment relationship was established between the overlap threshold and the inspection distance normalization factor, inspection speed normalization factor, and real-time alignment confidence. When the inspection distance is long, the speed is fast, or the registration quality is poor, the threshold relaxation increases, and the overlap threshold increases accordingly; when the inspection conditions are good, the threshold remains close to the base value. This dynamic adjustment mechanism can optimize the strictness of the judgment according to the actual inspection conditions.

[0072] S500: Based on the fused feature map, detect temperature abnormality areas and visual defect areas, calculate the degree of overlap, and when the degree of overlap is greater than the overlap threshold, determine that there is a composite hidden danger.

[0073] Existing technologies typically analyze visible light and infrared images independently, lacking a quantitative description of the spatial relationship between the two types of abnormal regions, making it difficult to effectively distinguish between complex anomalies caused by the same hidden danger and independent single-modal anomalies.

[0074] Step S500 in the method provided in this embodiment of the invention includes: The currently generated fusion feature map is input into the multi-task analysis model to obtain the temperature anomaly prediction probability map and the visual defect prediction probability map. Regions with a probability greater than the overlap threshold in the temperature anomaly prediction probability map are identified as temperature anomaly regions. Regions with a probability greater than the overlap threshold in the visual defect prediction probability map are identified as visual defect regions. Calculate the intersection-union ratio (IU / UG) of the temperature anomaly region and the visual defect region as the degree of overlap; If the degree of overlap is greater than the dynamically adjusted threshold for the degree of overlap, it is determined that there is a composite hazard, and the composite hazard type with the highest probability in the hazard type prediction results output by the multi-task analysis model is directly obtained as the hazard type.

[0075] The specific implementation method is as follows: First, the fused feature map generated in step S300 is input into the trained multi-task analysis model to obtain a temperature anomaly prediction probability map and a visual defect prediction probability map. These two probability maps have the same spatial dimensions as the input infrared and visible light images. Each pixel has a value between 0 and 1; a higher value indicates a higher probability of the corresponding type of anomaly existing at that pixel location.

[0076] Furthermore, temperature anomaly regions are extracted from the temperature anomaly prediction probability map. Pixels with a predicted probability greater than a preset probability threshold in the probability map are identified as temperature anomaly pixels, and the continuous region formed by all temperature anomaly pixels constitutes the temperature anomaly region. Similarly, visual defect regions are extracted from the visual defect prediction probability map. The preset probability threshold here is a different parameter from the dynamically adjusted overlap threshold in step S400. The probability threshold is used to extract regions from the binary representation of the probability map; for example, it can be set to 0.5.

[0077] Furthermore, the intersection-union ratio (IUGR) of the temperature anomaly region and the visual defect region is calculated as the degree of overlap. The IUGR is the ratio of the intersection area to the union area of ​​the two regions. The intersection area refers to the number of pixels that simultaneously belong to both the temperature anomaly region and the visual defect region, while the union area refers to the total number of all non-repeating pixels belonging to either the temperature anomaly region or the visual defect region.

[0078] Finally, the calculated overlap degree is compared with the dynamically adjusted overlap degree threshold obtained in step S400. If the overlap degree is greater than the overlap degree threshold, a composite hazard is identified. A composite hazard refers to a hazard type in which temperature anomalies and visual defects coexist on the same component, and the two are highly correlated in spatial location. In this case, the hazard type prediction probability distribution output by the hazard type output head of the multi-task analysis model is directly obtained, and the category with the highest probability from the composite hazard categories is selected as the final output hazard type.

[0079] It should be noted that when the overlap threshold dynamically adjusted in step S400 is 1, the cross-union ratio (CUI) must reach 1 to be considered a composite hazard, meaning the temperature anomaly area and the visual defect area must completely overlap. However, in practical applications, when extracting regions from the probability map using a method with a probability greater than the threshold, there are no pixels with a probability greater than 1. If the above judgment condition is still used, no composite hazards will be detected. To avoid this situation, when the dynamically adjusted overlap threshold is 1, the overlap is directly set to 0, and it is not considered a composite hazard. This processing method, combined with the upper limit truncation of the CUI threshold, ensures that the rationality of the judgment process is not affected by overly strict thresholds under extreme working conditions.

[0080] Step S500 in the method provided in this embodiment of the invention further includes: When the degree of overlap is not greater than the dynamically adjusted overlap threshold, including: Obtain the hazard type prediction results output by the multi-task analysis model, and take the hazard type prediction results with the highest confidence among the hazard types other than the composite hazard type as the single hazard type; When the confidence level of a single hazard type is greater than a preset confidence level threshold for a single hazard, it is determined that there is a single-modality dominant hazard corresponding to the single hazard type.

[0081] When the degree of overlap is not greater than the dynamically adjusted overlap threshold, it indicates that the temperature anomaly area and the visual defect area have not reached the degree of overlap required for a composite hazard in space. There are two possibilities: one is that the component has a hazard in only a single mode, and the other is that although the two anomalies exist at the same time, they are caused by different reasons and are not related in spatial location.

[0082] First, obtain the predicted probability distribution of hazard types output from the hazard type output header of the multi-task analysis model. After excluding all composite hazard categories from the probability distribution, select the category with the highest predicted probability from the remaining single-type hazard categories as the single hazard type. The predicted probability is the confidence level of this hazard type.

[0083] If two or more single hazard types have the same and maximum confidence level, then all of these single hazard types with equal confidence levels will be output as a single hazard type. This avoids missed detections caused by the multi-task analysis model predicting probabilities evenly distributed across multiple categories, ensuring that all possibilities are recorded and reported when the multi-task analysis model has similar confidence levels for multiple single hazard types, for further verification by operations and maintenance personnel.

[0084] Furthermore, the confidence level of a single hazard type is compared with a preset single hazard confidence threshold. The single hazard confidence threshold is a minimum confidence requirement set to prevent false alarms due to low confidence levels; for example, it can be set to 0.6. When the confidence level is greater than this threshold, a single-modal dominant hazard corresponding to that single hazard type is determined to exist and output; when the confidence level is not greater than this threshold, the current detection result is considered to have insufficient confidence, and no hazard determination result is output.

[0085] Finally, by introducing a single hazard confidence threshold, when the overlap does not meet the criteria for a composite hazard, the system does not simply determine that there is no hazard. Instead, it further assesses the probability of a single-modal hazard, forming a hierarchical judgment mechanism that first judges composite hazards and then single-modal hazards. The confidence threshold filters out unreliable results with low model prediction probabilities, reducing the false alarm rate. The parallel output mechanism when multiple confidence levels are equal further reduces the risk of missed alarms caused by the uncertainty in multi-task analysis model predictions.

[0086] The following technical effects were achieved through this step: First, a quantitative calculation method for the spatial overlap between temperature anomaly areas and visual defect areas was established. By calculating the intersection-union ratio of the two areas, the spatial correlation degree is transformed into a comparable numerical index, which is then quantitatively compared with a dynamically adjusted overlap threshold, thus achieving standardization and automation of the judgment of complex hazards. Special handling for the extreme case where the overlap threshold is 1 ensures the rationality and stability of the judgment process under various operating conditions.

[0087] Second, a hierarchical judgment mechanism was constructed. When the overlap exceeds a threshold, it is judged as a composite hazard, and the corresponding hazard type is directly output. When the overlap does not exceed the threshold, the probability of a single-modal hazard is further evaluated, and unreliable results are filtered out using a confidence threshold. This mechanism can effectively distinguish between composite hazards and single-modal dominant hazards, reducing the false negative rate of composite hazards and the false positive rate of single-modal hazards. The mechanism of parallel output when multiple single hazard types have equal confidence levels further reduces the risk of false negatives caused by model prediction uncertainty.

[0088] Example 2

[0089] like Figure 2 As shown, based on the same inventive concept as the online detection method for overhead line hazards by drone inspection provided in Embodiment 1, this embodiment of the invention also provides an online detection system for overhead line hazards by drone inspection. The system includes an inspection parameter acquisition module, an image registration module, a feature fusion module, a threshold acquisition module, and a hazard determination module. The system includes: The inspection parameter acquisition module 11 is used to acquire the modal sensitivity and standard detection distance of the component to be inspected, and to acquire the measured distance and inspection speed between the UAV and the component to be inspected, and to calculate and acquire the inspection distance normalization factor and the inspection speed normalization factor. Image registration module 12 is used to register the visible light image and infrared image of the component to be detected and calculate the real-time alignment confidence. The feature fusion module 13 is used to dynamically adjust the fusion weights of visible light features and infrared features based on the modal sensitivity of the component, the normalization factor of the inspection distance, and the normalization factor of the inspection speed, and generate a fused feature map. The threshold acquisition module 14 is used to acquire the overlap threshold of the hidden danger judgment based on the inspection distance normalization factor, the inspection speed normalization factor and the real-time alignment confidence. The hazard determination module 15 is used to detect temperature abnormality areas and visual defect areas based on the fused feature map, calculate the degree of overlap, and determine that there is a composite hazard when the degree of overlap is greater than the degree of overlap threshold.

[0090] In one embodiment, the inspection parameter acquisition module 11 is further configured to identify the component type of the component to be inspected, acquire the component modal sensitivity and standard detection distance corresponding to the component type, acquire the measured distance between the UAV and the component to be inspected, calculate the inspection distance normalization factor, and acquire the inspection speed normalization factor based on the inspection speed of the UAV, including: Identify the component to be detected from the visible light image to obtain the component type; From the pre-established component type and attribute mapping table, query and obtain the component modal sensitivity and standard detection distance corresponding to the component type, wherein the component modal sensitivity is the ratio of the distinguishability of the hidden danger features of the component type in the visible light mode to the distinguishability of the hidden danger features in the infrared mode; Obtain the measured distance between the drone and the component to be inspected, divide the measured distance by the standard detection distance, and obtain the inspection distance normalization factor; The inspection speed of the UAV within the time window during which it identifies the component to be inspected is obtained. The inspection speed is then divided by the standard inspection speed to obtain the inspection speed normalization factor. The inspection speed is obtained based on the average inspection speed during the identification of the component to be inspected.

[0091] In one embodiment, the image registration module 12 is further configured to register the visible light image and the infrared image of the component to be detected, and calculate the real-time alignment confidence, including: Feature points are extracted from the visible light image and the infrared image respectively, and the two sets of feature points are matched to obtain a set of matching point pairs; Count the number of internal point pairs in the set of matching point pairs that satisfy the preset geometric consistency constraint, and obtain the total number of matching point pairs in the set of matching point pairs; Divide the number of internal point pairs by the total number of matched point pairs to obtain the real-time alignment confidence.

[0092] In one embodiment, the feature fusion module 13 is further configured to dynamically adjust the fusion weights of visible light features and infrared features based on the component modal sensitivity, the inspection distance normalization factor, and the inspection speed normalization factor, to generate a fused feature map, including: Obtain the basic visible light fusion weight and the basic infrared fusion weight, wherein the sum of the basic visible light fusion weight and the basic infrared fusion weight is 1; The product of the basic visible light fusion weights and the modal sensitivity of the component is recorded as the sensitivity adjustment value. Divide the sensitivity adjustment value by the inspection distance normalization factor to obtain the first adjustment value; Divide the first adjustment value by the inspection speed normalization factor to obtain the visible light fusion weight adjustment value; Divide the visible light fusion weight adjustment value by the sum of the visible light fusion weight adjustment value and 1 to obtain the visible light fusion weight; Calculate 1 and subtract the visible light fusion weight to obtain the infrared fusion weight; A visible light feature map is extracted from the visible light image, and an infrared feature map is extracted from the infrared image, wherein the visible light feature map and the infrared feature map have the same feature dimension; Each channel of the visible light feature map extracted from the visible light image is multiplied by the visible light fusion weight to obtain a weighted visible light feature map; Each channel of the infrared feature map extracted from the infrared image is multiplied by the infrared fusion weight to obtain a weighted infrared feature map; The weighted visible light feature map and the weighted infrared feature map are concatenated along the channel dimension to generate a fused feature map.

[0093] Extracting visible light feature maps from the visible light image and extracting infrared feature maps from the infrared image includes: Collect visible light and infrared sample images of the components to be inspected along the same route from multiple historical inspections, and label each set of sample images with temperature anomaly area mask, visual defect area mask, and the hazard type label corresponding to the sample component to be inspected, to obtain a sample training set. The hazard type label includes composite hazard type and single hazard type. Construct a multi-task analysis model; The total loss function is constructed by combining the deviation between the temperature anomaly prediction probability map and the temperature anomaly area mask, the deviation between the visual defect prediction probability map and the visual defect area mask, and the deviation between the hazard type prediction result and the hazard type label. By minimizing the total loss function, the feature extraction branch, feature fusion branch, and multi-task analysis branch of the multi-task analysis model are jointly trained using the sample training set to obtain the trained multi-task analysis model.

[0094] The multi-task analysis model includes a feature extraction branch, a feature fusion branch, and a multi-task analysis branch. The feature extraction branch includes a visible light feature extraction branch and an infrared feature extraction branch, which take the visible light sample image and the infrared sample image as inputs and the original visible light feature map and the original infrared feature map as outputs, respectively. The feature fusion branch takes the original visible light feature map and the original infrared feature map as inputs and the fused feature map as output. The multi-task analysis branch takes the fused feature map as input and outputs a temperature anomaly prediction probability map, a visual defect prediction probability map, and a hazard type prediction result.

[0095] In one embodiment, the threshold acquisition module 14 is further configured to acquire an overlap threshold for hazard determination based on the inspection distance normalization factor, the inspection speed normalization factor, and the real-time alignment confidence level, including: Obtain the preset basic overlap threshold; Calculate 1 and subtract the real-time alignment confidence to obtain the alignment deviation. The mean of the inspection distance normalization factor and the inspection speed normalization factor is calculated and multiplied by the alignment deviation to obtain the threshold relaxation amount. Calculate the basic overlap threshold multiplied by the sum of the threshold relaxation amount to obtain the dynamically adjusted overlap threshold. When the overlap threshold is greater than 1, the dynamically adjusted overlap threshold is corrected to 1.

[0096] In one embodiment, the hazard determination module 15 is further configured to detect temperature anomaly regions and visual defect regions based on the fused feature map, calculate the degree of overlap, and determine the existence of a composite hazard when the degree of overlap is greater than the overlap threshold, including: The currently generated fusion feature map is input into the multi-task analysis model to obtain the temperature anomaly prediction probability map and the visual defect prediction probability map. Regions with a probability greater than the overlap threshold in the temperature anomaly prediction probability map are identified as temperature anomaly regions. Regions with a probability greater than the overlap threshold in the visual defect prediction probability map are identified as visual defect regions. Calculate the intersection-union ratio (IU / UG) of the temperature anomaly region and the visual defect region as the degree of overlap; If the degree of overlap is greater than the dynamically adjusted threshold for the degree of overlap, it is determined that there is a composite hazard, and the composite hazard type with the highest probability in the hazard type prediction results output by the multi-task analysis model is directly obtained as the hazard type.

[0097] When the degree of overlap is not greater than the dynamically adjusted overlap threshold, the method further includes: Obtain the hazard type prediction results output by the multi-task analysis model, and take the hazard type prediction results with the highest confidence among the hazard types other than the composite hazard type as the single hazard type; When the confidence level of a single hazard type is greater than a preset confidence level threshold for a single hazard, it is determined that there is a single-modality dominant hazard corresponding to the single hazard type.

[0098] It should be noted that the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some implementations, multitasking and parallel processing are possible or may be advantageous.

[0099] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0100] This specification and accompanying drawings are merely illustrative examples of the invention and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its scope. Therefore, if such modifications and modifications fall within the scope of the invention and its equivalents, the invention is intended to include these modifications and modifications.

Claims

1. A method for online detection of potential hazards in overhead power lines using unmanned aerial vehicle (UAV) inspection, characterized in that, include: The modal sensitivity and standard detection distance of the component to be tested are obtained, and the measured distance and inspection speed between the UAV and the component to be tested are obtained. The inspection distance normalization factor and the inspection speed normalization factor are calculated and obtained. The visible light image and infrared image of the component to be detected are registered, and the real-time alignment confidence is calculated. Based on the component modal sensitivity, the inspection distance normalization factor, and the inspection speed normalization factor, the fusion weights of visible light features and infrared features are dynamically adjusted to generate a fused feature map. Based on the inspection distance normalization factor, the inspection speed normalization factor, and the real-time alignment confidence, the overlap threshold for hazard judgment is obtained. Based on the fused feature map, temperature abnormality areas and visual defect areas are detected, and the degree of overlap is calculated. When the degree of overlap is greater than the overlap threshold, it is determined that there is a composite hidden danger.

2. The method for online detection of potential hazards in overhead power lines by unmanned aerial vehicle (UAV) inspection according to claim 1, characterized in that, The modal sensitivity and standard detection distance of the component to be inspected are obtained, and the measured distance and inspection speed between the UAV and the component to be inspected are obtained. The inspection distance normalization factor and the inspection speed normalization factor are calculated and obtained, including: Identify the component to be detected from the visible light image to obtain the component type; From the pre-established component type and attribute mapping table, query and obtain the component modal sensitivity and standard detection distance corresponding to the component type, wherein the component modal sensitivity is the ratio of the distinguishability of the hidden danger features of the component type in the visible light mode to the distinguishability of the hidden danger features in the infrared mode; Obtain the measured distance between the drone and the component to be inspected, divide the measured distance by the standard detection distance, and obtain the inspection distance normalization factor; The inspection speed of the UAV within the time window during which it identifies the component to be inspected is obtained. The inspection speed is then divided by the standard inspection speed to obtain the inspection speed normalization factor. The inspection speed is obtained based on the average inspection speed during the identification of the component to be inspected.

3. The method for online detection of potential hazards in overhead power lines by unmanned aerial vehicle (UAV) inspection according to claim 1, characterized in that, The registration of the visible light image and infrared image of the component to be detected, and the calculation of the real-time alignment confidence, include: Feature points are extracted from the visible light image and the infrared image respectively, and the two sets of feature points are matched to obtain a set of matching point pairs; Count the number of internal point pairs in the set of matching point pairs that satisfy the preset geometric consistency constraint, and obtain the total number of matching point pairs in the set of matching point pairs; Divide the number of internal point pairs by the total number of matched point pairs to obtain the real-time alignment confidence.

4. The method for online detection of potential hazards in overhead power lines by unmanned aerial vehicle (UAV) inspection according to claim 1, characterized in that, The process of dynamically adjusting the fusion weights of visible light and infrared features based on the component modal sensitivity, the inspection distance normalization factor, and the inspection speed normalization factor to generate a fused feature map includes: Obtain the basic visible light fusion weight and the basic infrared fusion weight, wherein the sum of the basic visible light fusion weight and the basic infrared fusion weight is 1; The product of the basic visible light fusion weights and the modal sensitivity of the component is recorded as the sensitivity adjustment value. Divide the sensitivity adjustment value by the inspection distance normalization factor to obtain the first adjustment value; Divide the first adjustment value by the inspection speed normalization factor to obtain the visible light fusion weight adjustment value; Divide the visible light fusion weight adjustment value by the sum of the visible light fusion weight adjustment value and 1 to obtain the visible light fusion weight; Calculate 1 and subtract the visible light fusion weight to obtain the infrared fusion weight; A visible light feature map is extracted from the visible light image, and an infrared feature map is extracted from the infrared image, wherein the visible light feature map and the infrared feature map have the same feature dimension; Each channel of the visible light feature map extracted from the visible light image is multiplied by the visible light fusion weight to obtain a weighted visible light feature map; Each channel of the infrared feature map extracted from the infrared image is multiplied by the infrared fusion weight to obtain a weighted infrared feature map; The weighted visible light feature map and the weighted infrared feature map are concatenated along the channel dimension to generate a fused feature map.

5. The method for online detection of potential hazards in overhead power lines by unmanned aerial vehicle (UAV) inspection according to claim 4, characterized in that, Extracting visible light feature maps from the visible light image and extracting infrared feature maps from the infrared image includes: Collect visible light and infrared sample images of the components to be inspected along the same route from multiple historical inspections, and label each set of sample images with temperature anomaly area mask, visual defect area mask, and the hazard type label corresponding to the sample component to be inspected, to obtain a sample training set. The hazard type label includes composite hazard type and single hazard type. Construct a multi-task analysis model; The total loss function is constructed by combining the deviation between the temperature anomaly prediction probability map and the temperature anomaly area mask, the deviation between the visual defect prediction probability map and the visual defect area mask, and the deviation between the hazard type prediction result and the hazard type label. By minimizing the total loss function, the feature extraction branch, feature fusion branch, and multi-task analysis branch of the multi-task analysis model are jointly trained using the sample training set to obtain the trained multi-task analysis model.

6. The method for online detection of potential hazards in overhead power lines by unmanned aerial vehicle (UAV) inspection according to claim 5, characterized in that, The multi-task analysis model includes a feature extraction branch, a feature fusion branch, and a multi-task analysis branch. The feature extraction branch includes a visible light feature extraction branch and an infrared feature extraction branch, which take the visible light sample image and the infrared sample image as inputs and the original visible light feature map and the original infrared feature map as outputs, respectively. The feature fusion branch takes the original visible light feature map and the original infrared feature map as inputs and the fused feature map as output. The multi-task analysis branch takes the fused feature map as input and outputs a temperature anomaly prediction probability map, a visual defect prediction probability map, and a hazard type prediction result.

7. The method for online detection of potential hazards in overhead power lines by unmanned aerial vehicle (UAV) inspection according to claim 1, characterized in that, The method of obtaining the overlap threshold for hazard judgment based on the inspection distance normalization factor, the inspection speed normalization factor, and the real-time alignment confidence level includes: Obtain the preset basic overlap threshold; Calculate 1 and subtract the real-time alignment confidence to obtain the alignment deviation. The mean of the inspection distance normalization factor and the inspection speed normalization factor is calculated and multiplied by the alignment deviation to obtain the threshold relaxation amount. Calculate the basic overlap threshold multiplied by the sum of the threshold relaxation amount to obtain the dynamically adjusted overlap threshold. When the overlap threshold is greater than 1, the dynamically adjusted overlap threshold is corrected to 1.

8. The method for online detection of potential hazards in overhead power lines by unmanned aerial vehicle (UAV) inspection according to claim 5, characterized in that, The method involves detecting temperature anomaly regions and visual defect regions based on the fused feature map, calculating the degree of overlap, and determining the existence of a complex hidden danger when the degree of overlap exceeds a threshold. This includes: The currently generated fusion feature map is input into the multi-task analysis model to obtain the temperature anomaly prediction probability map and the visual defect prediction probability map. Regions with a probability greater than the overlap threshold in the temperature anomaly prediction probability map are identified as temperature anomaly regions. Regions with a probability greater than the overlap threshold in the visual defect prediction probability map are identified as visual defect regions. Calculate the intersection-union ratio (IU / UG) of the temperature anomaly region and the visual defect region as the degree of overlap; If the degree of overlap is greater than the dynamically adjusted threshold for the degree of overlap, it is determined that there is a composite hazard, and the composite hazard type with the highest probability in the hazard type prediction results output by the multi-task analysis model is directly obtained as the hazard type.

9. A method for online detection of potential hazards in overhead power lines by unmanned aerial vehicle (UAV) inspection according to claim 8, characterized in that, When the degree of overlap is not greater than the dynamically adjusted overlap threshold, the method further includes: Obtain the hazard type prediction results output by the multi-task analysis model, and take the hazard type prediction results with the highest confidence among the hazard types other than the composite hazard type as the single hazard type; When the confidence level of a single hazard type is greater than a preset confidence level threshold for a single hazard, it is determined that there is a single-modality dominant hazard corresponding to the single hazard type.

10. An online detection system for potential hazards in overhead power lines inspected by unmanned aerial vehicles (UAVs), characterized in that, The system is used to implement the online detection method for potential hazards of overhead power lines by unmanned aerial vehicle (UAV) inspection as described in any one of claims 1-9, the system comprising: The inspection parameter acquisition module is used to acquire the modal sensitivity and standard detection distance of the component to be inspected, and to acquire the measured distance and inspection speed between the UAV and the component to be inspected, and to calculate and acquire the inspection distance normalization factor and the inspection speed normalization factor. The image registration module is used to register the visible light image and the infrared image of the component to be detected, and to calculate the real-time alignment confidence. The feature fusion module is used to dynamically adjust the fusion weights of visible light features and infrared features based on the modal sensitivity of the component, the normalization factor of the inspection distance, and the normalization factor of the inspection speed, and to generate a fused feature map. The threshold acquisition module is used to acquire the overlap threshold of the hidden danger judgment based on the inspection distance normalization factor, the inspection speed normalization factor and the real-time alignment confidence. The hazard determination module is used to detect temperature abnormality areas and visual defect areas based on the fused feature map, calculate the degree of overlap, and determine that there is a composite hazard when the degree of overlap is greater than the overlap threshold.

Images