Target identification and accurate photographing method and system for power transmission tower inspection

Abstract

The present application relates to the technical field of power inspection, in particular to a target identification and accurate photographing method and system for transmission tower inspection. It comprises: obtaining a camera preview frame and extracting multi-scale fusion features, extracting context features through longitudinal and transverse strip pooling, generating tower body main structure heat map, cross arm structure heat map and wire linear region heat map, performing gate re-weighting according to the structure heat map to obtain structure guided features, performing target detection to output detection frames and confidence of six types of targets, correcting the confidence by sampling the response value of the center point of the detection frame on the corresponding structure heat map, selecting key shooting targets according to the corrected confidence and generating a pan-tilt attitude adjustment instruction and a zoom parameter, and triggering photographing when continuous multiple frames meet the centering, stability and clarity conditions. The present application realizes the closed-loop control of airborne real-time detection and accurate photographing, and improves the quality and efficiency of the inspection photos.

Description

Technical Field

[0001] This invention relates to the field of power line inspection technology, specifically to a method and system for target identification and precise photography during transmission tower inspection. Background Technology

[0002] With the development of drone technology, drone inspection has been widely used for the inspection of power transmission lines, towers, and power equipment. Existing technologies use cameras for image acquisition and deep learning for target recognition, laying the foundation for the automation and intelligence of drone inspection.

[0003] However, existing technologies have significant shortcomings. General detection models lack explicit modeling of the longitudinal and transverse structures of towers and the linear features of conductors, leading to significant false positives and false negatives in complex environments. Under limited airborne computing power, it is difficult to balance real-time performance and accuracy, and lightweight models exhibit unstable accuracy. The lack of structural consistency constraints makes it difficult to directly drive closed-loop control with detection results. The inference phase relies on complex rules, making parameter tuning difficult and maintenance costs high. The separation of target recognition and image capture processes prevents dynamic adjustments to composition and focus based on recognition results, resulting in target deviation, out-of-focus, or insufficient clarity at high zoom levels, severely impacting the quality of inspection photos and the efficiency of subsequent defect detection. Summary of the Invention

[0004] This invention provides a method and system for target identification and accurate photography during transmission tower inspection, aiming to solve the problems of unstable target detection, insufficient airborne real-time performance, and poor imaging quality caused by the disconnect between identification and photography control in existing transmission tower inspections.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] This invention relates to a target identification and precise photographing method for power transmission tower inspection, comprising:

[0007] S100: Acquire camera preview frames during the drone inspection process and extract multi-scale fusion features;

[0008] S200: Perform vertical strip pooling and horizontal strip pooling on the fused features respectively, extract the vertical context features and horizontal context features and fuse them with the original fused features, generate multiple structural heat maps including the main structure heat map of the tower, the crossarm structure heat map and the conductor linear region heat map based on the fused features, and perform gating reweighting on the fused features according to the multiple structural heat maps to obtain structural guidance features.

[0009] S300: Target detection is performed based on the structural guidance features, and the output includes the detection frame, category and confidence level of the entire tower, crossarm, insulator string, hardware connection area, clamp and conductor;

[0010] S400: The response value of the center point of the sampling detection box on the structural heatmap corresponding to its category. The confidence level is weighted and calculated with the response value to obtain the structurally corrected confidence level.

[0011] S500: Selects key shooting targets from the detection results based on the confidence level after structural correction, generates gimbal attitude adjustment instructions based on the offset between the center of the target detection box and the center of the image, and adjusts the camera zoom parameters based on the ratio of the area of ​​the target detection box to the area of ​​the image.

[0012] S600: When multiple consecutive frames simultaneously meet the conditions of target centering, detection stability, and image sharpness, control the camera to capture high-resolution photos.

[0013] As a preferred embodiment of the present invention, in S200, the steps of longitudinal strip pooling and transverse strip pooling include:

[0014] The fused features are segmented into multiple vertical strips along the vertical axis of the image, and the features within each vertical strip are pooled to obtain the vertical context features;

[0015] The fused features are segmented into multiple horizontal strips along the horizontal direction of the image, and the features within each horizontal strip are pooled to obtain the horizontal context features.

[0016] As a preferred embodiment of the present invention, in S200, the gated reweighting step includes:

[0017] A gated weight map is generated based on the heat map of the main tower structure, the heat map of the crossarm structure, and the heat map of the linear region of the conductor.

[0018] The structure-guided features are obtained by multiplying the gating weight map element-wise with the fused features.

[0019] As a preferred embodiment of the present invention, in S400, the structural heatmap corresponding to the category is as follows:

[0020] Crossarm category and corresponding crossarm structure heatmap;

[0021] Heatmap of the linear region corresponding to the conductor type;

[0022] The categories of insulator strings, hardware connection areas, and clamps correspond to the heat maps of the main tower structure, crossarm structure, or conductor linear areas.

[0023] The overall category of the tower corresponds to the heat map of the main structure of the tower.

[0024] As a preferred embodiment of the present invention, in S400, the weighted calculation step is as follows:

[0025] The correction factor is obtained by multiplying the response value by the preset weighting coefficient and then adding 1.

[0026] Multiply the confidence level by the correction factor to obtain the confidence level after structural correction.

[0027] As a preferred embodiment of the present invention, in S500, the step of selecting the key shooting target includes:

[0028] Based on the priority order of insulator strings, clamps, hardware connection areas, crossarms, towers as a whole, and conductors, the target with the highest confidence level after structural correction is selected as the key shooting target from the test results.

[0029] As a preferred embodiment of the present invention, in S600:

[0030] The target centering condition is that the offset between the center of the target detection box and the center of the image is less than a preset centering threshold.

[0031] The detection stability condition is that the change in the center position and the change in the area of ​​the target detection box are both less than a preset stability threshold in multiple consecutive frames.

[0032] The image sharpness condition is that the image gradient magnitude of the target area is greater than a preset sharpness threshold.

[0033] As a preferred embodiment of the present invention, the method for generating the supervision signal of the structural heatmap during training of the model for target detection in S300 is as follows:

[0034] The monitoring signal for the main structure heat map of the tower is generated based on the center line of the marked tower overall detection frame;

[0035] The monitoring signal for generating a heat map of the crossarm structure is generated based on the long side direction of the marked crossarm detection frame;

[0036] A supervisory signal is generated from the marked wire detection frame area to create a heat map of the wire linear region.

[0037] As a preferred embodiment of the present invention, the model for object detection in S300 is trained using structural consistency training constraints, wherein the structural consistency training constraints include:

[0038] Category-Structure Matching Loss: When the response value of the predicted detection box center point on the corresponding structure heatmap is lower than the first preset threshold, the loss is calculated based on the difference between the threshold and the response value.

[0039] Hierarchical feasible domain loss: When the center point of the predicted detection frame for the crossarm category, insulator string category, hardware connection area category, or clamp category exceeds the extended area of ​​the overall predicted detection frame of the tower, the loss is calculated based on the distance between the center point and the boundary of the area.

[0040] Component topology relationship loss: The loss is calculated based on the distance between the predicted detection frames of the preset key component pairs, which is normalized by the area of ​​the overall detection frame of the tower. When the normalized distance exceeds the second preset threshold, the loss is calculated.

[0041] The preset key component pair includes at least one of the following: the connection area between the insulator string and the fitting, the clamp and the conductor, and the insulator string and the crossarm.

[0042] This invention also proposes a target recognition and precise photography system for power transmission tower inspection, comprising:

[0043] The image acquisition module is used to acquire camera preview frames during the drone inspection process;

[0044] The feature extraction module is used to extract multi-scale fusion features from the preview frame;

[0045] The tower structure prior module is used to perform vertical strip pooling and horizontal strip pooling on the fused features respectively, extract the vertical context features and horizontal context features and fuse them with the original fused features, generate multiple types of structural heat maps based on the fused features, including the tower main structure heat map, the crossarm structure heat map and the conductor linear region heat map, and perform gated reweighting on the fused features according to the multiple structural heat maps to obtain structural guidance features;

[0046] The target detection module is used to perform target detection based on the structural guidance features, and outputs the detection frame, category and confidence level of the entire tower, crossarm, insulator string, hardware connection area, clamp and conductor;

[0047] The confidence correction module is used to sample the response value of the center point of the detection box on the structural heatmap corresponding to its category, and to calculate the confidence value by weighting the confidence value with the response value to obtain the structure-corrected confidence value.

[0048] The image capture control module is used to select key shooting targets from the detection results based on the confidence level after structural correction, generate gimbal attitude adjustment commands based on the offset between the center of the target detection box and the center of the image, adjust the camera zoom parameters based on the ratio of the area of ​​the target detection box to the area of ​​the image, and control the camera to capture high-resolution photos when the target centering condition, detection stability condition, and image sharpness condition are met simultaneously in multiple consecutive frames.

[0049] The beneficial effects of this invention are:

[0050] 1. This invention extracts the longitudinal and lateral contextual features of towers through directional strip pooling, generates a structural heatmap, and performs gated reweighting. It explicitly incorporates the tower structure priors into the feature representation and combines structural consistency training constraints to ensure that the detection results conform to the spatial regularity of tower components. This improves detection stability and reduces false positive rates under complex backgrounds and multi-scale target conditions. During the inference phase, only a minimalist structure is used to guide reweighting, avoiding the stacking of complex rules, reducing latency, and minimizing the parameter tuning costs of cross-scenario deployment.

[0051] 2. This invention establishes a target recognition-driven, precise image capture closed-loop control. Based on the confidence level after structural correction, it selects the target for shooting and generates gimbal attitude adjustment commands and zoom parameters in real time. Image capture is triggered when the detection is stable. The combination of structural consistency constraints and closed-loop control ensures stable and reliable detection output, improves target centering and clarity, reduces out-of-focus shots, and significantly enhances the usability of inspection photos and inspection efficiency. Attached Figure Description

[0052] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0053] Figure 1 This is a flowchart illustrating the target identification and precise photography method for power transmission tower inspection according to the present invention.

[0054] Figure 2 This is a schematic diagram of the thermal output of the structure of this invention;

[0055] Figure 3 This is a schematic diagram of the prior lightweight target detection model of the present invention;

[0056] Figure 4 This is a flowchart of the closed-loop process of the recognition-driven precise image capture of the present invention;

[0057] Figure 5 This is a schematic diagram of the structural consistency constraints of the present invention;

[0058] Figure 6 This is a schematic diagram of the target identification and precise photography system for power transmission tower inspection according to the present invention;

[0059] Figure 7 This is a schematic diagram of the application scenario and system boundary of the present invention. Detailed Implementation

[0060] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0061] Example 1: As Figure 1As shown, the present invention provides a target identification and precise photographing method for power transmission tower inspection, comprising:

[0062] S100: Acquire camera preview frames during the drone inspection process and extract multi-scale fusion features;

[0063] Specifically, during the drone's inspection flight, the gimbal camera continuously outputs image frames in real-time preview mode, and the current frame is obtained from the preview frame stream as input. The obtained camera preview frame is adjusted to a fixed resolution, which can be 640×640, 960×960, or 1280×720, to balance real-time performance and small target detection capability, and the pixel values ​​are normalized to the [0,1] range.

[0064] A lightweight feature backbone network is employed to extract multi-scale feature maps from preprocessed images. The backbone network is constructed using depthwise separable convolutions, outputting three feature layers P3, P4, and P5 at different scales. P3 is a high-resolution feature layer used to detect small-scale targets such as insulator strings, hardware connection areas, and wire clamps; P4 is a medium-resolution feature layer used to detect medium-scale targets; and P5 is a low-resolution feature layer used to detect large-scale targets such as the entire tower. The multi-scale features P3, P4, and P5 are then input into a multi-scale fusion network for feature fusion.

[0065] The fusion network employs a bidirectional fusion approach, combining top-down and bottom-up methods. The top-down path upsamples high-level semantic information to lower levels, while the bottom-up path downsamples low-level detail information to higher levels. The fusion process utilizes additive fusion or channel attention-weighted fusion to maintain low latency. After fusion, the output consists of fusion features F3, F4, and F5 corresponding to P3, P4, and P5, respectively. These fusion features contain both semantic and detail information, providing a foundation for subsequent processing.

[0066] S200: Perform vertical strip pooling and horizontal strip pooling on the fused features respectively, extract the vertical context features and horizontal context features and fuse them with the original fused features, generate multiple structural heat maps including the main structure heat map of the tower, the crossarm structure heat map and the conductor linear region heat map based on the fused features, and perform gating reweighting on the fused features according to the multiple structural heat maps to obtain structural guidance features.

[0067] Furthermore, the steps of vertical strip pooling and horizontal strip pooling include:

[0068] The fused features are segmented into multiple vertical strips along the vertical axis of the image, and the features within each vertical strip are pooled to obtain the vertical context features;

[0069] The fused features are segmented into multiple horizontal strips along the horizontal direction of the image, and the features within each horizontal strip are pooled to obtain the horizontal context features.

[0070] Specifically, directional strip pooling is performed on at least one of the fusion features F3, F4, and F5 (preferably F3 or F4). The fusion feature is segmented into multiple vertical strips along the image's longitudinal direction, and features within each strip are pooled to obtain a vertical context feature, which captures the strong vertical structural information of the tower's main structure. Similarly, the fusion feature is segmented into multiple horizontal strips along the image's transverse direction, and features within each strip are pooled to obtain a horizontal context feature, which captures the strong horizontal structural information of the crossarm. The extracted vertical and horizontal context features are then fused with the original fusion feature. The fusion method can be feature concatenation followed by convolution to adjust the number of channels or additive fusion to obtain enhanced features containing directional structural information.

[0071] Three types of structural heatmaps are generated based on enhanced features. For example... Figure 2 As shown, a low-resolution thermal map is output through a lightweight convolutional layer, including: thermal map of the main structure of the tower. Used to characterize the main axial region of the tower; heat map of the crossarm structure. Used to characterize the transverse structural region of the crossarm; heat map of the linear region of the conductor. This is used to characterize the linear distribution area of ​​the conductor. The numerical range of each structure heatmap is [0,1], and the higher the value, the greater the probability that the location belongs to the corresponding structure.

[0072] Furthermore, the gated reweighting step includes:

[0073] A gated weight map is generated based on the heat map of the main tower structure, the heat map of the crossarm structure, and the heat map of the linear region of the conductor.

[0074] The structure-guided features are obtained by multiplying the gating weight map element-wise with the fused features.

[0075] Specifically, the enhancement features are gated and reweighted based on the generated multi-type structural heatmaps. This is based on the heatmap of the main tower structure. Crossarm structure heat map and linear region heat map A gated weight map is generated by weighting and combining the three types of structural heatmaps or taking the maximum value to obtain a comprehensive structural region response map. The gated weight map is then multiplied element-wise with the enhancement features to enhance features in areas related to the tower structure while suppressing background features, resulting in structural guidance features F3', F4', and F5'. The above-mentioned vertical strip pooling, horizontal strip pooling, structural heatmap generation, and gated reweighting operations constitute the Tower Structure Prior Module (TSPM). The output of the TSPM is the structural guidance features and the three types of structural heatmaps. , , .

[0076] S300: Target detection is performed based on the structural guidance features, and the output includes the detection frame, category and confidence level of the entire tower, crossarm, insulator string, hardware connection area, clamp and conductor;

[0077] Specifically, such as Figure 3 As shown, the structure-prior lightweight target detection model TSL-Det consists of an input preprocessing module, a lightweight feature backbone network, a multi-scale fusion network, a tower structure prior module (TSPM), and a decoupled detection head connected in series.

[0078] The input preprocessing module scales and normalizes the pixels of the camera preview frame, and outputs a normalized image of a fixed size with selectable resolutions of 640×640, 960×960, or 1280×720.

[0079] The lightweight feature backbone network is constructed using depthwise separable convolutions as basic units. It performs multi-level downsampling on the preprocessed image and outputs feature layers P3, P4, and P5 at three scales. P3 has the highest resolution and is used to detect small-scale targets such as insulator strings, hardware connection areas, and wire clamps. P4 has a medium resolution and is used to detect medium-scale targets. P5 has the lowest resolution and is used to detect large-scale targets such as the entire tower.

[0080] The multi-scale fusion network receives P3, P4, and P5, and adopts a bidirectional fusion approach of top-down and bottom-up to output fusion features F3, F4, and F5 that simultaneously contain high-level semantic information and low-level detail information.

[0081] The Tower Structure Prior Module (TSPM) receives fused features F3, F4, and F5, and comprises three sub-units: a directional strip context extraction unit, a structural heatmap output head, and a structural gated fusion unit. These sub-units perform the following operations sequentially: The directional strip context extraction unit performs strip pooling on the fused features along both the horizontal and vertical directions, extracting horizontal and vertical context features, and then fuses these features with the original fused features to obtain enhanced features containing directional structural information. The structural heatmap output head outputs three single-channel structural heatmaps based on these enhanced features through lightweight convolutional layers: a heatmap of the main tower structure. Crossarm structure heat map and linear region heat map The numerical range of each heatmap is: The larger the value, the higher the probability that the location belongs to the corresponding structure. Three structural heatmaps are simultaneously transmitted to the structural guidance reweighting unit for use; the structural gating fusion unit will... , , The weighted combination or the maximum value of each element is used to generate a comprehensive gating weight map, which is then multiplied element by element with the enhanced features to output the structure-guided features F3', F4', and F5'.

[0082] The decoupled detection head receives structure-guided features F3', F4', and F5', and performs detection at three different scales. Each scale's detection head contains three decoupled parallel prediction branches: a classification branch (Cls), a bounding box regression branch (Reg), and a target confidence branch (Obj). F3' primarily detects small-scale targets such as insulator strings, hardware connection areas, and clamps; F4' primarily detects medium-scale targets; and F5' primarily detects large-scale targets such as the entire tower, crossarms, and conductors. The outputs of the three scale detection heads are combined to obtain a set of detection boxes for six target categories: the entire tower, crossarms, insulator strings, hardware connection areas, clamps, and conductors. Each detection box (bbox) contains the bounding box's position coordinates (center point coordinates, width, height), its class, and a confidence score, collectively denoted as {class, bbox, score}. The confidence score ranges from [value missing in original text]. A higher value indicates a higher reliability of the test results.

[0083] S400: The response value of the center point of the sampling detection box on the structural heatmap corresponding to its category. The confidence level is weighted and calculated with the response value to obtain the structurally corrected confidence level.

[0084] Furthermore, the structural heatmap corresponding to the category is as follows:

[0085] Crossarm category and corresponding crossarm structure heatmap;

[0086] Heatmap of the linear region corresponding to the conductor type;

[0087] The categories of insulator strings, hardware connection areas, and clamps correspond to the heat maps of the main tower structure, crossarm structure, or conductor linear areas.

[0088] The overall category of the tower corresponds to the heat map of the main structure of the tower.

[0089] Specifically, for each detection box output by S300, the corresponding structural heatmap is determined according to its predicted category, and the response value of the center point of the detection box on the structural heatmap is sampled to correct the confidence level for structural consistency.

[0090] The correspondence between category and structure heatmap is as follows: Crossarm category corresponds to crossarm structure heatmap. Heatmap of linear regions corresponding to conductor types The overall category of the tower corresponds to the heat map of the main structure of the tower. For insulator string type, hardware connection area type, and clamp type, the center point of their detection frame is sampled in the thermal images of the three types of structures. Response value on , , (subscript) , , (Separately label the heat map of the main tower structure, the crossarm structure, and the linear area of ​​the conductor), and select the maximum value among them as the response value of the detection box. .

[0091] For each detection frame, let the coordinates of its center point be... The predicted category is The original confidence level is Because the structural heatmap is a low-resolution heatmap, the coordinates of the center point of the detection box are... The structure heatmap is scaled according to the resolution ratio of the original image to obtain the corresponding coordinates on the structure heatmap, and then the response value at that coordinate is sampled. The sampling method can be nearest neighbor sampling or bilinear interpolation.

[0092] Furthermore, the weighted calculation steps are as follows:

[0093] The correction factor is obtained by multiplying the response value by the preset weighting coefficient and then adding 1.

[0094] Multiply the confidence level by the correction factor to obtain the confidence level after structural correction.

[0095] Specifically, the response value With preset weighting coefficients Multiply by 1 and add 1 to obtain the correction factor. Then, use the original confidence scores... Multiply by the correction factor to obtain the confidence level after structural correction. The calculation formula is:

[0096] ;

[0097] in, For the first Confidence level of a detection box after structural correction, subscript For the detection box index; For the first The original confidence level of each detection box; The preset weighting coefficient is a small constant with a value range of 0.1 to 0.5, used to control the degree of influence of the structural heatmap on the confidence level correction. For the first The response value of the center point of each detection box on the corresponding structural heatmap ranges from 0 to 1.

[0098] When the center point of the detection box is located in the corresponding structural region, the response value High confidence level after structural correction The response value is increased accordingly; when the center point of the detection box deviates from the corresponding structural region. Lower confidence level after structural correction The improvement is relatively small. This correction mechanism allows detection frames that conform to the structural characteristics of the tower to obtain higher confidence, while reducing the priority of false detection frames in the background area.

[0099] After confidence correction for all detection boxes, non-maximum suppression (NMS) is performed to remove overlapping redundant detection boxes, and the detection box with the highest confidence after structure correction is retained as the final detection result.

[0100] S500: Selects key shooting targets from the detection results based on the confidence level after structural correction, generates gimbal attitude adjustment instructions based on the offset between the center of the target detection box and the center of the image, and adjusts the camera zoom parameters based on the ratio of the area of ​​the target detection box to the area of ​​the image.

[0101] Furthermore, the selection step of the key shooting target includes:

[0102] Based on the priority order of insulator strings, clamps, hardware connection areas, crossarms, towers as a whole, and conductors, the target with the highest confidence level after structural correction is selected as the key shooting target from the test results.

[0103] Specifically, see Figure 4 Based on the confidence level obtained from the structure correction of S400, key shooting targets are selected from the detection results, and gimbal attitude adjustment commands and camera zoom parameters are generated according to the position and size of the target detection box to achieve composition optimization.

[0104] The system selects key targets for imaging from the inspection results according to a preset priority order. The priority order is: insulator strings, clamps, hardware connection areas, crossarms, the entire tower, and conductors. Within each priority category, the target with the highest confidence level after structural correction is selected as the key imaging target. Specifically, it first checks if there is an inspection frame for the insulator string category in the inspection results. If so, the inspection frame with the highest confidence level in that category is selected as the key imaging target. If there is no insulator string, it checks in turn if there are inspection frames for the clamps, hardware connection areas, crossarms, the entire tower, and conductor categories, and selects the first inspection frame with the highest confidence level in that category. This priority order ensures that more critical small component targets are photographed first during the inspection.

[0105] The gimbal attitude adjustment command is generated based on the offset between the center of the detection box of the key target and the image center. Let the image center coordinates be... The center coordinates of the key target detection bounding box are: Calculate the horizontal offset and vertical offset The offset is compared with a preset centering threshold. When the horizontal offset is... When the centering threshold is exceeded, a gimbal yaw angle adjustment command is generated. The adjustment direction is consistent with the offset direction, and the adjustment magnitude is proportional to the offset amount. When the vertical offset... When the centering threshold is exceeded, a gimbal tilt angle adjustment command is generated. The adjustment direction is consistent with the offset direction, and the adjustment magnitude is proportional to the offset amount. The centering threshold can be set to 2% to 8% of the image width or height. By fine-tuning the gimbal attitude, the target is gradually moved to the center area of ​​the image, achieving target-centered composition.

[0106] Adjust camera zoom parameters based on the ratio of the key target detection bounding box area to the image area. Calculate the target detection bounding box area. (subscript) (Identifying the target detection box) and the total image area (subscript) The ratio of the identifier image ,in The ratio of the area of ​​the target detection bounding box to the image area, with a value range of [value range missing]. , the ratio Compare with the preset target area percentage range (which can be set to 8% to 25%).

[0107] when When the target area is less than the lower limit of the target area occupancy range, a zoom command is generated to increase the camera's focal length and thus increase the target's occupancy in the image; when... When the target area exceeds the upper limit of the target area occupancy range, a zoom-out command is generated to reduce the camera's focal length, thus decreasing the target's occupancy in the image; when... When the target area is within the target area percentage range, keep the current zoom parameters unchanged. The target area percentage range can be set to 8% to 25% to ensure that the target is neither too small (resulting in unclear details) nor too large (resulting in cropping). Zoom adjustment range and ratio The degree of deviation from the target range is directly proportional.

[0108] The generated gimbal attitude adjustment commands and camera zoom parameters are sent to the gimbal controller and camera controller to execute the corresponding attitude adjustment and zoom operations. After the adjustment is completed, a new preview frame is acquired and the detection and control process from S100 to S500 is repeated to achieve closed-loop iterative optimization until the photo-taking trigger condition is met.

[0109] S600: When multiple consecutive frames simultaneously meet the conditions of target centering, detection stability, and image sharpness, control the camera to capture high-resolution photos.

[0110] Furthermore, the target centering condition is that the offset between the center of the target detection box and the center of the image is less than a preset centering threshold;

[0111] The detection stability condition is that the change in the center position and the change in the area of ​​the target detection box are both less than a preset stability threshold in multiple consecutive frames.

[0112] The image sharpness condition is that the image gradient magnitude of the target area is greater than a preset sharpness threshold.

[0113] Specifically, see Figure 4 After the S500 performs gimbal attitude adjustment and camera zoom, it continuously acquires new preview frames and repeats the detection and control process from S100 to S500, determining whether each frame meets the photo-taking trigger conditions. The photo-taking trigger conditions include target centering, detection stability, and image sharpness. Only when all three conditions are met simultaneously for multiple consecutive frames will the camera be controlled to take a high-resolution photo.

[0114] For the key target in the current frame, calculate the center coordinates of the key target detection box. Image center coordinates The offset. Calculate the horizontal offset. and vertical offset Determine the horizontal offset. Is it less than the preset centering threshold and the vertical offset? If the image size is less than a preset centering threshold, and both conditions are met, the current frame is considered to meet the target centering condition. The preset centering threshold is set according to the image resolution and can be set to 2% to 8% of the image width or height.

[0115] Maintain a fixed-length frame buffer to store the key target detection bounding boxes of the most recent N frames, where N ranges from 3 to 5 frames. For the same target detection bounding boxes in N consecutive frames, calculate the change in the center position and the change in the area of ​​the detection bounding box between adjacent frames. Let the th frame be... The center coordinates of the frame target detection box are The area is Calculate the change in center position between adjacent frames and area change ,in For the first Frame and the The change in Euclidean distance at the center of the detection box between frames, index For frame indexing; For the first The center coordinates of the target detection bounding box in the frame; For the first The center coordinates of the target detection bounding box in the frame; where For the first Frame and the The relative change in the area of ​​the detection box between frames; For the first Area of ​​the target detection bounding box in the frame; For the first Frame target detection bounding box area. Determine the change in center position across all adjacent frames in N consecutive frames. Are all values ​​less than the preset stable threshold and the area change? If all values ​​are less than the preset stability threshold, the detection stability condition is considered met. The preset stability threshold is set according to the target scale and image resolution. The threshold for center position change can be set to 1% to 3% of the image width or height, and the threshold for area change can be set to 5% to 15%.

[0116] For the key target detection bounding box region in the current frame, the image gradient magnitude of that region is calculated as a sharpness evaluation metric. For each pixel within the detection bounding box region, the horizontal gradient is calculated using the Sobel operator or other gradient operators. (subscript) (Identifying horizontal and vertical gradients) (subscript) (Identify the vertical direction), calculate the gradient magnitude. ,in This represents the gradient magnitude of the pixel, reflecting the intensity of local changes in the image at that point.

[0117] The average gradient magnitude of all pixels within the detection bounding box is calculated as the sharpness evaluation value for that region. The sharpness evaluation value is then checked against a preset sharpness threshold; if it is, the image sharpness condition is considered met. The preset sharpness threshold is set based on the camera type and imaging conditions and can be determined by statistically analyzing the gradient magnitude distribution of sharp images on the training or validation set.

[0118] When N consecutive frames simultaneously meet the above three conditions, the current composition, zoom, and focus state are determined to be optimal, triggering the camera to capture a high-resolution photo. The camera switches to photo mode, captures a photo at the set highest resolution, and saves it to the storage device. When saving the photo, the system records the category of key targets, detection box coordinates, confidence level, shooting time, and UAV location information for subsequent image management and defect identification. After taking the photo, the system continues to inspect and capture the next target or flies to the next shooting point according to a preset flight path.

[0119] Furthermore, this embodiment also includes the following: during the training of the model used for object detection in S300, the supervision signal for the structural heatmap is generated in the following way:

[0120] The monitoring signal for the main structure heat map of the tower is generated based on the center line of the marked tower overall detection frame;

[0121] The monitoring signal for generating a heat map of the crossarm structure is generated based on the long side direction of the marked crossarm detection frame;

[0122] A supervisory signal is generated from the marked wire detection frame area to create a heat map of the wire linear region.

[0123] Specifically, during the model training phase, to reduce annotation costs, a weakly supervised approach is used to generate supervisory signals for structural heatmaps from bounding box annotations. For the six types of target bounding boxes annotated in the training images, corresponding structural heatmap supervisory signals are generated based on their category and geometric features.

[0124] For the marked overall inspection frame of the tower, the centerline of its bounding box is extracted as an approximate representation of the tower's main axis. The coordinates of the center point and the height direction of the inspection frame are calculated, and a longitudinal line segment is generated along the height direction, with the segment length equal to the height of the inspection frame. Using this line segment as the center, a strip-shaped heatmap is generated using a Gaussian kernel function; the width of the heatmap is set according to the actual width of the tower. The generated heatmap serves as the heatmap of the tower's main structure. The supervisory signal, in which the superscript Indicates truth value label, subscript The main structure of the tower is marked, and the heat map value is the largest at the center of the line segment, gradually decreasing to 0 towards both sides.

[0125] For the labeled crossarm detection frame, the long side direction of its bounding box is extracted as an approximate representation of the crossarm's transverse structure. The coordinates of the center point and the long side direction of the detection frame are calculated, and a transverse line segment is generated along the long side direction, with the segment length equal to the long side length of the detection frame. A strip-shaped heatmap is generated using a Gaussian kernel function centered on this line segment; the heatmap width is set according to the actual width of the crossarm. The generated heatmap serves as the heatmap of the crossarm structure. The supervisory signal, in which the superscript The value is indicated by the true value label. The subscript C indicates the crossarm structure. The heat map value is the largest at the center of the line segment, and gradually decreases to 0 towards both sides.

[0126] For the labeled wire detection boxes, their bounding box regions are extracted as an approximate representation of the wire's linear distribution. Since the wires are linear and may have curvature, a linear heatmap with width is directly generated from the detection box regions. Higher heatmap values ​​are set for pixels within the detection box regions, and lower heatmap values ​​are set for pixels outside the detection box regions. Optionally, Gaussian blurring is applied to the detection box boundaries to generate a smooth heatmap. The generated heatmap serves as the heatmap for the linear region of the wire. The supervisory signal, in which the superscript Indicates truth value label, subscript Identify the conductors. Optionally, a line detection algorithm is used to process the conductor image, extract the conductor centerline and generate a coarse line mask, and then convert the mask into a heat map to improve the accuracy of the supervision signal.

[0127] The generated three types of structural heatmap monitoring signals , Downsample to the same resolution as the structure heatmap output by the model. During training, the structure heatmap output by the model is calculated. The loss between the structure heatmap and the corresponding monitoring signal is calculated using either mean squared error loss or binary cross-entropy loss.

[0128] ;

[0129] in, For structural heatmap monitoring loss; Let the mean square error function be used. The output of the model is a heat map of the main structure of the tower. (Subscript) Identify the main structure of the tower; For monitoring signals of the heat map of the main structure of the tower, superscript Indicates truth value labeling; The heatmap of the crossarm structure output by the model, with subscripts... Identification crossbeam structure; For monitoring signals of the heat map of the crossarm structure, superscript Indicates truth value labeling. For the heatmap of the wire-shaped region output by the model, the subscript is... Identify the wires; The superscript is the supervisory signal for the heatmap of the linear region of the conductor. This indicates a truth label.

[0130] Furthermore, this embodiment also includes, in S300, the model used for object detection employs structural consistency training constraints during training, the structural consistency training constraints including:

[0131] Category-Structure Matching Loss: When the response value of the predicted detection box center point on the corresponding structure heatmap is lower than the first preset threshold, the loss is calculated based on the difference between the threshold and the response value.

[0132] Hierarchical feasible domain loss: When the center point of the predicted detection frame for the crossarm category, insulator string category, hardware connection area category, or clamp category exceeds the extended area of ​​the overall predicted detection frame of the tower, the loss is calculated based on the distance between the center point and the boundary of the area.

[0133] Component topology relationship loss: The loss is calculated based on the distance between the predicted detection frames of the preset key component pairs, which is normalized by the area of ​​the overall detection frame of the tower. When the normalized distance exceeds the second preset threshold, the loss is calculated.

[0134] The preset key component pair includes at least one of the following: the connection area between the insulator string and the fitting, the clamp and the conductor, and the insulator string and the crossarm.

[0135] Specifically, structural consistency training constraints are introduced during the model training phase. By incorporating class-structure matching loss, hierarchical feasible region loss, and component topology relationship loss into the loss function, the detection results output by the model conform to the structural characteristics of the tower. The total loss function is defined as:

[0136] ;

[0137] in, This is the total loss function; To detect the loss, subscript Object detection is characterized by classification loss, bounding box regression loss, and IoU loss. The subscript represents the weighting coefficient of the structural heatmap supervision loss. The heatmap of the identifier structure has a value range of 0.5 to 2. For structural heatmap monitoring loss; The subscript represents the weighting coefficient of the structural consistency constraint loss. The identifier structure consistency constraint has a value range of 0.5 to 2; The loss is due to structural consistency constraints.

[0138] Structural consistency constraint loss It consists of three parts:

[0139] ;

[0140] in, The loss is due to structural consistency constraints; for The weighting coefficient ranges from 0.5 to 2. For category-structure matching loss, subscript Identify the matching relationship between categories and structures; for The weighting coefficient ranges from 0.5 to 2. For hierarchical feasible region loss, subscript Constraints on the location of the marking components within the overall frame of the tower; for The weighting coefficient ranges from 0.5 to 2. For component topology loss, subscript Identify the topological relationships between components.

[0141] For each detection box predicted by the model, let its center point coordinates be... The predicted category is The corresponding structural heatmap is determined based on the predicted category, and the response value of the center point of the sampling detection box on the corresponding structural heatmap is used. For the crossarm category, the corresponding structural heatmap is the crossarm structural heatmap. For each type of conductor, the corresponding structural heatmap is a heatmap of the conductor's linear region. For insulator string type, hardware connection area type, and clamp type, the corresponding structural heat map is the tower main structure heat map. Crossarm structure heat map or linear region heat map At least one of the following; for the overall category of towers, the corresponding structural heatmap is the heatmap of the main structure of the tower body. Set categories The corresponding structure heatmap index is Then the first The response value of the center point of each predicted detection box on the corresponding structural heatmap is .

[0142] Calculate category-structure matching loss For each predicted detection box, determine the response value of its center point on the corresponding structural heatmap. Is it below the first preset threshold? If the value is below a threshold, the loss is calculated based on the difference between the threshold and the response value. The category-structure matching loss is defined as:

[0143] ;

[0144] in, For category-structure matching loss; To correct the linear unit function, output this value when the input is greater than 0, otherwise output 0; The first preset threshold value ranges from 0.3 to 0.6. For category The corresponding structural heatmap, For category Mapping index to the structure heatmap; For the first The center point coordinates of each predicted detection box, index For the detection box index.

[0145] Calculate the feasible region loss at each level. See also Figure 5 For the prediction detection frames of crossarm category, insulator string category, hardware connection area category, and clamp category, determine whether their center points exceed the expansion area of ​​the overall tower prediction detection frame. Let the overall tower prediction detection frame be... Its extended area By The boundary is expanded outward by a certain proportion, which can be set to 10% to 30%. For each component class, the predicted detection box is calculated, and its center point is determined. To the extended area The distance to the boundary. If the center point is within the extended region, the distance is 0; if the center point is outside the extended region, the distance is the Euclidean distance from the center point to the nearest boundary, denoted as . The hierarchical feasible region loss is defined as:

[0146] ;

[0147] in, The loss is the hierarchical feasible region loss; These respectively represent the crossarm, insulator string, hardware connection area, and clamp type; For the first Center point of each prediction detection box To the extended area The distance to the boundary is 0 when the center point is within the extended area, and Euclidean distance from the center point to the nearest boundary when it is outside the extended area. Overall prediction detection frame for tower The extended area, superscript Identify extended areas, Subscript The entire signpost tower.

[0148] Calculate the topology relationship loss of components See also Figure 5 For preset key component pairs, the normalized distance between their predicted detection frames is calculated. When the normalized distance exceeds a second preset threshold, the loss is calculated. The preset key component pairs include the insulator string and hardware connection area, and the clamp and conductor.

[0149] For the insulator string and fitting connection area component pair, let the center coordinates of the insulator string (i.e., the C3 category detection frame) be... The center coordinates of the hardware connection area, i.e., the C4 category detection frame, are: The subscripts correspond to the category numbers mentioned above, and the Euclidean distance between them is calculated. and the overall detection frame area of ​​the tower Normalize the square root to obtain the normalized distance. subscript The entire signpost tower.

[0150] For the wire clamp and wire component pair, let the center coordinates of the wire clamp (C5 category detection frame) be... Calculate the shortest distance from the center of the wire clamp detection frame to the conductor area. ,in The area covered by the conductor inspection frame is normalized by the square root of the area of ​​the entire tower inspection frame to obtain the normalized distance. .

[0151] The component topology relationship loss is defined as:

[0152] ;

[0153] in, For component topology loss; To correct the linear unit function; Center of the inspection frame for the connection area between the insulator string and the fittings and Normalized distance between them, subscript and Corresponding to categories C3 and C4; The second preset threshold for the normalized distance between the insulator string and the fitting connection area, with subscript... The maximum permissible distance for marking is 0.05 to 0.25. Center of the wire clamp detection frame Normalized shortest distance to the wire detection frame area, subscript and Corresponding to C5 category and wire; The second preset threshold is the normalized distance between the clamp and the conductor, with a value range of 0.05 to 0.25.

[0154] During model training, the total loss function Backpropagation is performed, and model parameters are updated using stochastic gradient descent or the Adam optimizer. Structural consistency training constraints enable the model to learn the structural patterns of the towers during training, thus allowing it to output stable detection results that conform to these patterns during inference without complex post-processing.

[0155] Example 2: A power company is responsible for maintaining a 220kV high-voltage transmission line in a mountainous area. The line is approximately 150 kilometers long and traverses complex terrain, including mountains, forests, and valleys. Traditional manual inspection methods are inefficient, costly, and pose safety risks in adverse weather and complex terrain conditions. To improve inspection efficiency and quality, the company decided to adopt unmanned aerial vehicle (UAV) intelligent inspection technology. Therefore, the target recognition and precise photography system for transmission tower inspection of this invention was introduced, with the structure as follows: Figure 6 As shown.

[0156] See Figure 7This system is deployed on a UAV platform, with the onboard edge computing unit running the TSL-Det lightweight target detection model based on structural priors. After the UAV flies to the vicinity of the tower along a preset route, the gimbal camera begins acquiring real-time preview frames. The image acquisition module transmits the preview frames to the feature extraction module to extract multi-scale fused features. The tower structure prior module performs vertical and horizontal strip pooling on the fused features, extracting vertical and horizontal contextual features and fusing them with the original fused features. Subsequently, it generates heatmaps of the main tower structure, crossarm structure, and conductor linear regions. Then, based on these structural heatmaps, the fused features are gated and reweighted to obtain structural guidance features.

[0157] The target detection module detects targets such as the entire tower, crossarms, insulator strings, hardware connection areas, clamps, and conductors in real time based on structural guidance features, and outputs the detection frame, category, and confidence level. The confidence level correction module samples the response value of the center point of each detection frame on the structural heatmap corresponding to its category, and calculates the confidence level by weighting the response value to obtain the structurally corrected confidence level.

[0158] The image capture control module selects key targets based on the confidence level after structural correction. Taking an insulator string as an example, once the insulator string is detected, the module calculates the offset between the center of its detection frame and the image center, generates gimbal yaw and pitch adjustment commands, and controls the gimbal to fine-tune its attitude to gradually center the target. Simultaneously, it adjusts the camera zoom parameters based on the ratio of the target detection frame area to the image area, ensuring the target occupies a suitable proportion in the frame. The module continuously monitors the target position, detection stability, and image sharpness across multiple frames. When these conditions are simultaneously met, the camera is triggered to capture a high-resolution image and record information such as the target category and location.

[0159] During the two-month trial operation, the system performed stably in tower inspection tasks along the line. Through real-time target recognition and precise image capture closed-loop control, the centering and clarity of key components such as insulator strings and hardware connection areas in the photos were improved, and out-of-focus and misaligned shots were significantly reduced. The airborne edge computing unit was able to complete target detection inference with low latency, meeting the real-time inspection requirements. Due to the introduction of the structural prior module, the model's detection stability in complex backgrounds such as mountains and forests was improved, and frequent gimbal adjustments and accidental image captures were reduced.

[0160] The system operates normally under various tower types and weather conditions, requiring minimal parameter adjustments when deployed across different routes. Compared to traditional route-based fixed-point photography, this system improves the usability of inspection photos through recognition-driven closed-loop control, reducing the workload of subsequent manual verification.

[0161] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for target identification and precise photography for power transmission tower inspection, characterized in that, include: S100: Acquire camera preview frames during UAV inspection and extract multi-scale fusion features; wherein, a lightweight feature backbone network is used to extract multi-scale feature maps from the camera preview frames, outputting three feature layers P3, P4, and P5 at different scales, and inputting the feature layers P3, P4, and P5 into the fusion network. The fusion network adopts a bidirectional fusion method of top-down and bottom-up, and outputs fusion features F3, F4, and F5 corresponding to P3, P4, and P5. The fusion features contain both semantic information and detail information. S200: Perform vertical strip pooling and horizontal strip pooling on the fused features respectively, extract the vertical context features and horizontal context features, and fuse them with the original fused features to obtain enhanced features containing directional structural information. Based on the fused enhanced features, generate multiple types of structural heat maps including tower main structure heat map, crossarm structure heat map and conductor linear region heat map. Based on the multiple types of structural heat maps, perform gating weighting on the fused enhanced features to obtain structural guidance features. The steps of the gate weighting process include: A comprehensive gating weight map is generated by weighting and combining the heat map of the main tower structure, the heat map of the crossarm structure, and the heat map of the linear region of the conductor, or by taking the maximum value bit by bit. The integrated gating weight map is multiplied element-wise with the fused enhanced features to obtain the structure-guided features; S300: Target detection is performed based on the structural guidance features, and the output includes the detection frame, category and confidence level of the entire tower, crossarm, insulator string, hardware connection area, clamp and conductor; S400: The response value of the center point of the sampling detection box on the structural heatmap corresponding to its category. The confidence level is weighted and calculated with the response value to obtain the structurally corrected confidence level. S500: Selects key shooting targets from the detection results based on the confidence level after structural correction, generates gimbal attitude adjustment instructions based on the offset between the center of the target detection box and the center of the image, and adjusts the camera zoom parameters based on the ratio of the area of ​​the target detection box to the area of ​​the image. S600: When multiple consecutive frames simultaneously meet the conditions of target centering, detection stability, and image sharpness, control the camera to capture high-resolution photos.

2. The target identification and precise photography method for transmission tower inspection according to claim 1, characterized in that, In S200, the steps of vertical strip pooling and horizontal strip pooling include: The fused features are segmented into multiple vertical strips along the vertical axis of the image, and the features within each vertical strip are pooled to obtain the vertical context features; The fused features are segmented into multiple horizontal strips along the horizontal direction of the image, and the features within each horizontal strip are pooled to obtain the horizontal context features.

3. The target identification and precise photography method for transmission tower inspection according to claim 1, characterized in that, In S400, the structural heatmap corresponding to the category is as follows: Crossarm category and corresponding crossarm structure heatmap; Heatmap of the linear region corresponding to the conductor type; The categories of insulator strings, hardware connection areas, and clamps correspond to the heat maps of the main tower structure, crossarm structure, or conductor linear areas. The overall category of the tower corresponds to the heat map of the main structure of the tower.

4. The target identification and precise photography method for transmission tower inspection according to claim 1, characterized in that, In S400, the weighted calculation steps are as follows: The correction factor is obtained by multiplying the response value by the preset weighting coefficient and then adding 1. Multiply the confidence level by the correction factor to obtain the confidence level after structural correction.

5. The target identification and precise photography method for transmission tower inspection according to claim 1, characterized in that, In S500, the selection step for the key shooting target includes: Based on the priority order of insulator strings, clamps, hardware connection areas, crossarms, towers as a whole, and conductors, the target with the highest confidence level after structural correction is selected as the key shooting target from the test results.

6. The target identification and precise photography method for transmission tower inspection according to claim 1, characterized in that, In S600: The target centering condition is that the offset between the center of the target detection box and the center of the image is less than a preset centering threshold. The detection stability condition is that the change in the center position and the change in the area of ​​the target detection box are both less than a preset stability threshold in multiple consecutive frames. The image sharpness condition is that the image gradient magnitude of the target area is greater than a preset sharpness threshold.

7. The target identification and precise photography method for transmission tower inspection according to claim 1, characterized in that, This also includes the method for generating the supervision signal for the structure heatmap during training of the model used for object detection in S300: The monitoring signal for the main structure heat map of the tower is generated based on the center line of the marked tower overall detection frame; The monitoring signal for generating a heat map of the crossarm structure is generated based on the long side direction of the marked crossarm detection frame; A supervisory signal is generated from the marked wire detection frame area to create a heat map of the wire linear region.

8. The target identification and precise photography method for transmission tower inspection according to claim 1, characterized in that, It also includes the fact that the model used for object detection in S300 employs structural consistency training constraints during training, the structural consistency training constraints including: Category-Structure Matching Loss: When the response value of the predicted detection box center point on the corresponding structure heatmap is lower than the first preset threshold, the loss is calculated based on the difference between the threshold and the response value. Hierarchical feasible domain loss: When the center point of the predicted detection frame for the crossarm category, insulator string category, hardware connection area category, or clamp category exceeds the extended area of ​​the overall predicted detection frame of the tower, the loss is calculated based on the distance between the center point and the boundary of the area. Component topology relationship loss: The loss is calculated based on the distance between the predicted detection frames of the preset key component pairs, which is normalized by the area of ​​the overall detection frame of the tower. When the normalized distance exceeds the second preset threshold, the loss is calculated. The preset key component pair includes at least one of the following: the connection area between the insulator string and the fitting, the clamp and the conductor, and the insulator string and the crossarm.

9. A target recognition and precise photography system for power transmission tower inspection, characterized in that, The system is used to execute the target identification and precise photography method for transmission tower inspection as described in any one of claims 1-8, the system comprising: The image acquisition module is used to acquire camera preview frames during the drone inspection process; The feature extraction module is used to extract multi-scale fusion features from the preview frame; The tower structure prior module is used to perform vertical strip pooling and horizontal strip pooling on the fused features respectively, extract the vertical context features and horizontal context features and fuse them with the original fused features, generate multiple types of structural heat maps based on the fused features, including the tower main structure heat map, the crossarm structure heat map and the conductor linear region heat map, and perform gated reweighting on the fused features according to the multiple structural heat maps to obtain structural guidance features; The target detection module is used to perform target detection based on the structural guidance features, and outputs the detection frame, category and confidence level of the entire tower, crossarm, insulator string, hardware connection area, clamp and conductor; The confidence correction module is used to sample the response value of the center point of the detection box on the structural heatmap corresponding to its category, and to calculate the confidence value by weighting the confidence value with the response value to obtain the structure-corrected confidence value. The image capture control module is used to select key shooting targets from the detection results based on the confidence level after structural correction, generate gimbal attitude adjustment commands based on the offset between the center of the target detection box and the center of the image, adjust the camera zoom parameters based on the ratio of the area of ​​the target detection box to the area of ​​the image, and control the camera to capture high-resolution photos when the target centering condition, detection stability condition, and image sharpness condition are met simultaneously in multiple consecutive frames.

Images