A power distribution line equipment identification method based on prior guidance and multi-modal fusion

Abstract

This invention discloses a method for identifying power distribution line equipment based on prior guidance and multimodal fusion, relating to the field of power grid inspection technology. The method includes: collecting heterogeneous data; constructing a priori library of power equipment structures and generating a priori heatmaps; constructing a priori-guided dual-branch multimodal segmentation network, introducing a cross-modal attention fusion mechanism at the bottom layer of the dual-branch encoder, injecting the prior heatmaps into the upsampling process of the dual-branch encoder in a multi-channel format, and outputting multi-task results; and performing discrimination operations according to anomaly discrimination logic to generate structured alarm information. This invention embeds prior knowledge into the segmentation network architecture and also includes a dual-branch multimodal segmentation network, enabling infrared thermal anomaly features to dynamically guide the visible light segmentation process. This facilitates high-precision pixel-level segmentation of unobstructed areas and allows inference of the equipment's state when it is obstructed through infrared thermal signals. The invention also includes anomaly discrimination logic to facilitate the generation of structured alarm information.

Description

Technical Field

[0001] This invention relates to the field of power grid inspection technology, specifically a method for identifying distribution line equipment based on prior guidance and multimodal fusion. Background Technology

[0002] With the accelerated construction of new power systems, intelligent inspection technology for power distribution lines is undergoing a profound transformation from "manual-based" to "machine-autonomous." Leveraging its advantages such as live-line operation, close proximity to conductors, and high-resolution imaging, power distribution line inspection robots have become core equipment for the operation and maintenance of ultra-high voltage and inter-regional interconnected lines. Key equipment such as insulators, fittings, conductors, guy wires, and surge arresters are scattered and vary significantly in size (e.g., towers can reach over ten meters in height, while line clamps are only a few centimeters). Their critical details (such as cracked skirts, missing bolts, and overheated contact points) determine the safety status of the line, necessitating pixel-level precision identification. Robots can autonomously move along ground or phase lines, collecting multimodal data that carries both structural and thermal information about the equipment. This information forms the basis for automatically identifying fine-grained anomalies such as damage, loosening, missing parts, and overheating. The accuracy of this identification directly determines the reliability and practical value of the inspection system.

[0003] However, in practical applications, the working environment of power distribution line inspection robots is complex, and their perception data faces multiple challenges: 1. Severe visual occlusion: The robot body (wheels, robotic arm), vibration damper, spacer bars, and other key components often obscure the view, resulting in partial or complete loss of visible light information; 2. Hidden fault states: High-risk defects such as loose clamps and poor connections do not show obvious changes in appearance under visible light, only manifesting as localized temperature anomalies, making it difficult to capture with a single mode; 3. Complex equipment structure and large scale: Insulators have a periodic awning structure, and hardware needs to be installed in pairs symmetrically. If the model lacks such prior knowledge, it is easy to misidentify isolated bolts as valid targets or generate discontinuous segments of damaged areas; 4. Strong background interference: Towers, conductors, and vegetation are highly similar to the equipment in texture and color, further increasing the difficulty of recognition. All of these factors combined result in insufficient robustness of traditional inspection methods in real-world scenarios, making it difficult to support high-reliability operation and maintenance requirements.

[0004] Existing power distribution network equipment identification technologies also have some shortcomings: 1. They rely on a single visible light mode and cannot use infrared thermal features to penetrate obstructions or sense hidden overheating, resulting in "visibility to the eye means unknowability of the state"; 2. They use general segmentation networks (such as U-Net and DeepLab) without embedding prior knowledge of the spatial topology, geometry, and installation specifications of power equipment, resulting in segmentation results lacking physical rationality and easily leading to unreliable engineering outputs such as isolated fittings and fracture boundaries; 3. They do not explicitly model obstructions, lack obstruction confidence output, cannot distinguish between "real equipment missing" and "visual obstruction," and cannot trigger intelligent decisions such as autonomous re-inspection based on this, thus limiting the system's intelligence level.

[0005] Based on this, a method for identifying power distribution line equipment based on prior guidance and multimodal fusion is provided, which can eliminate the drawbacks of existing technical solutions. Summary of the Invention

[0006] The purpose of this invention is to provide a method for identifying power distribution line equipment based on prior guidance and multimodal fusion, so as to solve the problems in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A method for identifying power distribution line equipment based on prior guidance and multimodal fusion specifically includes the following steps:

[0009] Step S1: Collect heterogeneous data synchronously through several sensors mounted on the power distribution line inspection robot. The heterogeneous data includes visible light images, infrared thermal images, and robot pose data. The heterogeneous data is aligned by hardware-level timestamps.

[0010] Step S2: Construct a priori library of power equipment structures. The priori library of power equipment structures is used to store the spatial positional relationships, geometric structural characteristics and component combination logic of power distribution line equipment. Based on the priori library of power equipment structures and robot pose data, a priori heat map is generated.

[0011] Step S3: Construct a priori-guided dual-branch multimodal segmentation network. The dual branches are two parallel visible light branch encoders and infrared branch encoders. A cross-modal attention fusion mechanism is introduced at the bottom layer of the dual-branch encoder. The prior heatmap is injected into the upsampling process of the dual-branch encoder in a multi-channel form, and the output is a multi-task result.

[0012] Step S4: Perform the discrimination operation according to the anomaly discrimination logic. If the anomaly is discerned, output the abnormal equipment and anomaly type of the power distribution line, and generate structured alarm information.

[0013] The anomaly detection logic specifically includes:

[0014] Insulator damage assessment based on the number of connected components and boundary roughness;

[0015] Hardware loosening and overheating judgment based on the average temperature of the overlapping area and the integrity of the component;

[0016] Anomaly detection of pull lines based on mask length, fitting angle, and bottom temperature;

[0017] Anomaly detection of conductors based on skeleton continuity and sag height.

[0018] Preferably, step S2 further includes: using a lightweight U-Net coarse segmentation network, wherein the lightweight U-Net coarse segmentation network uses MobileNetV2 as the encoder backbone to output a preliminary semantic segmentation mask for the device, so as to obtain the preliminary spatial location of the power distribution line equipment and optimize the spatial projection accuracy of the prior heat map.

[0019] Preferably, the process of generating the prior heatmap in step S2 specifically includes:

[0020] Based on robot pose data, the preliminary spatial position of power distribution line equipment is back-projected from image coordinates to world coordinates, and the expected world position is calculated based on prior knowledge in the power equipment structure prior library.

[0021] By reprojecting the expected world location onto the image coordinates, the expected distribution area or trajectory of the power distribution line equipment can be obtained.

[0022] For each type of device, an independent channel linear Gaussian heatmap is generated based on the expected image coordinates. Each pixel value in the linear Gaussian heatmap represents the prior probability that a device of the corresponding category exists at that location, ultimately forming a multi-channel prior heatmap consistent with the number of device categories.

[0023] Preferably, in step S3, the prior-guided bi-branch multimodal segmentation network includes an input module, a bi-branch encoder, a cross-modal attention fusion module, a prior heatmap injection module, and a segmentation head module;

[0024] The input module is used to input infrared images and visible light images;

[0025] The dual-branch encoder uses a parallel architecture to process different modal input features. The visible light branch encoder uses EfficientNet-B0 as the backbone network, takes a visible light image as input, and outputs a 4-level multi-scale feature map to capture the texture, edge and structural details of the device. The infrared branch encoder uses MobileNetV2 as the backbone network, takes an infrared thermal image as input, and outputs an infrared feature map to focus on encoding temperature anomaly areas on the surface of the device.

[0026] The cross-modal attention fusion module is connected to the output of the dual-branch encoder. It fuses features through the cross-modal attention fusion mechanism, using visible light features as the query and infrared features as the key and value, to enhance the response intensity in the infrared thermal anomaly region.

[0027] The prior heatmap injection module injects the prior heatmap into the encoder's upsampling process in a multi-channel form. After each skip link and before the upsampling convolution, the prior heatmaps of the corresponding scale are fused as additive biases.

[0028] The segmentation head includes an output layer that outputs multi-task results, which outputs semantic segmentation mask, occlusion mask and infrared anomaly mask respectively.

[0029] Preferably, the specific expression of the cross-modal attention fusion mechanism includes:

[0030] ;

[0031] ;

[0032] ;

[0033] ;

[0034] ;

[0035] in, , , For learnable linear projection, Features after fusion This is the attention weight matrix. For querying Query, For key, Value This is the feature map output from the lowest level of the visible light branch encoder. This is the infrared feature map output by the infrared branch encoder. For feature dimension, This is the normalization function.

[0036] Preferably, the formula for calculating the number of connected components in the insulator damage determination is as follows:

[0037] ;

[0038] in, This is the binarized image. Image pixel coordinates, For the insulator channel in the semantic segmentation mask, the insulator damage detection uses OpenCV's 8-neighborhood connected component labeling. , obtain the number of connected components ;

[0039] The formula for calculating the boundary roughness is:

[0040] ;

[0041] in, For boundary roughness, The total length of the outer boundary of the connected region. This represents the total number of pixels within the connected region.

[0042] Preferably, the formula for calculating the average temperature of the overlapping area in the hardware loosening and overheating judgment includes:

[0043] ;

[0044] ;

[0045] in, This is the overlapping area between the hardware area and the infrared anomaly area. For the hardware channel in the semantic segmentation mask, For infrared anomaly mask, The average temperature of the overlapping region. For the original infrared thermal image in pixels Temperature value at that location;

[0046] The integrity of the component is determined as follows: if and Then it is determined that a component is missing, among which, This is a priori heat map.

[0047] Preferably, the formula for calculating the mask length in the wire anomaly detection includes:

[0048] ;

[0049] ;

[0050] in, This is the binarized image. For the drawn line channel in the semantic segmentation mask, For mask length, The skeleton length calculation function is used, and the line anomaly detection uses OpenCV's cv2.findContours to extract the main contour of the line;

[0051] The formula for calculating the fitting angle is:

[0052] ;

[0053] in, To fit the angle, , This represents the difference between the ordinate and abscissa of the drawn line profile. It is the arctangent function;

[0054] The bottom temperature is obtained by taking the bottom 10% of the area of ​​the pull-out mask region and calculating the average infrared temperature of that area.

[0055] Preferably, in the conductor anomaly detection, the skeleton continuity is obtained by skeletonizing the feature map using cv2.ximgproc.thinning and simultaneously calculating the number of endpoints in the skeleton. The sag height is obtained by fitting the conductor center curve, obtaining the ordinate of the lowest point of the conductor center curve, and converting it into the ground height by combining it with the robot pose data.

[0056] Preferably, the robot pose data in step S1 includes three-dimensional position coordinates, pitch angle, and yaw angle.

[0057] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0058] This invention constructs a priori library of power equipment structures, systematically encodes installation specifications and geometric constraints, and transforms them into learnable prior heatmaps. It embeds this priori knowledge of power equipment structures into a segmentation network architecture, achieving pixel-level recognition driven by physical constraints, thus improving the geometric consistency and engineering reliability of the results. Furthermore, this invention designs a cross-modal aligned bi-branch multimodal segmentation network, enabling infrared thermal anomaly features to dynamically guide the visible light segmentation process. The segmentation network synchronously outputs semantic segmentation masks, occlusion masks, and infrared anomaly masks, allowing the inspection system to not only perform high-precision pixel-level segmentation of unobstructed areas but also infer the state of equipment through infrared thermal signals when it is obstructed. In addition, this invention includes end-to-end anomaly discrimination logic, facilitating the discrimination of insulator damage, hardware loosening and overheating, guy wire anomalies, and conductor anomalies, and supporting the generation of structured alarm information. This effectively solves the shortcomings of existing technologies in terms of physical rationality, occlusion robustness, and state discriminability. Attached Figure Description

[0059] Figure 1 This is a schematic diagram illustrating the steps of the identification method of the present invention.

[0060] Figure 2 This is a flowchart illustrating the identification method of the present invention.

[0061] Figure 3 This is a schematic diagram illustrating the process of generating the prior heat map according to the present invention.

[0062] Figure 4 This is a schematic diagram of the lightweight U-Net coarse segmentation network of the present invention.

[0063] Figure 5 This is a schematic diagram of the priori-guided bi-branch multimodal segmentation network of the present invention.

[0064] Figure 6 This is a schematic diagram of the anomaly detection process of the present invention. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0066] Multimodal fusion refers to the collaborative processing, feature alignment, and semantic integration of information from different physical sensors or data sources to build a more comprehensive and robust perception system. In this invention, multimodal refers to three types of heterogeneous information: visible light images (providing texture, color, and structural details), infrared thermal images (reflecting the temperature distribution of the device and penetrating some visual obstructions), and robot pose data (including position, pitch angle, etc., used for spatial context modeling). The fusion process uses a cross-modal attention mechanism to dynamically guide the visible light segmentation based on infrared thermal anomalies, enabling the model to make state inferences based on thermal features even in areas where visual information is missing, thereby improving the robustness of the inspection system in complex working conditions such as obstruction, nighttime, and strong reflection.

[0067] Prior knowledge refers to domain-specific knowledge in power systems regarding equipment installation specifications, geometric shapes, and spatial topological relationships. In this invention, prior knowledge specifically includes spatial positional relationships (such as "line clamps should be installed 1.0-1.5 meters below the insulator"), geometric structural characteristics (such as "composite insulator skirts are arranged in a periodic axisymmetric manner" and "the angle between the guy wire and the ground is usually 45°±15°"), and component combination logic (such as "suspension line clamps must appear in pairs and clamp both sides of the conductor"). This prior knowledge is systematically encoded into prior heatmaps or topological constraint rules and embedded as soft or hard constraints in the training and reasoning process of the segmented network. For example, in the decoding stage, the prior heatmap, as an additive bias, guides the feature response to focus on physically reasonable areas, thereby effectively avoiding misjudging isolated bolts as valid hardware or generating discontinuous segmentation boundaries for damaged insulators.

[0068] Semantic segmentation is a pixel-level computer vision task that aims to assign a semantic category label (such as "insulator", "fitting", "wire", "background") to each pixel in an input image and output a classification mask of the same size as the input image. Semantic segmentation can provide precise contour and internal structure information of a device.

[0069] Example 1

[0070] In this embodiment, as Figures 1-6 As shown, a method for identifying power distribution line equipment based on prior guidance and multimodal fusion specifically includes the following steps:

[0071] Step S1: Collect heterogeneous data synchronously through several sensors mounted on the power distribution line inspection robot. The heterogeneous data includes visible light images, infrared thermal images and robot pose data. The heterogeneous data is aligned by hardware-level timestamps.

[0072] Specifically, since existing methods often rely on a single visible light mode, they cannot detect latent faults. Therefore, this invention constructs a three-modal collaborative perception framework of visible light, infrared, and pose, and designs a cross-modal attention fusion module to dynamically guide the visible light segmentation process using infrared thermal anomaly features. The network synchronously outputs three key results: semantic segmentation mask, occlusion mask, and infrared anomaly mask. This enables not only high-precision pixel-level segmentation of unoccluded areas, but also the inference of the device's status through infrared thermal signals when the device is occluded by the inspection robot or anti-vibration hammer.

[0073] Specifically, the mathematical expression of a visible light image is as follows: ,in, Represents the set of real numbers, with subscripts This indicates that the image is a three-channel color image with red, green, and blue channels. Indicates the height of the image. The width, height and width of the image are all represented in pixels. The visible light image is acquired by a high-definition visible light camera carried by the inspection robot. It is used to provide information on the texture, color, edge and structural details of the equipment, and is the main basis for identifying appearance features such as insulator skirts, hardware bolts, and wire corrosion.

[0074] Specifically, the mathematical expression of an infrared thermal image is as follows: , where subscript This indicates that the image is an infrared single-channel thermal image. Indicates the height of the image. The width, height, and unit of measurement for the image are all pixels. Infrared thermal images are synchronously acquired by an infrared thermal imager, reflecting the temperature distribution on the equipment surface. They can effectively detect latent faults such as loose wire clamps and poor contact at connection points, which only manifest as localized overheating. Furthermore, they have the ability to penetrate some visual obstructions (such as thin fog or slight icing). In addition, since the resolution of infrared images is usually lower than that of visible light images, they need to be upsampled to have the same spatial dimension as visible light images. This invention uses a bilinear interpolation method for upsampling to obtain the upsampled infrared image. Its definition is: ,in, This indicates bilinear interpolation upsampling;

[0075] Specifically, the robot pose data includes three-dimensional position coordinates, pitch angle, and yaw angle, and its mathematical expression is: ,in, , , This represents the robot's three-dimensional position coordinates in the world coordinate system, in meters. This indicates the robot's pitch angle, which is the angle between the robot's camera optical axis and the horizontal plane (>0 indicates tilting down, <0 indicates tilting up). The yaw angle represents the robot's forward direction, which is the angle between the robot's forward direction and true north. The units for both pitch and yaw angles are radians. The robot's pose data is fused from the inertial measurement unit (IMU) and the walking encoder, and includes the robot's position coordinates and attitude angles in space. This data is used to establish the spatial context of the device in the image and support the accurate projection of the prior position.

[0076] In this embodiment, the three types of heterogeneous data are strictly aligned using hardware-level timestamps to ensure consistency in the time dimension, that is, for any collection time... There exists a unique triple: ;

[0077] Step S2: Construct a priori library of power equipment structures. The priori library of power equipment structures is used to store the spatial positional relationships, geometric structural characteristics and component combination logic of power distribution line equipment. Based on the priori library of power equipment structures and robot pose data, a priori heat map is generated.

[0078] Specifically, existing segmentation models (such as U-Net and DeepLab) treat power distribution network equipment identification as a general image segmentation task, lacking modeling of the inherent laws of the power system, resulting in physically unreasonable segmentation results. To improve the physical rationality and engineering credibility of the segmentation results, this invention constructs a priori library of power equipment structures, systematically encodes installation specifications and geometric constraints such as "the clamp is located 1.0-1.5 meters below the insulator", "the insulator skirts are periodically axially symmetric", and "the angle between the guy wire and the ground is 45°±15°", and transforms them into learnable prior heatmaps, which are dynamically injected into the network decoding process as spatial biases, thereby forcing the segmentation model to segment within physically reasonable areas, improving the geometric consistency and engineering credibility of the results.

[0079] Specifically, the prior knowledge is derived from power industry standards (DL / T standards), equipment installation specifications, and expert experience. It is common engineering knowledge known to those skilled in the art. Spatial relationships are used to describe the relative installation positions of different devices. For example, the clamp should be installed below the insulator. At a distance of meters, where The value range is [1.0, 1.5] meters; geometric structural characteristics are used to describe the shape and symmetry of a single device, for example: "The sheds of the composite insulator are arranged in a periodic axisymmetric manner", "The angle between the guy wire and the ground is usually [30°-60°]"; component combination logic is used to describe the dependencies between devices, for example: "Suspension clamps must appear in pairs and symmetrically clamp the conductors on both sides"; the constructed power equipment structure prior library is shown in Tables 1, 2, and 3 below:

[0080] Table 1 - Schematic diagram of spatial location relationships

[0081]

[0082] Table 2 - Schematic diagram of geometric structure characteristics

[0083]

[0084] Table 3 - Component Combination Logic Diagram

[0085]

[0086] Step S3: Construct a priori-guided dual-branch multimodal segmentation network. The dual branches are two parallel visible light branch encoders and infrared branch encoders. A cross-modal attention fusion mechanism is introduced at the bottom layer of the dual-branch encoder. The prior heatmap is injected into the upsampling process of the dual-branch encoder in a multi-channel form, and the output is a multi-task result.

[0087] Step S4: Perform the discrimination operation according to the anomaly discrimination logic. If the anomaly is discerned, output the abnormal equipment and anomaly type of the power distribution line, and generate structured alarm information.

[0088] The anomaly detection logic specifically includes:

[0089] Insulator damage assessment based on the number of connected components and boundary roughness;

[0090] Hardware loosening and overheating judgment based on the average temperature of the overlapping area and the integrity of the component;

[0091] Anomaly detection of pull lines based on mask length, fitting angle, and bottom temperature;

[0092] Conductor anomaly detection based on skeleton continuity and sag height;

[0093] Specifically, since existing technologies can only output device location or masks and cannot automatically determine fine-grained anomalies such as "whether it is damaged, loose, or missing," this invention is based on a joint anomaly detection logic of multi-task output results. This logic directly transforms low-level pixel information into high-level operational semantics, supporting the automatic generation of structured alarms on the inspection robot. The structured alarm information is as follows: Figure 6 As shown in the example: if a fault is detected (such as an insulator crack), the output is: {Device: "Insulator", Abnormality: "Crack"}. If further confirmation is needed (such as obstruction), a re-inspection is performed (e.g., move back 0.3m and inspect again). If no abnormality is found, the inspection continues.

[0094] Among them, such as Figure 2 and Figure 4As shown, step S2 also includes: using a lightweight U-Net coarse segmentation network, with MobileNetV2 as the encoder backbone, to output a preliminary semantic segmentation mask for the device, so as to obtain the preliminary spatial location of the power distribution line equipment and optimize the spatial projection accuracy of the prior heat map.

[0095] Specifically, to obtain the preliminary spatial location of distribution network equipment to support the generation of prior heatmaps, this invention employs a lightweight U-Net coarse segmentation network. This network is based on the U-Net architecture, and its encoder part adopts the MobileNetV2 backbone network, which is divided into four stages, namely the inverted residual block stages 1 to 4 (MBConv Stage 1~4). MBConvStage 1~4 represent multiple inverted residual blocks (Mobile Inverted Bottleneck Convolution, MBConv) in the MobileNetV2 encoder. Inverted residual blocks 1~4 (MBConv1~MBConv4) are used to progressively extract mid-to-low-level features of the image, representing different levels in the corresponding network structure. Upsampling 1~4 (Upsample1~4) are upsampling at the corresponding levels, using bilinear interpolation upsampling (MBConv). Stages 1-4 output features at different scales. The decoder consists of three upsampling layers and skip connections, outputting a semantic segmentation mask. The input image is processed by a MobileNetV2 encoder to extract four levels of multi-scale features (16, 24, 32, and 96 channels respectively), corresponding to equipment structure information at different scales. The decoder concatenates the encoder features with bilinear upsampling. UpSamples 1-4 are the upsampling modules of the decoder, which magnify the features of each stage of the encoder to the corresponding resolution, gradually restoring the spatial resolution. Finally, through an independent 1×1 convolutional head OutConv, five types of equipment semantic segmentation masks are generated: insulators, hardware, wires, guy wires, and towers. The coarse segmentation results output by the lightweight U-Net coarse segmentation network are not fused with prior guidance, but can provide the initial spatial location and category information of the equipment, providing necessary support for subsequent prior heatmap generation and fine segmentation.

[0096] Among them, such as Figure 2 and Figure 3 As shown, the specific process for generating the prior heatmap in step S2 includes:

[0097] Based on robot pose data, the preliminary spatial position of power distribution line equipment is back-projected from image coordinates to world coordinates, and the expected world position is calculated based on prior knowledge in the power equipment structure prior library.

[0098] By reprojecting the expected world location onto the image coordinates, the expected distribution area or trajectory of the power distribution line equipment can be obtained.

[0099] For each type of device, an independent channel of linear Gaussian heatmap is generated based on the expected image coordinates. Each pixel value in the linear Gaussian heatmap represents the prior probability that a device of the corresponding category exists at that location, and finally a multi-channel prior heatmap is formed with the same number of device categories.

[0100] Specifically, based on the prior library of power equipment structures and the current robot pose data A priori thermogram was obtained. ,in, For the prior heat map, This represents the equipment category, and 5 represents the number of channels. Each channel represents a priori heatmap for the corresponding category. and These represent the height and width of the image, in pixels, respectively, and are the values ​​of each pixel in the heatmap. Indicates the coordinates in the image The prior probability of a device of a certain category exists at a given point, in terms of the pull line. For example, based on prior knowledge, we know that the guy wires extend from the top of the tower, are usually symmetrical, and have an angle of α∈[45°, 60°] with the ground. We extract the tower mask from the same image, take the centroid coordinates of the top 10% of the tower mask, and back-project them to world coordinates using robot pose data, camera intrinsics, and centroid coordinates. Based on the world coordinates and the range of angles with the ground, we calculate the expected world position and reproject the world position to the image coordinates to obtain the guy wire trajectory. For each guy wire trajectory, we generate a linear Gaussian heatmap, i.e., a priori heatmap. The coordinate system transformation method is: based on camera intrinsics and extrinsics (derived from robot pose data), we back-project the image coordinates to the world coordinate system through perspective transformation.

[0101] Specifically, the prior heatmap is essentially a spatial attention mask. Its high-response regions (such as the expected location of a clamp) enhance the feature activation values ​​of the corresponding pixels, thereby improving the segmentation confidence of physically reasonable regions, suppressing missegmentation of the background or unreasonable locations, and compensating for information loss caused by occlusion. In this embodiment, the prior heatmap, as a formal expression of prior knowledge, does not directly participate in loss calculation, but rather serves as a soft constraint signal to dynamically guide the spatial rationality of the segmentation results. Its application runs through the multi-scale feature fusion stage of the decoder, and the operation is implemented as follows:

[0102] Prior heat map After upsampling at each stage of the U-Net decoder, feature fusion is performed with the skip links from the encoder, let the th... The input to the level decoder is: ,in, For the first Input features of the level decoder Indicates the first The features of the level decoder are upsampled. For the first The output characteristics of the level encoder (transmitted via jumper links). For convolution operations, the prior heatmap is injected using an additive bias, represented as: ,in, To inject prior and post-fusion features, Indicates the a priori heatmap Adjusted to match encoder features Same size, For coefficients, Indicates the first Level encoder features Size information (as) (Target size of the operation);

[0103] Example 2

[0104] Among them, such as Figure 5 As shown, in order to fully utilize the complementarity of visible light and infrared modes synchronously collected by the power distribution line inspection robot, this invention also provides a priori-guided dual-branch multimodal segmentation network, namely the EPOWERSEG fine segmentation network. This network is designed with a dual-branch encoder structure and introduces a cross-modal attention fusion mechanism at the feature level, thereby realizing deep collaborative perception of "infrared-guided visible light". The segmentation network specifically includes an input module, a dual-branch encoder, a cross-modal attention fusion module, a priori heatmap injection module, and a segmentation head module.

[0105] The input module is used to input infrared and visible light images;

[0106] The dual-branch encoder employs a parallel architecture to process different modal input features separately. The visible light branch encoder uses EfficientNet-B0 as the backbone network, with the Spatial Transform Module (STM) and inverse residual blocks 1-4 (MBConv1-4) as its internal modules. The input is a visible light image. The output is a 4-level multi-scale feature map. The number of channels is [16, 24, 32, 96], used to capture the texture, edge, and structural details of the device. The infrared branch encoder uses MobileNetV2 as the backbone network, and the output module (ConvBlock), inverted residual structure module, etc. are all internal modules. The input is an infrared thermal image. The output is an infrared feature map. Used to address temperature anomalies on the surface of key coding equipment;

[0107] The Cross-Modal Fusion Block connects to the output of the dual-branch encoder and fuses features through a cross-modal attention fusion mechanism. Visible light features are used as queries, and infrared features are used as keys and values ​​to enhance the response intensity in infrared thermal anomaly regions.

[0108] Specifically, the specific expressions for the cross-modal attention fusion mechanism include:

[0109] ;

[0110] ;

[0111] ;

[0112] ;

[0113] ;

[0114] in, , , For learnable linear projection, Features after fusion This is the attention weight matrix. For querying Query, For key, Value This is the feature map output from the lowest level of the visible light branch encoder. This is the infrared feature map output by the infrared branch encoder. For feature dimension, This is the normalization function;

[0115] The prior heatmap injection module (upsampling UpSample1~4 pairs of fused features and simultaneously injecting prior heatmaps) renders the prior heatmaps in multi-channel format. In the upsampling process of the injection encoder, after each skip link and before the upsampling convolution, the prior heatmaps of the corresponding scale are used as additive bias fusion.

[0116] The segmentation head (output convolutional layers OutConv1~3) includes output layers that output the results of multiple tasks, and output semantic segmentation mask, occlusion mask and infrared anomaly mask respectively;

[0117] Example 3

[0118] like Figure 2 and Figure 6As shown, based on the multi-task results output by the segmentation network, this invention also constructs an end-to-end anomaly detection logic that requires no post-processing detector and no manual intervention. This anomaly detection logic directly parses the segmentation head output, realizing automatic mapping from pixel-level information to operation and maintenance-level semantics. The parameters and values ​​in the status diagram table are all derived from power industry standards (DL / T standards), equipment installation specifications, and expert experience, which are common engineering knowledge known to those skilled in the art. This invention also provides specific discrimination formulas and processes for the anomaly detection logic, as detailed below:

[0119] In this embodiment, the input for insulator damage detection is the insulator channel in the semantic segmentation mask. By calculating the number of connected components and boundary roughness To determine if an insulator is abnormal, consult the insulator status table, as shown in Table 4 below. The formula for calculating the number of connected components in the insulator damage assessment is:

[0120] ;

[0121] in, This is the binarized image, where 0 represents a non-insulator and 1 represents an insulator. Image pixel coordinates, For insulator channels in the semantic segmentation mask, insulator damage detection uses OpenCV's 8-neighborhood connected component labeling. , obtain the number of connected components The 8-neighborhood connectivity criterion states that if two pixels have the same value of 1 and their coordinates satisfy the 8-neighborhood relationship, they are considered to be the same connected region. This method can effectively avoid pseudo-connected region splitting caused by serrations or slight breaks at the edge of the insulator skirt, and improve the robustness of damage detection.

[0122] The formula for calculating boundary roughness is:

[0123] ;

[0124] in, For boundary roughness, The total length of the outer boundary of the connected region. This represents the total number of pixels within the connected region.

[0125] Table 4 - Insulator Status Diagram

[0126]

[0127] In this embodiment, the input for determining hardware loosening and overheating is the hardware channel in the semantic segmentation mask. and infrared anomaly mask By calculating the average temperature of the overlapping region To determine if there are any abnormalities in the hardware condition diagram table, which is shown in Table 5 below, the calculation formula for the average temperature of the overlapping area in the hardware loosening and overheating judgment includes:

[0128] ;

[0129] ;

[0130] in, This is the overlapping area between the hardware area and the infrared anomaly area. For the hardware channel in the semantic segmentation mask, For infrared anomaly mask, The average temperature of the overlapping region. For the original infrared thermal image in pixels Temperature value at that location;

[0131] Component integrity is determined as follows: if and Then it is determined that a component is missing, among which, A priori thermogram;

[0132] Table 5 - Hardware Condition Diagram

[0133]

[0134] In this embodiment, the input for pull-line anomaly detection is the pull-line channel in the semantic segmentation mask. By calculating the mask length Fitting angle and bottom temperature To determine if a pull-string status is abnormal, consult the pull-string status illustration table, as shown in Table 6 below. The formula for calculating the mask length in pull-string anomaly detection includes:

[0135] ;

[0136] ;

[0137] in, This is the binarized image; 0 represents no string, and 1 represents a string. For the drawn line channel in the semantic segmentation mask, For mask length, The function for calculating skeleton length is provided, and OpenCV's cv2.findContours is used to extract the main contour of the drawn lines for line anomaly detection.

[0138] The formula for calculating the fitting angle is:

[0139] ;

[0140] in, To fit the angle, , This represents the difference between the ordinate and abscissa of the drawn line profile. The arctangent function is used to obtain the fitting angle with the horizontal direction by fitting a straight line to the pull-out mask.

[0141] The bottom temperature is obtained by taking the bottom 10% of the area of ​​the pull-out mask region and calculating the average infrared temperature of that area.

[0142] Table 6 - Diagram of Wire Pulling Status

[0143]

[0144] In this embodiment, the input for wire anomaly detection is the wire channel in the semantic segmentation mask. By calculating the continuity of the skeleton With sag height The system queries the conductor status table to determine if there are any anomalies. The conductor status table is shown in Table 7 below. In the conductor anomaly detection, the skeleton continuity is determined by using cv2.ximgproc.thinning to skeletonize the feature map and simultaneously calculate the number of endpoints in the skeleton. The sag height is obtained by fitting the center curve of the conductor, obtaining the ordinate of the lowest point of the center curve, and converting it into the ground height by combining it with the robot pose data.

[0145] Table 7 - Schematic diagram of conductor status

[0146]

[0147] In summary, this invention constructs a recognition method specifically tailored for power distribution line inspection robots by using prior knowledge guidance, multimodal fusion, and end-to-end anomaly design. It effectively solves the inherent defects of existing technologies in terms of physical rationality, occlusion robustness, and state discriminability, and improves the recognition accuracy and robustness in complex scenarios such as occlusion, hidden faults, and multi-scale conditions. It is suitable for operation and maintenance scenarios with real working conditions such as visual occlusion, background interference, and complex lighting.

[0148] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for identifying power distribution line equipment based on prior guidance and multimodal fusion, characterized in that, Specifically, the following steps are included: Step S1: Collect heterogeneous data synchronously through several sensors mounted on the power distribution line inspection robot. The heterogeneous data includes visible light images, infrared thermal images, and robot pose data. The heterogeneous data is aligned by hardware-level timestamps. Step S2: Construct a priori library of power equipment structures. The priori library of power equipment structures is used to store the spatial positional relationships, geometric structural characteristics and component combination logic of power distribution line equipment. Based on the priori library of power equipment structures and robot pose data, a priori heat map is generated. Step S3: Construct a priori-guided dual-branch multimodal segmentation network. The dual branches are two parallel visible light branch encoders and infrared branch encoders. A cross-modal attention fusion mechanism is introduced at the bottom layer of the dual-branch encoder. The prior heatmap is injected into the upsampling process of the dual-branch encoder in a multi-channel form, and the output is a multi-task result. Step S4: Perform the discrimination operation according to the anomaly discrimination logic. If the anomaly is discerned, output the abnormal equipment and anomaly type of the power distribution line, and generate structured alarm information. The anomaly detection logic specifically includes: Insulator damage assessment based on the number of connected components and boundary roughness; Hardware loosening and overheating judgment based on the average temperature of the overlapping area and the integrity of the component; Anomaly detection of pull lines based on mask length, fitting angle, and bottom temperature; Conductor anomaly detection based on skeleton continuity and sag height; The process of generating the prior heatmap in step S2 specifically includes: Based on robot pose data, the preliminary spatial position of power distribution line equipment is back-projected from image coordinates to world coordinates, and the expected world position is calculated based on prior knowledge in the power equipment structure prior library. By reprojecting the expected world location onto the image coordinates, the expected distribution area or trajectory of the power distribution line equipment can be obtained. For each type of device, an independent channel of linear Gaussian heatmap is generated based on the expected image coordinates. Each pixel value in the linear Gaussian heatmap represents the prior probability that a device of the corresponding category exists at that location, thus forming a multi-channel prior heatmap consistent with the number of device categories. In step S3, the prior-guided bi-branch multimodal segmentation network includes an input module, a bi-branch encoder, a cross-modal attention fusion module, a prior heatmap injection module, and a segmentation head module. The input module is used to input infrared images and visible light images; The dual-branch encoder uses a parallel architecture to process different modal input features. The visible light branch encoder uses EfficientNet-B0 as the backbone network, takes a visible light image as input, and outputs a 4-level multi-scale feature map to capture the texture, edge and structural details of the device. The infrared branch encoder uses MobileNetV2 as the backbone network, takes an infrared thermal image as input, and outputs an infrared feature map to focus on encoding temperature anomaly areas on the surface of the device. The cross-modal attention fusion module is connected to the output of the dual-branch encoder. It fuses features through the cross-modal attention fusion mechanism, using visible light features as the query and infrared features as the key and value, to enhance the response intensity in the infrared thermal anomaly region. The prior heatmap injection module injects the prior heatmap into the encoder's upsampling process in a multi-channel form. After each skip link and before the upsampling convolution, the prior heatmaps of the corresponding scale are fused as additive biases. The segmentation head includes an output layer that outputs multi-task results, and outputs semantic segmentation mask, occlusion mask and infrared anomaly mask respectively; The specific expression of the cross-modal attention fusion mechanism includes: ； ； ； ； ； in, , , For learnable linear projection, Features after fusion Here is the attention weight matrix. For querying Query, For key, Value This is the feature map output from the lowest level of the visible light branch encoder. This is the infrared feature map output by the infrared branch encoder. For feature dimension, This is the normalization function.

2. The method for identifying power distribution line equipment based on prior guidance and multimodal fusion according to claim 1, characterized in that, Step S2 further includes: using a lightweight U-Net coarse segmentation network, which uses MobileNetV2 as the encoder backbone, to output a preliminary semantic segmentation mask for the device, thereby obtaining the preliminary spatial location of the power distribution line equipment and optimizing the spatial projection accuracy of the prior heat map.

3. The method for identifying power distribution line equipment based on prior guidance and multimodal fusion according to claim 1, characterized in that, The formula for calculating the number of connected components in the insulator damage assessment is as follows: ； in, This is the binarized image. Image pixel coordinates, For the insulator channel in the semantic segmentation mask, the insulator damage detection uses OpenCV's 8-neighborhood connected component labeling. , obtain the number of connected components ; The formula for calculating the boundary roughness is: ； in, For boundary roughness, The total length of the outer boundary of the connected region. This represents the total number of pixels within the connected region.

4. The method for identifying power distribution line equipment based on prior guidance and multimodal fusion according to claim 1, characterized in that, The formula for calculating the average temperature of the overlapping area in the hardware loosening and overheating detection includes: ; ； in, This is the overlapping area between the hardware area and the infrared anomaly area. For the hardware channel in the semantic segmentation mask, For infrared anomaly mask, The average temperature of the overlapping region. For the original infrared thermal image in pixels Temperature value at; The integrity of the component is determined as follows: if and Then it is determined that a component is missing, among which, This is a priori heat map.

5. The method for identifying power distribution line equipment based on prior guidance and multimodal fusion according to claim 1, characterized in that, The formula for calculating the mask length in the wire anomaly detection includes: ； ； in, This is the binarized image. For the drawn line channel in the semantic segmentation mask, For mask length, The skeleton length calculation function is used, and the line anomaly detection uses OpenCV's cv2.findContours to extract the main contour of the line; The formula for calculating the fitting angle is: ； in, To fit the angle, , This represents the difference between the ordinate and abscissa of the drawn line profile. It is the arctangent function; The bottom temperature is obtained by taking the bottom 10% of the area of ​​the pull-out mask region and calculating the average infrared temperature of that area.

6. The method for identifying power distribution line equipment based on prior guidance and multimodal fusion according to claim 1, characterized in that, In the conductor anomaly detection, the skeleton continuity is obtained by skeletonizing the feature map using cv2.ximgproc.thinning and simultaneously calculating the number of endpoints in the skeleton. The sag height is obtained by fitting the conductor center curve, obtaining the ordinate of the lowest point of the conductor center curve, and converting it into the ground height by combining it with the robot pose data.

7. The method for identifying power distribution line equipment based on prior guidance and multimodal fusion according to claim 1, characterized in that, The robot pose data in step S1 includes three-dimensional position coordinates, pitch angle, and yaw angle.

Images