Electric power facility intelligent operation and maintenance system and method based on multi-modal information fusion

Abstract

The invention discloses an electric power facility intelligent operation and maintenance system and method based on multi-modal information fusion, and relates to the technical field of electric power facility operation and maintenance, an unmanned aerial vehicle is used for obtaining visible light and infrared thermal imaging images of an insulator chain, the collected images are preprocessed, then fine segmentation is carried out on the images, and single-modal features are extracted. And then, based on the extracted insulator sheet features and the spatial adjacency matrix, constructing an insulator graph fused with multi-modal information, and fully mining internal association and spatial structure information among the multi-modal features by using a pre-trained graph neural network. And finally, accurately identifying the defect condition of the insulator sheet through a classifier to obtain a defect identification result of the electric power facility. Thus, the advantages of visible light and infrared thermal imaging can be effectively fused, intelligence, automation and high efficiency in the operation and maintenance process of electric power facilities are achieved, and powerful technical support is provided for safe and stable operation of an electric power system.

Description

Technical Field

[0001] The present application relates to the technical field of power facility operation and maintenance, and more specifically, to an intelligent power facility operation and maintenance system and method based on multimodal information fusion. Background Art

[0002] With the rapid development of smart grids, power facilities are becoming increasingly large and complex, placing higher demands on the intelligent and precise operation and maintenance of these facilities. The safe and stable operation of power facilities is directly related to the reliability and quality of the power grid. Defect detection of key components such as insulators and cable joints is a core part of operation and maintenance, and the level of technical expertise directly impacts the safety and economic efficiency of the power system. Traditional power facility operation and maintenance relies heavily on manual inspections or single sensor detection, which not only consumes significant manpower and material resources but also suffers from low detection efficiency, high missed detection rates, and poor real-time performance. This makes it difficult to adapt to the modern power grid's demand for rapid inspection and precise diagnosis of large-scale facilities.

[0003] Existing technologies primarily rely on manual visual inspections or environmental parameters collected by single sensors, ignoring the structural complexity and diverse manifestations of damage. Most rely on data collected at the time of inspections, making it difficult to capture dynamic changes and hidden defects during equipment operation. Furthermore, isolated analysis of single-modal information (such as visible light images or temperature) is susceptible to environmental influences and data noise, leading to significant misjudgments and missed detections, making it difficult to meet the high-precision, time-sensitive, and intelligent demands of modern power operation and maintenance.

[0004] Therefore, there is an urgent need for an intelligent operation and maintenance system and method for power facilities based on multimodal information fusion. Summary of the Invention

[0005] In order to solve the above technical problems, this application is proposed.

[0006] According to one aspect of the present application, a method for intelligent operation and maintenance of power facilities based on multimodal information fusion is provided, which includes: Obtain images of insulator strings taken by a visible light camera on a drone and thermal images of insulator strings taken by an infrared camera on a drone; Preprocessing the insulator string image and the insulator string thermal image to obtain a preprocessed insulator string image and a preprocessed insulator string thermal image; performing insulator component segmentation and single-modal feature extraction on the preprocessed insulator string image and the preprocessed insulator string thermal image respectively to obtain a list of visible light feature vectors of insulator segments, a list of infrared feature vectors of insulator segments, and a spatial adjacency matrix between insulator segments; constructing an insulator graph based on the list of visible light feature vectors of the insulator slices, the list of infrared feature vectors of the insulator slices, and the spatial adjacency matrix between the insulator slices; Inputting the insulator graph into a pre-trained graph neural network model to obtain a list of insulator slice node embedding vectors; Each insulator segment node embedding vector in the list of insulator segment node embedding vectors is input into a classifier to obtain a defect recognition result of each insulator segment.

[0007] According to another aspect of the present application, there is provided an intelligent operation and maintenance system for electric power facilities based on multimodal information fusion, comprising: an infrared image acquisition module, used to acquire an image of the insulator string taken by a visible light camera of the UAV and a thermal image of the insulator string taken by an infrared camera of the UAV; an image preprocessing module, configured to preprocess the insulator string image and the insulator string thermal image to obtain a preprocessed insulator string image and a preprocessed insulator string thermal image; a feature extraction and segmentation module, configured to perform insulator component segmentation and single-modal feature extraction on the preprocessed insulator string image and the preprocessed insulator string thermal image, respectively, to obtain a list of visible light feature vectors of insulator segments, a list of infrared feature vectors of insulator segments, and a spatial adjacency matrix between insulator segments; an insulator graph construction module, configured to construct an insulator graph based on the list of visible light feature vectors of the insulator slices, the list of infrared feature vectors of the insulator slices, and the spatial adjacency matrix between the insulator slices; A neural network inference module, configured to input the insulator graph into a pre-trained graph neural network model to obtain a list of insulator slice node embedding vectors; The defect recognition module is used to input each insulator segment node embedding vector in the list of insulator segment node embedding vectors into a classifier to obtain a defect recognition result of each insulator segment.

[0008] Compared with the existing technology, the present application provides an intelligent operation and maintenance system and method for power facilities based on multimodal information fusion, which uses drones to obtain visible light and infrared thermal imaging images of insulator strings, and pre-processes the collected images, and then finely segments them and extracts single-modal features. Then, based on the extracted insulator sheet features and the spatial adjacency matrix, an insulator graph that integrates multimodal information is constructed, and the pre-trained graph neural network is used to fully explore the intrinsic correlation and spatial structure information between multimodal features. Finally, the defect conditions of the insulator sheets are accurately identified through a classifier to obtain the defect identification results of the power facilities. In this way, the advantages of visible light and infrared thermal imaging can be effectively integrated to achieve intelligence, automation and efficiency in the operation and maintenance of power facilities, providing strong technical support for the safe and stable operation of the power system. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0010] Figure 1 This is a flowchart of an intelligent operation and maintenance method for power facilities based on multimodal information fusion according to an embodiment of the present application.

[0011] Figure 2 This is a data flow diagram of the intelligent operation and maintenance method of power facilities based on multimodal information fusion according to an embodiment of the present application.

[0012] Figure 3 This is a flowchart of sub-step S2 of the intelligent operation and maintenance method of power facilities based on multimodal information fusion according to an embodiment of the present application.

[0013] Figure 4 This is a flowchart of sub-step S3 of the intelligent operation and maintenance method of power facilities based on multimodal information fusion according to an embodiment of the present application.

[0014] Figure 5 This is a flowchart of sub-step S4 of the intelligent operation and maintenance method of power facilities based on multimodal information fusion according to an embodiment of the present application.

[0015] Figure 6 This is a block diagram of an intelligent operation and maintenance system for power facilities based on multimodal information fusion according to an embodiment of the present application. DETAILED DESCRIPTION

[0016] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. While the drawings illustrate certain embodiments of the present disclosure, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0017] In response to the problems in the above-mentioned background technology, this application proposes an intelligent operation and maintenance method for power facilities based on multimodal information fusion. Figure 1 This is a flowchart of an intelligent operation and maintenance method for power facilities based on multimodal information fusion according to an embodiment of the present application. Figure 2 Figure 1 is a data flow diagram of the intelligent operation and maintenance method of power facilities based on multimodal information fusion according to an embodiment of the present application. Figure 1 and Figure 2 As shown, the intelligent operation and maintenance method of electric power facilities based on multimodal information fusion includes the following steps: S1, acquiring an insulator string image taken by a visible light camera of a drone and a thermal image of the insulator string taken by an infrared camera of a drone; S2, preprocessing the insulator string image and the thermal image of the insulator string to obtain a preprocessed insulator string image and a preprocessed insulator string thermal image; S3, performing insulator component segmentation and single-modal feature extraction on the preprocessed insulator string image and the preprocessed thermal image of the insulator string respectively to obtain visible light features of the insulator piece. S4, constructing an insulator graph based on the list of visible light feature vectors of the insulator slices, the list of infrared feature vectors of the insulator slices and the spatial adjacency matrix between the insulator slices; S5, inputting the insulator graph into a pre-trained graph neural network model to obtain a list of insulator slice node embedding vectors; S6, inputting each insulator slice node embedding vector in the list of insulator slice node embedding vectors into a classifier respectively to obtain a defect recognition result for each insulator slice.

[0018] In the aforementioned intelligent operation and maintenance method for power facilities based on multimodal information fusion, step S1 involves acquiring an image of the insulator string captured by a drone's visible light camera and a thermal image of the insulator string captured by a drone's infrared camera. It should be understood that due to the low efficiency, high missed detection rate, and poor real-time performance of traditional manual inspections, they are difficult to adapt to the modern power grid's demand for rapid inspection and precise diagnosis of large-scale facilities. Therefore, the present application utilizes a drone to simultaneously acquire both images of the insulator string and thermal images of the insulator string. By using a drone equipped with a dual-modal camera, non-contact, large-scale inspections of transmission line insulator strings are achieved, significantly improving inspection coverage efficiency compared to manual inspections while avoiding the risks of personnel working at height. Specifically, the insulator string image can clearly reflect the physical defects of the insulator, such as damage and cracks, while the thermal image can capture temperature anomalies caused by corona and poor contact. The two modal data complement each other, effectively improving the comprehensiveness and accuracy of defect identification, providing high-quality, multi-dimensional raw data for subsequent preprocessing, and ensuring that the multimodal fusion algorithm can fully exploit the inherent correlations between the data.

[0019] In particular, in one possible embodiment, step S1 is implemented as follows: First, a drone platform with a flight time of at least two hours and a stable flight control system is selected. The visible light camera must be at least 20 megapixels and support autofocus. The infrared thermal imaging camera must have a temperature resolution of 0.1°C and a thermal image resolution of 320×240 or higher. The dual cameras are mounted using a mechanical bracket with parallel optical axes, ensuring a field of view overlap exceeding 90%. During commissioning, the infrared camera is calibrated with blackbody calibration at three temperatures: 25°C, 50°C, and 80°C, while the visible light camera is also calibrated for distortion. Second, an inspection route is planned based on the transmission line GIS coordinate data. Aerial survey software is used to generate an S-shaped reciprocating route. The drone is controlled to fly 5-10 meters from the insulator string at a speed of 3-5 meters per second. Automatic waypoint switching logic is configured for complex terrain. Before each tower inspection, the camera is triggered to perform a pre-capture shot to cover the entire insulator string. During data collection, the drone's automatic flight mode was activated, and the ground station synchronized the triggering of the two cameras. A capture rate of 2-3 frames per second was set, ensuring at least 30% overlap between adjacent images. Camera parameters were dynamically adjusted based on ambient lighting conditions. The visible light camera used an aperture of F8, a shutter speed of 1 / 1000 second, and an ISO of 100 on sunny days, and an aperture of F5.6, a shutter speed of 1 / 500 second, and an ISO of 200 on cloudy days. The infrared camera also enabled scene temperature adaptive gain. The two cameras were connected via a hardware trigger cable, ensuring a capture time difference of less than 10 milliseconds. Image metadata included GPS coordinates with an accuracy of ≤1 meter and a timestamp of ≤1 millisecond. Data was stored in dual backup format: JPEG images of the insulator string and radiometrically calibrated RAW thermal images were stored simultaneously on the drone's memory card and on the ground station terminal. Finally, the ground station previewed the dual-modality images in real time to check the clarity and overexposure of the insulator string images, as well as the temperature uniformity of the thermal images. Any anomalies triggered an immediate return to flight for retakes. After the task is completed, the original data is batch verified, and the image resolution, GPS coordinates and timestamp integrity are tested through scripts, and unqualified files are eliminated.

[0020] In the above-mentioned intelligent operation and maintenance method of power facilities based on multimodal information fusion, the step S2 pre-processes the insulator string image and the insulator string thermal image to obtain a pre-processed insulator string image and a pre-processed insulator string thermal image. It should be understood that since the insulator string image and the insulator string thermal image obtained by the drone often have problems such as noise interference, insufficient contrast, and inaccurate temperature data, the present application improves the data quality through targeted processing and provides standardized input for subsequent segmentation, feature extraction and other links. Specifically, denoising the insulator string image can eliminate the noise points generated by sensor noise or environmental interference during the shooting process, the enhancement processing can highlight key features such as the edge contour of the insulator, temperature calibration of the thermal image can convert the original thermal signal into an accurate temperature value, and pseudo-color mapping improves the recognizability of temperature abnormality areas through visual encoding, thereby establishing a reliable data foundation for multimodal feature fusion.

[0021] In particular, in one possible embodiment, Figure 3 FIG is a flowchart of sub-step S2 of the intelligent operation and maintenance method of power facilities based on multimodal information fusion according to an embodiment of the present application. Figure 3 As shown, the step S2 includes: S21, performing image denoising and image enhancement on the insulator string image to obtain the preprocessed insulator string image; S22, performing temperature calibration and pseudo-color mapping on the insulator string thermal image to obtain the preprocessed insulator string thermal image.

[0022] Specifically, step S21 performs image denoising and image enhancement on the insulator string image to obtain the preprocessed insulator string image. Specifically, image denoising requires suppressing random noise while preserving the edge details of the insulators, and image enhancement requires enhancing the grayscale difference between the insulator string and the background in the image, making physical defects easier for the algorithm to identify, thereby achieving a preprocessing transition from a noisy original image to a structured feature image. After denoising, the Gaussian noise and salt-and-pepper noise in the image are significantly reduced, and the contour edges of the insulator segments remain clear, avoiding misjudgments by the segmentation model due to noise interference. After image enhancement, the contrast between the texture details of the insulator string and the background is significantly improved, highlighting defect features in dark areas, such as subtle cracks covered by shadows, enabling subsequent target detection models to more accurately locate the insulator segment ROI area. Furthermore, the preprocessed image has a more uniform grayscale distribution, reducing feature extraction bias caused by uneven illumination and providing more consistent visual input for feature alignment and graph neural network analysis between the visible light and infrared modalities.

[0023] In particular, in one possible embodiment, step S21 is implemented as follows: First, a bilateral filtering algorithm is used for image denoising. By constructing a kernel that is the product of spatial distance weights and pixel value similarity weights, this algorithm smooths noise while preserving edges. In the specific implementation, the spatial domain standard deviation is set to 50 pixels to ensure noise suppression at the 3-5 pixel scale. The range standard deviation is set to 20 grayscale levels, so that when the grayscale difference between adjacent pixels exceeds 20, the weight decays rapidly, thereby preserving the step-like grayscale changes at the edges of the insulator segments. For an insulator string image with a resolution of 4000×3000, the filter value is calculated point by point using a 5×5 pixel window. After processing, the noise density is reduced from 2.3% to below 0.5%.

[0024] Secondly, image enhancement uses contrast-limited adaptive histogram equalization (AHE) technology. The image is divided into 16×16 pixel sub-blocks, and histogram equalization is performed on each sub-block with a contrast threshold of 40. To address backlit scenes, which are common in insulator string images, gamma correction is performed to increase overall brightness before contrast-limited adaptive histogram equalization, with γ=0.8. The cumulative distribution function is then calculated for each sub-block. When mapping pixel values ​​to a grayscale range of 0–255, gradients exceeding the threshold are clipped to prevent noise amplification.

[0025] Finally, quality control is performed. The peak signal-to-noise ratio (PSNR) of the pre- and post-processing images is calculated, requiring an increase from 25dB to over 32dB after denoising and a further increase to 35dB after enhancement. The structural similarity (SSIM) metric is then used, with SSIM ≥ 0.92 after denoising and ≥ 0.95 after enhancement, to ensure the preservation of image structural information. A visual inspection of the pre-processed insulator string images is then performed, focusing on residual noise at the interface between the insulator cap and the porcelain body, and the elimination of jagged artifacts at the shed edges. The pre-processed insulator string images are stored in PNG format, retaining 8-bit grayscale information. The spatial domain standard deviation and range standard deviation parameters of the bilateral filter, as well as the block size and contrast threshold of the CLAHE, are recorded in the file metadata to facilitate subsequent algorithm debugging and tracing.

[0026] Specifically, in step S22, the thermal image of the insulator string is temperature calibrated and pseudo-color mapped to obtain the pre-processed thermal image of the insulator string. That is, the thermal signal of the insulator string is converted into an accurate temperature value through temperature calibration, and the recognizability of the temperature distribution is enhanced through pseudo-color mapping. Specifically, after temperature calibration, the pixel value of the thermal image can accurately reflect the actual temperature of each part of the insulator, and the temperature measurement error is controlled within ±0.5°C to avoid misjudgment due to the nonlinear response of the sensor. Pseudo-color mapping uses color gradient changes to intuitively present temperature differences. The adjacent 2°C temperature difference is distinguishable in color, and the color contrast between the high-temperature area and the normal area is significantly enhanced, such as local overheating at the damaged insulator, so as to facilitate the subsequent convolutional network to extract temperature anomaly features.

[0027] In particular, in one possible embodiment, the implementation process of step S22 is as follows: First, temperature calibration is performed using the blackbody radiation source calibration method. The blackbody furnace is set to five calibration temperature points: -20°C, 0°C, 25°C, 50°C, and 80°C, with an accuracy of ±0.5°C. After each temperature point stabilizes, the drone is controlled to capture thermal images at a distance of 10 meters from the blackbody furnace, and the original digital value corresponding to each temperature point, i.e., the DN value, is recorded. The five sets of DN values ​​are fitted to the actual temperature data using a quadratic polynomial using the least squares method to obtain a temperature conversion model T(DN) = a × DN² + b × DN + c, where the coefficients a, b, and c are solved through matrix operations, and T(DN) is the actual temperature value, so that the fitting mean square error is less than 0.3°C.

[0028] Secondly, pseudo-color mapping uses the Jet color table encoding strategy shown in Table 1, dividing the temperature range into three visual intervals: low temperatures (≤25°C) are mapped to blue; medium temperatures (25-50°C) are mapped to blue to green; and high temperatures (≥50°C) are mapped to green to orange. During mapping, each pixel's temperature value, T, is normalized and linearly mapped to the [0, 1] interval. Color conversion is then performed by looking up the RGB values ​​in the Jet color table. Outliers with temperatures exceeding 100°C are uniformly mapped to white to highlight them. For example, in an insulator thermal defect, the pseudo-color mapping shows the hot area (65°C) in an orange-yellow color, creating a clear color boundary with the surrounding green area (50°C). Compared to the grayscale display of the original thermal image, defect recognition is improved by over 40%.

[0029] Table 1 Jet color table:

[0030] Finally, the quality control process includes quantitative verification and visual inspection. In terms of quantitative verification, 10 known temperature points are randomly selected for error verification. The temperature points can be selected from ambient temperature, normal operating temperature of the insulator, etc. The absolute temperature error is required to be ≤0.5°C, and the temperature gradient change is consistent with the actual physical law. In terms of visualization, check whether the temperature distribution of the insulator string in the pseudo-color image conforms to the principles of thermodynamics. For example, the temperature at the iron cap connection is slightly higher than that of the porcelain body, and whether the color transition in the high-temperature area is smooth to avoid color block jumps. The final pre-processed thermal image is stored in TIFF format. Each pixel retains the RGB color value and the calibrated temperature metadata, which facilitates subsequent visual analysis and quantitative temperature calculations. The blackbody calibration parameters and pseudo-color mapping interval are recorded in the file header to ensure data traceability.

[0031] In the above-mentioned intelligent operation and maintenance method of power facilities based on multimodal information fusion, the step S3 performs insulator component segmentation and single-modal feature extraction on the pre-processed insulator string image and the pre-processed insulator string thermal image, respectively, to obtain a list of visible light feature vectors of the insulator slices, a list of infrared feature vectors of the insulator slices, and a spatial adjacency matrix between insulator slices. It should be understandable that since the pre-processed insulator string thermal image is still pixel-level raw data, it cannot be directly used for subsequent defect identification and multimodal fusion analysis. Therefore, the present application performs insulator component segmentation on the pre-processed insulator string image and the pre-processed insulator string thermal image, accurately separates each insulator slice, and then extracts single-modal features to obtain detailed information under different modes, and at the same time calculates the spatial adjacency matrix between the insulator slices based on the insulator slice visible ROI mask. In this way, the lists of visible light feature vectors and infrared feature vectors of the insulator sheets are obtained, which enables the refined extraction of the features of the insulator sheets in different modes. Combined with the spatial adjacency matrix, the multimodal features of the insulator sheets are combined with their spatial structural relationships, providing structured data for the subsequent use of graph neural networks for feature fusion and defect identification.

[0032] In particular, in one possible embodiment, Figure 4 FIG is a flowchart of sub-step S3 of the intelligent operation and maintenance method of power facilities based on multimodal information fusion according to an embodiment of the present application. Figure 4As shown, step S3 includes: S31, performing image registration on the preprocessed insulator string image and the preprocessed insulator string thermal image; S32, inputting the preprocessed insulator string image into a target detection and instance segmentation model to obtain an insulator sheet visible ROI mask; S33, mapping the insulator sheet visible ROI mask to the registered preprocessed insulator string image and the preprocessed insulator string thermal image to obtain a list of insulator sheet visible ROI images and a list of insulator sheet infrared ROI images; S34, performing convolutional coding-based visual feature extraction on each insulator sheet visible ROI image in the list of insulator sheet visible ROI images and each insulator sheet infrared ROI image in the list of insulator sheet infrared ROI images to obtain a list of insulator sheet visible light feature vectors and a list of insulator sheet infrared feature vectors.

[0033] Specifically, step S31 performs image registration on the pre-processed insulator string image and the pre-processed insulator string thermal image. Specifically, image registration aligns the two types of pre-processed images in a spatial coordinate system through geometric transformation, so that the insulator pieces at the same physical position have a consistent pixel coordinate correspondence in the two modal images. Through image registration, it is possible to achieve precise alignment of the spatial positions of the pre-processed insulator string image and the pre-processed insulator string thermal image, and the geometric position deviation of the same insulator piece in the two types of images is controlled within the pixel-level accuracy range. This allows the visible light features and infrared features subsequently extracted based on the registered image to accurately correspond to the same physical object, avoiding the feature misassociation phenomenon caused by inter-modal position offset. In addition, the registered image provides an accurate spatial position relationship for the calculation of the spatial adjacency matrix between the insulator pieces, so that the adjacency matrix can truly reflect the structural connectivity of the insulator string, thereby improving the ability of the graph model to express the spatial structural information of the insulator.

[0034] In particular, in one possible embodiment, step S31 is implemented as follows: First, a scale-invariant feature transform algorithm is used to extract texture edge points, metal connector corners, and boundary points with significant temperature gradients from the preprocessed insulator string image, forming a cross-modal feature point set. Specifically, the scale-invariant feature transform (SIFT) algorithm is an algorithm that can stably extract local image features under different scales, rotations, and illumination conditions. It first performs Gaussian convolution on the image at different scales to generate a scale-space pyramid. Key points are located by detecting extreme points in the scale space, ensuring scale invariance. Based on the gradient direction distribution in the key point's neighborhood, a main direction is assigned to each key point, ensuring rotation invariance. Finally, a gradient direction histogram is calculated within the key point's neighborhood to generate a cross-modal feature point set containing position, scale, and direction information. Second, a fast nearest neighbor approximation algorithm is used for initial feature matching. Based on the structural prior of the linear arrangement of the insulator string, a random sampling consensus algorithm is used to remove outliers, ensuring that matching point pairs satisfy the spatial arrangement constraints of the insulator string. Next, based on the matched feature point pairs, an affine transformation (suitable for small perspective changes) or perspective transformation (suitable for large perspective distortion) model is selected. Affine transformation is suitable for small perspective changes, while perspective transformation is suitable for large perspective distortion. The transformation matrix is ​​solved using the least squares method to map the infrared image to the coordinate system of the visible light image. Bilinear interpolation or cubic spline interpolation is then used to perform a geometric transformation on the infrared image to eliminate missing pixels after coordinate mapping and ensure image smoothness. Finally, the registration accuracy is evaluated by calculating the root mean square error (RMS) and mutual information (MUT). The coordinate deviation at key locations of the insulator string must be less than 1 pixel.

[0035] Specifically, in step S32, the pre-processed insulator string image is input into a target detection and instance segmentation model to obtain a visible ROI mask of the insulator slice. In one possible embodiment, the target detection and instance segmentation model is a Mask R-CNN model. It should be understood that the ROI mask can be used to independently segment each insulator slice from the complex background, avoiding feature confusion between different insulator slices, and at the same time providing an accurate spatial positioning reference for mapping the corresponding area of ​​the thermal image of the insulator string. Through the instance segmentation processing of the Mask R-CNN model, a pixel-level accurate visible ROI mask of the insulator slice can be obtained. Specifically, the visible ROI mask of the insulator slice can be directly mapped to the corresponding area of ​​the infrared thermal image, ensuring that the spatial position of the same insulator slice in the two modalities is strictly aligned, so that the subsequent texture features and temperature features extracted based on the ROI can accurately correspond to the same insulator slice, providing a reliable regional division basis for multimodal fusion, and ultimately improving the accuracy and robustness of defect recognition.

[0036] In particular, in one possible embodiment, step S32 is implemented as follows: First, a training dataset suitable for insulator string detection is constructed. Images of insulator strings under different operating conditions and environments are collected. Each insulator segment is annotated with polygons using tools such as LabelMe, generating training samples containing category labels (e.g., normal segment, damaged segment) and pixel-level masks. To enhance model generalization, the dataset is augmented with data augmentation techniques such as rotation, scaling, brightness adjustment, and the addition of Gaussian noise. Ultimately, a training set consisting of at least 5,000 annotated images is constructed, with 20% of these images used as a validation set.

[0037] Next, the Mask R-CNN model architecture was built. ResNet-50 was selected as the backbone network to extract multi-level features from the insulator string image. The Region Proposal Network (RPN) generates candidate regions based on the feature maps output by the backbone network. Anchor boxes (using anchor boxes of varying scales and ratios, such as 1:2 and 1:3 for the insulator segments) were used to improve the recall of the candidate regions. The instance segmentation head employed an FCN (fully convolutional network) architecture, combining a classification branch with a bounding box regression branch to perform category prediction, bounding box optimization, and mask generation for each candidate region. The model was trained using an end-to-end multi-task learning strategy, with a loss function consisting of a classification loss (cross entropy), a bounding box regression loss (smoothed L1), and a mask loss (binary cross entropy). The network parameters were adjusted using the Adam optimizer. The initial learning rate was set to 0.001, decaying by a factor of 0.1 every 10 epochs, and training was performed until the loss function converged (total loss below 0.5).

[0038] During the inference phase, the preprocessed insulator string image is fed into the trained Mask R-CNN model. The backbone network first extracts feature maps. The RPN slides across the feature maps to generate approximately 2,000 candidate regions. Non-maximum suppression (NMS, with a threshold of 0.7) is then used to select approximately 100 high-quality candidate regions. Feature pooling (RoIAlign) is then performed on each candidate region, mapping regions of varying sizes to feature maps of a fixed size. These regions are then fed into the classification and segmentation branches, respectively. The classification branch outputs the probability that the candidate region belongs to an insulator segment (with a threshold of 0.8). The bounding box regression branch optimizes the region coordinates. The segmentation branch generates a binary mask with a resolution of 28×28. This mask is then upsampled to the original image size to obtain the final visible ROI mask for the insulator segment. To address the potential for overlapping insulator segments within an insulator string, the model uses pixel-by-pixel predictions from the mask branch to effectively distinguish the pixels in overlapping areas. For example, by leveraging prior knowledge of the insulator segment orientation (typically vertical), the model performs morphological operations (erosion and dilation) on the overlapping mask in post-processing to ensure the integrity of the ROI mask for each insulator segment. Finally, the generated ROI mask is saved as a binary image, with each mask corresponding to a specific insulator segment instance. This provides precise spatial region identification for subsequent mapping to the insulator string thermal image and feature extraction.

[0039] Specifically, step S33 maps the insulator segment visible ROI mask to the registered preprocessed insulator string image and preprocessed insulator string thermal image to obtain a list of insulator segment visible ROI images and a list of insulator segment infrared ROI images. In other words, by mapping the mask to the preprocessed insulator string image and preprocessed insulator string thermal image, precise instance-level alignment of multimodal data is achieved, ensuring strict correspondence between feature extraction regions for the same insulator segment in both modalities and avoiding feature misassociation caused by regional division deviations. Furthermore, generating independent ROI image lists can isolate individual insulator segments from a complex background, eliminating interference from adjacent components and environmental noise, allowing subsequent convolutional network-based feature extraction to better focus on the target object itself.

[0040] In particular, in one possible embodiment, step S33 is implemented as follows: First, the insulator segment visual ROI mask output by Mask R-CNN is binarized, with the target region pixel value set to 1 and the background pixel value set to 0. For overlapping regions, a 3×3 kernel erosion operation is performed, combining prior knowledge of the vertical arrangement of the insulator strings to optimize the overlapping boundaries, ensuring that each mask corresponds to a single insulator segment. Next, the preprocessed and registered insulator string image is pixel-wise multiplied with the binarized mask. Pixels with a mask value of 1 are retained, and the remaining pixels are set as background. The visual ROI image is extracted, and the instance number is recorded to correspond to the infrared ROI image. Subsequently, the processed visual ROI mask is mapped to the registered infrared thermal image, and the infrared ROI image is directly extracted by performing a pixel-by-pixel dot product with the same mask, preserving the temperature metadata and pseudo-color RGB values ​​of each pixel to avoid loss of temperature calibration data. To ensure ROI image consistency and processing efficiency, the extracted visible and infrared ROI images are standardized: images of varying sizes are uniformly scaled to 224×224 pixels using bilinear interpolation, keeping the insulator slice centered. The edges of the scaled images are filled with black or mean grayscale to make all ROI images consistent in size, forming a standardized list. Finally, the ROI image list quality is verified: a visual check is performed to ensure that the corresponding insulator slices are fully included, and to check whether there are any incomplete segmentation or background residues. The temperature distribution of the infrared ROI image is verified to comply with thermodynamic laws and to ensure that there are no abnormal jumps or omissions. After passing the verification, the visible and infrared ROI images are stored one-to-one according to the instance number to form structured data, providing standardized materials for subsequent single-modal feature extraction. The entire process is batch processed through automated scripts to meet the real-time requirements of large-scale operation and maintenance.

[0041] Specifically, step S34 performs convolutional coding-based visual feature extraction on each insulator-slice visible ROI image in the list of insulator-slice visible ROI images and each insulator-slice infrared ROI image in the list of insulator-slice infrared ROI images to obtain a list of visible light feature vectors of the insulator-slice and a list of infrared feature vectors of the insulator-slice, and calculates the spatial adjacency matrix between the insulator-slices based on the insulator-slice visible ROI mask. That is, fine-grained features of the insulator-slice in visible light and infrared modalities are extracted through convolutional coding, and the structural topological relationship of the insulator string is constructed based on the spatial adjacency matrix. Specifically, the feature extraction based on convolutional coding can capture texture detail features such as insulator damage and cracks from the insulator-slice visible ROI images, and extract temperature anomaly distribution features from the insulator-slice infrared ROI images. The generated insulator-slice visible light feature vectors and insulator-slice infrared feature vectors have unified dimensions and can effectively characterize the modal specificity of different defect types. The spatial adjacency matrix between insulator segments accurately reflects the series structure of the insulator string. The element value of the adjacency matrix corresponds to whether the insulator segments are adjacent, so that the graphical model can retain the physical arrangement rules of the insulator string.

[0042] In particular, in one possible embodiment, step S34 is implemented as follows: When constructing a convolutional neural network architecture for insulator segment feature extraction, a pre-trained ResNet-50 is selected as the backbone network, taking into account the multi-scale characteristics of insulator defects. Residual connections are used to address the training degradation problem of deep networks. The visible light feature extraction branch adjusts the ResNet-50 input layer to receive a 224×224×3 standardized visual ROI image. The first four residual blocks are retained to extract mid- and low-level features. After removing the classification layer, global average pooling and 1×1 convolution are performed for dimensionality reduction to obtain a 512-dimensional visible light feature vector. Furthermore, the infrared feature extraction branch adopts a dual-channel input strategy, with one input channel receiving a 224×224×3 pseudo-color RGB image and the other inputting a temperature-normalized 224×224×1 single-channel image. The pseudo-color branch uses a pre-trained ResNet-50 with the same structure as the visible light branch to extract visual features of the temperature distribution. The temperature branch uses a lightweight CNN consisting of three 3×3 convolutional blocks, followed by global average pooling and a fully connected layer, outputting a 256-dimensional temperature feature vector. The 512-dimensional features of the pseudo-color branch and the 256-dimensional features of the temperature branch are concatenated to form a 768-dimensional infrared feature vector. Before feature extraction, the visible light ROI image is normalized according to the ImageNet standard, the infrared pseudo-color image is normalized with the same parameters, and the single-channel temperature image is normalized to [0, 1]. After preprocessing, these images are batched and fed into the network for feature extraction. To enhance robustness, dropout is applied to the infrared temperature branch with a ratio of 0.3. Gradient visualization is used to verify defect responses in the visible light features. Transfer learning is used to fine-tune the last two residual blocks of the ResNet-50 for weakly responsive defects. Finally, a 512-dimensional visible light feature vector and a 768-dimensional infrared feature vector are obtained for the insulator segments. These are stored by instance number and their spatial location information is recorded, providing a basis for constructing the insulator map. Feature extraction is accelerated by GPU batch processing, with a single image processing time of ≤50ms, meeting the needs of large-scale inspections. When calculating the spatial adjacency matrix, based on the geometric position information of the insulator segment visible ROI mask, the Euclidean distance between the centers of the minimum circumscribed rectangle of each pair of insulator segments is first calculated. If the distance is less than 1.2 times the average spacing between adjacent insulator segments, and a benchmark value determined by historical data statistics is used, the segments are considered adjacent. A mask pixel overlap threshold is also introduced: if the overlap exceeds 5%, the segments are also considered adjacent to handle tilted or partially overlapping insulator segments. This ultimately generates an N×N adjacency matrix, where N is the number of insulator segments and the matrix element value is 1 or 0, indicating whether the corresponding insulator segment is adjacent, accurately reflecting the physical connection structure of the insulator string.

[0043] In the above-mentioned intelligent operation and maintenance method for power facilities based on multimodal information fusion, step S4 constructs an insulator graph based on the list of visible light feature vectors of the insulator segments, the list of infrared feature vectors of the insulator segments, and the spatial adjacency matrix between the insulator segments. It should be understood that traditional single-modal feature extraction ignores the spatial arrangement patterns of insulator strings, such as the defect propagation effect of adjacent insulator segments, and simple feature splicing cannot explicitly express structural associations. By constructing an insulator graph, each insulator segment can be used as a graph node and the physical adjacency relationship as a graph edge, so that the data has both multi-dimensional feature expression and topological structure information, thereby solving the problem of isolated feature analysis and lack of structural correlation in traditional methods, and providing a structured data foundation for graph neural networks to mine the intrinsic dependencies of cross-modal features, such as the correlation pattern between morphological defects and temperature anomalies, and spatial propagation patterns, such as the defect impact of adjacent insulator segments.

[0044] In particular, in one possible embodiment, Figure 5 FIG4 is a flowchart of sub-step S4 of the intelligent operation and maintenance method of power facilities based on multimodal information fusion according to an embodiment of the present application. Figure 5 As shown, the step S4 includes: S41, fusing each group of corresponding insulator-slice visible light feature vectors and insulator-slice infrared feature vectors in the list of the insulator-slice visible light feature vectors and the list of the insulator-slice infrared feature vectors to obtain an insulator-slice multimodal feature encoding vector as an insulator-slice node of the insulator graph; S42, adding edges between adjacent insulator-slice nodes based on the spatial adjacency matrix between the insulator-slices, wherein the edges are structural connection relationships between adjacent insulator-slices.

[0045] Specifically, the step S41 fuses each corresponding group of insulator sheet visible light feature vectors and insulator sheet infrared feature vectors in the list of the insulator sheet visible light feature vectors and the insulator sheet infrared feature vectors to obtain an insulator sheet multimodal feature encoding vector as the insulator sheet node of the insulator graph. Specifically, the visible light feature vector captures the physical defects of the insulator, such as cracks and damage, and the infrared feature vector reflects temperature anomalies, such as local overheating caused by corona. Independent analysis of the two is prone to missing cross-modal correlation information, such as the coordinated appearance of morphological damage and temperature anomalies in a certain area. Through the fusion operation, the two types of features can be mapped to the same semantic space, so that the graph nodes have both morphological and temperature attributes, providing a basis for the graph neural network to mine the intrinsic dependencies between multimodal features, such as the correlation between the degree of damage and temperature increase, thereby solving the technical problems of incomplete defect identification and insufficient cross-modal correlation mining in traditional single-modal analysis, and realizing the upgrade from single feature analysis to cross-modal collaborative representation.

[0046] In particular, in one possible embodiment, the implementation process of step S41 is as follows: First, the 512-dimensional visible light feature vector of the insulator sheet and the 768-dimensional infrared feature vector of the insulator sheet are concatenated to generate a 1280-dimensional feature fusion vector. In order to balance the modal weights, a learnable weight matrix is ​​introduced. and ,pass To achieve weighted fusion, is the feature fusion vector, is the weight matrix of the visible light characteristics of the insulator sheet, is the visible light eigenvector of the insulator sheet, is the weight matrix of the infrared characteristics of the insulator sheet, is the infrared feature vector of the insulator sheet, and the weight is initialized to the unit matrix. Secondly, the feature fusion vector is L2 normalized to eliminate the scale difference of different modes. The formula is ,in is the normalized feature fusion vector, To calculate the L2 norm. Then, the cross-modal attention mechanism is introduced through Calculate the attention weight, where is the cross-modal attention weight, is the sigmoid activation function, is the weight matrix of the attention mechanism, is the bias term. Then we weight the feature fusion vector to get ,in The multimodal features of the insulator segments are encoded into vectors to strengthen the association of complementary features. To optimize computational efficiency, principal component analysis is used to reduce the 1280-dimensional features to 512 dimensions, retaining 95% of the variance. Whitening is then performed to eliminate feature correlation. Finally, t-SNE visualization verifies the distribution of the fused features, ensuring that normal and defective samples form independent clusters. Gradient backpropagation verifies the contribution of each modality, ultimately generating an N × 512-dimensional multimodal feature matrix as input to the insulator graph nodes.

[0047] Specifically, step S42 adds edges between adjacent insulator segment nodes based on the spatial adjacency matrix between insulator segments. These edges represent the structural connections between adjacent insulator segments. It should be understood that since an insulator string is a linear series structure, defects in adjacent insulator segments may have synergistic effects. For example, overheating in one segment may be transmitted to adjacent segments. Therefore, by adding edges based on the spatial adjacency matrix, this physical adjacency relationship is converted into a graph message passing path, enabling the model to optimize defect detection at the current node using the feature information of adjacent nodes. The edge structure of the insulator graph after adding edges strictly corresponds to the physical connections between the insulator segments, allowing the graph neural network to capture the spatial propagation effects of defects through adjacent node feature aggregation (such as the attention mechanism of a graph attention network). For example, if the damage of an insulator segment is accompanied by a temperature anomaly in an adjacent segment, the model can link the features of these two segments through edge transfer, avoiding missed detections in single-node analysis. Furthermore, the spatial prior knowledge introduced by the edge structure (such as linear arrangement patterns) can constrain the learning process of the graph neural network, reducing false positives caused by feature noise and improving the robustness of defect detection in complex scenarios. After this processing step, the structural representation capability of the insulator graph enables the model to improve the accuracy of collaborative defect recognition of adjacent insulator segments by about 29%, especially in locating defects in the middle area of ​​long strings of insulators.

[0048] In particular, in one possible embodiment, the implementation process of step S42 is as follows: First, based on the calculated N×N-dimensional spatial adjacency matrix between insulator pieces, all non-zero element coordinates are extracted to generate an undirected edge list. Since the insulator string is a linear structure, there is only one adjacent edge between the first and last nodes, so the edge list size is 2N-2. In order to clarify the bidirectional connectivity of the edge, a reverse edge is added to each undirected edge to form a directed edge index matrix with a dimension of 2×M, where M is the number of directed edges. Secondly, structural features are assigned to the edge, and the Euclidean distance between the centroids of the two insulator pieces connected by each edge is calculated and normalized to the interval [0,1] as the spatial distance feature of the edge. It is spliced ​​with the edge weight to form a 2-dimensional edge feature vector and stored in the edge feature matrix. Then, the Data class of PyTorchGeometric is used to construct the graph data structure, integrate the node feature matrix, edge index matrix, and edge feature matrix, and add metadata such as the node position index. Finally, the edge structure is optimized and the normalized Laplacian matrix is ​​calculated, which serves as the propagation operator in the graph neural network to improve message transmission efficiency. For long insulator strings, neighborhood sampling techniques are used to reduce computational complexity, such as retaining 50 adjacent nodes per layer to ensure model training efficiency. The resulting graph data strictly corresponds to the physical structure of the insulator string, providing structured input for subsequent graph neural network mining of spatial correlation features.

[0049] In the above-mentioned intelligent operation and maintenance method of power facilities based on multimodal information fusion, in step S5, the insulator graph is input into a pre-trained graph neural network model to obtain a list of insulator slice node embedding vectors. In a possible embodiment, the pre-trained graph neural network model is a Graph Attention Network. Specifically, the graph neural network model calculates the importance of each node feature through adaptive attention weights, strengthens defect-related features and suppresses noise, such as the association between crack texture and overheating temperature, and at the same time encodes the physical series structure of the insulator string based on the adjacency matrix, so that the model can learn the mutual influence between adjacent insulator slices, such as the temperature conduction effect. The pre-trained graph neural network model can map the original multimodal features to a high-dimensional semantic space, providing a more discriminative defect representation for the subsequent classifier.

[0050] In the aforementioned intelligent operation and maintenance method for power facilities based on multimodal information fusion, step S6 involves inputting each insulator segment node embedding vector in the list of insulator segment node embedding vectors into a classifier to obtain defect identification results for each insulator segment. Specifically, the insulator segment node embedding vectors extracted by the graph neural network (which fuses multimodal features and spatial structure information) are converted into specific defect category labels, achieving a semantic mapping from feature representation to defect diagnosis.

[0051] In particular, in one possible embodiment, step S6 is implemented as follows: First, a three-layer fully connected neural network is used as the classifier. The input layer dimension is consistent with the node embedding vector (e.g., 512 dimensions), the hidden layer dimensions are 256 and 128 dimensions, respectively, and the output layer dimension is the number of defect categories (e.g., four categories: normal, damaged, overheated, and combined defects). A ReLU activation function is used between the input and hidden layers, and a Softmax activation is used in the output layer to obtain the probability distribution of each category. During the training phase, a cross-entropy loss function and the Adam optimizer are used (with an initial learning rate of 0.001, decaying by a factor of 0.5 every 10 epochs). The batch size is set to 32, and training is performed until the validation set loss converges (the loss value is less than 0.3).

[0052] First, a 256-dimensional feature vector is generated through the first fully connected layer. This feature vector is activated by ReLU and then fed into the second fully connected layer. ReLU activation is performed again to obtain a 128-dimensional feature vector. Finally, a 4-dimensional probability vector is generated through the output layer. The category corresponding to the maximum probability is the defect type. To improve generalization, a dropout layer (ratio 0.3) is added after the hidden layer to prevent overfitting. If an insulator segment is classified as "overheated," the classification results of its neighboring nodes are checked. If none of the neighboring nodes have temperature anomalies and the temperature gradient in the infrared signature of the node does not exceed the threshold, a secondary verification mechanism is triggered (raising the classification probability threshold to 0.85) to avoid misclassification due to infrared noise. The final defect recognition output includes the category label, confidence level, and the spatial location index of the corresponding insulator segment. It is stored in a structured data format such as JSON for easy integration into power operation and maintenance management systems.

[0053] Specifically, for the set of insulator segment node embedding vectors obtained by inputting the pre-trained graph neural network model into the insulator graph, due to the spatial adjacency topological associations represented by the graph neural network for the node feature vectors, the set of insulator segment node embedding vectors becomes a set of graph structure primitive patterns related to the topological edge connection structure. However, because the node representation corresponding to the spatial adjacency matrix incorporates image visual features from different visual dimensions, namely the visible light dimension and the infrared dimension, this results in non-uniform topological edge connections in spatial adjacency. For example, the association between a pair of adjacent insulators may be primarily defined by shadow or contamination continuity in the visible light image, while the association between another pair may be dominated by a subtle temperature gradient in the infrared image. This is further amplified by the graph structure, resulting in a weakened overall connection between the primitive patterns, affecting the consistency and accuracy of defect recognition results for each insulator segment obtained by inputting the node embedding vectors into the classifier. For example, the embedding vector of a normal insulator that appears to be overheated due to direct sunlight may be inappropriately pulled towards the feature space of a true overheating defect, resulting in misjudgment.

[0054] In one embodiment, each insulator segment node embedding vector in the list of insulator segment node embedding vectors is input into a classifier to obtain a defect recognition result of each insulator segment, including: Using each insulator slice node embedding vector in the list of insulator slice node embedding vectors as a row vector, construct an insulator slice node embedding matrix , and calculate the variational field value of each row vector in the insulator sheet node embedding matrix, expressed as: ,in, Indicates that the insulator node is embedded in the matrix row vectors and the row vectors, represents the two-norm of the vector, Indicates the The variational field value of the row vector.

[0055] First, the node embedding vectors of all insulator segments are organized into an insulator segment node embedding matrix, which provides a convenient mathematical form for subsequent global analysis. Next, the variational field value of each row vector in the insulator segment node embedding matrix—that is, each insulator segment embedding vector—is calculated. This variational field value quantifies the uniqueness or degree of discreteness of each insulator segment embedding vector in the entire high-dimensional feature space. For example, a vector with an extremely high variational field value often corresponds to an insulator segment with the most extreme state or the most significant characteristics. For example, a severely damaged insulator with severe discharge will be far away from all other normal or slightly defective counterparts in the feature space.

[0056] Then, the maximum value of the variational field is selected The corresponding insulator sheet node embedding vector As the anti-weakening connection vector, and based on the anti-weakening connection vector, the insulator piece node embedding matrix Perform inversion mapping to obtain the central stability constraint vector, which is expressed as: ;in, represents the insulator node embedding vector corresponding to the maximum variational field value, represents matrix multiplication, represents the insulator sheet node embedding matrix, represents the central stability constraint vector.

[0057] It should be understood that in order to accurately locate the anchor point that best represents the abnormal state, the embedding vector with the largest variational field value is selected as the anti-weakening connection vector. This vector is considered the key to breaking the current weak connection state and reestablishing a stable association. Based on this, a central stability constraint vector is generated by inverting the anti-weakening connection vector with the insulator node embedding matrix. In this way, a global stability benchmark centered on the most significant abnormal point is established, providing direction for subsequent adjustments.

[0058] Next, the variational field values ​​of the row vectors are arranged into variational field representation vectors , and embed the insulator node matrix based on the variational field representation vector Perform variational cluster inversion mapping to obtain the approximate variational field cluster distribution vector, thereby obtaining the approximate variational field cluster distribution in the graph structure topological space, which is expressed as: ;in, represents the variational field representation vector, represents matrix multiplication, represents the insulator sheet node embedding matrix, Represents the approximate variational field cluster distribution vector.

[0059] At the same time, to understand the overall distribution of the current feature space, the variational field values ​​of the row vectors are arranged into a variational field representation vector, which is used to perform a variational cluster inversion mapping on the insulator slice node embedding matrix, thereby obtaining an approximate variational field cluster distribution vector. This depicts how individual insulator slices spontaneously form clusters based on their feature similarities within the current topological space, specifically which insulators are characteristically close to each other. This reflects the initial, but potentially biased, pattern clustering results learned by the graph neural network.

[0060] Then, based on the approximate variational field cluster distribution vector and the central stability constraint vector, the insulator sheet node embedding matrix is ​​subjected to variational cluster density propagation derivation to obtain a propagation derivation correction vector, which is expressed as: ;in, represents the central stability constraint vector, represents the approximate variational field cluster distribution vector, represents matrix multiplication, represents the insulator sheet node embedding matrix, Represents vector subtraction represents the propagation-derived correction vector.

[0061] This method uses a density backpropagation feedback mechanism involving a set of graph structure primitive patterns containing spatially adjacent topological associations to use the variationally stable primitive connection domain within the high-dimensional graph structure topological space as the pattern connection stabilization point for density propagation of variational clusters under central stability constraints. Specifically, the most significant outlier (central stability constraint) is used as the source to propagate influence throughout the feature space (approximate variational field clusters). Based on the density and position of each cluster, the precise correction to be applied to each insulator segment embedding vector is calculated. In this way, vectors that are mistakenly pulled closer due to noise or non-uniform information sources are pushed apart, while vectors that should belong to the same class but are pulled apart are brought closer together, thereby reshaping the topological structure of the entire feature space.

[0062] Next, the propagation-derived correction vector is used to derive and optimize each insulator-slice node embedding vector in the list of insulator-slice node embedding vectors to obtain an optimized list of insulator-slice node embedding vectors, which is expressed as: ;in, represents the propagation-derived correction vector, Indicates that the insulator node is embedded in the matrix row vector, i.e., the first row in the list of embedding vectors of the insulator slice nodes. Insulator slice node embedding vectors, Indicates point multiplication by position, Indicates the An optimized insulator slice node embedding vector.

[0063] In this way, the propagation-derived correction vector that condenses the global optimization information is used to derive and optimize each original insulator slice node embedding vector one by one. The list of optimized insulator slice node embedding vectors has significantly enhanced internal consistency and separability between different categories. A normal insulator that is overheated due to sunlight reflection will have its vector pulled back to the normal cluster due to the stability constraints of a large number of normal companions around it; an insulator with a hidden crack will have its vector pushed more clearly to the defect cluster due to density propagation. In other words, the above topological variational transformation expression can be used to improve the multi-dimensional anti-weakening coupling and hybrid primitive mode central connection aggregation of the topological space variation cluster based on the accuracy of density propagation derivation, thereby improving the consistency accuracy between the defect identification results of each insulator slice obtained by inputting the node embedding vector of each insulator slice into the classifier.

[0064] Finally, each list of optimized insulator segment node embedding vectors in the list of optimized insulator segment node embedding vectors is input into the classifier respectively to obtain the defect recognition result of each insulator segment, thereby ensuring that the defect recognition result of each insulator segment is obtained after fully considering its own multimodal information, series structure relationship and global state consistency, thereby greatly improving the comprehensive accuracy and reliability of the defect diagnosis of the entire string of insulators.

[0065] In summary, the intelligent operation and maintenance method of power facilities based on multimodal information fusion based on the embodiment of the present application is explained, which uses a drone to obtain visible light and infrared thermal imaging images of the insulator string, and pre-processes the acquired images, and then finely segments them and extracts single-modal features. Then, based on the extracted insulator sheet features and the spatial adjacency matrix, an insulator graph that integrates multimodal information is constructed, and the pre-trained graph neural network is used to fully explore the intrinsic correlation and spatial structure information between multimodal features. Finally, the defect conditions of the insulator sheet are accurately identified by the classifier to obtain the defect identification results of the power facility. In this way, the advantages of visible light and infrared thermal imaging can be effectively integrated to realize the intelligence, automation and efficiency of the operation and maintenance process of power facilities, providing strong technical support for the safe and stable operation of the power system.

[0066] Figure 6 FIG is a block diagram of an intelligent operation and maintenance system for electric power facilities based on multimodal information fusion according to an embodiment of the present application. Figure 6As shown, according to an embodiment of the present application, an intelligent operation and maintenance system 100 for electric power facilities based on multimodal information fusion includes: an infrared image acquisition module 110, for acquiring an insulator string image taken by a visible light camera of a drone and a thermal image of an insulator string taken by an infrared camera of a drone; an image preprocessing module 120, for preprocessing the insulator string image and the thermal image of the insulator string to obtain a preprocessed insulator string image and a preprocessed insulator string thermal image; a feature extraction and segmentation module 130, for performing insulator component segmentation and single-modal feature extraction on the preprocessed insulator string image and the preprocessed insulator string thermal image respectively to obtain an insulator component image. a list of visible light feature vectors of insulator sheets, a list of infrared feature vectors of insulator sheets and a spatial adjacency matrix between insulator sheets; an insulator graph construction module 140, for constructing an insulator graph based on the list of visible light feature vectors of insulator sheets, the list of infrared feature vectors of insulator sheets and the spatial adjacency matrix between insulator sheets; a neural network inference module 150, for inputting the insulator graph into a pre-trained graph neural network model to obtain a list of insulator sheet node embedding vectors; a defect recognition module 160, for inputting each insulator sheet node embedding vector in the list of insulator sheet node embedding vectors into a classifier respectively to obtain a defect recognition result for each insulator sheet.

[0067] As described above, the intelligent operation and maintenance system 100 for electric power facilities based on multimodal information fusion according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with an equipment asset operation and maintenance management algorithm based on intelligent networking. In one possible implementation, the intelligent operation and maintenance system 100 for electric power facilities based on multimodal information fusion according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or hardware module. For example, the intelligent operation and maintenance system 100 for electric power facilities based on multimodal information fusion can be a software module in the operating system of the wireless terminal, or can be an application developed for the wireless terminal; of course, the intelligent operation and maintenance system 100 for electric power facilities based on multimodal information fusion can also be one of the many hardware modules of the wireless terminal.

[0068] Alternatively, in another example, the intelligent operation and maintenance system 100 of electric power facilities based on multimodal information fusion and the wireless terminal may also be separate devices, and the intelligent operation and maintenance system 100 of electric power facilities based on multimodal information fusion may be connected to the wireless terminal through a wired and / or wireless network, and transmit interactive information in accordance with an agreed data format.

[0069] Here, those skilled in the art will understand that the specific operations of each step in the above-mentioned intelligent operation and maintenance system for power facilities based on multimodal information fusion have been referred to above. Figures 1 to 5It has been introduced in detail in the description of the intelligent operation and maintenance method of power facilities based on multimodal information fusion, and therefore, its repeated description will be omitted.

Claims

1. A method for intelligent operation and maintenance of power facilities based on multimodal information fusion, characterized in that: include: Obtain images of insulator strings taken by a visible light camera on a drone and thermal images of insulator strings taken by an infrared camera on a drone; Preprocessing the insulator string image and the insulator string thermal image to obtain a preprocessed insulator string image and a preprocessed insulator string thermal image; performing insulator component segmentation and single-modal feature extraction on the preprocessed insulator string image and the preprocessed insulator string thermal image respectively to obtain a list of visible light feature vectors of insulator segments, a list of infrared feature vectors of insulator segments, and a spatial adjacency matrix between insulator segments; constructing an insulator graph based on the list of visible light feature vectors of the insulator slices, the list of infrared feature vectors of the insulator slices, and the spatial adjacency matrix between the insulator slices; Inputting the insulator graph into a pre-trained graph neural network model to obtain a list of insulator slice node embedding vectors; Each insulator segment node embedding vector in the list of insulator segment node embedding vectors is input into a classifier to obtain a defect recognition result of each insulator segment.

2. The intelligent operation and maintenance method for electric power facilities based on multimodal information fusion according to claim 1 is characterized in that: Preprocessing the insulator string image and the insulator string thermal image to obtain a preprocessed insulator string image and a preprocessed insulator string thermal image, including: performing image denoising and image enhancement on the insulator string image to obtain the preprocessed insulator string image; Temperature calibration and pseudo-color mapping are performed on the thermal image of the insulator string to obtain the pre-processed thermal image of the insulator string.

3. The intelligent operation and maintenance method for electric power facilities based on multimodal information fusion according to claim 1 is characterized in that: Performing insulator component segmentation and single-modal feature extraction on the preprocessed insulator string image and the preprocessed insulator string thermal image respectively to obtain a list of visible light feature vectors of insulator segments, a list of infrared feature vectors of insulator segments, and a spatial adjacency matrix between insulator segments, including: performing image registration on the preprocessed insulator string image and the preprocessed insulator string thermal image; Inputting the preprocessed insulator string image into an object detection and instance segmentation model to obtain an insulator segment visual ROI mask; Mapping the insulator segment visible ROI mask to the registered pre-processed insulator string image and the pre-processed insulator string thermal image to obtain a list of insulator segment visible ROI images and a list of insulator segment infrared ROI images; Visual feature extraction based on convolution coding is performed on each insulator sheet visible ROI image in the list of the insulator sheet visible ROI images and each insulator sheet infrared ROI image in the list of the insulator sheet infrared ROI images to obtain a list of the insulator sheet visible light feature vectors and a list of the insulator sheet infrared feature vectors.

4. The intelligent operation and maintenance method for electric power facilities based on multimodal information fusion according to claim 3 is characterized in that: performing insulator component segmentation and single-modal feature extraction on the preprocessed insulator string image and the preprocessed insulator string thermal image respectively to obtain a list of visible light feature vectors of insulator segments, a list of infrared feature vectors of insulator segments, and a spatial adjacency matrix between insulator segments, further comprising: Based on the insulator slice visible ROI mask, the spatial adjacency matrix between the insulator slices is calculated.

5. The intelligent operation and maintenance method for electric power facilities based on multimodal information fusion according to claim 3 is characterized in that: The object detection and instance segmentation model is a Mask R-CNN model.

6. The intelligent operation and maintenance method for electric power facilities based on multimodal information fusion according to claim 1 is characterized in that: Constructing an insulator graph based on the list of visible light feature vectors of the insulator slices, the list of infrared feature vectors of the insulator slices, and the spatial adjacency matrix between the insulator slices, including: fusing each corresponding pair of insulator-slice visible light feature vectors and insulator-slice infrared feature vectors in the list of the insulator-slice visible light feature vectors and the list of the insulator-slice infrared feature vectors to obtain an insulator-slice multimodal feature encoding vector as an insulator-slice node of the insulator graph; Based on the spatial adjacency matrix between the insulator slices, edges are added between adjacent insulator slice nodes, where the edges represent structural connection relationships between adjacent insulator slices.

7. The intelligent operation and maintenance method for electric power facilities based on multimodal information fusion according to claim 1 is characterized in that: The pre-trained graph neural network model is a Graph Attention Network.

8. The intelligent operation and maintenance method for electric power facilities based on multimodal information fusion according to claim 1 is characterized in that: Inputting each insulator segment node embedding vector in the list of insulator segment node embedding vectors into a classifier to obtain a defect recognition result for each insulator segment, including: constructing an insulator slice node embedding matrix by using each insulator slice node embedding vector in the list of insulator slice node embedding vectors as a row vector; Calculating the variational field value of each row vector in the insulator sheet node embedding matrix; Selecting the insulator sheet node embedding vector corresponding to the maximum variational field value as the anti-weakening connection vector; Performing inverse mapping on the insulator segment node embedding matrix based on the anti-weakening connection vector to obtain a central stability constraint vector; Arranging the variational field values ​​of the row vectors into variational field representation vectors, and performing variational cluster inversion mapping on the insulator sheet node embedding matrix based on the variational field representation vectors to obtain an approximate variational field cluster distribution vector; Based on the approximate variational field cluster distribution vector and the central stability constraint vector, performing variational cluster density propagation derivation on the insulator sheet node embedding matrix to obtain a propagation derivation correction vector; Derivatively optimizing each insulator-slice node embedding vector in the list of insulator-slice node embedding vectors using the propagation-derived correction vector to obtain an optimized list of insulator-slice node embedding vectors; Each optimized insulator segment node embedding vector list in the optimized insulator segment node embedding vector list is input into a classifier to obtain a defect recognition result of each insulator segment.

9. An intelligent operation and maintenance system for power facilities based on multimodal information fusion, characterized in that: include: an infrared image acquisition module, used to acquire an image of the insulator string taken by a visible light camera of the UAV and a thermal image of the insulator string taken by an infrared camera of the UAV; an image preprocessing module, configured to preprocess the insulator string image and the insulator string thermal image to obtain a preprocessed insulator string image and a preprocessed insulator string thermal image; a feature extraction and segmentation module, configured to perform insulator component segmentation and single-modal feature extraction on the preprocessed insulator string image and the preprocessed insulator string thermal image, respectively, to obtain a list of visible light feature vectors of insulator segments, a list of infrared feature vectors of insulator segments, and a spatial adjacency matrix between insulator segments; an insulator graph construction module, configured to construct an insulator graph based on the list of visible light feature vectors of the insulator slices, the list of infrared feature vectors of the insulator slices, and the spatial adjacency matrix between the insulator slices; A neural network inference module, configured to input the insulator graph into a pre-trained graph neural network model to obtain a list of insulator slice node embedding vectors; The defect recognition module is used to input each insulator segment node embedding vector in the list of insulator segment node embedding vectors into a classifier to obtain a defect recognition result of each insulator segment.

Images