Power equipment metal corrosion detection method and device based on multi-channel multi-mode imaging

Abstract

The invention discloses an electrical equipment metal corrosion detection method and device based on multi-channel and multi-mode imaging, and relates to the field of metal corrosion detection. The method comprises the following steps: acquiring a multi-modal image synchronously acquired by multi-channel imaging equipment, segmenting a local detection area of a key component based on an equipment structure model, and standardizing each channel image to obtain a multi-spectral image; initial features are extracted through an independent encoder, and modal semantic association is learned through a cross-modal attention fusion module to obtain fusion features; and in combination with the environment illumination information and the pre-stored statistics, after the features are normalized by the environment adaptive correction module, residual fusion is carried out, final unified features are generated, and the corrosion condition is judged according to the final unified features. According to the method, complex illumination influences such as strong light, weak light and color temperature change can be overcome, stable fusion feature distribution is ensured, fine and early corrosion features are recognized, the requirement of power grid operation and maintenance for defect detection accuracy is met, and the reliability and stability of outdoor scene detection are greatly improved.

Description

Technical Field

[0001] This invention relates to the field of metal corrosion technology for power equipment, and in particular to a method and apparatus for detecting metal corrosion in power equipment based on multi-channel multimodal imaging. Background Technology

[0002] Metal components of power grids are exposed to harsh outdoor environments for extended periods, and their corrosion directly threatens grid safety, potentially leading to equipment failure or even major accidents. Therefore, developing a metal corrosion detection technology that enables early, accurate detection without disrupting normal grid operation has become an urgent need to ensure the safe and stable operation of power systems.

[0003] Currently, metal corrosion detection technologies are mainly divided into contact and non-contact methods. Contact methods (such as ultrasonic testing) require power outages, resulting in low efficiency and narrow coverage. Non-contact optical imaging technologies suffer from inherent limitations with single-modal approaches: visible light imaging is greatly affected by illumination, easily overexposing in strong light and losing detail in weak light; short-wave infrared imaging is sensitive to early corrosion, but its specific wavelengths are susceptible to moisture interference, leading to decreased reliability in humid weather. To address these shortcomings, multimodal fusion technology has been proposed, combining visible light and short-wave infrared image information to improve detection capabilities. However, existing general multimodal fusion schemes lack adaptability to the complex outdoor environment when applied to the specific scenario of outdoor power equipment corrosion detection. The fusion process fails to dynamically correlate with the changing illumination conditions during acquisition, resulting in unstable detection performance under different weather conditions and at different times. Summary of the Invention

[0004] To address the problem in existing technologies where equipment corrosion images fail to dynamically correlate with the varying lighting conditions at the time of acquisition, leading to unstable detection performance under different weather conditions and time periods, this invention provides a method and apparatus for detecting metal corrosion in power equipment based on multi-channel multimodal imaging. This method overcomes the influence of complex lighting conditions such as strong sunlight on sunny days, weak light on cloudy days, and color temperature changes during dawn and dusk, ensuring that the fusion feature distribution of data acquired under different weather conditions and times remains stable. This significantly improves the reliability and stability of detection results in real outdoor scenarios. The specific technical solution is as follows: In a first aspect, the present invention provides a method for detecting metal corrosion in power equipment based on multi-channel multimodal imaging, comprising the following steps: S1. Acquire multimodal image data of the surface of the power equipment to be inspected by a multi-channel imaging device simultaneously. The multimodal image data includes image data in the visible spectrum and image data in the short-wave infrared spectrum. S2. Based on a predefined power equipment structure model, segment out a local detection region including at least one key component from the multimodal image data; S3. Perform standardized preprocessing on the image data of each channel within the local detection area to obtain the spectral image data corresponding to each channel; S4. Use multiple independent encoders to extract features from each of the spectral image data to obtain an initial feature representation for each mode; S5. Input the initial feature representations of each modality into the cross-modal attention fusion module, learn the semantic associations between each modality through a multi-head self-attention mechanism, generate and fuse cross-modal modality-invariant features, and obtain the fused features after cross-modal information correction; S6. Input the fused features into the environment adaptive feature correction module, normalize and calibrate the features based on the ambient lighting information during acquisition and the pre-stored statistics, and generate the final unified feature representation through residual fusion. S7. Based on the unified feature representation, determine whether there is metal corrosion on the surface of the key component within the local detection area.

[0005] Preferably, the image data in the visible spectrum includes three independent channels located in the blue, green, and red band regions; the image data in the shortwave infrared spectrum includes multiple independent channels uniformly distributed within different wavelength ranges.

[0006] Preferably, the preprocessing further includes ambient light compensation, specifically: Based on the ambient light intensity recorded during acquisition, adaptive contrast enhancement or brightness equalization is performed on the image data of each channel.

[0007] Preferably, the cross-modal attention fusion module performs the following operations: S51. Concatenate the initial feature representations of each modality to form a unified feature sequence; S52. Map the feature sequence into a query matrix, a key matrix, and a value matrix, respectively; S53. Calculate the attention weight of each position in the feature sequence to all positions through a multi-head self-attention mechanism, and perform a weighted summation of the value matrix to obtain the attention output; S54. Perform linear projection on the attention output to generate modality-invariant features; S55. The modality-invariant features are weighted and fused with the corresponding initial feature representations in a residual manner, and the corrected fused features are output.

[0008] Preferably, the environmental adaptive feature correction module performs the following operations: S61. Obtain feature statistics that match the current ambient lighting conditions. The statistics include mean and variance, and different sets of statistics are pre-stored for different lighting levels. S62. Using the matched statistics, center and normalize the variance of the input fusion features. S63. Perform residual weighted fusion with the original fused features to output the final unified feature representation; wherein, the fusion weights of the residual weighted fusion process are adjusted according to the ambient lighting conditions.

[0009] Preferably, the statistical set of different light levels is obtained by simulating or actually collecting samples under different weather conditions and time periods during the training phase and calculating them in groups.

[0010] Preferably, determining whether there is metal corrosion on the surface of the key component within the local detection area based on the unified feature representation includes: S71. Input the unified feature representation into the classification network to obtain the erosion probability map of each pixel or region within the local detection area; S72. Combining the prior knowledge of the spatial structure of the key components, spatial constraint aggregation is performed on the corrosion probability map to obtain the aggregation result; S73. Determine the corrosion state based on the polymerization result and output the range information of the corrosion area.

[0011] Preferably, the multiple independent encoders adopt a hybrid architecture of shared low-level parameters and independent high-level parameters; wherein, the low-level parameters are used to extract basic edge and texture features, and the high-level parameters are used to extract semantic features of the corresponding channel spectral characteristics.

[0012] Preferably, the local detection area is segmented, and different spatial context ranges are used for different types of key components of power equipment.

[0013] Secondly, the present invention also provides a metal corrosion detection device for power equipment based on multi-channel multimodal imaging, which, using the aforementioned method, includes: The image data acquisition module is used to acquire multimodal image data of the surface of the power equipment to be inspected, which is synchronously acquired by the multi-channel imaging device. The multimodal image data includes image data in the visible spectrum and image data in the short-wave infrared spectrum. The image segmentation module is used to segment local detection regions, including at least one key component, from the multimodal image data based on a predefined power equipment structure model. The image processing module is used to perform standardized preprocessing on the image data of each channel within the local detection area to obtain spectral image data corresponding to each channel; An initial feature extraction module is used to extract features from each of the spectral image data using multiple independent encoders to obtain an initial feature representation for each mode. The cross-modal attention fusion module is used to learn the semantic associations between the initial feature representations of each modality through a multi-head self-attention mechanism, generate and fuse cross-modal modality-invariant features, and obtain fused features after cross-modal information correction; The environment adaptive feature correction module is used to normalize and calibrate the fused features based on the ambient light information at the time of acquisition and pre-stored statistics, and generate the final unified feature representation through residual fusion. The corrosion determination module is used to determine whether there is metal corrosion on the surface of the key component within the local detection area based on the unified feature representation.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. The present invention provides a method for detecting metal corrosion in power equipment based on multi-channel multimodal imaging. This method utilizes an environmental adaptive feature correction module to dynamically correlate feature processing with real-time ambient lighting conditions. This enables the detection process to overcome the influence of complex lighting conditions such as strong sunlight on sunny days, weak light on cloudy days, and color temperature changes during dawn and dusk. It ensures that the fused feature distribution of data collected under different weather conditions and at different times remains stable, significantly improving the reliability and stability of detection results in real outdoor scenarios.

[0015] 2. The method for detecting metal corrosion in power equipment based on multi-channel multimodal imaging of the present invention also utilizes target segmentation technology based on the structural model of the power equipment to accurately locate key components to be inspected from complex backgrounds, concentrating computational resources on the core area. This effectively eliminates interference from irrelevant backgrounds, improves the effectiveness and efficiency of detection, and makes it more sensitive to the identification of fine, early corrosion features.

[0016] 3. This invention leverages the structural detail advantages of visible light imaging and the material sensitivity advantages of short-wave infrared imaging. While suppressing background noise and illumination interference, it deeply integrates complementary information across modes, ultimately achieving higher-precision identification of metal corrosion in critical component areas, meeting the requirements of power grid operation and maintenance for accurate defect detection. Attached Figure Description

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0018] Figure 1 This is a flowchart of a method for detecting metal corrosion in power equipment based on multi-channel multimodal imaging according to the present invention.

[0019] Figure 2 This is a flowchart illustrating the execution process of the cross-modal attention fusion module of the present invention.

[0020] Figure 3 This is a flowchart illustrating the execution process of the environmental adaptive feature correction module of the present invention.

[0021] Figure 4 This is a flowchart of the corrosion determination process of the present invention.

[0022] Figure 5 This is a schematic diagram of a power equipment metal corrosion detection system based on multi-channel multimodal imaging according to the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] It should be understood that, when used in this specification, the terms “comprising” and “including” indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0025] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0026] It should also be further understood that the term "and / or" as used in this specification refers to any combination of one or more of the associated listed items, as well as all possible combinations, and includes such combinations.

[0027] Please refer to the following examples. Figures 1 to 5 .

[0028] This application provides a method for detecting metal corrosion in power equipment based on multi-channel multimodal imaging, comprising the following steps: S1. Acquire multimodal image data of the surface of the power equipment to be inspected by a multi-channel imaging device simultaneously. The multimodal image data includes image data in the visible spectrum and image data in the short-wave infrared spectrum. Multi-channel imaging equipment can be part of an airborne or unmanned aerial vehicle (UAV) platform. It typically consists of two detectors: a visible light imaging detector, capable of imaging the visible spectrum, equipped with RGB wavelength filters to acquire image data in three modes; and a short-wave infrared detector, capable of imaging the 900nm-1700nm short-wave infrared spectrum. For example, this detector may have eight different wavelength filters to acquire image data in eight modes. In other specific implementations, the detector can be configured with different numbers of filters of varying wavelengths to acquire image data in different wavelength modes.

[0029] The dual-detector integrated system is mounted on a UAV inspection platform. The visible light detector and the short-wave infrared detector share the same incident light path through optical design, enabling physically synchronized imaging of the same field of view. In terms of software control, trigger signals are sent to both detectors simultaneously.

[0030] The image data in the visible spectrum includes three independent channels located in the blue, green, and red band regions; the image data in the shortwave infrared spectrum includes multiple independent channels uniformly distributed within different wavelength ranges.

[0031] In this embodiment, the multimodal image data comprises 11 modalities, including three visible light spectral modalities: R-mode data (630 nm spectral wavelength), G-mode data (530 nm spectral wavelength), and B-mode data (460 nm spectral wavelength). This combination covers the human eye's visible spectrum and can reflect changes in the color, texture, and macroscopic morphology of the device surface. Metal corrosion is often accompanied by color changes and increased surface roughness; these three channels can effectively identify this type of information.

[0032] In this embodiment, the short-wave infrared (SWIR) image data uses image data corresponding to the center wavelength within a certain wavelength range. SWIR1 mode data: image data with a spectral center wavelength of 900 nm; SWIR2 mode data: image data with a spectral center wavelength of 1000 nm; SWIR3 mode data: image data with a spectral center wavelength of 1100 nm; SWIR4 mode data: image data with a spectral center wavelength of 1200 nm; SWIR5 mode data: image data with a spectral center wavelength of 1300 nm; SWIR6 mode data: image data with a spectral center wavelength of 1400 nm; SWIR7 mode data: image data with a spectral center wavelength of 1500 nm; SWIR8 mode data: image data with a spectral center wavelength of 1600 nm. This spectral band is sensitive to molecular vibrations. Corrosion products of metals exhibit characteristic absorption spectra in this region that differ from the metal bulk. Selecting multiple discrete channels aims to construct a low-resolution spectral fingerprint covering key characteristic wavelengths, used to distinguish different types of early corrosion or quantify the degree of corrosion. For example, specific channels are sensitive to water molecules and can be used to assess the degree of surface moisture, which is a key factor accelerating corrosion.

[0033] It should be noted that in some other embodiments, the image data of the shortwave infrared spectrum may also be image data in a non-center wavelength range, or image data corresponding to a wavelength range that deviates from the center wavelength, and there is no specific limitation.

[0034] S2. Based on a predefined power equipment structure model, segment out a local detection region including at least one key component from the multimodal image data; When segmenting the local detection area, different spatial context ranges are used for different types of key components of power equipment. For example, for conductors and fittings, the context range is set to 1.5-2 times the area of ​​the component itself; for towers and transformer housings, the context range is set to a multi-scale area of ​​the component itself, so as to simultaneously capture local corrosion and structural deformation.

[0035] Among them, the predefined power equipment structure model is a digital knowledge base that contains standard 3D CAD models of target equipment, such as insulators, clamps, and connecting hardware, multi-view template images, and semantic tags and 3D spatial location information of key components, such as metal flanges, steel feet, and crimped pipes.

[0036] The predefined knowledge base for power equipment structural models not only stores the geometric models of components, but also labels each key component type with its contextual analysis strategy. For example: Class A components: These include delicate, localized parts such as wires, connecting hardware, and spacers. They are characterized by their relatively small size, with corrosion typically occurring on the surface or at specific contact points. They have low dependence on the surrounding background, but require some context for accurate identification of component boundaries and orientation. A component-body expansion strategy is employed. Centered on the component detection frame, the area is expanded outwards to 1.5 to 2 times its side length, forming a moderately spacious local window. This area is sufficient to clearly encompass the component and its directly connected structures while effectively excluding irrelevant background elements.

[0037] Category B components: These include large structural components such as main / auxiliary materials for poles and towers, transformer housings, and circuit breaker casings. They are characterized by their large size and complex structure; corrosion may manifest as localized pitting and rust spots, or it may be accompanied by regional coating peeling or overall structural deformation. A multi-scale context-aware strategy is employed. This involves not only focusing on the component itself but also constructing multiple analytical scales: Local fine scale: A high-resolution fine window surrounding suspected corrosion points or high-stress areas, used to analyze pitting corrosion characteristics.

[0038] Component global scale: A complete view of the entire component or its main parts, used to assess the overall pattern of corrosion distribution and regional defects.

[0039] Structural Relational Scale: A slightly larger view including the component and its adjacent connected components, used to assess the impact of corrosion or deformation on its structural function.

[0040] In actual inspection image processing, the following steps are performed to achieve adaptive segmentation: Using a trained object detection model, identify electrical equipment components in an image and output their category labels and initial bounding boxes.

[0041] Based on the identified component category, the corresponding context strategy is automatically invoked; For a type A component, the expanded bounding box = initial bounding box * scaling factor. The scaling factor can be dynamically adjusted based on the component's pixel size in the image.

[0042] If it is a Class B component, generate a multi-scale region proposal: Fine-scale: Sliding windows are created on the surface of the component or multiple potential regions of interest are generated based on attention mechanisms.

[0043] Global scale: Directly use the precise segmentation mask of the component or the bounding box that is expanded by a certain multiple.

[0044] Structural scale: Based on the projection of the 3D model or the graph relationship model, the component regions directly connected to the current component are merged to form a combined analysis region.

[0045] Based on one or more region bounding boxes calculated above, corresponding image patches are simultaneously cropped from the 11-channel aligned multispectral image obtained in step S1.

[0046] By employing target segmentation technology based on power equipment structural models, key components to be inspected can be accurately located from complex backgrounds, concentrating computational resources on core areas. This effectively eliminates interference from irrelevant backgrounds, improving the effectiveness and efficiency of inspection and making it more sensitive to the identification of subtle, early-stage corrosion features.

[0047] S3. Perform standardized preprocessing on the image data of each channel within the local detection area to obtain the spectral image data corresponding to each channel; Each channel image after preprocessing is standardized separately. Natural scene statistics are used for the visible light channel, while statistics from samples of metal surfaces of power equipment are used for the short-wave infrared channel. This eliminates numerical distribution differences caused by varying response curves and gain settings of different detectors.

[0048] The preprocessing also includes ambient light compensation, specifically: Based on the ambient light intensity recorded during acquisition, adaptive contrast enhancement or brightness equalization is performed on the image data of each channel.

[0049] The drone platform integrates a light sensor that records the ambient light intensity in real time at the moment of data acquisition. If the light intensity is not higher than a preset threshold, adaptive histogram equalization is applied to the visible light channels (R, G, B) to enhance local contrast and highlight texture details in low light. If the light intensity is higher than the preset threshold, an algorithm based on Retinex theory is applied to the visible light channels to estimate and separate the light components, and the reflection components are enhanced to suppress overexposure of highlights and restore details in shadow areas.

[0050] Among them, the short-wave infrared channel is less affected by the intensity of visible light, and this light intensity value is retained as metadata.

[0051] S4. Use multiple independent encoders to extract features from each of the spectral image data to obtain an initial feature representation for each mode; The multiple independent encoders adopt a hybrid architecture of shared low-level parameters and independent high-level parameters; wherein, the low-level parameters are used to extract basic edge and texture features, and the high-level parameters are used to extract semantic features of the corresponding channel spectral characteristics.

[0052] By employing 11 encoder branches with identical structures but independent parameters, images from 11 channels are processed in parallel.

[0053] Shared underlying layers: The first 1-2 convolutional layers of all branches share weights. This is responsible for extracting basic, modality-independent low-level features such as edges, corners, and general textures. Shared parameters significantly reduce the number of model parameters, improve training efficiency, and force the network to learn low-level representations that are general across modalities.

[0054] Independent high-level layers: The parameters of subsequent convolutional layers or Transformer modules in each branch are independent. They are responsible for learning and extracting channel-specific high-level semantic features. For example, the high-level network of the visible light branch focuses on learning feature patterns related to color and complex textures. The high-level network of the short-wave infrared branch learns spectral response patterns related to material chemical composition and microscopic surface state.

[0055] The input to each encoder branch is a normalized single-channel image patch. After processing by the hybrid architecture, each branch outputs a global feature vector as the initial feature representation for that modality. Ultimately, 11 feature vectors with the same dimensions are obtained, which learn both general structural information from the shared lower layers and channel-specific spectral semantic information from independent higher layers.

[0056] S5. Input the initial feature representations of each modality into the cross-modal attention fusion module, learn the semantic associations between each modality through a multi-head self-attention mechanism, generate and fuse cross-modal modality-invariant features, and obtain the fused features after cross-modal information correction; The cross-modal attention fusion module performs the following operations: S51. Concatenate the initial feature representations of each modality to form a unified feature sequence; The 11 independent feature vectors are concatenated in a predetermined fixed order to form a matrix. Let the input be the initial feature representations of the 11 modalities extracted in step S4: in, Indicates the first Feature vectors of each modality This represents the dimension of the model's hidden layers.

[0057] This matrix can be viewed as a feature sequence, where each element or position represents global feature information of a specific modality.

[0058] S52. Map the feature sequence into a query matrix, a key matrix, and a value matrix, respectively; Three core matrices need to be derived from the unified feature sequence described above: the query matrix, the key matrix, and the value matrix. This is achieved through three independent linear transformation layers. Specifically, the concatenated feature sequence is multiplied by three different learnable weight matrices, thereby projecting it into three new vector spaces to obtain the query matrix, the key matrix, and the value matrix, respectively. The query matrix... Key matrix Sum matrix Represented as: Wherein, the projection weight matrix , usually set .therefore, .

[0059] S53. Calculate the attention weight of each position in the feature sequence to all positions through a multi-head self-attention mechanism, and perform a weighted summation of the value matrix to obtain the attention output; The query, key, and value matrices obtained in the previous step are computed in parallel across the feature dimension. The dot product between the transposes of the query matrix and the key matrix is ​​calculated to obtain the original attention score matrix, which reflects the initial correlation strength between any two modality features. This score matrix is ​​then scaled and normalized to obtain an attention weight matrix. Each row of this weight matrix represents the distribution of attention a modality gives to all modalities; a higher weight indicates more information is extracted from the corresponding modality during fusion. Finally, this normalized attention weight matrix is ​​multiplied by the value matrix to obtain the new feature representation after weighted aggregation of cross-modal information. After this step, the feature vectors of each modality are no longer isolated but incorporate complementary information from 10 other modalities, filtered by weights. The specific process is as follows: Let the number of attention heads be Dimensions of each head .

[0060] Will Uniformly split along the feature dimension into Group: in, Indicates the first The submatrix corresponding to each head.

[0061] Calculate the attention output for a single head, for the . Size, calculate attention score matrix : right Perform Softmax normalization row by row to obtain the attention weight matrix. : The output of this header is: Concatenate the outputs of all heads along the feature dimension: By outputting the projection matrix Will Map back Dimensions: in, This is the final output of multi-head self-attention, where each row corresponds to a feature representation of a modality after cross-modal interaction.

[0062] S54. Perform linear projection on the attention output to generate modality-invariant features; A dedicated linear projection layer maps the features corresponding to each mode of the attention output to a new feature space. This projection process strips away and filters out components that are overly correlated with the physical properties of the mode itself, such as visible light reflecting color texture or infrared reflecting material spectrum, while retaining and enhancing common semantic information that is directly related to the corrosion state.

[0063] Extracting mode-invariant features through a linear projection layer: in, The projection weight matrix is... is the dimension of the modality-invariant feature. each line That is, the first The mode-invariant features corresponding to each mode.

[0064] S55. The modality-invariant features are weighted and fused with the corresponding initial feature representations in a residual manner, and the corrected fused features are output.

[0065] Considering that completely replacing the original features might lead to the loss of valuable modality-specific information, this module employs residual learning for the final fusion. Specifically, the modality-invariant features generated in step S54 are added element-wise to the original initial feature representation from step S51. Before addition, the modality-invariant features are typically multiplied by a fusion weight coefficient between 0 and 1 to adjust the strength of the cross-modal correction information. This coefficient can be a preset fixed value or a learnable parameter. Through this residual connection method, the final output is the corrected fused feature. The specific process is as follows: The modality-invariant features are combined with the original features in a residual manner. Fusion. If First, you need to Mapped to via an adapter layer Dimensions: Otherwise, directly order Let the fusion weight be... , which can be a fixed value or a learnable parameter, then the corrected fusion feature output is: In addition, the addition is done element by element. each line That is, the first The final fused feature after cross-modal information correction of each modality.

[0066] S6. Input the fused features into the environment adaptive feature correction module, normalize and calibrate the features based on the ambient lighting information during acquisition and the pre-stored statistics, and generate the final unified feature representation through residual fusion. During the model training phase, a mapping library between ambient lighting levels and feature statistics needs to be pre-built. This is the foundation for the adaptive correction module to achieve adaptive correction. The adaptive correction module performs the following operations: S61. Obtain feature statistics that match the current ambient lighting conditions. The statistics include mean and variance, and different sets of statistics are pre-stored for different lighting levels. When acquiring each training sample image, key environmental parameters, including ambient light intensity and average scene brightness, are simultaneously recorded or extracted from image metadata. Based on light intensity, the environment is divided into multiple discrete light levels. The statistical sets for different light levels are obtained during the training phase by simulating or actually acquiring samples under different weather conditions and time periods, and then grouping and calculating them. All samples in the training dataset are grouped according to their respective light levels. For each light level group, the statistical measures of the training features for each modality's feature channel are calculated. The mean represents the typical central location of the modality's features under that light condition. The variance represents the degree of dispersion of the modality's feature distribution under that light condition. Finally, a mapping library is formed, with the light level as the key and the corresponding mean and variance statistics as the values. The specific implementation process is as follows: Suppose the training dataset contains N samples, and the ambient lighting level L∈{L1,L2,…,LK} is recorded for each sample during acquisition, with a total of K preset levels. For each sample, before the input to the environment adaptive feature correction module after forward propagation through the network, its original feature representations of 11 modalities can be obtained. During the training phase, statistics are calculated and stored as follows: Grouping by conditions: All training samples are grouped according to their illumination level L and modality number m (m=1,…,11).

[0067] Calculate the within-group statistic. For each group (L,m), calculate the mean of the m-th modal feature of all samples in that group. and variance : in, Let |·| represent the set of all feature vectors belonging to illumination level L and mode m; |·| represents the size of the set.

[0068] Build the mapping library StatMap:(L,m)→( , ).

[0069] Let the current input be the fused feature matrix corrected by the cross-modal attention fusion module: The measured ambient light level Lcurr corresponds to the currently acquired image.

[0070] To obtain the matching statistics for each modality m=1,2,…,11, retrieve the statistics for its corresponding illumination level Lcurr from the mapping database: If Lcurr is not in the preset level, the nearest neighbor level or interpolation method can be used to obtain an approximate statistic.

[0071] S62. Using the matched statistics, center and normalize the variance of the input fusion features. Using the matched statistics, each modal feature in the input is standardized modally to eliminate distribution shifts caused by illumination. The mean of the corresponding illumination level is subtracted from the feature vector of each modality, eliminating the overall numerical shift of features caused by different illumination intensities. The centered features are then divided by the standard deviation of their corresponding illumination level's variance, unifying the scale of feature distributions under different illumination conditions to a similar range and reducing feature scale instability caused by illumination variations.

[0072] S63. Perform residual weighted fusion with the original fused features to output the final unified feature representation; wherein, the fusion weights of the residual weighted fusion process are adjusted according to the ambient lighting conditions.

[0073] To eliminate illumination interference while preserving the effective semantic information in the original features to the greatest extent possible, a residual fusion strategy is adopted.

[0074] Let the dynamic fusion weight be... Its value depends on the current light level. The extreme degree is calculated. The final output uniform feature representation is then: In addition, the addition is done element by element. This refers to the final characteristic that has undergone environmental adaptive correction and is robust to strong illumination.

[0075] By conditionally standardizing and weighting the residuals of the features, the influence of changes in ambient lighting on the feature distribution is effectively suppressed, thereby improving the model's detection capability in complex outdoor scenes.

[0076] S7. Based on the unified feature representation, determine whether there is metal corrosion on the surface of the key component within the local detection area.

[0077] The step of determining whether there is metal corrosion on the surface of the key component within the local detection area based on the unified feature representation includes: S71. Input the unified feature representation into the classification network to obtain the erosion probability map of each pixel or region within the local detection area; A lightweight decoder network is used as the classification network. The input to this network is the unified feature representation output by S6. The network typically consists of fully connected layers and / or deconvolutional layers. The network integrates feature information from all modalities through fully connected layers, and then progressively upsamples the space through a series of deconvolutional layers, ultimately restoring the feature map to a resolution similar to or the same as the original local detection region image. The final layer uses the sigmoid activation function to generate a scalar value between 0 and 1 for each pixel location in the output feature map. This value represents the confidence probability that the pixel belongs to the erosion category. Finally, an erosion probability heatmap corresponding to the local region is obtained.

[0078] S72. Combining the prior knowledge of the spatial structure of the key components, spatial constraint aggregation is performed on the corrosion probability map to obtain the aggregation result; Since probabilistic graphs may contain noise or scattered low-probability points, direct thresholding can lead to fragmented results. This paper proposes an intelligent aggregation method that incorporates prior knowledge of the spatial structure of power equipment components.

[0079] Application of prior knowledge of structure: Component geometric constraints: Based on the type of component being inspected, its inherent geometric model is applied. For example, for cylindrical wires or fittings, the corrosion area should appear as a strip-like distribution extending along the component's axial direction on the probability map; for planar connecting plates or flanges, corrosion may appear as sheet-like or dot-like clusters.

[0080] Prior knowledge of stress and vulnerable areas: The knowledge base pre-stores information on the easily corroded areas of different components. In these known high-risk areas, the probability values ​​can be weighted to enhance their impact, so that even if the absolute probability value is not high, they can still be given priority consideration because they are located in critical positions.

[0081] Spatial continuity constraint: By applying connected component analysis, pixels that are spatially adjacent and whose probability values ​​exceed the primary threshold are aggregated into several candidate erosion regions.

[0082] For each candidate region formed by aggregation, its aggregation result is calculated, including the total area of ​​the region, the average probability of the region, the maximum probability of the region, and the spatial density of the region. This comprehensively describes the degree of corrosion suspicion of the candidate region.

[0083] S73. Based on the polymerization results, determine the corrosion state and degree, and output the location range information of the corrosion area.

[0084] Based on the aggregated results, a final binary decision is made. A final decision threshold is set. The judgment rule is usually not singular, but rather a comprehensive logic combining multiple aggregated indicators. For example: if the average probability of a region is greater than the first threshold T1 and the region area is greater than the minimum salient area, then the region is judged to have corrosion. Or, if the maximum probability of a region is greater than the second threshold T2, even if the area is small, it may be judged to have pitting corrosion. Here, the first threshold T1 is less than the second threshold T2.

[0085] Output the overall corrosion status of the local detection area, such as whether corrosion is present or no obvious corrosion is observed. Range information includes: Location information is used to highlight the areas identified as eroded in the original input image or local detection area image using bounding boxes or polygonal outlines.

[0086] The area information outputs the pixel area of ​​the eroded region, which can be converted into the actual physical size by combining it with the known imaging scale.

[0087] The severity information, based on the area proportion of the corroded area and the average probability, provides a level assessment information of mild, moderate, and severe corrosion.

[0088] The final output corrosion detection results include information such as component ID, component type, corrosion status, corrosion degree, corrosion area coordinates / outline, corrosion area, timestamp, and ambient light level, which are directly connected to the power grid asset management system.

[0089] This invention presents a method for detecting metal corrosion in power equipment based on multi-channel multimodal imaging. Through an environmentally adaptive feature correction module, feature processing is dynamically correlated with real-time ambient lighting conditions. This enables the detection process to overcome the influence of complex lighting conditions such as strong sunlight on sunny days, weak light on cloudy days, and color temperature changes during dawn and dusk. It ensures that the fused feature distribution of data collected under different weather conditions and at different times remains stable, significantly improving the reliability and stability of detection results in real outdoor scenarios. Furthermore, through target segmentation technology based on power equipment structural models, key components to be inspected can be accurately located from complex backgrounds, concentrating computational resources on core areas. This effectively eliminates interference from irrelevant backgrounds, improving the effectiveness and efficiency of detection, and making the identification of fine, early corrosion features more sensitive. This invention leverages the structural detail advantages of visible light imaging and the material sensitivity advantages of short-wave infrared imaging. While suppressing background noise and lighting interference, it deeply integrates complementary information across modalities, ultimately achieving higher-precision identification of metal corrosion in key component areas, meeting the requirements of power grid operation and maintenance for accurate defect detection.

[0090] This application also provides a device for detecting metal corrosion in power equipment based on multi-channel multimodal imaging, which applies the aforementioned method and includes: The image data acquisition module is used to acquire multimodal image data of the surface of the power equipment to be inspected, which is synchronously acquired by the multi-channel imaging device. The multimodal image data includes image data in the visible spectrum and image data in the short-wave infrared spectrum. The image segmentation module is used to segment local detection regions, including at least one key component, from the multimodal image data based on a predefined power equipment structure model. The image processing module is used to perform standardized preprocessing on the image data of each channel within the local detection area to obtain spectral image data corresponding to each channel; An initial feature extraction module is used to extract features from each of the spectral image data using multiple independent encoders to obtain an initial feature representation for each mode. The cross-modal attention fusion module is used to learn the semantic associations between the initial feature representations of each modality through a multi-head self-attention mechanism, generate and fuse cross-modal modality-invariant features, and obtain fused features after cross-modal information correction; The environment adaptive feature correction module is used to normalize and calibrate the fused features based on the ambient light information at the time of acquisition and pre-stored statistics, and generate the final unified feature representation through residual fusion. The corrosion determination module is used to determine whether there is metal corrosion on the surface of the key component within the local detection area based on the unified feature representation.

[0091] The functional explanation of each unit in this embodiment is the same as that of a method for detecting metal corrosion in power equipment based on multi-channel multimodal imaging, and the technical effects are the same, so it will not be repeated here.

[0092] Those skilled in the art will recognize that the units of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the invention.

[0093] In the embodiments provided by the present invention, it should be understood that the division of units is only a logical functional division. In actual implementation, there may be other division methods, such as multiple units can be combined into one unit, one unit can be split into multiple units, or some features can be ignored.

[0094] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0095] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the specification of the present invention.

Claims

1. A method for detecting metal corrosion in power equipment based on multi-channel multimodal imaging, characterized in that, Includes the following steps: S1. Acquire multimodal image data of the surface of the power equipment to be inspected by a multi-channel imaging device simultaneously. The multimodal image data includes image data in the visible spectrum and image data in the short-wave infrared spectrum. S2. Based on a predefined power equipment structure model, segment out a local detection region including at least one key component from the multimodal image data; S3. Perform standardized preprocessing on the image data of each channel within the local detection area to obtain the spectral image data corresponding to each channel; S4. Use multiple independent encoders to extract features from each of the spectral image data to obtain an initial feature representation for each mode; S5. Input the initial feature representations of each modality into the cross-modal attention fusion module, learn the semantic associations between each modality through a multi-head self-attention mechanism, generate and fuse cross-modal modality-invariant features, and obtain the fused features after cross-modal information correction; S6. Input the fused features into the environment adaptive feature correction module, normalize and calibrate the features based on the ambient lighting information during acquisition and the pre-stored statistics, and generate the final unified feature representation through residual fusion. S7. Based on the unified feature representation, determine whether there is metal corrosion on the surface of the key component within the local detection area.

2. The method for detecting metal corrosion in power equipment based on multi-channel multimodal imaging according to claim 1, characterized in that, The image data in the visible spectrum includes three independent channels located in the blue, green, and red band regions; the image data in the shortwave infrared spectrum includes multiple independent channels uniformly distributed within different wavelength ranges.

3. The method for detecting metal corrosion in power equipment based on multi-channel multimodal imaging according to claim 1, characterized in that, The preprocessing also includes ambient light compensation, specifically: Based on the ambient light intensity recorded during acquisition, adaptive contrast enhancement or brightness equalization is performed on the image data of each channel.

4. The method for detecting metal corrosion in power equipment based on multi-channel multimodal imaging according to claim 1, characterized in that, The cross-modal attention fusion module performs the following operations: S51. Concatenate the initial feature representations of each modality to form a unified feature sequence; S52. Map the feature sequence into a query matrix, a key matrix, and a value matrix, respectively; S53. Calculate the attention weight of each position in the feature sequence to all positions through a multi-head self-attention mechanism, and perform a weighted summation of the value matrix to obtain the attention output; S54. Perform linear projection on the attention output to generate modality-invariant features; S55. The modality-invariant features are weighted and fused with the corresponding initial feature representations in a residual manner, and the corrected fused features are output.

5. The method for detecting metal corrosion in power equipment based on multi-channel multimodal imaging according to claim 1, characterized in that, The environmental adaptive feature correction module performs the following operations: S61. Obtain feature statistics that match the current ambient lighting conditions. The statistics include mean and variance, and different sets of statistics are pre-stored for different lighting levels. S62. Using the matched statistics, center and normalize the variance of the input fusion features. S63. Perform residual weighted fusion with the original fused features to output the final unified feature representation; wherein, the fusion weights of the residual weighted fusion process are adjusted according to the ambient lighting conditions.

6. The method for detecting metal corrosion in power equipment based on multi-channel multimodal imaging according to claim 5, characterized in that, The statistical set of different light levels is obtained by simulating or actually collecting samples under different weather conditions and time periods during the training phase and calculating them in groups.

7. The method for detecting metal corrosion in power equipment based on multi-channel multimodal imaging according to claim 1, characterized in that, The step of determining whether there is metal corrosion on the surface of the key component within the local detection area based on the unified feature representation includes: S71. Input the unified feature representation into the classification network to obtain the erosion probability map of each pixel or region within the local detection area; S72. Combining the prior knowledge of the spatial structure of the key components, spatial constraint aggregation is performed on the corrosion probability map to obtain the aggregation result; S73. Determine the corrosion state based on the polymerization result and output the range information of the corrosion area.

8. The method for detecting metal corrosion in power equipment based on multi-channel multimodal imaging according to claim 1, characterized in that, The multiple independent encoders adopt a hybrid architecture of shared low-level parameters and independent high-level parameters; wherein, the low-level parameters are used to extract basic edge and texture features, and the high-level parameters are used to extract semantic features of the corresponding channel spectral characteristics.

9. The method for detecting metal corrosion in power equipment based on multi-channel multimodal imaging according to claim 1, characterized in that, The local detection area is segmented, and different spatial context ranges are used for different types of key components of power equipment.

10. A device for detecting metal corrosion in power equipment based on multi-channel multimodal imaging, characterized in that, The method described by any one of claims 1 to 9 includes: The image data acquisition module is used to acquire multimodal image data of the surface of the power equipment to be inspected, which is synchronously acquired by the multi-channel imaging device. The multimodal image data includes image data in the visible spectrum and image data in the short-wave infrared spectrum. The image segmentation module is used to segment local detection regions, including at least one key component, from the multimodal image data based on a predefined power equipment structure model. The image processing module is used to perform standardized preprocessing on the image data of each channel within the local detection area to obtain spectral image data corresponding to each channel; An initial feature extraction module is used to extract features from each of the spectral image data using multiple independent encoders to obtain an initial feature representation for each mode. The cross-modal attention fusion module is used to learn the semantic associations between the initial feature representations of each modality through a multi-head self-attention mechanism, generate and fuse cross-modal modality-invariant features, and obtain fused features after cross-modal information correction; The environment adaptive feature correction module is used to normalize and calibrate the fused features based on the ambient light information at the time of acquisition and pre-stored statistics, and generate the final unified feature representation through residual fusion. The corrosion determination module is used to determine whether there is metal corrosion on the surface of the key component within the local detection area based on the unified feature representation.

Images