Iron tower corrosion detection and quantitative evaluation method and system based on multi-modal fusion

Abstract

The method and system for tower corrosion detection and quantitative evaluation based on multi-modal fusion relate to the technical field of tower corrosion detection. The method comprises the following steps: collecting an RGB image of the tower, and using a zero-shot detection model for preliminary identification to obtain a rust candidate region bounding box; performing pixel-level segmentation on the candidate region corresponding to the rust candidate region bounding box to obtain a rust region mask; after obtaining the rust region mask, obtaining a depth map and a polarization map that are accurately aligned with the RGB image by collecting data on the region corresponding to the rust region mask through a depth camera and a polarization camera; within the region defined by the rust region mask, calculating a depth feature according to the depth map and a polarization feature according to the polarization map; discriminating the rust candidate region according to the depth feature and the polarization feature, and when in a fuzzy interval, comprehensively judging in combination with a visual feature; and performing multi-index weighted scoring on the region determined as true rust to obtain a rust grade and a corresponding processing suggestion.

Description

Technical Field

[0001] This invention relates to the field of tower corrosion detection technology, and more specifically, to a method and system for tower corrosion detection and quantitative evaluation based on multimodal fusion. Background Technology

[0002] Corrosion of metal components in power transmission line towers is one of the main causes of structural weakness and even fracture. Traditional methods relying on manual climbing for inspection are inefficient, risky, and subject to subjective assessments. To address this challenge, automated inspection technology based on computer vision has become a key area of ​​research and application in the industry, with deep learning methods currently being the mainstream approach.

[0003] Existing technical solutions primarily focus on using drones equipped with visible light cameras to acquire images and then analyzing them based on various deep learning models. Specifically, existing solutions can be categorized as follows: locating and identifying corroded areas using object detection models such as the YOLO series; performing pixel-level segmentation and calculating the proportion of corroded area using semantic segmentation models such as Mask R-CNN; employing semi-supervised learning schemes to reduce reliance on labeled data; and using cloud-edge collaborative processing architectures to balance computational power and accuracy. Furthermore, some research attempts to combine traditional image processing techniques or improve network structures to enhance model performance.

[0004] However, the aforementioned existing technical solutions share several core drawbacks. First, they generally rely heavily on large amounts of precisely labeled data for supervised learning, but rust data itself is scarce due to the difficulty of collection and the high cost of professional labeling, resulting in insufficient model generalization ability. Second, existing methods rely only on a single RGB image modality, making judgments based solely on color and texture features, failing to effectively distinguish between rust, dirt, and watermarks that appear similar, leading to a high false positive rate. Third, their quantitative assessment of rust severity is extremely limited, typically relying on simple grading based on the proportion of rust area, failing to obtain physical parameters related to structural safety, such as surface roughness and rust depth, making it difficult to support scientific maintenance decisions. Finally, when deployed at the edge, existing solutions may be limited by computational and power consumption constraints, only able to use lightweight models to process single-modality data, thus limiting functionality. They may also require cloud collaboration and cannot work independently when network conditions are poor. Summary of the Invention

[0005] This invention provides a method and system for detecting and quantitatively evaluating corrosion of iron towers based on multimodal fusion, in order to improve at least one of the above-mentioned technical problems.

[0006] In a first aspect, the present invention provides a method for detecting and quantitatively evaluating corrosion of iron towers based on multimodal fusion, which includes steps S1 to S7.

[0007] S1. Acquire RGB image of the tower.

[0008] S2. Use a zero-shot detection model to perform preliminary identification on the RGB image to obtain the bounding box of the corrosion candidate region.

[0009] S3. Perform pixel-level segmentation on the candidate region corresponding to the bounding box of the corrosion candidate region to obtain the corrosion region mask.

[0010] S4. After obtaining the rusted area mask, acquire the depth map and polarization map that are precisely aligned with the RGB image, obtained by data acquisition of the area corresponding to the rusted area mask using a depth camera and a polarization camera.

[0011] S5. Within the area defined by the rusted area mask, calculate the depth features based on the depth map and calculate the polarization features based on the polarization map.

[0012] S6. Based on the depth features and polarization features, the candidate rust regions are distinguished as genuine or fake, and when they are in the fuzzy range, a comprehensive judgment is made by combining visual features.

[0013] S7. For areas determined to be true rust, perform multi-index weighted scoring to obtain the rust level and corresponding treatment recommendations.

[0014] As a further aspect of the present invention, the zero-sample detection model is an improved version of YOLO-World.

[0015] S2 specifically includes: normalizing the RGB image, and then inputting it along with the hierarchical prompt words into the improved YOLO-World to output bounding boxes for rust candidate regions. The RGB image normalization specifically includes: adjusting the RGB image to 640×640 pixels and then normalizing it.

[0016] As a further aspect of the present invention, the improvements of the improved YOLO-World include a corrosion feature enhancement module, layered prompt words, a multi-scale detection head, a multi-scale corrosion sensitivity adjustment mechanism, and an adaptive confidence threshold module.

[0017] Preferably, the rust feature enhancement module is used to process RGB images. The rust feature enhancement module includes color-sensitive convolution, texture enhancement convolution, and edge enhancement modules.

[0018] The color-sensitive convolution converts the RGB image to the HSL color space using a dynamic convolution kernel and calculates a color attention map to obtain a color-sensitive feature map.

[0019] .

[0020] .

[0021] In the formula, This is a color attention map. This is the activation function. This is the result of convolution processing on an HSL color space image. It is a dynamic convolution kernel. It is the basic convolution kernel. This represents the operation performed in conjunction with the color attention map.

[0022] The texture enhancement convolution uses a Gabor filter bank to extract multi-directional textures to obtain texture enhancement feature maps. The direction set is... For the direction set of the Gabor filter bank, , , , These represent the texture extraction directions.

[0023] The edge enhancement module uses edge-sensitive convolution to enhance the boundary of the rusted area to obtain an edge enhancement feature map.

[0024] The color-sensitive feature map, texture enhancement feature map, and edge enhancement feature map are fused using concatenation and 1×1 convolution to obtain rust enhancement features.

[0025] Preferably, the layered cue words include color cue words, texture cue words, location cue words, and negative sample cue words.

[0026] Preferably, the multi-scale corrosion sensitivity adjustment mechanism includes: increasing the detection sensitivity and lowering the confidence threshold for small corrosion areas smaller than 32×32 pixels, using standard detection sensitivity for medium-sized corrosion areas from 32×32 pixels to 128×128 pixels, and reducing false alarm sensitivity and increasing the confidence threshold for large corrosion areas larger than 128×128 pixels.

[0027] Preferably, the multi-scale detection head includes a P3 detection head for detecting small-area corrosion, a P4 detection head for detecting medium-area corrosion, and a P5 detection head for detecting large-area corrosion.

[0028] Preferably, the adaptive confidence threshold module dynamically adjusts the threshold according to the lighting conditions, contrast and signal-to-noise ratio, wherein the threshold for ideal quality images is 0.5, the threshold for images under normal conditions is 0.4, and the threshold for images under severe conditions is 0.3.

[0029] As a further aspect of the present invention, S3 specifically includes S31 to S34.

[0030] S31. Adjust the candidate region to 224×224 pixels.

[0031] S32. Use the center of the bounding box of the rust candidate region output by YOLO-World as the prompt point.

[0032] S33. Input the candidate region and the prompt point into MobileSAM for pixel-level segmentation and output a pixel-level segmentation mask.

[0033] S34. Perform morphological operations on the pixel-level segmentation mask to remove noise and fill holes, obtaining the rust region mask. The rust region mask is used to define the calculation area for subsequent depth and polarization features.

[0034] As a further aspect of the present invention, S5 specifically includes S51 to S56.

[0035] S51. Perform point cloud preprocessing on the 3D point cloud data within the rust area mask acquired by the depth camera to remove outliers.

[0036] S52. Fit the reference plane using the least squares method and calculate the height deviation of each point from the fitted plane.

[0037] S53. Calculate the surface roughness based on the height deviation. .

[0038] .

[0039] In the formula, This represents the number of points in the point cloud within the masked area. For the first The height deviation of each point from the fitted plane.

[0040] S54, Calculate the average depth .

[0041] .

[0042] In the formula, This represents the number of points in the point cloud within the masked area. For the first The perpendicular distance from a point to the reference plane; a positive value indicates that the point is below the reference plane.

[0043] S55. Calculate the Stokes parameters based on the polarization images of the four polarization angles of 0°, 45°, 90°, and 135°.

[0044] .

[0045] .

[0046] .

[0047] In the formula, This is the first Stokes parameter. This is the second Stokes parameter. This is the third Stokes parameter. This represents the pixel intensity value of an image captured by a polarization camera at a polarization angle of 0°. This represents the pixel intensity value of an image captured by a polarization camera at a polarization angle of 45°. This represents the pixel intensity value of an image captured by a polarization camera at a 90° polarization angle. This represents the pixel intensity value of an image captured by a polarization camera at a polarization angle of 135°.

[0048] S56. Calculate the polarization degree of each pixel within the mask region. .

[0049] .

[0050] S57. Calculate the average polarization degree within the mask region. .

[0051] .

[0052] In the formula, This represents the number of pixels within the mask area. For the first The degree of polarization of each pixel.

[0053] As a further aspect of the present invention, the following is defined: Surface roughness. This is the high roughness threshold. This is the low roughness threshold. The low polarization threshold. This is the high polarization threshold. This represents the average degree of polarization.

[0054] The authenticity determination includes:

[0055] when When the rust candidate area is determined to be prone to true rust, it is determined that the rust candidate area is prone to true rust.

[0056] when When the corrosion candidate area is determined to be prone to pseudo-corrosion.

[0057] when When the rust candidate area is determined to be prone to true rust, it is determined that the rust candidate area is prone to true rust.

[0058] when When the corrosion candidate area is determined to be prone to pseudo-corrosion.

[0059] like and If it is found to be true corrosion, it will be subject to quantitative assessment.

[0060] like or If the result is false rust, it will be identified as "stain / water stain" and output as "stain / water stain".

[0061] When depth and polarization features are in the fuzzy range, a comprehensive score is obtained by combining color and texture features.

[0062] .

[0063] In the formula, The overall score is based on visual features. For color weights, To determine the score based on color characteristics, For texture weights, The score is determined based on the texture features.

[0064] when At that time, it was determined to be genuine rust and entered into a quantitative assessment.

[0065] when When the condition is found to be false rust, the output will be "stain / water stain".

[0066] As a further aspect of the present invention, the multi-index weighted scoring of S7 includes: Calculate the percentage of rusted area .

[0067] .

[0068] .

[0069] .

[0070] In the formula, This represents the number of pixels in the rust mask area. This represents the number of pixels in the component mask area. coordinates The mask value for the rusted area. coordinates The component area mask value at that location.

[0071] Calculate area score Roughness score Depth Score And polarization score .

[0072] .

[0073] .

[0074] .

[0075] .

[0076] In the formula, Represents a logarithmic function. Surface roughness. This is the reference value for maximum roughness. This represents the average depth. This is the maximum depth reference value. This represents the average degree of polarization.

[0077] Calculate the overall score .

[0078] .

[0079] In the formula, Area weight. This represents the roughness weight. For depth weights. This is the polarization degree weight.

[0080] when At that time, it was determined to be a level 1 light corrosion, and a treatment suggestion of regular monitoring and no treatment at present was issued.

[0081] when At that time, it was determined to be a level 2 moderate rust, and treatment recommendations for surface rust removal and anti-corrosion coating were provided.

[0082] when When the corrosion is classified as Level 3 severe corrosion, a treatment suggestion to replace the component is output.

[0083] As a further aspect of the present invention, a method for detecting and quantitatively evaluating corrosion of iron towers based on multimodal fusion adopts a progressive on-demand data acquisition mechanism.

[0084] Phase 1: The progressive on-demand acquisition mechanism includes: first acquiring RGB images and performing preliminary YOLO-World recognition.

[0085] Phase 2: When a candidate rust region is detected in Phase 1, MobileSAM is triggered to perform pixel-level precise segmentation to obtain the rust region mask.

[0086] Phase 3: After obtaining the rust region mask in Phase 3, the depth camera and polarization camera are triggered to acquire depth and polarization images. If no rust candidate region is detected in Phase 1, the acquisition of depth and polarization images is not triggered. The depth and polarization features are calculated only within the rust region mask to reduce the overall image computation and achieve region alignment between the RGB image, depth image, and polarization image.

[0087] As a further aspect of the present invention, the RGB image is acquired by an RGB camera mounted on a drone platform.

[0088] The unmanned aerial vehicle (UAV) platform includes a hexacopter UAV, an RGB camera, a depth camera, a polarization camera, a positioning unit, an inertial measurement unit, and a computing platform.

[0089] The RGB camera is used to acquire visual inspection images of the metal components of the iron tower.

[0090] The depth camera is used to acquire three-dimensional depth information and surface roughness information of the rusted area.

[0091] The polarization camera is used to acquire polarization information in four directions: 0°, 45°, 90°, and 135°.

[0092] The positioning unit is used to record the geographical location where corrosion occurs.

[0093] The inertial measurement unit is used to provide UAV attitude information to achieve image attitude correction and stabilization.

[0094] The computing platform is used to perform coarse screening, fine segmentation, authenticity determination, and grade evaluation at the edge.

[0095] Secondly, the present invention provides a multimodal fusion-based system for detecting and quantitatively evaluating corrosion of iron towers, which includes a drone and a server connected by communication.

[0096] The server is used to execute a method for detecting and quantitatively evaluating corrosion of iron towers based on multimodal fusion, as described in any paragraph of the first aspect.

[0097] By adopting the above technical solution, the present invention can achieve the following technical effects: This invention requires no large amount of labeled data. Through a two-stage collaborative process of zero-shot detection model and pixel-level fine segmentation, it can achieve rapid screening and accurate mask extraction of rusted areas, significantly reducing data collection and labeling costs and significantly improving the model's generalization ability in new scenarios.

[0098] Based on this, the present invention innovatively integrates depth features and polarization features, and uses surface roughness and polarization degree to construct a dual threshold discrimination rule, which can effectively distinguish between real rust and defects with similar appearances such as dirt and watermarks, reducing the false detection rate from 25% to 40% of traditional methods to less than 5%, and significantly improving the detection accuracy.

[0099] Meanwhile, this invention adopts a multi-index weighted scoring system, which comprehensively evaluates four dimensions: rust area ratio, surface roughness, rust depth, and polarization degree. This enables scientific classification of the degree of rust and outputs targeted treatment suggestions, avoiding over-repair or under-repair.

[0100] Furthermore, by employing a progressive on-demand acquisition mechanism and pixel-level masking to limit the calculation area, this invention triggers the acquisition and feature calculation of depth and polarization data only when a candidate region is detected. This significantly reduces the amount of data acquisition and edge-end computational overhead, enabling precise alignment of RGB, depth, and polarization modal data and efficient collaborative work at the edge. It can independently complete the entire process of detection and evaluation without relying on cloud communication, thus combining functional completeness and real-time performance. Attached Figure Description

[0101] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the specific embodiments of the present invention will be briefly introduced below. It should be understood that the following drawings only show some specific embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0102] Figure 1 This is a flowchart illustrating the method for detecting and quantitatively assessing corrosion in iron towers.

[0103] Figure 2 This is a diagram of the edge hardware system architecture.

[0104] Figure 3 This is a structural diagram of the five improvements to the improved YOLO-World model.

[0105] Figure 4 This is a flowchart of the testing process for the first stage of coarse screening.

[0106] Figure 5 This is a flowchart for determining authenticity.

[0107] Figure 6 This is a flowchart for assessing the level of rust. Detailed Implementation

[0108] The technical solutions of the present invention will now be clearly and completely described with reference to the accompanying drawings in the embodiments of the present invention.

[0109] Example 1, as Figure 1 As shown, the multimodal fusion-based method for detecting and quantitatively evaluating corrosion of iron towers according to the present invention includes the following three stages.

[0110] Phase 1: Coarse screening. Zero-shot detection model is used to perform zero-shot detection on RGB images to quickly screen for rust candidate areas.

[0111] Phase 2: Fine segmentation. A pixel-level segmentation model is used to perform pixel-level precise segmentation of the candidate regions to obtain the accurate mask of the rusted areas.

[0112] Phase 3: Authenticity verification + quantitative evaluation. Depth and polarization data are collected within the segmented mask area, multimodal features are calculated, and authenticity is determined and graded.

[0113] Please see Figures 1 to 6 The first embodiment of the present invention provides a method for detecting and quantitatively evaluating corrosion of iron towers based on multimodal fusion, which includes steps S1 to S7.

[0114] S1. Acquire RGB image of the tower.

[0115] Preferably, the RGB image is captured by an RGB camera mounted on a drone platform.

[0116] The unmanned aerial vehicle (UAV) platform includes a hexacopter UAV, an RGB camera, a depth camera, a polarization camera, a positioning unit, an inertial measurement unit, and a computing platform.

[0117] The RGB camera is used to acquire visual inspection images of the metal components of the iron tower.

[0118] The depth camera is used to acquire three-dimensional depth information and surface roughness information of the rusted area.

[0119] The polarization camera is used to acquire polarization information in four directions: 0°, 45°, 90°, and 135°.

[0120] The positioning unit is used to record the geographical location where corrosion occurs.

[0121] The inertial measurement unit is used to provide UAV attitude information to achieve image attitude correction and stabilization.

[0122] The computing platform is used to perform coarse screening, fine segmentation, authenticity determination, and grade evaluation at the edge.

[0123] S2. Use a zero-shot detection model to perform preliminary identification on the RGB image to obtain the bounding box of the corrosion candidate region.

[0124] Phase 1, coarse screening (zero-sample rapid detection): zero-sample detection is performed on the collected RGB images of transmission line towers to quickly screen for candidate areas of corrosion.

[0125] Preferably, this embodiment uses an improved version of YOLO-World for zero-shot detection. That is, the zero-shot detection model is an improved version of YOLO-World.

[0126] like Figure 4 As shown, S2 specifically includes: normalizing the RGB image, and then inputting it along with the hierarchical prompt words into the improved YOLO-World to output the bounding boxes of the rust candidate regions. Specifically, normalizing the RGB image includes: adjusting the RGB image to 640×640 pixels and then normalizing it.

[0127] The specific implementation steps of S2 include S21 to S26.

[0128] S21. Image preprocessing. Adjust the image size to 640×640 pixels and normalize it.

[0129] S22, Rust Feature Enhancement. The image is processed using color-sensitive convolution, texture enhancement convolution, and edge enhancement modules.

[0130] S23, YOLO-World detection. Input the enhanced image and layered cue words, output the bounding boxes of the rust candidate regions.

[0131] S24. Multi-scale sensitivity adjustment. Automatically adjusts detection sensitivity based on candidate region size.

[0132] S25. Confidence Filtering. Filter candidate regions based on adaptive thresholds.

[0133] S26. List of candidate rust regions, usually 3-15.

[0134] Based on the above embodiments, in an optional embodiment of the present invention, such as Figure 3 As shown, the improvements in the improved version of YOLO-World include a corrosion feature enhancement module, layered prompt words, a multi-scale detection head, a multi-scale corrosion sensitivity adjustment mechanism, and an adaptive confidence threshold module.

[0135] Preferably, the rust feature enhancement module is used to process RGB images. The rust feature enhancement module includes color-sensitive convolution, texture enhancement convolution, and edge enhancement modules.

[0136] The color-sensitive convolution converts the RGB image to the HSL color space using a dynamic convolution kernel and calculates a color attention map to obtain a color-sensitive feature map.

[0137] .

[0138] .

[0139] In the formula, This is a color attention map. This is the activation function. This is the result of convolution processing on an HSL color space image. It is a dynamic convolution kernel. It is the basic convolution kernel. This represents the operation performed in conjunction with the color attention map.

[0140] The texture enhancement convolution uses a Gabor filter bank to extract multi-directional textures to obtain texture enhancement feature maps. The direction set is... For the direction set of the Gabor filter bank, , , , These represent the texture extraction directions.

[0141] The edge enhancement module uses edge-sensitive convolution to enhance the boundary of the rusted area to obtain an edge enhancement feature map.

[0142] The color-sensitive feature map, texture enhancement feature map, and edge enhancement feature map are fused using concatenation and 1×1 convolution to obtain rust enhancement features.

[0143] Preferably, the layered cue words include color cue words, texture cue words, location cue words, and negative sample cue words.

[0144] Color-coded terms include: reddish-brown metal, orange-brown surface, rust-colored steel, etc. Texture cues include: rough metal surface, flaky corrosion, and crusted iron.

[0145] Location-related keywords (for steel towers) include: rust on tower, corrosion on angle steel, rusty metal connection, etc.

[0146] Negative sample prompts (excluding false positives) include prompts such as: smooth paint, clean metal, and shiny steel.

[0147] Preferably, the multi-scale detection head includes a P3 detection head for detecting small-area corrosion, a P4 detection head for detecting medium-area corrosion, and a P5 detection head for detecting large-area corrosion.

[0148] Specifically, the P3 detection head is used to detect small-area corrosion (8×8 pixels). Output: Small anchor frame: [8, 16, 32]. The P4 detection head is used to detect medium-area corrosion (16×16 pixels). Output: Medium anchor frame: [16, 32, 64]. The P5 detection head is used to detect large-area corrosion (32×32 pixels). Output: Large anchor frame: [32, 64, 128]. Different scales are then connected through an FPN structure.

[0149] Preferably, the multi-scale corrosion sensitivity adjustment mechanism includes: increasing the detection sensitivity and decreasing the confidence threshold for small-area corrosion regions smaller than 32×32 pixels to enhance the feature weight of small targets; applying standard detection sensitivity to medium-area corrosion regions from 32×32 pixels to 128×128 pixels to balance precision and recall; and decreasing false alarm sensitivity and increasing the confidence threshold for large-area corrosion regions larger than 128×128 pixels to avoid misclassifying large areas of dirt as corrosion.

[0150] Specifically, the sensitivity parameter is dynamically adjusted: the sensitivity level is automatically selected based on the candidate region size, and the NMS (non-maximum suppression) threshold is adaptively adjusted.

[0151] Preferably, the adaptive confidence threshold module dynamically adjusts the threshold according to the lighting conditions, contrast and signal-to-noise ratio, wherein the threshold for ideal quality images is 0.5, the threshold for images under normal conditions is 0.4, and the threshold for images under severe conditions is 0.3.

[0152] Specifically, the threshold is dynamically adjusted based on image quality: Image quality indicators are analyzed: lighting conditions (average brightness), contrast, and signal-to-noise ratio (SNR). A high-quality image is defined as having uniform brightness, sufficient contrast, and a high SNR (no noise, no overexposure / underexposure, and a clear image). The confidence threshold for a high-quality image is 0.5. A medium-quality image is defined as having average brightness / contrast and a moderate SNR (slight noise, slight overexposure / underexposure, and identifiable rust). The confidence threshold for a medium-quality image is 0.4. A low-quality image is defined as having uneven brightness (too dark / too bright), poor contrast, and a low SNR (severe noise, blur, occlusion, and motion blur). The confidence threshold for a low-quality image is 0.3.

[0153] S3. Perform pixel-level segmentation on the candidate region corresponding to the bounding box of the corrosion candidate region to obtain the corrosion region mask. Based on the above embodiments, in an optional embodiment of the present invention, S3 specifically includes S31 to S34.

[0154] S31. Adjust the candidate region to 224×224 pixels.

[0155] S32. Use the center of the bounding box of the rust candidate region output by YOLO-World as the prompt point.

[0156] S33. Input the candidate region and the prompt point into MobileSAM for pixel-level segmentation and output a pixel-level segmentation mask.

[0157] S34. Perform morphological operations on the pixel-level segmentation mask to remove noise and fill holes, obtaining the rust region mask. The rust region mask is used to define the calculation area for subsequent depth and polarization features.

[0158] Specifically, the second stage of fine segmentation (precise segmentation) performs pixel-level precise segmentation on the candidate regions selected in the first stage to obtain a precise mask of the rusted region, providing accurate region boundaries for subsequent calculation of depth and polarization features within the masked region.

[0159] This embodiment uses MobileSAM for accurate segmentation. The specific steps are as follows: Image preprocessing. Adjust the candidate region size to 224×224 pixels. Cue point generation. Use the center of the bounding box output by YOLO-World as the cue point. MobileSAM segmentation. Input the image and cue points, output a pixel-level segmentation mask. Post-processing. Morphological operations remove noise and fill holes. Output: A mask of accurately segmented rusted regions (this mask is used for subsequent depth and polarization feature calculations to define the region).

[0160] S4. After obtaining the rusted area mask, acquire the depth map and polarization map that are precisely aligned with the RGB image, obtained by data acquisition of the area corresponding to the rusted area mask using a depth camera and a polarization camera.

[0161] Phase 3: Authenticity verification + quantitative evaluation (multimodal feature discrimination and rating).

[0162] Function Description: Within the precise mask area obtained by MobileSAM segmentation, depth and polarization data are collected, multimodal features are calculated, and authenticity is determined and corrosion level is assessed.

[0163] Data acquisition process: After stage two completes fine segmentation and obtains the accurate mask, the system triggers the depth camera and polarization camera to align with the area for data acquisition. At this time, the acquired depth map and polarization map are precisely aligned with the RGB image, ensuring that all feature calculations are performed within the same mask area.

[0164] S5. Within the area defined by the rust region mask, calculate the depth feature based on the depth map and calculate the polarization feature based on the polarization map. Based on the above embodiments, in an optional embodiment of the present invention, S5 specifically includes S51 to S56.

[0165] S51. Perform point cloud preprocessing on the 3D point cloud data within the rust area mask acquired by the depth camera to remove outliers.

[0166] S52. Fit the reference plane using the least squares method and calculate the height deviation of each point from the fitted plane.

[0167] S53. Calculate the surface roughness based on the height deviation. .

[0168] .

[0169] In the formula, This represents the number of points in the point cloud within the masked area. For the first The height deviation of each point from the fitted plane.

[0170] S54, Calculate the average depth .

[0171] .

[0172] In the formula, This represents the number of points in the point cloud within the masked area. For the first The perpendicular distance from a point to the reference plane; a positive value indicates that the point is below the reference plane.

[0173] The physical principle behind deep feature analysis is that rusted surfaces are rough and uneven, while normal metal surfaces are relatively smooth. Dirt and watermarks adhere to smooth surfaces without changing their roughness.

[0174] The specific implementation steps include: acquiring 3D point cloud data of the masked region using a depth camera (only calculating the point cloud within the mask); preprocessing the point cloud to remove outliers; fitting a reference plane using the least squares method; calculating the height deviation of each point from the fitted plane; and calculating the surface roughness. (Arithmetic mean deviation of the profile) and average depth.

[0175] S55. Calculate the Stokes parameters based on the polarization images of the four polarization angles of 0°, 45°, 90°, and 135°.

[0176] .

[0177] .

[0178] .

[0179] In the formula, This is the first Stokes parameter. This is the second Stokes parameter. This is the third Stokes parameter. This represents the pixel intensity value of an image captured by a polarization camera at a polarization angle of 0°. This represents the pixel intensity value of an image captured by a polarization camera at a polarization angle of 45°. This represents the pixel intensity value of an image captured by a polarization camera at a 90° polarization angle. This represents the pixel intensity value of an image captured by a polarization camera at a polarization angle of 135°.

[0180] S56. Calculate the polarization degree of each pixel within the mask region. .

[0181] .

[0182] S57. Calculate the average polarization degree within the mask region. .

[0183] .

[0184] In the formula, This represents the number of pixels within the mask area. For the first The degree of polarization of each pixel.

[0185] The physical principle of polarization characteristic analysis is as follows: metal surfaces exhibit specular reflection and have a high degree of polarization. Corroded surfaces exhibit diffuse reflection and have a low degree of polarization.

[0186] The specific implementation steps include: acquiring polarization images at four polarization angles (0°, 45°, 90°, and 135°); calculating the Stokes parameters; and calculating the degree of polarization of each pixel within the mask region based on the Stokes parameters. And the average polarization degree within the mask region.

[0187] S6. Based on the depth features and polarization features, the candidate rust regions are distinguished as genuine or fake, and when they are in the fuzzy range, a comprehensive judgment is made by combining visual features.

[0188] Based on the above embodiments, in an optional embodiment of the present invention, the following is defined: Surface roughness. This is the high roughness threshold. This is the low roughness threshold. The low polarization threshold. This is the high polarization threshold. This represents the average degree of polarization.

[0189] like Figure 5 As shown, the authenticity determination includes: when When this occurs, the candidate corrosion areas are determined to be likely to be true corrosion. Preferably, The typical value is 30-40 μm.

[0190] when When the candidate rust area is determined to be prone to pseudo-rust (stains / water stains), it is preferred that... Typical values ​​are 15-25 μm.

[0191] when When the corrosion candidate area is determined to be prone to true corrosion (diffuse reflection), it is determined that the corrosion candidate area tends to be true corrosion. The typical value is 0.35-0.45.

[0192] when When the rust candidate area is determined to be prone to pseudo-rust (specular reflection), it is determined that the rust candidate area tends to be pseudo-rust (specular reflection). The typical value is 0.45-0.55.

[0193] like and If it is found to be true corrosion, it will be subject to quantitative assessment.

[0194] like or If the result is false rust, it will be identified as "stain / water stain" and output as "stain / water stain".

[0195] Preferably, the above thresholds can be dynamically adjusted according to the application scenario. For example, in a coastal high-salt-fog environment: Reduced to 25-30μm. Increase to 0.40-0.50. Industrial pollution zone: Increased to 35-45μm. Decreased to 0.30-0.40. Arid and low-rainfall areas: Reduce to 0.40-0.50 to avoid misjudging water stains.

[0196] Other situations, such as when or At this time, depth features and polarization features are in the fuzzy range, and it is necessary to combine color features and texture features for comprehensive scoring.

[0197] Specifically, when depth features and polarization features are in the fuzzy range, the following visual feature discrimination rules are first adopted.

[0198] Color feature extraction and determination: Calculate the color features within the mask area in the HSL color space.

[0199] .

[0200] .

[0201] .

[0202] In the formula: It represents the average hue. The average saturation. This represents the average brightness. This represents the number of pixels within the mask area. , and The first The hue, saturation, and brightness of each pixel.

[0203] The color determination rules are as follows: like and and It tends to be true rust (reddish brown, moderate saturation, darker).

[0204] like or or If so, it tends to be a pseudo-rust (non-rust color, too low saturation, too bright).

[0205] In other cases, the process proceeds to texture feature determination.

[0206] Texture feature extraction and determination are performed using the gray-level co-occurrence matrix (GLCM). ) Calculate the texture features of the masked region. Wherein, To statistically analyze gray levels and In distance ,direction The probability of both appearing simultaneously.

[0207] Texture feature calculation includes: Contrast: .

[0208] energy: .

[0209] entropy: .

[0210] The texture determination rules are as follows: like and and It tends to be true rust (high roughness, irregular texture).

[0211] like or or It tends to be pseudo-rust (smooth surface, regular texture).

[0212] In other cases, a comprehensive score is given based on a combination of color and texture features.

[0213] Computational visual feature comprehensive score: .

[0214] In the formula, The overall score is based on visual features. For color weights. The score is determined based on the color characteristics. This represents the texture weight. The score is determined based on the texture features.

[0215] when At that time, it was determined to be genuine rust and entered into a quantitative assessment.

[0216] when When the condition is found to be false rust, the output will be "stain / water stain".

[0217] Preferred, . . The value range is 0-100 points. The value range is (0-100 points).

[0218] True corrosion conditions include: Condition C1: (Reddish-brown area). Condition C2: (Moderate saturation). Condition C3: (Darker).

[0219] The scoring rule is: if conditions C1, C2, and C3 are all satisfied simultaneously, then... Points. If both conditions are met, then... Points. If one condition is met, then... Points. If none of them are satisfied, then... point.

[0220] True corrosion conditions include: Condition T1: (High contrast, high roughness). Condition T2: (Low energy, irregular texture). Condition T3: (High entropy, complex texture).

[0221] The scoring rule is: if conditions T1, T2, and T3 are all satisfied simultaneously, then... Points. If both conditions are met, then... Points. If one condition is met, then... Points. If none of them are satisfied, then... point.

[0222] S7. For areas determined to be true rust, perform multi-index weighted scoring to obtain the rust level and corresponding treatment recommendations.

[0223] Based on the above embodiments, in an optional embodiment of the present invention, such as Figure 6 As shown, for areas confirmed as genuine corrosion, a multi-index weighted scoring method is used for grade assessment.

[0224] The S7 multi-indicator weighted score includes: S71. Calculate the percentage of rusted area. Surface roughness, average depth, and degree of polarization.

[0225] .

[0226] .

[0227] .

[0228] In the formula, This represents the number of pixels in the rust mask area. This represents the number of pixels in the component mask area. coordinates The mask value for the rusted area. coordinates The component area mask value at that location.

[0229] Specifically, the percentage of rusted area = (number of pixels in the rusted area / number of pixels in the component area) × 100%. Number of pixels in the rusted area: the total number of pixels in the rusted area mask obtained through MobileSAM segmentation. Number of pixels in the component area: the total number of pixels in the metal component area obtained through object detection or semantic segmentation.

[0230] S72, Calculate the area score Roughness score Depth Score And polarization score A piecewise linear scoring method is used, with scores ranging from 0 to 100.

[0231] .

[0232] .

[0233] .

[0234] .

[0235] In the formula, Represents a logarithmic function. Surface roughness. This is the reference value for maximum roughness. This represents the average depth. This is the maximum depth reference value. This represents the average degree of polarization.

[0236] S73. Calculate the overall score .

[0237] .

[0238] In the formula, Area weight. This represents the roughness weight. For depth weights. This is the polarization degree weight.

[0239] Default weight settings: =0.3. =0.3. =0.2. =0.2.

[0240] S74. Level Determination and Handling Recommendations.

[0241] when At that time, it was determined to be a level 1 light corrosion, and a treatment suggestion of regular monitoring and no treatment at present was issued.

[0242] when At that time, it was determined to be a level 2 moderate rust, and treatment recommendations for surface rust removal and anti-corrosion coating were provided.

[0243] when When the corrosion is classified as Level 3 severe corrosion, a treatment suggestion to replace the component is output.

[0244] Based on the above embodiments, in an optional embodiment of the present invention, the method for detecting and quantitatively evaluating corrosion of iron towers based on multimodal fusion adopts a progressive on-demand data acquisition mechanism.

[0245] Phase 1: The progressive on-demand acquisition mechanism includes: first acquiring RGB images and performing preliminary YOLO-World recognition.

[0246] Phase 2: When a candidate rust region is detected in Phase 1, MobileSAM is triggered to perform pixel-level precise segmentation to obtain the rust region mask.

[0247] Phase 3: After obtaining the rust region mask in Phase 3, the depth camera and polarization camera are triggered to acquire depth and polarization images. If no rust candidate region is detected in Phase 1, the acquisition of depth and polarization images is not triggered. The depth and polarization features are calculated only within the rust region mask to reduce the overall image computation and achieve region alignment between the RGB image, depth image, and polarization image.

[0248] The multimodal fusion-based method for detecting and quantitatively evaluating corrosion in iron towers proposes for the first time a two-stage collaborative process of YOLO-World coarse screening and MobileSAM fine segmentation. Combining zero-sample detection capability and pixel-level segmentation accuracy, it enables precise location of corrosion areas without the need for labeled data.

[0249] Five improvements have been made to YOLO-World for corrosion detection: First, a corrosion feature enhancement module including color-sensitive convolution, Gabor texture enhancement, and edge enhancement. Second, a multi-scale sensitivity adjustment mechanism that adjusts detection sensitivity for corrosion regions of different sizes. Third, a layered cue word engineering system with four layers of cue words: color, texture, location, and negative samples. Fourth, a multi-scale detection head system with three different scales: P3, P4, and P5. Fifth, an adaptive confidence threshold that dynamically adjusts the detection threshold based on image quality. By quickly filtering candidate corrosion regions, a large number of negative samples can be eliminated (approximately 80% of images without corrosion do not need to proceed to stage two), significantly improving recognition speed.

[0250] YOLO-World offers fast filtering, while MobileSAM only processes candidate regions, significantly reducing computational cost. Without accurate segmentation, multimodal feature computation across the entire image is required, which is unsustainable at edge deployments. MobileSAM's lightweight design is well-suited for edge deployments. The combination of YOLO-World and MobileSAM achieves an optimal balance between speed, accuracy, and computational cost.

[0251] This invention employs a two-stage process of coarse screening followed by fine segmentation, balancing speed and accuracy. YOLO-World enables rapid screening, while MobileSAM provides precise segmentation. The segmentation mask lays the foundation for multimodal fusion at the edge. Existing technologies either cannot achieve pixel-level segmentation or suffer from excessive computational demands, making edge deployment impractical.

[0252] This invention presents a novel method for detecting and quantitatively evaluating corrosion in iron towers based on multimodal fusion. It proposes for the first time to combine surface roughness and polarization degree in the determination of genuine and counterfeit corrosion, constructing a "roughness-polarization degree dual-threshold discrimination rule." The originality of this method is reflected in three aspects: feature selection, discrimination rule design, and feature combination.

[0253] First, the originality of the feature selection: This is the first time that significant differences have been discovered between rust, dirt, and watermarks in terms of surface roughness and polarization, and this discovery has been verified through extensive experiments. No existing patents have used such features to distinguish genuine rust from counterfeit rust.

[0254] Second, the originality of the discrimination rule: a "roughness-polarization degree dual threshold cross-verification discrimination rule" is proposed, which realizes the true and false judgment through the mutual verification of two physical features.

[0255] Third, the originality of the feature combination: For the first time, depth and polarization features are combined to form a complete system for distinguishing between genuine and counterfeit products. A single feature may lead to misjudgment: rough rust stains versus smooth rust. Dual-feature cross-validation: improves the accuracy of the judgment. Ambiguous region supplementation: combines visual features for comprehensive judgment.

[0256] This invention utilizes depth and polarization sensor information to explicitly model the differences between corrosion and similar defects from both geometric and optical dimensions, giving the features clear physical meaning and interpretability. In contrast, existing technologies rely on a single RGB mode, judging solely based on appearance similarity. This invention, through the fusion of depth and polarization features, reduces the false detection rate from 25-40% to below 5%.

[0257] The multimodal fusion-based method for detecting and quantitatively evaluating corrosion in iron towers proposes for the first time a multimodal progressive on-demand data acquisition mechanism. This mechanism dynamically determines whether to acquire the next level of data based on the results of the preceding processing, rather than acquiring all modal data at the beginning.

[0258] Progressive data acquisition strategy: Instead of acquiring all modal data at the beginning, the decision to continue is made based on the results of previous processing. Only RGB data is acquired when there is no corrosion, reducing the amount of data and computational overhead. Depth and polarization acquisition are only triggered when there are candidate regions.

[0259] On-demand feature extraction: Depth and polarization features are calculated only within the masked area, avoiding full-image calculation and improving computational efficiency. MobileSAM segmentation masks provide accurate regions for subsequent processing.

[0260] The three-stage collaborative mechanism consists of three phases: Phase 1, rapid screening to eliminate a large number of negative samples; Phase 2, precise segmentation to obtain candidate region boundaries; and Phase 3, multimodal verification to ensure accurate discrimination.

[0261] This invention significantly reduces data acquisition volume and computational overhead (by approximately 60-80%) by progressive, on-demand data collection, while maintaining detection accuracy. Existing technologies either collect all data, leading to resource waste, or rely solely on RGB, which is insufficient for authenticity verification.

[0262] This invention establishes a four-dimensional corrosion level assessment system for the first time. It comprehensively evaluates the degree of corrosion using a weighted scoring method, resulting in a more complete and accurate assessment that allows for the scientific formulation of maintenance strategies. In contrast, existing single-dimensional assessments cannot accurately reflect the impact of corrosion on structural safety.

[0263] This invention proposes for the first time a two-stage collaborative process of coarse screening using a zero-shot detection model and pixel-level fine segmentation. The zero-shot detection model achieves rapid zero-shot rust screening through five improvements (rust feature enhancement, multi-scale sensitivity adjustment, hierarchical cue word engineering, multi-scale detection head, and adaptive confidence threshold). The pixel-level segmentation model performs pixel-level precise segmentation of candidate regions to obtain rust region boundary masks. The two work together, balancing speed and accuracy, with the segmentation mask providing a crucial foundation for multimodal fusion at the edge.

[0264] This invention is the first to combine surface roughness and polarization degree in the identification of genuine and counterfeit rust. By quantitatively describing the differences between rust and similar defects in both geometric and optical dimensions, it achieves an interpretable mechanism for distinguishing genuine from counterfeit rust. These features transform physical metallurgy knowledge into a computable mathematical model, effectively solving the technical problem of misidentifying defects such as dirt and watermarks as rust.

[0265] This invention designs a progressive on-demand acquisition mechanism, which does not collect all modal data at the beginning, but first uses RGB to quickly filter and confirm candidate regions before collecting depth and polarization data as needed, which greatly reduces the amount of data collected and computational overhead.

[0266] This invention establishes a corrosion level assessment system based on multi-index weighted scoring, which integrates four indicators: area, roughness, depth, and polarization, providing a more accurate basis for structural strength assessment and maintenance decisions.

[0267] The aforementioned innovations enable this invention to significantly outperform existing technologies in terms of reduced false detection rate, comprehensive evaluation dimensions, and efficient edge-end fusion, demonstrating outstanding substantive features and significant progress.

[0268] This invention employs a two-stage collaborative process: a zero-shot detection model for coarse screening and pixel-level fine segmentation. The zero-shot detection model achieves rapid screening of rust through zero-shot learning, requiring no labeled data. The pixel-level segmentation model performs pixel-level precise segmentation of candidate regions, obtaining boundary masks for the rusted areas. Existing technologies, such as supervised learning models like YOLO, require large amounts of labeled data, or segmentation models like Mask R-CNN suffer from excessive computational costs. This invention allows for rapid adaptation to new scenarios, significantly reduces data acquisition costs, and simultaneously achieves pixel-level segmentation accuracy.

[0269] This invention employs a pixel-level segmentation model to perform precise pixel-level segmentation of candidate regions, obtaining boundary masks for rusted areas. This mask enables precise alignment of RGB, depth, and polarization modal data to the same region, calculating depth and polarization features only within the masked area. Existing technologies, however, only output bounding boxes, failing to achieve pixel-level alignment and requiring full-image multimodal feature computation. This invention significantly reduces computational load (approximately 80%), making it a key technology for achieving edge-side multimodal fusion.

[0270] This invention distinguishes between genuine and counterfeit items by using depth features (surface roughness) and polarization features, explicitly modeling the differences between rust, dirt, and watermarks in both geometric and optical dimensions. In contrast, existing technologies rely solely on the color and texture features of RGB images, lacking a dedicated design for authenticity detection. This significantly reduces the false positive rate and dramatically improves the accuracy of this invention.

[0271] This invention employs depth and polarization sensors to calculate surface roughness, depth, and polarization degree, and establishes a four-dimensional evaluation system based on the proportion of rust area. Existing technologies, however, can only assess the rust area. This allows for the development of scientific maintenance strategies, avoiding over- or under-maintenance.

[0272] This invention achieves efficient collaborative fusion of multimodal data at the edge by employing a progressive on-demand acquisition mechanism (first filtering RGB data, then acquiring depth and polarization), pixel-level precise segmentation (achieving multimodal data alignment), and feature calculation within the masked region (significantly reducing computational load). Existing edge deployment solutions all utilize a single RGB camera or simple decision-level fusion. This allows the invention to combine functional completeness with edge independence, enabling it to operate even in network-free areas and providing excellent real-time performance.

[0273] Example 2: The present invention provides a multimodal fusion-based system for detecting and quantitatively evaluating corrosion of iron towers, which includes a drone and a server connected by communication.

[0274] The server is used to execute a method for detecting and quantitatively evaluating corrosion of iron towers based on multimodal fusion, as described in any paragraph of Embodiment 1.

[0275] like Figure 2As shown, the tower corrosion detection and quantitative assessment system consists of three parts: a drone platform, a computing platform, and a multimodal sensor.

[0276] The drone platform includes drones, RGB cameras, depth cameras, and polarization cameras.

[0277] The drone is a DJI Matrice 350 RTK (M350 RTK), a six-rotor drone with a flight time of about 45 minutes, a maximum takeoff weight of 9.2 kg, and an operating altitude of 10-80 meters.

[0278] The RGB camera is a DJI Zenmuse H20T, with 4K resolution, a focal length of 24-200mm, and 30x optical zoom, used for visual inspection of rusted areas.

[0279] The depth camera is an Intel RealSense D435i with a depth resolution of 640×480, a measurement accuracy of ±0.1mm, and a measurement distance of 0.3-10m. It is used to acquire three-dimensional depth information and calculate surface roughness of the rusted area.

[0280] The polarization camera is a Freescale Bayer IXPUG-5280MP with a resolution of 1920×1080 and four-directional polarization (0° / 45° / 90° / 135°), used to acquire polarization information for authenticity verification.

[0281] The drone has a built-in positioning unit. This unit includes a DJI D-RTK 2 high-precision positioning module and a built-in inertial measurement unit (IMU). The DJI D-RTK 2 high-precision positioning module, with an accuracy of ±2cm, records the precise geographical location of the corrosion, facilitating subsequent maintenance. The built-in IMU, with an attitude angle accuracy of ±0.1°, provides the drone's attitude information, enabling attitude correction and stabilization of the image.

[0282] The computing platform is an NVIDIA Jetson Orin NX (16GB). Specifications include 256GB eMMC, 70 TOPS INT8 computing power, and a power consumption of 15-25W.

[0283] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting and quantitatively assessing corrosion of iron towers based on multimodal fusion, characterized in that, Include: S1. Acquire RGB images of the tower; S2. Use a zero-shot detection model to perform preliminary identification on the RGB image to obtain the bounding box of the corrosion candidate region; S3. Perform pixel-level segmentation on the candidate region corresponding to the bounding box of the corrosion candidate region to obtain the corrosion region mask; S4. After obtaining the rusted area mask, acquire a depth map and a polarization map that are precisely aligned with the RGB image, obtained by data acquisition of the area corresponding to the rusted area mask using a depth camera and a polarization camera; S5. Within the area defined by the rust region mask, calculate the depth features based on the depth map and calculate the polarization features based on the polarization map; S6. Based on the depth features and polarization features, the candidate rust regions are distinguished as genuine or false, and when they are in the fuzzy range, visual features are combined for comprehensive judgment. S7. Perform multi-index weighted scoring on areas determined to be true rust to obtain the rust level and corresponding treatment recommendations; S5 specifically includes: Point cloud preprocessing is performed on the 3D point cloud data within the rust area mask acquired by the depth camera to remove outliers; The reference plane is fitted using the least squares method, and the height deviation from the fitted plane at each point is calculated. Surface roughness is calculated based on the height deviation. ; ； In the formula, The number of points in the point cloud within the masked area; For the first The height deviation of each point from the fitted plane; Calculate the average depth ; ； In the formula, The number of points in the point cloud within the masked area; For the first The perpendicular distance from each point to the reference plane, with positive values ​​indicating that the point is below the reference plane; Calculate the Stokes parameters based on the polarization images at four polarization angles: 0°, 45°, 90°, and 135°. ； ； ； In the formula, The first Stokes parameter; For the second Stokes parameter; This is the third Stokes parameter; This represents the pixel intensity value of an image captured by a polarization camera at a polarization angle of 0°. The pixel intensity value of the image captured by the polarization camera at a polarization angle of 45°; The pixel intensity value of the image captured by the polarization camera at a polarization angle of 90°; The pixel intensity value of the image captured by the polarization camera at a polarization angle of 135°; Calculate the polarization degree of each pixel within the mask region. ; ； Calculate the average polarization degree within the mask region ; ； In the formula, This represents the number of pixels within the mask area; For the first The degree of polarization of each pixel.

2. The method for detecting and quantitatively evaluating corrosion of iron towers based on multimodal fusion according to claim 1, characterized in that, The zero-shot detection model is an improved version of YOLO-World. The RGB image is initially identified using a zero-shot detection model, specifically including: normalizing the RGB image and then inputting it along with hierarchical prompt words into the improved YOLO-World to output bounding boxes of rust candidate regions; The improved YOLO-World includes a corrosion feature enhancement module, layered prompt words, a multi-scale detection head, a multi-scale corrosion sensitivity adjustment mechanism, and an adaptive confidence threshold module; The rust feature enhancement module is used to process RGB images; the rust feature enhancement module includes color-sensitive convolution, texture enhancement convolution, and edge enhancement modules. The color-sensitive convolution converts the RGB image to the HSL color space using a dynamic convolution kernel and calculates a color attention map to obtain a color-sensitive feature map. ； ； In the formula, Color attention map; For activation functions; This is the result of convolution processing on an HSL color space image; Dynamic convolution kernel; Based on the convolutional kernel; This indicates the operations performed in conjunction with the color attention map; The texture enhancement convolution uses a Gabor filter bank to extract multi-directional textures to obtain texture enhancement feature maps; where the direction set is... ; For the direction set of the Gabor filter bank, , , , These represent the texture extraction directions; The edge enhancement module uses edge-sensitive convolution to enhance the boundary of the rusted area in order to obtain an edge enhancement feature map. The color-sensitive feature map, texture enhancement feature map, and edge enhancement feature map are fused using concatenation and 1×1 convolution to obtain rust enhancement features.

3. The method for detecting and quantitatively evaluating corrosion of iron towers based on multimodal fusion according to claim 2, characterized in that, Normalizing the RGB image specifically includes adjusting the RGB image to 640×640 pixels and then normalizing it. Layered cue words include color cue words, texture cue words, location cue words, and negative sample cue words; The multi-scale corrosion sensitivity adjustment mechanism includes: increasing the detection sensitivity and lowering the confidence threshold for small corrosion areas smaller than 32×32 pixels; using the standard detection sensitivity for medium-sized corrosion areas from 32×32 pixels to 128×128 pixels; and reducing the false alarm sensitivity and increasing the confidence threshold for large corrosion areas larger than 128×128 pixels. The multi-scale detection head includes the P3 detection head for detecting small-area corrosion, the P4 detection head for detecting medium-area corrosion, and the P5 detection head for detecting large-area corrosion. The adaptive confidence threshold module dynamically adjusts the threshold based on lighting conditions, contrast, and signal-to-noise ratio. The threshold for ideal quality images is 0.5, for images under normal conditions it is 0.4, and for images under severe conditions it is 0.

3.

4. The method for detecting and quantitatively evaluating corrosion of iron towers based on multimodal fusion according to claim 1, characterized in that, The candidate regions corresponding to the bounding boxes of the corrosion candidate regions are segmented at the pixel level, specifically including: The candidate region is adjusted to 224×224 pixels; Use the center of the bounding box of the rust candidate region output by YOLO-World as the cue point; The candidate region and the cue point are input into MobileSAM for pixel-level segmentation, and a pixel-level segmentation mask is output. Morphological operations are performed on the pixel-level segmentation mask to remove noise and fill holes, thereby obtaining the rust region mask; wherein, the rust region mask is used to define the calculation area for subsequent depth features and polarization features.

5. The method for detecting and quantitatively evaluating corrosion of iron towers based on multimodal fusion according to claim 1, characterized in that, definition: For surface roughness; High roughness threshold; Low roughness threshold; Low polarization threshold; High polarization threshold; The average degree of polarization; The authenticity determination includes: when When the candidate rust area is determined to be likely to be true rust, it is determined that the candidate rust area is likely to be true rust. when When the candidate rust area is determined to be prone to pseudo-rust, it is determined that the candidate rust area is prone to pseudo-rust. when When the candidate rust area is determined to be likely to be true rust, it is determined that the candidate rust area is likely to be true rust. when When the candidate rust area is determined to be prone to pseudo-rust, it is determined that the candidate rust area is prone to pseudo-rust. like and If so, it is determined to be true corrosion and enters the quantitative assessment stage; like or If so, it is judged as pseudo-rust and output as "stain / water stain"; When depth and polarization features are in the fuzzy range, a comprehensive score is obtained by combining color and texture features. ； In the formula, The overall score is based on visual features. For color weights, To determine the score based on color characteristics, For texture weights, The score is determined based on the texture features. when At that time, it was determined to be genuine rust and entered into a quantitative assessment; when When the time comes, it is judged as fake rust and outputs "stain / water stain".

6. The method for detecting and quantitatively evaluating corrosion of iron towers based on multimodal fusion according to claim 1, characterized in that, The multi-indicator weighted score includes: Calculate the percentage of rusted area ; ； ； ； In the formula, This represents the number of pixels in the rust mask area. The number of pixels in the component mask area; coordinates The mask value for the rusted area; coordinates The component region mask value at the location; Calculate area score Roughness score Depth Score And polarization score ; ； ； ； ； In the formula, Represents a logarithmic function; For surface roughness; This is the reference value for maximum roughness; Average depth; This is the maximum depth reference value; The average degree of polarization; Calculate the overall score ; ； In the formula, Area weighting; For roughness weight; Depth weights; Polarization degree weight; when At that time, it was determined to be a level 1 light corrosion, and a treatment suggestion of regular monitoring and no treatment for the time being was issued; when At that time, it was determined to be a level 2 moderate rust, and treatment recommendations for surface rust removal and anti-corrosion coating were provided; when When the corrosion is classified as Level 3 severe corrosion, a treatment suggestion to replace the component is output.

7. A method for detecting and quantitatively evaluating corrosion of iron towers based on multimodal fusion according to any one of claims 1 to 6, characterized in that, The method employs a progressive, on-demand data collection mechanism; Phase 1: The progressive on-demand acquisition mechanism includes: first acquiring RGB images and performing preliminary YOLO-World recognition; Phase 2: When a candidate rust region is detected in Phase 1, MobileSAM is triggered to perform pixel-level precise segmentation to obtain the rust region mask; Phase 3: After obtaining the rust region mask in Phase 3, the depth camera and polarization camera are triggered to acquire depth and polarization images; if no rust candidate region is detected in Phase 1, the acquisition of depth and polarization images is not triggered; wherein, the depth features and polarization features are calculated only within the rust region mask to reduce the computational load of the entire image and achieve region alignment of the RGB image, depth image and polarization image.

8. A method for detecting and quantitatively evaluating corrosion of iron towers based on multimodal fusion according to any one of claims 1 to 6, characterized in that, The RGB images were captured by an RGB camera mounted on a drone platform. The unmanned aerial vehicle (UAV) platform includes a hexacopter UAV, an RGB camera, a depth camera, a polarization camera, a positioning unit, an inertial measurement unit, and a computing platform. The RGB camera is used to acquire visual inspection images of the metal components of the iron tower; The depth camera is used to acquire three-dimensional depth information and surface roughness information of the corroded area; The polarization camera is used to acquire polarization information in four directions: 0°, 45°, 90°, and 135°. The positioning unit is used to record the geographical location where corrosion occurs; The inertial measurement unit is used to provide UAV attitude information to achieve image attitude correction and stabilization; The computing platform is used to perform coarse screening, fine segmentation, authenticity determination, and grade evaluation at the edge.

9. A system for detecting and quantitatively evaluating corrosion of iron towers based on multimodal fusion, characterized in that, This includes drones and servers with communication connections; The server is used to execute a method for detecting and quantitatively evaluating corrosion of iron towers based on multimodal fusion as described in any one of claims 1 to 8.

Images