Foggy power transmission line image defect detection method based on CLIP semantic guidance

Abstract

The application provides a foggy day power transmission line image defect detection method based on CLIP semantic guidance, comprising: obtaining a to-be-detected foggy day image and performing depth estimation; identifying a region and assigning a material weight to generate a mask, and then combining an atmospheric scattering model to calculate an initial transmittance; extracting visual features, and inputting multi-level text prompts into a text encoder to extract features; calculating the spatial semantic similarity of visual and text features to generate a defect saliency map, and optimizing the initial transmittance to obtain semantic transmittance; inputting a depth map, an atmospheric light value and the semantic transmittance into a physical semantic joint dehazing network to reconstruct a clear image; inputting the dehazed image into a defect detection network to output a result; and establishing a bidirectional gradient interaction mechanism and applying a truncated gate operation in the training stage. The application effectively establishes a semantic closed loop of dehazing and detection, and improves the distinguishability and detection accuracy of key defect regions in the foggy day power transmission line image.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and intelligent inspection of power transmission lines, and in particular to a method for detecting image defects in power transmission lines in foggy weather based on CLIP semantic guidance. Background Technology

[0002] As a critical infrastructure of the power system, the operating status of transmission lines directly affects the stability of the power grid. With the development of computer vision technology and UAV inspection equipment, automated defect detection based on inspection images has become a routine technique. Images acquired in foggy conditions are easily affected by scattering from atmospheric water droplets, resulting in contrast degradation, color shift, and blurred details, which poses challenges to feature extraction in downstream defect detection algorithms.

[0003] Existing image dehazing techniques are mainly based on physical priors or single-pixel-level deep learning. These traditional dehazing algorithms often take minimizing pixel-level reconstruction errors as an independent optimization objective. When dealing with transmission line scenarios, they struggle to adequately address the specific requirements of downstream inspection tasks for the identifiability of subtle defects in critical components such as insulators, fittings, and conductors. Furthermore, current cascaded architectures typically optimize the dehazing network and the detection module independently, meaning the dehazing network cannot receive gradient feedback from downstream detectors regarding contrast in key areas.

[0004] Furthermore, fixed atmospheric scattering models have limitations when adapting to the light reflection characteristics of slender structures such as conductors and special material surfaces. After processing, they may suffer from insufficient texture detail in key areas or halo artifacts. In addition, transmission line defect types exhibit a long-tailed distribution, and conventional detection models often require large-scale labeled data for training support when expanding to new defect categories. Summary of the Invention

[0005] To address the problems in existing technologies, such as the disconnect between physical models and high-level semantic information caused by independent optimization of defogging and defect detection modules for transmission line images in foggy weather, and the reliance of conventional detection models on large amounts of data for expanding long-tail defect categories, this invention provides a CLIP-based semantic guidance method for defect detection in transmission line images in foggy weather.

[0006] This invention provides a method for detecting defects in transmission line images in foggy weather based on CLIP semantic guidance, comprising the following steps: Acquire images of the transmission line to be inspected in foggy weather and perform depth estimation to obtain a depth map; The insulator region, conductor region and background region in the foggy transmission line image are identified, and corresponding material weights are assigned accordingly to generate a material weight mask. The pixel of the foggy power transmission line image is weighted using the material weight mask to reduce interference from bright metal parts and calculate the atmospheric light value. Then, the initial transmittance is calculated by combining the atmospheric scattering model. The image of the transmission line in foggy weather is input into a pre-trained visual encoder to extract visual features, and multi-level text prompts containing normal and defective parts are input into a text encoder to extract multi-level text features. The similarity between the visual features and the text features of normal and defective components in the multi-level text features is calculated for each spatial location. The difference between the similarity of all defective component text features and the similarity of normal component text features at the same spatial location is compared. The maximum value of the difference is taken and activated to generate a defect saliency map for that spatial location. The initial transmittance is optimized using the defect saliency map. The initial transmittance is then added to a semantic enhancement term constructed based on the defect saliency map and the fog concentration term to obtain a semantic transmittance that can improve the defogging effect in the defect area. The depth map, atmospheric light value, and semantic transmittance are injected as physical condition features into the physical-semantic joint dehazing network to reconstruct a clear dehazing image. The clear, dehazed image is input into the defect detection network, which then outputs the defect detection results. During the training phase, a bidirectional gradient interaction mechanism is established to calculate the gradient of the detection loss of the defect detection network with respect to the original parameters of the physical-semantic joint dehazing network. A truncation gating operation based on a preset threshold is applied to the original gradient, and the parameters of the physical-semantic joint dehazing network are updated using the truncated effective gradient, so as to establish a semantic closed loop of dehazing and detection while preventing distortion of physical reconstruction.

[0007] Optionally, the step of using the material weighting mask to weight the pixels of the foggy power transmission line image to reduce interference from bright metal components and calculate the atmospheric light value, and then combining it with an atmospheric scattering model to calculate the initial transmittance, includes: When generating the material weight mask, the insulator region is assigned the highest weight, the background region is assigned the lowest weight, and the conductor region is assigned a center weight; the set of brightest pixels in the dark channel image of the foggy transmission line is obtained, and the pixel values ​​of the pixels in the set are weighted and averaged using the material weight mask to obtain the atmospheric light value; The initial physical transmittance is derived based on the atmospheric light value and truncated to a lower limit. The grayscale image of the transmission line in foggy weather is used as a guiding image. The truncated initial physical transmittance is smoothed by guiding filtering to preserve its edges, and the final initial transmittance is output.

[0008] Optionally, the step of inputting multi-level text prompts containing normal and defective components into a text encoder to extract multi-level text features includes: Construct a three-tiered text prompt library that includes scene-level prompts, normal component-level prompts, and defective component-level prompts; A learnable context vector is inserted before the text content embedding of the text encoder for dynamic cue learning. The learnable context vector is optimized and updated during the training phase through backpropagation of the detection loss. The initial text features, generated through dynamic prompting learning, are aligned to the same dimension as the visual features using a linear projection operation to obtain the final multi-level text features.

[0009] Optionally, adding the initial transmittance to a semantic enhancement term constructed based on the defect saliency map and the fog concentration term to obtain a semantic transmittance capable of improving the defogging degree of the defect region includes: The fog concentration term, reflecting the degree of fog concentration, is calculated based on the initial transmittance. The semantic enhancement term is obtained by multiplying the saliency value at the corresponding position of the defect saliency map, the fog concentration term, and the preset semantic enhancement coefficient. The initial transmittance is summed with the semantic enhancement term, and the range is truncated to prevent the value from going out of bounds. Finally, the semantic transmittance is output.

[0010] Optionally, the step of injecting the depth map, the atmospheric light value, and the semantic transmittance as physical condition features into the physical-semantic joint dehazing network to reconstruct a clear dehazed image includes: The depth map, the atmospheric light value broadcast to the full map size, and the semantic transmittance are stitched together along the channel dimension, and the modulation parameters are predicted by a multilayer perceptron. The modulation parameters are then applied to the encoder features extracted by the dehazing network encoder. In the skip connection between the encoder and decoder of the dehazing network, the visual features after projection alignment are stitched together with the encoder features to generate a gating signal. The ratio of feature retention to introduction is determined according to the value of the gating signal. Then, scaling dot product attention fusion is performed on the encoder features and the visual features, and the fusion result is input to the decoder to reconstruct the clear dehazed image.

[0011] Optionally, during the training of the physical-semantic joint dehazing network, its optimization objective, in addition to receiving the truncated effective gradients, also includes minimizing the joint loss of the dehazing network itself, wherein the joint loss includes: The physical reconstruction loss of the constrained dehazed image, after being re-rendered using an atmospheric scattering model, is consistent with the input foggy transmission line image at the pixel level. The semantic consistency loss is constrained to have a high cosine similarity between the global features extracted by the visual encoder of the clear dehazed image and the features extracted by the text encoder of the clear scene text description. The feature consistency loss constrains the visual features of the clear dehazed image to maintain dimensional consistency with the visual features of the clear reference image.

[0012] Optionally, the defect detection network includes a backbone network and a multi-scale feature pyramid, and its detection process includes: The backbone network extracts features from the clear dehazed image and outputs them to the multi-scale feature pyramid; A cross-modal attention mechanism is introduced into each scale layer of the multi-scale feature pyramid. The projected text features of the defective component are used as global query vectors. The degree of matching between each spatial location and the semantics of the defect is calculated to generate an attention distribution. The attention distribution and a learnable scaling factor are used to perform residual weighted fusion of the spatial features at each scale.

[0013] Optionally, the process of the defect detection network outputting defect detection results further includes confidence enhancement based on semantic saliency: Obtain the raw confidence score output by the detection head for each target candidate box; Calculate the average saliency value of the defect saliency map contained within the target candidate box region; Using a preset balance coefficient, the original confidence level is fused with the weighted confidence level based on the average significance value, so as to maintain the original confidence level of high defect probability regions while reducing the confidence level of false detection boxes in normal background regions, and output the final detection result.

[0014] Optionally, during the training phase, the bidirectional gradient interaction mechanism employs an alternating training strategy: During each of the multiple forward propagations of the defogging network, only one bidirectional interactive operation is set to calculate the detection loss and backfeed the effective gradient to the defogging network. During the remaining forward propagation without performing bidirectional interaction, the parameters of the dehazing network are updated using only the loss function of the dehazing network itself.

[0015] Optionally, for newly added defect categories in small-sample scenarios, the following prototype extension mechanism is executed to update the defect detection network: The text description of the newly added defect category is concatenated with the trained learnable context vector and input into the text encoder to generate the semantic prototype vector of the new category. After feature projection, the semantic prototype vector is added as a new row vector and concatenated into the original classification head weight matrix of the defect detection network to initialize the classification weights for the new defect category. The defect detection network is fine-tuned and optimized using a small number of labeled images with newly added defect categories. A regularization term for the weights of the new categories is added to the fine-tuned loss function to continuously penalize the degree to which the weights of the new categories deviate from the semantic prototype vector, and / or the difference between the weights of the new categories and the semantic prototype vector is constrained to be less than a preset range through a weight projection mechanism.

[0016] The present invention has achieved the following beneficial effects: This invention establishes a joint optimization process for dehazing reconstruction and defect detection tasks by constructing a physical and semantic joint dehazing network and a bidirectional gradient interaction mechanism. This enables the dehazing network to receive gradient feedback from downstream detectors, effectively balancing physical models and high-level semantic information. Material weight masks are used to weight bright metal components, and a defect saliency map is introduced into the transmittance calculation for semantic guidance, improving the adaptability of fixed scattering models to special material surfaces and enhancing the preservation of texture details in key areas. Simultaneously, this method incorporates textual prompts and learnable context vectors from a pre-trained visual language model, constructing multi-level textual features for different component states. When facing long-tailed defect distributions in transmission lines and the expansion of new categories, this reduces the dependence on large-scale labeled data and improves the model's defect detection accuracy in complex foggy scenarios.

[0017] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0018] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of the image defect detection method for transmission lines in foggy weather based on CLIP semantic guidance in an embodiment of the present invention. Detailed Implementation

[0020] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0021] Transmission lines, as critical infrastructure of the power system, directly impact the stability of the power grid. With the development of computer vision technology and UAV inspection equipment, automated defect detection based on inspection images has become a routine technique. However, images captured by UAVs in foggy conditions suffer from severe contrast degradation, color shift, and detail blurring due to scattering by atmospheric water droplets, leading to a decline in the recognition performance of downstream defect detection algorithms. Existing dehazing techniques based on physical priors or single-pixel-level deep learning generally suffer from the limitation of a disconnect between the physical model and high-level semantic information. Traditional dehazing algorithms prioritize minimizing pixel-level reconstruction errors as an independent optimization objective, neglecting the need for downstream detection tasks to identify subtle defect areas in critical components such as insulators, fittings, and conductors. In existing cascaded architectures that prioritize dehazing before detection, the two modules are optimized independently. The dehazing network cannot receive gradient feedback from the downstream detector regarding the contrast of key areas, and the fixed atmospheric scattering model struggles to adapt to the light reflection characteristics of slender structures like conductors and special material surfaces, resulting in the loss of texture in key areas or the generation of halo artifacts after dehazing. Furthermore, transmission line defect types exhibit a long-tail distribution, and conventional detection models rely on large-scale labeled data when new defect categories are expanded. Based on this, this application provides a semantically guided method for detecting transmission line images in foggy weather using a contrastive learning language-image pre-trained model (CLIP). This method establishes a joint optimization process for defogging reconstruction and defect detection tasks by constructing a physical and semantic joint defogging network and a bidirectional gradient interaction mechanism.

[0022] like Figure 1 As shown in the figure, this application discloses a method for detecting defects in foggy transmission line images based on CLIP semantic guidance, including: acquiring a foggy transmission line image to be detected and performing monocular depth estimation to obtain a spatial depth map; identifying insulator regions, conductor regions, and background regions in the foggy transmission line image, and assigning corresponding material weights to generate a material weight mask; using the material weight mask to weight the pixels of the input image to calculate atmospheric light values, deriving the initial physical transmittance based on an atmospheric scattering model, and performing guided filtering optimization; extracting the visual features of the foggy transmission line image, A multi-level text prompt library is constructed and multi-level text features are extracted by combining learnable context vectors; the spatial semantic similarity between the visual features and the multi-level text features is calculated to generate a defect saliency map, and the initial physical transmittance is weighted and fused to obtain the semantic transmittance; the depth map, atmospheric light value and semantic transmittance are injected as physical condition features into a physical-semantic joint dehazing network to reconstruct a clear dehazed image; the clear dehazed image is input into a defect detection network to output the detection result; a bidirectional gradient interaction mechanism is established during the training phase, and the dehazing network parameters are updated after performing a truncation gating operation on the returned detection gradient.

[0023] Specifically, in the stage of estimating the physical scattering parameters for transmission lines, the original image of the transmission line to be detected in foggy weather, output by the image acquisition device, is obtained. A monocular depth estimation network is then used to extract spatial depth features from the foggy transmission line image to obtain the corresponding physical depth map. The specific depth estimation calculation process satisfies the following formula: ; In the above formula, This represents the normalized spatial depth feature map output after depth estimation of the input image; This represents the preprocessed monocular depth estimation network architecture; This represents the original image of the power transmission line to be detected in foggy weather. This represents the basic network weight parameters that are fixed during the pre-training phase of the depth estimation network, and these weight parameters remain frozen during the inference and training fine-tuning phases. The output depth feature map. Normalized mapping to Within the closed interval, the numerical value corresponding to each pixel position represents the actual physical vertical distance between the target surface and the camera optical center.

[0024] In the aforementioned depth estimation stage, the DepthAnythingV2 model employs a pre-trained depth estimation architecture, which uniformly scales the input image to a preset fixed resolution when extracting features, to ensure the consistency of spatial depth feature extraction.

[0025] Furthermore, in performing dynamic adaptive semantic feature extraction, the visual encoder specifically adopts a ViT-L / 14 network architecture. The input foggy transmission line image undergoes patch embedding processing, is divided into 14×14 pixel non-overlapping image blocks, and after additional positional encoding, is input into a visual encoder containing 24 Transformer blocks. Each Transformer block contains a multi-head self-attention mechanism (e.g., 16 attention heads) and a multilayer perceptron, ultimately outputting a high-level visual feature tensor with a dimension of 1024. This tensor is then mapped to 768 dimensions using a linear projection matrix to align with the modal dimension of the text encoder features.

[0026] The algorithm identifies insulator, conductor, and background regions in images of transmission lines in foggy weather and generates material weight masks. Specular reflections from bright metal components in the transmission line under illumination are easily misidentified as atmospheric light spots by traditional algorithms. A pre-trained semantic segmentation model is used to process the input image. Perform pixel-level category discrimination to generate material weight masks for corresponding spatial locations. The specific material weight allocation logic satisfies the following formula: ; In the above formula, This indicates the spatial location of the transmission line to be detected in the foggy weather image. The material weight value corresponding to the pixel at that location; Represents the two-dimensional spatial coordinates of the image; This represents the set of pixels in the image region that is identified as an insulator type by the semantic segmentation model. This represents the set of pixels in the image region that is identified as a wire type by the semantic segmentation model; This represents the set of image region pixels identified as background type by the semantic segmentation model, where the background type encompasses the sky and vegetation areas surrounding the power transmission line scene; when the spatial location Assign weights when the region belongs to the insulator region Weights are assigned when the region belongs to the conductor region. When it belongs to the background area, a weight is assigned. The remaining spatial locations not classified into the above three specific regions are assigned a default weight. .

[0027] Specifically, atmospheric light values ​​are calculated using the material weight mask. The input image is then used for calculation. From the dark channel image, extract the top-ranked brightness values. The pixel positions form the set of brightest pixels. A weighted average calculation based on material weights is performed on the original channel pixel values ​​of each pixel within this set to obtain the global atmospheric light vector. The specific calculation logic satisfies the following formula: ; In the above formula, This represents the global ambient atmospheric light vector value output after weighted optimization calculation using the material weight mask. This represents the set of spatial coordinates formed by the positions of the pixels with the highest pixel brightness values ​​in the dark channel image of the input image (0.1%). Represents the set of spatial coordinates The total number of pixels contained within; Represents the set The spatial coordinates of a single pixel in the image; This indicates that the input raw foggy image of the transmission line is in coordinates. The original channel pixel value at that location; Representing coordinates The corresponding material weight value.

[0028] The initial physical transmittance is derived using an atmospheric scattering model. The classical atmospheric scattering physical model satisfies the following formula: ; In the above formula, This indicates the spatial location of the original image of the transmission line under inspection in foggy weather, captured by the camera equipment. The observed pixel value at that location; This indicates the spatial location of the fog-free, sharp scene image to be restored. The actual physical reflection pixel value at that location; Indicates spatial location The transmittance of the medium to light, and its range of values ​​characterizes the proportion of light intensity attenuation when scene light penetrates the suspended fog to reach the camera. This represents the global ambient atmospheric light vector value; Indicates spatial location The proportion of environmental shading caused by ambient fog scattering.

[0029] Based on the above physical model, assuming the dark channel value within a local window of a clear image is approximately zero, a haze retention coefficient is introduced to derive the initial physical transmittance at each spatial coordinate position. The specific calculation process for the initial physical transmittance satisfies the following formula: ; In the above formula, This represents the spatial location calculated based on the prior assumption of the dark channel. Initial physical transmittance at the location; This indicates the preset fog-prevention adjustment coefficient; Indicates the spatial location of the input image The dark channel pixel value within a local neighborhood window; This represents the global ambient atmospheric light vector value derived above.

[0030] To prevent the calculated physical transmittance value from being too low in dense fog areas, which would amplify noise in the reconstructed image, a lower limit truncation of the transmittance is applied. The specific transmittance truncation logic satisfies the following formula: ; In the above formula, Indicates spatial location The initial physical transmittance at the point is truncated, and the transmittance value is then calculated. This represents the initial physical transmittance calculated before truncation; This represents the preset minimum transmittance threshold value; the parameter is set. .

[0031] To address the blocky artifacts generated by local neighborhood window calculations, guided filtering is used to smooth the truncated transmittance map while preserving its edges. The input image is then extracted. A single-channel grayscale image is used as the guiding image, and linear local guided smoothing is performed. The specific guided filtering calculation logic satisfies the following formula: ; In the above formula, Represents spatial coordinates The physical transmittance at a given point is processed by a guided filter to output a smooth physical transmittance. Represents the original input image The resulting single-channel grayscale image is used as the structural guide layer for guided filtering. This indicates the center coordinates determined by the guiding image. and neighborhood coordinates The local linear model weight kernel function between them; Representing neighborhood coordinates Physical transmittance after truncation; the window radius parameter is set to [value missing] in the calculation. The regularization parameter is set to The final output is the physical transmittance result. .

[0032] After physical parameter estimation, dynamic adaptive semantic feature extraction is performed. The input image size is adjusted to the input specifications of the pre-trained modality-aligned encoder, and then input into the visual encoder to extract global and local features. The specific visual feature extraction logic satisfies the following formula: ; In the above formula, This represents the image visual feature tensor output by the visual encoder. This indicates the operation of adjusting the resolution of the original image to the standard input size using bilinear interpolation. This represents the visual encoder module of a pre-trained modality alignment network. This represents the basic network weight parameters fixed during the pre-training phase of the visual encoder; the output feature tensor has a dimension of [missing value]. .

[0033] A three-tiered text prompt library is constructed. Tier 1 is scenario-level prompts, with the baseline text "transmission line inspection image" providing a global semantic benchmark; Tier 2 is normal component-level prompts, including... Level 1 provides standard equipment information, such as "healthy ceramic insulator string," "intact vibration damper," "rust-free suspension clamp," "normal tension clamp," and "intact connecting bolts." Level 3 provides defective component level information, including... The text describes defects such as "broken insulator," "flashover-damaged insulator," "dirty insulator," "rusted hardware," "deformed equalizing ring," "missing bolt," and "loose cotter pin." The standard text encoder extracts fixed-word features according to the following formula: ; In the above formula, This represents the static text feature vector output by a standard text encoder after feature encoding a fixed prompt text sequence; This represents the text encoder module of a pre-trained modality alignment network. Represents a given sequence of input prompt text; These represent the basic parameters that are fixed in the text encoder.

[0034] Introducing continuous learnable context vectors Dynamic prompting learning is performed before text word embedding. The specific dynamic word embedding encoding logic satisfies the following formula: ; In the above formula, This represents the dynamic text feature vector output after dynamic adaptive optimization. This represents a continuously learnable context vector tensor inserted at the beginning of the text word embedding sequence, initialized to satisfy... And parameters ; This indicates the specific meaning of the embedded content in the prompt text; This represents the concatenation operation of vectors along the length dimension of the sequence; This represents the parameters of the backbone network.

[0035] When the text feature dimensions and visual feature dimensions are inconsistent, feature alignment mapping is performed using a linear projection matrix. The feature projection process for normal component prompt text satisfies the following formula: ; In the above formula, Indicates the first Projection-aligned text features corresponding to a normal device baseline description; This represents mapping the text feature space to a linear projection matrix with the same dimensions as the visual features, where the matrix dimension is set to 1. ; Represents a learnable context vector; Indicates the first The original text prompt description for each normal component.

[0036] The feature projection process for defective component warning text satisfies the following formula: ; In the above formula, Indicates the first The projection-aligned text features corresponding to a specific defect description; Represents the transformation matrix of a linear transformation; Represents a learnable context vector; Indicates the first The original textual description of a specific defect target.

[0037] After acquiring semantic prior features, closed-loop guided transmittance optimization is performed. For visual features... Perform spatial interpolation upsampling to make its resolution consistent with the original input image. The specific spatial upsampling mapping formula satisfies: ; In the above formula, This represents the upsampled visual feature tensor output after spatial resolution stretching, where the feature size satisfies... ; Represents the upsampling mapping function in the bilinear interpolation space; This represents the low-resolution visual feature tensor of the original output of the visual encoder.

[0038] Calculate the cosine similarity between visual features and features of normal and defective text at each spatial location. (This is done for spatial coordinates.) The calculation process for normal semantic similarity satisfies the following formula: ; In the above formula, Indicates spatial location Visual features at the location and the first The cosine similarity scalar value between the text features of each normal component; Indicates the upsampled visual features in coordinates The eigenvector at that location; Indicates the first Text feature vectors of normal components; This represents the L2 norm function calculated for the input feature vector.

[0039] Regarding spatial coordinates The process of calculating the semantic similarity of defects satisfies the following formula: ; In the above formula, Indicates spatial location Visual features at the location and the first The cosine similarity scalar value between specific defective text features; Indicates the upsampled visual features in coordinates The eigenvector at that location; Indicates the first Each defect semantic feature vector.

[0040] Calculate the maximum similarity difference between all defects and normal warning pairs at each spatial location to generate a defect saliency map. The specific logic for generating the saliency map satisfies the following formula: ; In the above formula, Representing spatial coordinates The calculated probability value of the defect significance is used to generate the value. Indicates the location First The first defect warning and the first The semantic similarity difference between normal prompts; This represents the operation of iterating through all suggestion pairs, combining them, and taking the global maximum difference. This represents a nonlinear activation function.

[0041] The initial physical transmittance is weighted and optimized using the aforementioned defect saliency map. The specific semantic transmittance calculation process satisfies the following formula: ; In the above formula, This indicates the position after semantic saliency weighting optimization. The semantic transmittance value at the location; This represents the basic initial physical transmittance derived from step one; This represents the local fog obstruction concentration term determined by physical transmittance; The semantic moderating coefficient represents the control over the strength of significance enhancement; Representing spatial coordinates The probability value of the significance of the defect at that location.

[0042] Numerical truncation is applied to the optimized transmittance. The specific transmittance truncation formula satisfies: ; In the above formula, This represents the final semantic transmittance result after safe interval truncation. This represents the semantic transmittance value obtained before truncation; This represents the lower limit threshold of transmittance.

[0043] Construct a physical-semantic joint dehazing network. Encode the physical condition parameters as multi-channel feature tensors. The specific condition concatenation logic satisfies the following formula: ; In the above formula, This represents the physical condition feature tensor formed after channel fusion; This represents the final output single-channel semantic transmittance map; This indicates that the global atmospheric light vector is obtained by broadcasting and copying it in the spatial dimension. Feature tensor; Represents a monocular depth feature map; This represents the splicing function along the tensor channel dimension.

[0044] Adaptive instance normalization is used to inject physical conditions into the dehazing network encoder. The specific control formula satisfies: ; In the above formula, This represents the encoder modulation characteristics output after physical condition feature modulation. This represents the original visual features extracted by the dehazing network encoder at the current feature layer; and These respectively represent the original features The channel mean and standard deviation tensor calculated in the spatial dimension; and This indicates that the multilayer perceptron network is configured under the aforementioned physical conditions. The input is the channel bias term and scale term obtained from the prediction.

[0045] Cross-modal gated attention fusion is introduced in the skip connections between the encoder and decoder. The visual feature projection compression process satisfies the following formula: ; In the above formula, This represents the projected visual characteristics output after channel compression; Represents the visual features after spatial interpolation and stretching; This represents a linear convolution operation with a channel projection size of 1x1, and the output feature dimension is... .

[0046] The calculation logic for the feature-gated signal in the skip connection satisfies the following formula: ; In the above formula, This represents the cross-modal feature fusion gate tensor of the computational output; Indicates the encoder features of the current layer; Indicates the visual characteristics of projection; This represents the channel splicing function; This represents the feature transformation weight matrix that the parameters can learn; This represents a nonlinear activation function.

[0047] Cross-modal feature fusion is performed based on the gated signal. The specific skip fusion logic satisfies the following formula: ; In the above formula, This represents the jump connection feature output after fusing the underlying texture and high-level semantics; Represents the gate tensor; Indicate encoder features; This represents the element-wise multiplication operation of matrices; This represents the attention operator for cross-feature fusion; Represents the visual feature tensor.

[0048] The specific definition of the above scaled dot product attention operator satisfies the following formula: ; In the above formula, This represents the query matrix generated using the encoder features as query terms; This represents the key matrix generated using the visual features as key terms; This represents a value matrix generated using the visual features as value terms; This represents the tensor obtained after transposing the key matrix; This represents the dimension of the feature channels of the feature vector, and is used as a scaling factor in the denominator of the inner product calculation to avoid gradient saturation in subsequent derivative calculations. This represents the normalized exponential function.

[0049] The physical reconstruction loss function of a defogging network is defined by the following formula: ; In the above formula, This represents the physical reconstruction first-order norm loss scalar that constrains image rendering consistency during the training of the dehazing network. and Indicates the height and width of the image space; Represents pixel coordinates; This indicates the location of the input foggy image. The actual pixel value at that location; This represents the pixel value of the clear, fog-free image reconstructed by the dehazing network. Indicates semantic transmittance value; This represents the global atmospheric light value.

[0050] The semantic consistency loss function of a defogging network is defined according to the following formula: ; In the above formula, The semantic consistency loss scalar represents the constraint that prevents the macroscopic representation of the image from drifting. This represents the global feature vector extracted from the reconstructed dehazed image by the pre-trained image encoder; The feature vectors extracted by the text encoder represent the baseline textual description of a clear scene. This represents the operation function for calculating the cosine similarity between two vectors.

[0051] Feature consistency loss is activated when paired, clearly supervised data exists. The specific loss definition formula satisfies: ; In the above formula, A scalar representing the feature consistency error that constrains the fit of deep visual representation; Indicates the total number of dimensions of visual features; Indicates a specific channel feature index; Indicates the reconstruction of the dehazed image at the 1st... Dimensional visual features; Indicates the corresponding supervised reference image at the 1st Dimensional features; This represents the square 2 norm operation.

[0052] A defect detection network with bidirectional gradient interaction is constructed. Multi-scale features are extracted from a clear, dehazed input image. The feature pyramid generation logic satisfies the following formula: ; In the above formula, Let represent the set of multi-scale feature pyramid tensors output by the backbone of the object detection network, where This represents a small target feature layer for small parts. This represents a medium-sized target feature layer. Represents a large target feature layer; This represents the core feature extraction architecture of the detection model; This indicates a clear, dehazed image; This represents the learnable weight parameters of the detection network.

[0053] Cross-modal spatial attention is introduced at each level of the feature pyramid. The dedicated channel projection formula for text features satisfies: ; In the above formula, This represents the channel alignment defect text feature vector output after hierarchical projection mapping; This represents the initial defective text feature vector; Indicates the first A dedicated channel transformation projection matrix for the scale feature layer is used to map text features to the same channel dimension as the spatial features of the current layer. .

[0054] The spatial-semantic cross-matching attention response distribution calculation logic satisfies the following formula: ; In the above formula, The spatial-semantic response feature map represents the computational output; The flattened feature sequence is the first... Scale-space feature tensor; This represents the projected global query defect semantic vector; Indicates the size of the feature layer channels; This represents the spatial attention normalization operator.

[0055] Spatial-semantic attention is superimposed and fused into the original features. The specific residual fusion formula satisfies: ; In the above formula, This represents the optimized feature pyramid feature tensor output after cross-modal semantic guidance enhancement; Represents the original multi-scale spatial feature tensor; Represents the attention response feature map; Indicates initialization to Furthermore, the semantic scaling parameters are adaptively updated during fine-tuning.

[0056] Significance post-processing correction is performed on the detection confidence level. The specific confidence level quadratic fusion formula satisfies: ; In the above formula, This represents the final target detection confidence score after weighted correction using the saliency plot. This represents the confidence level of the detection head in the original classification prediction of the target candidate box; This represents the average defect significance value contained within the candidate box region; parameter This represents the weighting coefficient used to balance the addition of two items.

[0057] The calculation logic for the average defect significance of the above candidate box region satisfies the following formula: ; In the above formula, This represents the average defect significance value within the space defined by the target candidate box; This represents the pixel area occupied by the rectangular candidate box output by the object detection head in the two-dimensional image space; This represents the total number of pixels within the rectangular candidate box area; Indicates the pixel coordinates within the candidate box; Representing coordinates The probability value of the significance of the defect at that location.

[0058] Bidirectional gradient interaction is performed during the training phase. The forward propagation process of the dehazing network to generate dehazed images satisfies: ; In the above formula, This represents a clear, dehazed image generated by the forward propagation computation of the dehazing network. This represents the backbone architecture of the physical-semantic joint defogging network; This represents the original input image of a foggy day. This represents the total learnable weight parameters of the dehazing network.

[0059] The process of calculating the loss during the forward propagation of the detection network satisfies the following formula: ; In the above formula, This represents the scalar value of the total joint detection loss of multiple targets calculated by the detection network; This describes the target defect detection network architecture; Represents a dehazed image; Indicates the detector weight parameters; This represents the actual candidate bounding boxes and category labels in the dataset.

[0060] The original gradient of the detection loss backpropagation to the dehazing network parameters is calculated based on the chain rule: ; In the above formula, This represents the original detection gradient tensor that is fed back from the downstream detection task loss to the upstream defogging network parameters; The matrix representing the partial derivative of the total detection loss with respect to the intermediate tensor pixel values ​​of the dehazed image represents the detector's feedback prior to the feature contrast of the reconstructed image. This represents the partial derivative tensor of the dehazed image with respect to the weight parameters inside the dehazed network.

[0061] A truncation gating operation is performed on the original detection gradient. The specific truncation formula satisfies: ; In the above formula, This represents the valid backpropagation gradient tensor output after truncation and trimming of the original gradient; This represents the original detection gradient tensor returned; This indicates that the elements of the input matrix are constrained to... Numerical truncation function within a closed interval; This indicates the absolute value threshold parameter for truncation gate, set. .

[0062] The truncated effective gradients are used to update the weight parameters of the dehazing network. The specific parameter optimization formula satisfies: ; In the above formula, This represents all parameters within the defogging network that need to be optimized and updated. This represents the base learning rate parameter set for training the dehazing network; This represents the gradient of the physical reconstruction loss with respect to the weights of the defogging network; This represents the gradient of the semantic consistency loss with respect to the weights of the dehazing network; This represents the semantic loss weight balance coefficient; This represents the valid detection gradient term returned.

[0063] Specifically, the bidirectional gradient interaction employs a multi-task joint alternating training strategy. In four consecutive network forward propagation and optimization cycles, end-to-end bidirectional gradient interaction calculation and parameter fine-tuning are performed in the fourth forward propagation cycle; while backpropagation gradient flow is disabled in the first three forward propagation cycles. The dehazing network parameters are optimized independently using only the physical reconstruction loss and semantic consistency loss of the dehazing network itself. The target detection loss function of the detection task consists of three parts: ; In the above formula, This represents the total target loss for overall detector optimization; This represents the focus classification loss for handling class imbalance. The regression crossover ratio loss represents the regression crossover ratio loss that constrains the positioning accuracy of the candidate bounding box coordinates. This represents the target-oriented binary cross-entropy loss used to determine whether a real object exists within the candidate bounding box.

[0064] During the joint optimization phase, multi-task integrated loss constraints are applied. The specific task joint loss definition satisfies: ; In the above formula, This represents the target joint total loss scalar that drives the joint update of all network model parameters. , , and These represent the physical reconstruction loss, semantic consistency loss, downstream detection loss, and feature consistency reference loss, respectively. , , and These represent the weighting coefficients for each loss factor, respectively. , , , .

[0065] The training phase includes a warm-up phase and a joint fine-tuning phase. The warm-up phase occurs from the first to the fiftieth iteration, during which the large model context parameters are frozen, and the dehazing and detection networks converge independently. The joint fine-tuning phase occurs from the fifty-first to the two-hundredth iteration, during which a bidirectional interaction mechanism is activated, executing once every four steps. In the few-sample expansion mechanism for adding new categories of long-tail defects in transmission lines, the text prototype generation logic satisfies the following formula: ; In the above formula, This represents the semantic prototype vector of the newly added defect category output after processing by the pre-trained text encoder; This represents the learnable context vector that has been solidified after the aforementioned training convergence; A sequence of natural language text descriptive words representing the newly added defect category; This represents the concatenation operation function; This represents the text feature encoder module.

[0066] The initialization process of the classification head weight matrix when adding a new category satisfies the following formula: ; In the above formula, This represents the new class head weight matrix generated after splicing and expansion; This represents the converged and solidified basic classification weight matrix in the original detection network's classification head; The semantic prototype feature vector representing the newly added category; This represents a linear projection matrix that maps semantic prototype features to the detection head dimension; This represents the matrix row concatenation operator.

[0067] In the few-shot fine-tuning stage, no more than thirty new defect category images are provided for fine-tuning. The specific few-shot fine-tuning loss function is defined according to the following formula: ; In the above formula, This represents the scalar of the total few-sample loss when optimizing the classifier head parameters during the fine-tuning phase. This represents the target classification loss calculated for images of the new category; This indicates the dynamically updated category weights during fine-tuning; This represents the static basic semantic prototype weight term of the initial concatenation injection; Represents Euclidean norm terms; This represents the regularization weight constraint coefficient, and the parameter is set. .

[0068] In practical implementation, in addition to the soft constraint penalty mechanism provided by the L2 regularization term, this embodiment also introduces a hard constraint truncation mechanism based on weight projection to strictly ensure that the new category classification weights do not undergo severe semantic drift and that the difference distance strictly meets the condition of being less than a preset range. Specifically, after each backpropagation update in the network parameter fine-tuning phase, the algorithm performs an additional distance determination and range truncation operation: calculating the currently updated new category weights. With the static basic semantic prototype weights The Euclidean distance between them; when the Euclidean distance is determined to be greater than or equal to the preset maximum threshold value. At that time, feature space projection is performed, and the updated new category weights are scaled proportionally and reprojected onto a sphere centered on the semantic prototype weights. On a hypersphere with radius , the specific forced projection update formula satisfies: ; By combining the above-mentioned fine-tuning strategy of soft and hard constraints, this invention allows the model to learn new features while ensuring that the difference between the new category weights and the semantic prototype vectors is strictly limited to a safe preset range, effectively avoiding the risk of overfitting in small sample scenarios.

[0069] Specifically, in the joint training and validation phase of the physical-semantic dehazing network and the defect detection network, this invention utilizes a rich dataset. The training and testing data include 13,000 synthetic foggy images from the RESIDE dataset, 4,322 real-world foggy images from the RTTS dataset, and 5,000 real transmission line inspection images from the TLDD-5K dataset, specifically designed for power scenarios. To further enhance the model's generalization ability under extreme fog conditions, this embodiment also synthesizes an additional 25,000 foggy extended images using an atmospheric scattering model combined with depth maps. In terms of training strategy, the AdamW optimizer is used to update the model parameters, and the initial learning rate of the network is set to... It is supplemented by a cosine annealing decay strategy, with the batch size set to 16, and the entire joint training process consists of 200 iterations (epochs).

[0070] To verify the effectiveness of the method in the embodiments of this application, a synthetic test set and a real test set containing 25,000 images under different fog concentration conditions were constructed. The average accuracy of the baseline detection model under foggy conditions was tested. The highest percentage among the cascaded defogging detection comparison methods is 73.1%, while the method in this embodiment achieves a percentage of 81.6%.

[0071] To further demonstrate the synergistic gains and unexpected technical effects of the module combinations in this invention, a detailed ablation experiment was conducted in this embodiment. When only the classic atmospheric scattering model was used for dehazing and directly cascaded with the YOLOv8 detection network, artifacts were easily generated on bright metal parts such as insulators due to the lack of semantic guidance, and the model's mAP@0.5 was only 63.2%. When only CLIP multi-level text features were introduced to generate semantic saliency maps for unidirectional optimization of transmittance, the discernibility of subtle defect areas was initially improved, and mAP@0.5 increased to 74.5%. Finally, under the complete bidirectional gradient interaction mechanism of this invention, the dehazing network successfully received feedback from the downstream detector regarding the contrast of defect features, not only completely eliminating halo artifacts on special material surfaces, but also significantly increasing the recall rate for small target defects (such as loose cotter pins and missing bolts) at the detection end, thereby achieving the best performance of 81.6% mAP@0.5. This result fully demonstrates that the semantic guidance and physical reconstruction closed loop of this invention are not a simple patchwork of existing technologies, but rather a synergistic process that produces significant technical progress.

[0072] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for detecting image defects in transmission lines in foggy weather based on CLIP semantic guidance, characterized in that, Includes the following steps: Acquire images of the power transmission line to be detected in foggy weather and perform depth estimation to obtain a depth map; The insulator region, conductor region and background region in the foggy transmission line image are identified, and corresponding material weights are assigned accordingly to generate a material weight mask. The pixel of the foggy power transmission line image is weighted using the material weight mask to reduce interference from bright metal parts and calculate the atmospheric light value. Then, the initial transmittance is calculated by combining the atmospheric scattering model. The image of the transmission line in foggy weather is input into a pre-trained visual encoder to extract visual features, and multi-level text prompts containing normal and defective parts are input into a text encoder to extract multi-level text features. The similarity between the visual features and the text features of normal and defective components in the multi-level text features is calculated for each spatial location. The difference between the similarity of all defective component text features and the similarity of normal component text features at the same spatial location is compared. The maximum value of the difference is taken and activated to generate a defect saliency map for that spatial location. The initial transmittance is optimized using the defect saliency map. The initial transmittance is then added to a semantic enhancement term constructed based on the defect saliency map and the fog concentration term to obtain a semantic transmittance that can improve the defogging effect in the defect area. The depth map, atmospheric light value, and semantic transmittance are injected as physical condition features into the physical-semantic joint dehazing network to reconstruct a clear dehazing image. The clear, dehazed image is input into the defect detection network, which then outputs the defect detection results. During the training phase, a bidirectional gradient interaction mechanism is established to calculate the gradient of the detection loss of the defect detection network with respect to the original parameters of the physical-semantic joint dehazing network. A truncation gating operation based on a preset threshold is applied to the original gradient, and the parameters of the physical-semantic joint dehazing network are updated using the truncated effective gradient, so as to establish a semantic closed loop of dehazing and detection while preventing distortion of physical reconstruction.

2. The method for detecting image defects of transmission lines in foggy weather based on CLIP semantic guidance according to claim 1, characterized in that, The process of using the material weighting mask to weight pixels in the foggy power transmission line image to reduce interference from bright metal components and calculate atmospheric light values, and then combining this with an atmospheric scattering model to calculate the initial transmittance, includes: When generating the material weight mask, the insulator region is assigned the highest weight, the background region is assigned the lowest weight, and the conductor region is assigned a center weight; the set of brightest pixels in the dark channel image of the foggy transmission line is obtained, and the pixel values ​​of the pixels in the set are weighted and averaged using the material weight mask to obtain the atmospheric light value; The initial physical transmittance is derived based on the atmospheric light value and truncated to a lower limit. The grayscale image of the transmission line in foggy weather is used as a guiding image. The truncated initial physical transmittance is smoothed by guiding filtering to preserve its edges, and the final initial transmittance is output.

3. The method for detecting image defects of transmission lines in foggy weather based on CLIP semantic guidance according to claim 1, characterized in that, The step of inputting multi-level text prompts containing normal and defective components into a text encoder to extract multi-level text features includes: Construct a three-tiered text prompt library that includes scene-level prompts, normal component-level prompts, and defective component-level prompts; A learnable context vector is inserted before the text content embedding of the text encoder for dynamic cue learning. The learnable context vector is optimized and updated during the training phase through backpropagation of the detection loss. The initial text features, generated through dynamic prompting learning, are aligned to the same dimension as the visual features using a linear projection operation to obtain the final multi-level text features.

4. The method for detecting image defects of transmission lines in foggy weather based on CLIP semantic guidance according to claim 1, characterized in that, The step of adding the initial transmittance to a semantic enhancement term constructed based on the defect saliency map and the fog concentration term to obtain a semantic transmittance that can improve the defogging effect in the defect region includes: The fog concentration term, reflecting the degree of fog concentration, is calculated based on the initial transmittance. The semantic enhancement term is obtained by multiplying the saliency value at the corresponding position of the defect saliency map, the fog concentration term, and the preset semantic enhancement coefficient. The initial transmittance is summed with the semantic enhancement term, and the range is truncated to prevent the value from going out of bounds. Finally, the semantic transmittance is output.

5. The method for detecting image defects of transmission lines in foggy weather based on CLIP semantic guidance according to claim 1, characterized in that, The process of injecting the depth map, atmospheric light value, and semantic transmittance as physical condition features into the physical-semantic joint dehazing network to reconstruct a clear dehazed image includes: The depth map, the atmospheric light value broadcast to the full map size, and the semantic transmittance are stitched together along the channel dimension, and the modulation parameters are predicted by a multilayer perceptron. The modulation parameters are then applied to the encoder features extracted by the dehazing network encoder. In the skip connection between the encoder and decoder of the dehazing network, the visual features after projection alignment are stitched together with the encoder features to generate a gating signal. The ratio of feature retention to introduction is determined according to the value of the gating signal. Then, scaling dot product attention fusion is performed on the encoder features and the visual features, and the fusion result is input to the decoder to reconstruct the clear dehazed image.

6. The method for detecting image defects of transmission lines in foggy weather based on CLIP semantic guidance according to claim 1, characterized in that, During the training of the physical-semantic joint dehazing network, its optimization objective, in addition to receiving the truncated effective gradients, also includes minimizing the joint loss of the dehazing network itself, which includes: The physical reconstruction loss of the constrained dehazed image, after being re-rendered using an atmospheric scattering model, is consistent with the input foggy transmission line image at the pixel level. The semantic consistency loss is constrained to have a high cosine similarity between the global features extracted by the visual encoder of the clear dehazed image and the features extracted by the text encoder of the clear scene text description. The feature consistency loss constrains the visual features of the clear dehazed image to maintain dimensional consistency with the visual features of the clear reference image.

7. The method for detecting image defects of transmission lines in foggy weather based on CLIP semantic guidance according to claim 1, characterized in that, The defect detection network includes a backbone network and a multi-scale feature pyramid, and its detection process includes: The backbone network extracts features from the clear dehazed image and outputs them to the multi-scale feature pyramid; A cross-modal attention mechanism is introduced into each scale layer of the multi-scale feature pyramid. The projected text features of the defective component are used as global query vectors. The degree of matching between each spatial location and the semantics of the defect is calculated to generate an attention distribution. The attention distribution and a learnable scaling factor are used to perform residual weighted fusion of the spatial features at each scale.

8. The method for detecting image defects of transmission lines in foggy weather based on CLIP semantic guidance according to claim 7, characterized in that, The process of the defect detection network outputting defect detection results also includes confidence enhancement based on semantic saliency: Obtain the raw confidence score output by the detection head for each target candidate box; Calculate the average saliency value of the defect saliency map contained within the target candidate box region; Using a preset balance coefficient, the original confidence level is fused with the weighted confidence level based on the average significance value, so as to maintain the original confidence level of high defect probability regions while reducing the confidence level of false detection boxes in normal background regions, and output the final detection result.

9. The method for detecting image defects of transmission lines in foggy weather based on CLIP semantic guidance according to claim 1, characterized in that, During the training phase, the bidirectional gradient interaction mechanism employs an alternating training strategy: During each of the multiple forward propagations of the defogging network, only one bidirectional interactive operation is set to calculate the detection loss and backfeed the effective gradient to the defogging network. During the remaining forward propagation without performing bidirectional interaction, the parameters of the dehazing network are updated using only the loss function of the dehazing network itself.

10. The method for detecting image defects of transmission lines in foggy weather based on CLIP semantic guidance according to claim 3, characterized in that, For newly added defect categories in small sample scenarios, the following prototype extension mechanism is executed to update the defect detection network: The text description of the newly added defect category is concatenated with the trained learnable context vector and input into the text encoder to generate the semantic prototype vector of the new category. After feature projection, the semantic prototype vector is added as a new row vector and concatenated into the original classification head weight matrix of the defect detection network to initialize the classification weights for the new defect category. The defect detection network is fine-tuned and optimized using a small number of labeled images with newly added defect categories. A regularization term for the weights of the new categories is added to the fine-tuned loss function to continuously penalize the degree to which the weights of the new categories deviate from the semantic prototype vector, and / or the difference between the weights of the new categories and the semantic prototype vector is constrained to be less than a preset range through a weight projection mechanism.

Images