A power transmission line equipment defect identification method based on a multi-modal large model technology

Abstract

This invention discloses a method for defect identification of transmission line equipment based on multimodal large model technology, comprising the following steps: acquiring inspection images, defect annotation data, and contextualized prompts of transmission line equipment, and constructing image-prompt-annotation triplet data; performing preprocessing to generate standardized training samples; loading a pre-trained visual-language multimodal large model as the base model; fine-tuning the base model with few samples based on a reinforcement learning fine-tuning framework, combined with an enhanced intersection-over-union reward mechanism and a prompt-inference text alignment mechanism, to obtain a defect identification model; inputting the image to be detected and the corresponding prompts into the defect identification model to generate defect type, location information, and interpretable inference process text; decoding the model output to output a structured defect identification result; this invention improves the small-sample generalization ability.

Description

Technical Field

[0001] This invention relates to the field of road engineering technology, and specifically to a method for identifying defects in power transmission line equipment based on multimodal large model technology. Background Technology

[0002] With the rapid development of the power industry, transmission lines have become one of the most important infrastructures in my country's power grid. However, due to their long-term exposure to the natural environment and the influence of climate conditions, surrounding animal activities, and human activities, transmission line equipment is prone to defects. Existing technologies suffer from several problems: low accuracy in locating small defects: Current technologies use Intersection of Units (IoU) as the reward function, but IoU only focuses on the overlap area between the predicted and ground truth bounding boxes, neglecting the distance between their center points and aspect ratios. Consequently, the model cannot effectively learn the boundary features of small defects and tends to generate larger predicted boxes to ensure overlap, ultimately leading to low accuracy in locating small defects. Degradation of Think content: Existing technologies lack a Think-Prompt alignment mechanism. When generating Think outputs, the model relies solely on general pre-trained knowledge without contextualized logical constraints. Faced with specific Prompts in a transmission scenario (such as "detecting insulator cracks"), the model cannot associate them with the "insulator-crack" scenario features, easily generating irrelevant content (such as "there is a white object in the image") or degenerating into ellipses, resulting in an uninterpretable inference process that fails to meet the traceability requirements of power operation and maintenance. Poor generalization ability with small sample sizes: Existing technologies are trained on general datasets, which do not include features specific to power transmission equipment (such as insulator skirt texture and metal reflection of fittings) and environmental interference (such as wild vegetation). This makes it difficult for the model to achieve effective transfer learning on small sample power transmission datasets, easily misclassifying "insulator cracks" as "tree branch shadows", ultimately resulting in low recognition accuracy in small sample scenarios and inability to adapt to power transmission scenarios. Summary of the Invention

[0003] Purpose of the invention: The purpose of this invention is to provide a method for identifying defects in transmission line equipment based on multimodal large model technology, which solves the problem of sample scarcity in complex scenarios in the existing technology for identifying defects in transmission line equipment.

[0004] Technical solution: The present invention provides a method for defect identification of transmission line equipment based on multimodal large model technology, characterized by comprising the following steps: (1) Obtain inspection images, defect annotation data and contextualized prompts of transmission line equipment, and construct image-prompt-annotation triplet data; (2) Preprocess the triplet data to generate standardized training samples; (3) Load the pre-trained visual-language multimodal large model as the base model; based on the reinforcement learning fine-tuning framework, combined with the enhanced intersection-union reward mechanism and the prompt word-inference text alignment mechanism, fine-tune the base model with a small number of samples to obtain the defect recognition model; (4) Input the image to be detected and the corresponding prompt words into the defect recognition model to generate the defect type, location information and interpretable reasoning process text; (5) Decode the model output to output structured defect identification results.

[0005] Furthermore, the preprocessing steps also include: template-encapsulating the prompt words, fixing the format of the system prompt words and question templates, and ensuring that the input text structure is consistent between the training and inference stages.

[0006] Furthermore, the enhanced cross-union ratio (CUNR) reward mechanism is as follows: Based on CUNR, a center point distance penalty factor and an aspect ratio consistency constraint factor between the predicted bounding box and the true bounding box are introduced to improve the positioning accuracy of small-sized defect targets.

[0007] Furthermore, the prompt word-inference text alignment mechanism is as follows: Chinese word segmentation is performed on the prompt words input by the user and the inference text generated by the model to extract core words; similarity is calculated based on the longest common subsequence to evaluate the semantic matching degree between the inference text and the prompt words; the similarity is used as a reward signal to guide the model to generate inference content that is highly related to the contextual prompt words.

[0008] Furthermore, the structured defect identification results are output in the following formats: text format: directly outputting the defect type, bounding box coordinates, and inference text; image format: drawing the defect bounding box on the original image and labeling the defect type; JSON format: outputting the image name, defect type, bounding box coordinates, and inference text in a structured manner, which facilitates integration with the power operation and maintenance platform.

[0009] The present invention discloses a defect identification system for transmission line equipment based on multimodal large model technology, comprising: The data acquisition module is used to acquire inspection images, defect annotation data, and contextual prompts for power transmission line equipment. The data preprocessing module is used to standardize the collected data and generate training samples; The model fine-tuning module is used to load pre-trained visual-language multimodal large models and perform small-sample fine-tuning based on a reinforcement learning framework combined with an enhanced intersection-union reward mechanism and a cue word-inference text alignment mechanism. The inference output module is used to input the image to be detected and the prompt words into the fine-tuned model, and generate and output the defect type, location information and interpretable inference text.

[0010] An electronic device according to the present invention includes a memory and a processor. The memory stores a computer program, and the processor executes the program to implement the steps of the method.

[0011] The present invention discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method.

[0012] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: This invention designs a reward mechanism based on EIoU (Enhanced Intersection over Union), introduces center point distance penalty, aspect ratio constraint, and dynamic balance coefficient, which improves the average offset of bounding boxes for small defects; it constructs a dedicated prompt word library for power transmission scenarios and a structured logic of "localization → feature extraction → defect judgment", designs a semantic alignment loss function, solves the problem of Think content degradation, improves Think alignment rate, and meets the interpretability requirements of power operation and maintenance; it optimizes the adaptation strategy of Visual-RFT in power transmission scenarios and improves the generalization ability of small samples through scenario-based data processing. Attached Figure Description

[0013] Figure 1 This is a flowchart of the present invention; Figure 2 This is a flowchart of the EIoU reward calculation process of the present invention. Detailed Implementation

[0014] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0015] like Figure 1 As shown, this embodiment of the invention provides a method for defect identification of transmission line equipment based on multimodal large model technology, including the following steps: S101: Data Acquisition: Acquire inspection images of power transmission equipment, defect annotation data, and scenario-based prompts to form an "image-prompt-annotation" triple; Specific implementation method: (11) Inspection image acquisition: Use drones equipped with industrial cameras to take pictures along the power transmission line, sample the operating status of various equipment, and collect scenes covering sunny days, rainy days, and foggy days to avoid image blurring caused by extreme weather such as heavy rain and heavy snow. (12) Defect labeling data collection: Organize engineers with power operation and maintenance experience to use labeling tools to label defects. The labeling content includes: Bounding box: pixel-level coordinates (x1, y1, x2, y2). For example, for insulator cracks, the entire area from the start point to the end point needs to be marked to ensure that the bounding box accurately covers the defect. Defect attributes: Defect type (e.g., "insulator crack", "fitting deformation"); Scenario-based Prompt Design: Based on the actual needs of power transmission operation and maintenance, a two-tiered "component-defect" structure of prompts is designed. For example, "Broken silicone cylinder of a breather: The image shows a transparent or semi-transparent glass / polycarbonate cylinder filled with granular silica gel desiccant, with a metal flange connector at the top and a small oil seal cup at the bottom. This is typically a breather, vertically fixed to other equipment; obvious cracks or holes on the surface of the transparent cylinder indicate a broken silicone cylinder." "Broken meter dial: Typically protected by a circular or square casing, a meter contains a scale and one or more pointers used to display measured values. It can be installed on various other equipment to monitor physical quantities such as pressure, temperature, and current. Obvious cracks or damage to the meter dial, affecting the scale and pointer display, indicate a broken meter dial." Differentiated prompts are designed to cover various equipment types and defect scenarios, ensuring consistency between the prompts and operation and maintenance requirements. S102: Data preprocessing: Convert the annotation format to a COCO format that the model can read; specifically as follows: process the data into a standard format, such as {"image_path":insulator_001.jpg-bbox":[120,80,250,210]-"label":insulator crack}, to facilitate batch reading by the model. The format of the prompt words is standardized by integrating user input prompt words through a fixed system prompt word template, ensuring a consistent text input format during model training and inference. Specific implementation details are as follows: System prompt word template definition: Predefined fixed-format SYSTEM_PROMPT and QUESTION_TEMPLATE, where: `SYSTEM_PROMPT` is used to constrain the model's output structure, specifying the separation between the reasoning process and the answer: `SYSTEM_PROMPT = ("A dialogue between the user and the assistant. The user asks a question, and the assistant solves the problem. The assistant first thinks through the reasoning process in their mind, and then provides the answer to the user. The reasoning process and the answer are represented by...") <think> < / think> and <answer> < / answer> Enclosed in tags, i.e.: <think> This is the reasoning process.< / think> <answer> Here is the answer.< / answer> The QUESTION_TEMPLATE function is used to embed specific power transmission scenario detection requirements (user prompts) and specify the bounding box output format. QUESTION_TEMPLATE = "{Question} in <think>< / think> Output your thought process and your bounding box. Follow the instructions... <think> thought process< / think> \n <answer> (x1,y1),(x2,y2)< / answer> "Format." Prompt word integration and generation: Substitute the Prompt collected in step S101 into the {Question} placeholder of QUESTION_TEMPLATE, and then concatenate it with SYSTEM_PROMPT to form the complete input text: S103: Model Initialization: Load the Qwen2.5 VL Instruct model and configure the training environment; specifically: Base model selection: Qwen 2.5VL Instruct model was selected, which has general visual understanding capabilities; Model loading configuration: Load the model through the HuggingFace Transformers library, set torch_dtype=torch.float16 (to balance computational precision and GPU memory consumption), device_map="auto" (automatically allocate GPU resources, prioritizing devices with stronger computing power); load the AutoProcessor processor for image encoding (generating pixel_values) and text encoding (generating input_ids). S104: Visual-RFT Reinforcement Learning Fine-tuning: Includes: S1041: EIoU reward calculation, such as Figure 2 As shown, it includes 8 steps, the core of which is to improve the accuracy of small defect location through multi-dimensional constraints. The specific process is as follows: Obtaining coordinates: Parse the predicted bounding box b=(x1,y1,x2,y2) from the model output, and read the ground truth bounding box from the labeled data. Ensure that x1 < x2 and y1 < y2; Calculate IoU: First calculate the area of ​​intersection. Then calculate the area of ​​the prediction frame. Actual frame area ,final ; Calculate the squared distance ρ² between the center point and the predicted bounding box: The center point of the predicted bounding box is... The center point of the true bounding box is , , used to penalize bounding box offset; Calculate the diagonal length c of the minimum bounding rectangle: the coordinates of the bounding rectangle are... , This is used to normalize the distance and avoid the influence of image size. Calculate the aspect ratio index v: ,in (Prediction box width) (Predicted box height) (Actual frame width) (Actual frame height) is used to constrain aspect ratio consistency and adapt to different device shapes; Calculate EIoU: Substitute into the formula To avoid ineffective rewards; Mapped reward value: Substitute into the formula Ensure reward value Between 0 and 1; Output reward value: It is fed into the reinforcement learning framework as the core reward signal for model updates. S1042: Think-Prompt alignment optimization, updating model parameters through the GRPO algorithm; It comprises four steps, with the core being the use of structured constraints and semantic loss to address the Think content degradation problem, specifically: Input text: Obtain the user-input Prompt (e.g., "Detect cracks in 220kV line insulators") and the original Think text generated by the model (e.g., "The insulator skirt in the picture has a crack, about 5mm in length"). Chinese word segmentation preprocessing: The jieba word segmentation tool is called to segment the text (supporting the loading of power industry vocabulary, such as adding custom dictionaries like "umbrella skirt" and "metal fittings"). Prompt is segmented as "detect / 220kV / line / insulator / crack", and Think is segmented as "in the picture / insulator / umbrella skirt / has / crack / , / length / about / 5mm". Stop words (such as "in the picture", "has", "about") and punctuation marks are removed, and core words are retained: the core words of Prompt are "220kV / line / insulator / crack", and the core words of Think are "insulator / umbrella skirt / crack / length / 5mm". Rouge similarity calculation: Rouge-L (based on the longest common subsequence) is selected as the core indicator (adapting to the flexibility of Chinese word order). First, the longest common subsequence of the core word sequence is extracted (e.g., "insulator / crack" is the common subsequence of Prompt and Think); then, the Rouge-L score is calculated as: Prompt core word length + Think core word length 2 × common subsequence length. In the example, the score is 2 × 2 / (4 + 5) = 0.44. Output reward value: The Rouge-L score is fed into the reinforcement learning framework as a reward signal for model updates. S105: Model Evaluation: Use the validation set to evaluate mAP@0.5, small defect AP, and Think alignment rate. If there is no improvement for two consecutive epochs, stop training early. S106: Inference Test: Input the image to be detected and the prompt, preprocess it, and then feed it into the fine-tuned model to generate defect information and Think output; details are as follows: Input preparation: Receives the image and prompt input by the user; if the user does not input a prompt, the system automatically matches from the prompt word library (preliminarily identifies the device type through the image, such as matching the "detect insulator defect" type prompt if an insulator is identified); Inference preprocessing: Reuse the preprocessing logic from the training stage to ensure that the input features are consistent with the training data and avoid the decrease in accuracy caused by distribution shift; Model inference configuration: Set the generation parameters to max_new_tokens=256 (to ensure the Think text is complete and avoid truncation), num_beams=3 (beamsearch algorithm improves generation quality and reduces repetition), temperature=0.7 (to control randomness and balance accuracy and diversity), and no_repeat_ngram_size=3 (to avoid repeating 3 consecutive words, such as "crack crack crack"). Inference execution: In torch.no_grad() mode (disable gradient calculation to improve speed), call model.generate(pixel_values=pixel_values,input_ids=input_ids,**generate_config), and the output is a sequence of token_ids (text encoding format inside the model).

[0016] S107: Result Decoding: Parse the model output, restore the original image bounding box coordinates, and extract the structured Think text; specifically: Text decoding: The complete text is obtained by decoding using AutoProcessor.decode(token_ids,skip_special_tokens=False), such as " <think>Based on the description, the watch dial has obvious cracks and damage, affecting the display of the scale and hands. The watch dial in the picture has obvious cracks and damage, so it should be determined that the dial is damaged. <answer> (61,700)(391,923)< / answer> ”; Bounding box restoration: The bounding box values ​​need to be restored according to the original image resolution ratio. Think Text Extraction: Extract Think text using regular expressions.

[0017] S108: Output Results: Outputs defect type, confidence level, bounding box, and Think text in text, image, or JSON format, connecting to the power operation and maintenance platform. Supports three output formats to adapt to different operation and maintenance scenarios: Text format: Directly output key information, such as "Defect type: Insulator crack, original bounding box: (240,160,500,420)", which is convenient for quick manual review; Image format: Draw a red bounding box on the original image, label the defect type, and save it as a new image (e.g., "result_insulator_001.jpg") for easy visualization; JSON format: Used for integration with power operation and maintenance platforms, such as "{"image_name":"insulator_001.jpg","defect_type":"insulator crack","bbox_original":[61,700,391,923],"think":"According to the description, the meter dial has obvious cracks and damage, which affects the display of the scale and pointer. The meter dial in the picture has obvious cracks and damage, so it should be judged as dial damage."}", supporting batch data processing and analysis.< / think>

Claims

1. A power transmission line equipment defect identification method based on a multi-modal large model technology, characterized in that, Includes the following steps: (1) Obtain inspection images, defect annotation data and contextualized prompts of transmission line equipment, and construct image-prompt-annotation triplet data; (2) Preprocess the triplet data to generate standardized training samples; (3) Load a pre-trained visual-language multimodal large model as the base model; Based on the reinforcement learning fine-tuning framework, combined with the enhanced intersection-over-union reward mechanism and the cue word-inference text alignment mechanism, the basic model is fine-tuned with a few samples to obtain the defect recognition model; (4) Input the image to be detected and the corresponding prompt words into the defect recognition model to generate the defect type, location information and interpretable reasoning process text; (5) Decode the model output to output structured defect identification results.

2. The method for defect identification of transmission line equipment based on multimodal large model technology according to claim 1, characterized in that, The preprocessing steps also include: template-encapsulating the prompt words, fixing the format of system prompt words and question templates, and ensuring that the input text structure is consistent between the training and inference stages.

3. The method for defect identification of transmission line equipment based on multimodal large model technology according to claim 1, characterized in that, The enhanced cross-union ratio (CUNR) reward mechanism is as follows: Based on the CUNR, a center point distance penalty factor and an aspect ratio consistency constraint factor are introduced between the predicted bounding box and the true bounding box to improve the localization accuracy of small-sized defect targets.

4. The method for defect identification of transmission line equipment based on multimodal large model technology according to claim 1, characterized in that, The prompt-inference text alignment mechanism is as follows: Chinese word segmentation is performed on the prompts input by the user and the inference text generated by the model to extract core words; similarity is calculated based on the longest common subsequence to evaluate the semantic matching degree between the inference text and the prompts; the similarity is used as a reward signal to guide the model to generate inference content that is highly related to the contextual prompts.

5. The method for defect identification of transmission line equipment based on multimodal large model technology according to claim 1, characterized in that, The output of structured defect identification results includes: text format: directly outputting defect type, bounding box coordinates and inference text; image format: drawing defect bounding boxes on the original image and labeling the defect type; JSON format: structured output of image name, defect type, bounding box coordinates and inference text, which is convenient for integration with the power operation and maintenance platform.

6. A defect identification system for transmission line equipment based on multimodal large model technology, characterized in that, include: The data acquisition module is used to acquire inspection images, defect annotation data, and contextual prompts for power transmission line equipment. The data preprocessing module is used to standardize the collected data and generate training samples; The model fine-tuning module is used to load pre-trained visual-language multimodal large models and perform small-sample fine-tuning based on a reinforcement learning framework combined with an enhanced intersection-union reward mechanism and a cue word-inference text alignment mechanism. The inference output module is used to input the image to be detected and the prompt words into the fine-tuned model, and generate and output the defect type, location information and interpretable inference text.

7. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the program to implement the steps of the method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The device contains a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-5.

Images