Power transmission line defect detection method and device and medium

Abstract

The invention discloses a power transmission line defect detection method and device and a medium, and the method comprises the steps: carrying out the recognition and positioning of a power transmission line defect in image data according to a multi-mode large model, and obtaining a defect detection result; wherein the multi-modal large model is obtained by freezing a target trunk parameter in an initial multi-modal model, training a newly added low-rank parameter in the initial multi-modal model according to preset data, and performing back propagation optimization in combination with joint loss. According to the power transmission line defect detection method and device and the medium, model stability can be kept by freezing trunk parameters, precision can be improved and overfitting can be reduced through low-rank parameter training, multi-task performance is balanced through joint loss optimization, model robustness can be enhanced, and the detection accuracy is improved. Therefore, the defects of the power transmission line can be accurately identified by using the multi-modal large model, false detection and missing detection are reduced, and the problem that the precision and robustness of defect detection of the power transmission line are difficult to improve can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image detection, in particular to a power transmission line defect detection method, device and medium. BACKGROUND

[0002] The power transmission line is the core facility of the national power grid, responsible for long-distance and large-capacity power transmission tasks, and is crucial to the development of the national economy and the life of the people. However, these lines often span hundreds to thousands of kilometers and are exposed to harsh natural environments for a long time, and are vulnerable to lightning, strong winds, icing, pollution, bird damage, and external damage, thus producing a variety of physical defects such as conductor foreign matter, broken strands, insulator damage, hardware corrosion, tower tilt, bird nest building, tree barrier and mountain fire hazards. If these defects are not discovered and addressed in a timely manner, they will affect the efficiency of power transmission, and in severe cases, may cause line tripping, large-scale power outages, and even safety accidents, resulting in significant economic losses and social impact. Current defect detection techniques are mainly achieved through two technical paths: one is based on traditional image processing and machine learning methods, using feature extractors such as SIFT, SURF, HOG, and classifiers such as SVM and AdaBoost, to design artificial features and classify specific defect types; the other is based on deep learning methods, especially convolutional neural networks (CNN), such as Faster R-CNN, YOLO series, and Mask R-CNN models, which can automatically learn image features to achieve target detection and image segmentation tasks.

[0003] However, traditional image processing and machine learning methods have poor generalization ability, complex feature design, insufficient information utilization, and need to improve accuracy and robustness. Deep learning-based methods rely on large-scale labeled data, and the model's ability is limited, making it difficult to understand and utilize complex contextual information or perform natural language interactive queries, and often requires independent models for different tasks, making system integration complex and difficult to accurately measure power transmission line defects. SUMMARY

[0004] The present application provides a power transmission line defect detection method, device and medium to solve the problem of difficulty in improving the accuracy and robustness of power transmission line defect detection.

[0005] To achieve the above-mentioned purpose, the present application provides a power transmission line defect detection method, comprising:

[0006] Obtaining image data of the power transmission line;

[0007] According to the multi-modal large model, the defects of the power transmission line in the image data are recognized and positioned, and a defect detection result is obtained; wherein the multi-modal large model is obtained by freezing the target backbone parameters in the initial multi-modal model, training new low-rank parameters in the initial multi-modal model according to preset data, and combining joint loss for back propagation optimization.

[0008] The application accurately identifies and locates the defects of the power transmission line in the image data through the multi-modal large model, avoiding the false detection and missed detection problems that may exist in the traditional method. For the multi-modal large model, freezing the target backbone parameters in the initial multi-modal model helps to maintain the stable performance obtained in the previous learning task. These backbone parameters usually contain a large number of general features and knowledge, which are basic and important for the specific task of power transmission line defect detection. At the same time, by training the new low-rank parameters, the model can adaptively adjust to new image data without destroying the original stable structure, learn more specific defect features, and improve the detection accuracy. In addition, low-rank parameter updating has lower computational complexity and fewer parameter quantities than full parameter updating, which helps to reduce the risk of overfitting and improve the generalization ability of the model. Combined with joint loss for back propagation optimization, the loss functions of multiple tasks can be considered comprehensively to improve the overall performance. Moreover, joint loss optimization helps the model to find a balance point between multiple tasks, so that the model can perform higher precision and robustness on the power transmission line defect detection task.

[0009] Compared with the prior art, the application can maintain the stability of the model by freezing the backbone parameters, improve the accuracy and reduce the overfitting by training the low-rank parameters, and balance the performance of multiple tasks by joint loss optimization, thereby enhancing the robustness of the model. Therefore, using the multi-modal large model can accurately identify the defects of the power transmission line and reduce false detection and missed detection, so as to solve the problem of improving the precision and robustness of the power transmission line defect detection.

[0010] As a preferred scheme, the image data of the power transmission line is obtained, specifically:

[0011] The original image data of the power transmission line is obtained.

[0012] In the original image data, the boundary box of the defect area is labeled, the pre-defined class label is assigned to the image data, and the text description is attached to each defect instance by the preset model, and the comprehensive data is obtained.

[0013] The comprehensive data is divided into the image data including the training data set and the test data set.

[0014] The preferred scheme can accurately locate potential problems in the power transmission line by bounding box labeling of the defect area in the original image data, providing a basis for subsequent analysis and detection. Assigning predefined class labels to image data helps quickly identify and classify different types of defects, improving data processing efficiency. By attaching text descriptions to each defect instance using a pre-set model, the semantic information of the data is increased, which helps to better understand the defect situation.

[0015] As a preferred scheme, the multi-modal large model is obtained by freezing the target backbone parameters in the initial multi-modal model, training the newly added low-rank parameters in the initial multi-modal model according to the preset data, and combining the joint loss for back propagation optimization, specifically:

[0016] Freeze the target backbone parameters in the initial multi-modal model, and only open the newly added low-rank adaptive parameters in the initial multi-modal model to obtain an initialized multi-modal model;

[0017] According to the training data set, the joint loss is combined to iteratively optimize the newly added low-rank adaptive parameters in the initialized multi-modal model, to obtain the multi-modal large model.

[0018] In the preferred scheme, since only the newly added low-rank parameters are trained, the required computing resources and storage space are greatly reduced, and the newly added low-rank adaptive parameters enable the model to be fine-tuned for specific tasks without changing the overall structure of the pre-trained backbone network. This flexibility enables the model to better adapt to new data sets and tasks while effectively utilizing pre-trained knowledge. Since most model parameters are frozen and only the newly added low-rank parameters are optimized, this helps to maintain the stability of the training. During back propagation, the model is more likely to converge to a better solution because most of the weights have been optimized through pre-training.

[0019] As a preferred scheme, the joint loss is calculated according to the target detection loss, the classification loss, and the multi-modal contrast loss, specifically:

[0020] According to the predicted box coordinates output by the visual encoder in the initial multi-modal model, the real box label is combined to quantify the boundary box regression error through the GIoU loss, to establish the target detection loss;

[0021] According to the class probability distribution output by the model classification head in the initial multi-modal model, the classification error is quantified through the cross-entropy loss to obtain the classification loss;

[0022] According to the image-text feature vectors output by the model cross-modal interaction layer in the initial multi-modal model, the cosine similarity of the image and the text is calculated, and the multi-modal contrast loss is established in combination with the preset temperature coefficient;

[0023] The target detection loss, the classification loss and the multi-modal contrast loss are weighted to obtain the joint loss.

[0024] In this preferred embodiment, the joint loss integrates target detection, classification and multi-modal contrast tasks, enabling the model to master multiple skills simultaneously and enhancing generalization and practicality. The GIoU loss can accurately evaluate the gap between the predicted box and the true box, improving the accuracy of target positioning. The classification loss can effectively deal with class imbalance using cross-entropy, ensuring more accurate class probability distribution. The multi-modal contrast loss aligns the image and text features through cosine similarity and temperature coefficient, enhancing the understanding and association of cross-modal information and improving multi-modal learning effect.

[0025] As a preferred embodiment, the power transmission line defects in the image data are identified and located according to the multi-modal large model to obtain defect detection results, specifically:

[0026] The power transmission line defects in the image data are identified and located according to the multi-modal large model to obtain preliminary detection results.

[0027] The non-maximum suppression algorithm is used to eliminate repeated detection boxes for the same defect in the preliminary detection results, and the defect detection results with a confidence higher than a set threshold are selected.

[0028] This preferred embodiment eliminates repeated detection boxes for the same defect through the non-maximum suppression algorithm, avoiding multiple overlapping detection boxes in the output results, making the detection results clearer and more reliable. By setting a confidence threshold and filtering out detection results below the threshold, low-confidence false detection results can be excluded, thereby improving overall detection performance.

[0029] As a preferred embodiment, after the power transmission line defects in the image data are identified and located according to the multi-modal large model to obtain defect detection results, the following steps are further included:

[0030] The multi-dimensional data fusion technology is used to associate and map the defect detection results and the geographic location data of the power transmission line to obtain a structured inspection report.

[0031] The inspection report is displayed on a human-computer interaction interface.

[0032] This preferred embodiment integrates detection results and geographic location data through multi-dimensional data fusion technology to form a structured inspection report. This integration not only provides detailed information about defects, but also clearly identifies the specific location of defects on the power transmission line, making the data more complete and intuitive. Moreover, the structured inspection report is easy to understand and analyze, which helps maintenance personnel quickly grasp the health status of the power transmission line.

[0033] The application also provides a power transmission line defect detection device, comprising a data module and a detection module;

[0034] The data module is configured to obtain image data of the power transmission line.

[0035] The detection module is configured to identify and locate defects in the image data of the power transmission line according to a multi-modal large model, and obtain a defect detection result; wherein the multi-modal large model is obtained by freezing target backbone parameters in an initial multi-modal model, training new low-rank parameters in the initial multi-modal model according to preset data, and performing reverse propagation optimization combined with a joint loss.

[0036] As a preferred solution, the data module comprises an image unit, a preprocessing unit and a division unit.

[0037] The image unit is configured to obtain original image data of the power transmission line.

[0038] The preprocessing unit is configured to perform bounding box labeling on a defect area in the original image data, assign a predefined class label to the image data, and attach a text description to each defect instance through a preset model to obtain comprehensive data.

[0039] The division unit is configured to divide the comprehensive data into the image data comprising a training data set and a test data set.

[0040] As a preferred solution, the detection module comprises a freezing unit and a training unit.

[0041] The freezing unit is configured to freeze the target backbone parameters in the initial multi-modal model, and only open the newly added low-rank adaptive parameters in the initial multi-modal model to obtain an initialized multi-modal model.

[0042] The training unit is configured to perform reverse propagation iterative optimization on the newly added low-rank adaptive parameters in the initialized multi-modal model according to the training data set combined with the joint loss to obtain the multi-modal large model.

[0043] The application also provides a storage medium having a computer program stored thereon, wherein the computer program is invoked and executed by a computer to implement the power transmission line defect detection method. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is a flowchart of a power transmission line defect detection method provided by an embodiment of the application;

[0045] Figure 2is a high-efficiency fine-tuning process schematic diagram provided by the embodiment of the present application;

[0046] Figure 3 is a multi-modal large model power transmission line defect detection system architecture diagram based on high-efficiency fine-tuning provided by the embodiment of the present application;

[0047] Figure 4 is a power transmission line defect detection method flow chart based on a multi-modal large model of high-efficiency fine-tuning provided by the embodiment of the present application;

[0048] Figure 5 is a structural schematic diagram of a power transmission line defect detection device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0050] In the description of the present application, it should be understood that the terms "first" and "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first" and "second" can explicitly or implicitly include one or more features. In the description of the present application, unless otherwise specified, the meaning of "several" is two or more.

[0051] The power transmission line defect detection method provided by the embodiment of the present application aims to improve the precision, robustness and field adaptability of the multi-modal large model in the power transmission line defect detection by using the strong ability of VLM, the targeted fine-tuning strategy, the parameter-efficient fine-tuning technology, the data enhancement strategy, the specially designed loss function, and to realize unified and efficient detection and classification of various defects of power transmission line and provide more intelligent analysis results.

[0052] Embodiment one:

[0053] Please refer to Figure 1 The embodiment of the present application provides a power transmission line defect detection method, including S1-S2, and the specific implementation steps are as follows:

[0054] S1, acquiring image data of the power transmission line.

[0055] The step S1 of the embodiment of the present application comprises S1.1-S1.2; wherein S1.1 is a process of acquiring original image data of the power transmission line, and S1.2 is a process of preprocessing the original image data to acquire image data, specifically:

[0056] S1.1, based on the data acquisition module, through the unmanned aerial vehicle carrying high-definition visible light cameras and infrared thermal imagers and other sensors, according to the preset flight route or inspection plan, image and video data are collected, so as to acquire original image data of the power transmission line and its surrounding environment, which covers the original image and continuous video sequence of the power transmission line. In addition, the embodiment also supports accessing data sources from fixed monitoring cameras or other channels.

[0057] S1.2, based on the data preprocessing module, in the original image data, according to the data processing strategy, the defect area is marked with a bounding box, the image data is assigned a predefined class label, and a text description is added to each defect instance through a preset model, to obtain comprehensive data suitable for model input after preprocessing and labeling;

[0058] The comprehensive data is divided into image data including a training data set and a test data set.

[0059] Among them, the data processing strategy covers multiple key steps, aiming to optimize the quality and utility of the original image data. This includes: cleaning the data to remove impurities, enhancing the data to improve the model generalization ability; implementing geometric and color transformation, and simulating environmental changes to enrich data diversity; using CutMix / Mixup and other methods to synthesize new data; performing detailed data labeling to clearly define defect location and category; adding natural language description to enhance data semanticity; finally, format conversion is performed to meet the needs of different application scenarios. Each step is specifically:

[0060] ①Data cleaning: this step aims to remove low-quality images, including blurred, overexposed, severely occluded or information missing images, to ensure the accuracy and effectiveness of the training data.

[0061] ②Data enhancement: this step is the core link. According to the particularity of power transmission line monitoring, various data enhancement techniques are used, such as random flipping, rotation, color jittering, etc., to effectively expand the training sample set, alleviate the data scarcity problem, and enhance the generalization ability of the model.

[0062] ③Geometric transformation: random rotation, cropping, scaling and translation operations are applied to simulate power transmission line images from different perspectives, increasing the diversity of the data.

[0063] ④Color transformation: adjust the brightness, contrast and saturation of the image to adapt to image acquisition under different lighting conditions and improve the model's performance under different lighting conditions.

[0064] 5. Simulate environmental changes: By adding simulated weather effects such as fog, rain, and snow, as well as changes in lighting, the training data is further enriched, enabling the model to better handle complex and variable natural environments.

[0065] 6. Synthetic data generation: This step is optional. Advanced graphics techniques or Generative Adversarial Networks (GANs) are used to generate high-quality synthetic images containing specific power line defects such as conductor breakage, insulator contamination, etc., to supplement the lack of real-world data.

[0066] 7. CutMix / Mixup strategy: Advanced image mixing techniques such as Mixup (mixing enhancement) strategy and CutMix (cutting and mixing) strategy are used to fuse different images and their labels to generate new training samples, thereby improving the model's generalization ability and robustness.

[0067] 8. Detailed data labeling: Detailed labeling of images in the training set, including but not limited to:

[0068] Defect location marking: Use bounding boxes (Bounding Box) to accurately mark the area where the defect is located.

[0069] Defect category assignment: Assign each defect a predefined category label, such as "conductor foreign object", "insulator defect", "tower tilt", etc.

[0070] Natural language description: Provide detailed text descriptions for each defect instance through pre-set models or manual annotation, such as "insulator second umbrella skirt edge crack", which can be used for multi-modal training to enhance the model's understanding ability.

[0071] 9. Data format conversion: Convert the processed data into standard formats required for model training and inference, such as specific size image tensors, corresponding label vectors or text description files, to ensure data consistency and compatibility.

[0072] This embodiment S1.2 can accurately locate potential problems in the power transmission line by marking the defect area with a bounding box in the original image data, providing a basis for subsequent analysis and detection. Assigning predefined category labels to image data helps quickly identify and classify different types of defects, improving data processing efficiency. By adding text descriptions to each defect instance through pre-set models, the semantic information of the data is increased, which helps to better understand the defect situation.

[0073] S2, identifying and positioning defects in the power transmission line in the image data according to the multi-modal large model to obtain a defect detection result; wherein the multi-modal large model is obtained by freezing target backbone parameters in an initial multi-modal model, training new low-rank parameters in the initial multi-modal model according to preset data, and combining joint loss for back propagation optimization.

[0074] The embodiment of the application comprises steps S2.1-S2.3; wherein S2.1 is a process of constructing a multi-modal large model, S2.2 is a process of obtaining a preliminary detection result according to the multi-modal large model, and S2.3 is a process of post-processing the preliminary detection result to obtain a defect detection result, which is specifically:

[0075] S2.1, based on the core processing module, freezing the target backbone parameters θ0 in the initial multi-modal model, only opening the newly added low-rank adaptive parameters Δθ (|Δθ|<<|θ0|) in the initial multi-modal model, obtaining the initialized initial multi-modal model, i.e. the initialized multi-modal model. Wherein the initial multi-modal model adopts Qwen2.5-VL multi-modal large model (or other advanced multi-modal large model), which integrates the depth optimized visual encoder and high capacity autoregressive language model through multi-source data collaborative pre-training mechanism; through multi-granularity cross-modal alignment mechanism, it can realize the dynamic mapping of visual feature space and semantic concept space, which can not only maintain the geometric structure integrity of visual representation, but also realize the context perception generation of language mode; the synergistic enhancement of such dual modal ability, combined with fine-grained attention routing strategy, can construct a basic ability framework with strong generalization for downstream cross-modal reasoning tasks. And, the "newly added low-rank adaptive parameter" is a low-rank matrix parameter introduced additionally for task adaptation based on the initial multi-modal model.

[0076] Using the PEFT (Parameter Efficient Fine-Tuning) method, the newly added low-rank adaptive parameters in the initialized multi-modal model are iteratively optimized in reverse propagation combined with joint loss according to the training data set in the image data, and a multi-modal large model is obtained, i.e. the Qwen2.5-VL model after efficient fine-tuning.

[0077] Wherein, the PEFT method includes but is not limited to LoRA (Low-Rank Adaptation), AdapterTuning, Visual / Text Prompt Tuning and BitFit, and the following will specifically describe the way of training the initialized multi-modal model using each method:

[0078] ① LoRA (Low-Rank Adaptation):

[0079] 1) Parameter freezing stage: directly lock the target weight matrix W0 in the initialized multi-modal model (such as the Q / K / V projection layer or FFN layer in the Transformer), and keep its parameter value unchanged throughout the process to avoid the gradient propagation overhead brought by full fine-tuning.

[0080] 2) Low-rank adapter implantation: two trainable small matrices are inserted in parallel on both sides of W0: matrix A (dimension r x k): as an input feature encoder, which maps the original input x to a low-dimensional space; matrix B (dimension d x r): as an output feature decoder, which maps the low-dimensional representation back to the original dimension; where the rank r is usually taken as 4-64, which is much smaller than the original dimension d / k, such as d = 4096 and r = 8.

[0081] 3) Incremental calculation reconstruction: the update amount ΔW of the original weight matrix is decomposed into the product of two low-rank matrices: ΔW = B·A.

[0082] During forward propagation, the input feature x will simultaneously experience two calculation paths, one is the original path: x directly generates W0x through the frozen W0. The other is the incremental path: x is first reduced in dimension by A to obtain Ax, and then is raised in dimension by B to obtain B(Ax).

[0083] The final output is the sum of the results of the two paths: h = (W0+ΔW)x = W0x+B(Ax).

[0084] 4) Parameter update mechanism: only the following operations are performed during training: keep the gradient of W0 masked Calculate the gradient of A and B Update the parameters of A and B by the chain rule η is the learning rate).

[0085] With this strategy, the number of training parameters is significantly reduced from “d x k” to “r(d+k)”. Specifically, when “d = 4096” and “k = 4096”, if full fine-tuning is performed, about 16.7M parameters need to be updated. However, if the LoRA method of the present embodiment with “r = 8” is used, only about “8 x (4096 + 4096) ≈ 65K” parameters need to be adjusted, and the parameter reduction ratio is as high as 99.6%.

[0086] This process is essentially an approximation of the update direction of the high-dimensional parameter space by a low-dimensional manifold, which preserves the pre-trained knowledge while capturing the specific patterns of downstream tasks with a small number of trainable parameters, achieving a balance between computational resources and model performance.

[0087] ②Adapter Tuning: By cleverly embedding small and trainable "adapter" modules between the layers of the initialized multi-modal model, the core parameters of the original model are frozen, and only these adapter modules are trained, achieving efficient fine-tuning.

[0088] ③(Visual)Prompt Tuning: At the input end of the initialized multi-modal model, trainable visual prompts (Visual Prompts) or text prompts (Text Prompts) are introduced. These prompts act as guides, leading the model to focus on the core features of a specific task or domain without touching the complex structure or parameters inside the model.

[0089] ④BitFit: This strategy focuses on the bias parameters of the model. By fine-tuning these parameters, the performance can be optimized without interfering with the main structure of the model.

[0090] The above methods significantly reduce the dependence of the model on large-scale, high-quality labeled data (benefiting from the large amount of knowledge accumulated in pre-training, only a small number of parameters need to be fine-tuned to adapt to specific tasks), effectively solving the problem of data scarcity and high labeling cost in the power transmission field. In addition, by significantly reducing the demand for gradient calculation, optimizer state storage, etc., this method significantly reduces the computational resources (such as GPU memory and computing power) and time cost required for model training, making it more economical and efficient to use advanced large models for domain-specific adaptation.

[0091] Among them, the joint loss is calculated according to the target detection loss, classification loss and multi-modal contrast loss, specifically:

[0092] According to the predicted box coordinates output by the visual encoder in the initial multi-modal model, combined with the real box label, the boundary box regression error is quantified by the Generalized Intersection over Union (GIoU) loss, and the target detection loss is established. Specifically, in tasks involving defect positioning, the Focal Loss strategy is adopted to effectively solve the challenge of class distribution imbalance, and loss functions such as Generalized Intersection over Union (GIoU), Distance Intersection over Union (DIoU) and Complete Intersection over Union (CIoU) are introduced to focus on fine-tuning the regression accuracy of the boundary box; among them, FocalLoss strategy is a loss function used to solve the problem of class imbalance in target detection.

[0093] According to the class probability distribution output by the model classification head in the initial multi-modal model, the classification error is quantified by the Cross-Entropy Loss, and the classification loss is obtained;

[0094] According to the image-text feature vectors output by the model cross-modal interaction layer in the initial multi-modal model, the cosine similarity of the image and the text is calculated, and a multi-modal contrast loss is established in combination with a preset temperature coefficient. Specifically, if the training data set covers images and their corresponding text descriptions, a contrast learning loss (such as CLIP Loss) is used, which aims to pull closer the image-text pairs that match each other in the feature space, while pushing away the image-text pairs that do not match each other, thereby significantly enhancing the model's understanding and association ability of cross-modal information.

[0095] According to the importance or the number of samples of different defect types, the target detection loss, the classification loss and the multi-modal contrast loss are weighted and calculated to obtain a joint loss.

[0096] For the application of the embodiments of the present application, please refer to Figure 2 , Figure 2 is a high-efficiency fine-tuning process diagram provided by the embodiments of the present application, which shows the process of efficiently fine-tuning the pre-trained Qwen2.5-VL model. It mainly includes selecting a pre-trained model, preparing a field data set, selecting and implementing a parameter efficient fine-tuning (PEFT) method, defining a field-specific loss function, training and evaluating the model.

[0097] In the present embodiment S2.1, since only the newly added low-rank parameters are trained, the required computing resources and storage space are greatly reduced, and the newly added low-rank adaptive parameters enable the model to be fine-tuned for specific tasks without changing the overall structure of the pre-trained backbone network. This flexibility enables the model to better adapt to new data sets and tasks while maintaining the effective use of pre-trained knowledge. Since most of the model parameters are frozen and only the newly added low-rank parameters are optimized, this helps to maintain the stability of the training. During backpropagation, the model is more likely to converge to a better solution because most of the weights have been optimized through pre-training;

[0098] Moreover, the joint loss combines the target detection, classification and multi-modal contrast tasks, enabling the model to master multiple skills at the same time and enhancing its generalization and practicality. The GIoU loss can accurately evaluate the gap between the predicted box and the real box, improving the accuracy of target positioning. The classification loss can effectively deal with class imbalance using cross-entropy, ensuring more accurate class probability distribution. The multi-modal contrast loss aligns the image and text features through cosine similarity and temperature coefficient, enhancing the understanding and association of cross-modal information and improving the multi-modal learning effect.

[0099] Compared with traditional image processing techniques and early machine learning models with limited generalization ability, the present application relies on the powerful general visual understanding and cross-modal correlation ability of advanced multi-modal large models such as Qwen2.5-VL. Through parameter-efficient fine-tuning (such as LoRA) and field-specific data augmentation and loss function design, the present application successfully transfers and optimizes this general ability to the transmission line defect identification task, not only maintaining the model's broad applicability, but also enhancing its sensitivity to field-specific features such as defect morphology and complex background differences. This approach overcomes the adaptability challenges of general models in the power transmission field, enabling precise identification of diverse and subtle defects in complex backgrounds, significantly improving detection accuracy, enhancing robustness to lighting, angle, and weather changes, and greatly reducing the rate of missed and false detections.

[0100] S2.2, based on the core processing module, identifying and locating the transmission line defects in the test data set according to the multi-modal large model, obtaining the preliminary detection results; wherein the test data set is derived from image data.

[0101] Among them, the preliminary detection results cover multiple key information, including:

[0102] Defect location: provides accurate bounding box coordinates for each detected defect, clearly identifying the specific location of the defect in the image;

[0103] Defect category: accurately classifies each defect through classification labels (such as "wire strand breakage");

[0104] Confidence score: assigns a confidence score to each detection result, reflecting the model's degree of certainty about the detection result;

[0105] Natural language description: This description is an optional setting. According to user actual needs, the model can generate detailed text descriptions of the detected defects, providing more intuitive interpretation.

[0106] S2.3, based on the post-processing and result generation module, eliminating duplicate detection boxes for the same defect in the preliminary detection results through the non-maximum suppression algorithm (NMS), and filtering out detection parts with confidence scores below a certain threshold, obtaining the defect detection results. It is worth noting that in specific scenarios, such as using multi-model integration processing or analyzing video sequences, the module will further perform a result fusion step to integrate the detection data of each model or frame to generate the final defect detection results.

[0107] Through multi-dimensional data fusion technology, combined with UAV GPS data or line account information, the detected defects in the detection results and the geographic location data or tower number of the power transmission line are associated and mapped. On this basis, the processed defect information is arranged into a structured inspection report and defect list in a clear and orderly manner, including but not limited to tables or lists, and detailedly containing defect pictures, accurate positions, type classification, severity assessment (if conditions permit), and key information such as discovery time. It is worth mentioning that this geographic information association mapping is an optional enhancement function that can be flexibly applied according to actual needs.

[0108] Based on the user interaction and display module, the inspection report and defect list are displayed on the human-computer interaction interface of the desktop application or Web application, and the specific presentation mode is as follows: the detected defect frame and corresponding label are displayed on the original image, and the geographic location of the defect is accurately marked on the map. In addition, users can flexibly query and filter the results according to time, line, defect type, and other multiple conditions. Further, with the advanced interaction capability of VLM (Visual Language Model), users can ask questions in natural language, such as "Please enlarge and display the detailed situation of this insulator damage" or "Are there any other bird nests on this line?". At the same time, the embodiment also supports functions such as user-defined detection parameters and user permission management.

[0109] In this way, the embodiment can respond to user operation instructions in real time, quickly retrieve and display visual defect detection results, detailed reports, and system status information.

[0110] To apply the embodiment of the present application, please refer to Figures 3-4 ;

[0111] Figure 3 is the multi-modal large model power transmission line defect detection system architecture provided by the embodiment of the present application, which shows the main component modules of the system and their mutual relationship, and the system mainly includes a data acquisition module (101), a data preprocessing module (102), a core processing module (103), a post-processing and result generation module (104), and a user interaction and display module (105). The data flow starts from the acquisition stage, goes through preprocessing refinement, and then is input to the multi-modal large model for in-depth analysis. Finally, through the carefully designed post-processing step, the system generates defect detection results and displays them to the user through a user-friendly interface. And, Figure 3 The data acquisition module (101) to the user interaction and display module (105) shown in the embodiment are completely consistent with the modules used in the actual data processing process of the embodiment;

[0112] The data acquisition module (101) is configured to acquire raw data from the power transmission line site.

[0113] The data preprocessing module (102) is configured to clean, enhance, label and format the raw data, and prepare for model training and inference.

[0114] The core processing module (103) is configured to receive the preprocessed image and possible text prompts, use a multi-modal large model to identify and locate defects in the power transmission line image, and output preliminary detection results.

[0115] The post-processing and result generation module (104) is configured to optimize the preliminary detection results output by the model and generate a final report.

[0116] The user interaction and display module (105) is configured to provide a human-computer interaction interface to facilitate users to upload data, start detection, view results, manage reports, etc.

[0117] Figure 4 The power transmission line defect detection method based on the efficient fine-tuning multi-modal large model is provided in the embodiment of the present application, which shows the process of obtaining data, implementing defect detection, and displaying defect information, specifically as follows:

[0118] Step S301: Data acquisition and preprocessing. Start the data acquisition module (101) to capture image or video data of the power transmission line. The captured data is transmitted to the data preprocessing module (102) for cleaning, enhancement (only in the training stage), size adjustment and other preprocessing operations to ensure that the image data meets the detection requirements.

[0119] Step S302: Load fine-tuned model. Load the Qwen2.5-VL model parameters that have been efficiently fine-tuned from the core processing module (103) to prepare for the subsequent inference process.

[0120] Step S303: Model inference. Input the preprocessed image data into the loaded fine-tuned model. The model performs forward propagation and uses its learned professional knowledge to conduct in-depth analysis of the image.

[0121] Step S304: Output preliminary detection results. The model outputs preliminary detection results including potential defect locations (in the form of bounding boxes), class labels and confidence scores.

[0122] According to the requirements, the corresponding natural language description can also be output to provide more intuitive interpretation.

[0123] Step S305: result post-processing. The preliminary detection results are passed to the post-processing and result generation module (104). Non-maximum suppression (NMS), confidence filtering, and other operations are performed to further optimize the detection results and improve accuracy.

[0124] Step S306: information association and report generation, which is an optional operation. According to actual needs, the optimized defect results are associated with geographic location information or line asset information. Structured inspection reports are generated for subsequent analysis and processing.

[0125] Step S307: result display and user interaction. The final defect detection results and reports are sent to the user interaction and display module (105). Defect information is displayed intuitively on the user interface, and query, analysis, and interaction functions are provided to meet user needs.

[0126] The embodiment S2.3 eliminates repeated detection boxes for the same defect through the non-maximum suppression algorithm, avoiding multiple overlapping detection boxes in the output results, making the detection results clearer and more reliable. By setting a confidence threshold and filtering out detection results below the threshold, low-confidence false detection results can be excluded, thereby improving overall detection performance;

[0127] Moreover, through multi-dimensional data fusion technology, the detection results are closely integrated with geographic location data to form structured inspection reports. This integration not only provides detailed information about defects but also clearly identifies the specific location of defects on the transmission line, making the data more complete and intuitive. In addition, structured inspection reports are easy to understand and analyze, helping maintenance personnel quickly grasp the health status of the transmission line.

[0128] Overall, the embodiment has the following beneficial effects:

[0129] The application can accurately identify and locate defects in the power transmission line image data through a multi-modal large model, avoiding the false detection and missed detection problems that may exist in traditional methods. For the multi-modal large model, freezing the target backbone parameters in the initial multi-modal model helps to maintain the stable performance obtained in the previous learning task. These backbone parameters usually contain a large amount of general features and knowledge, which are basic and important for the specific task of power transmission line defect detection. At the same time, by training the newly added low-rank parameters, the model can adaptively adjust to new image data without destroying the original stable structure, and learn more specific defect features to improve the detection accuracy. In addition, the low-rank parameter update has lower computational complexity and fewer parameter quantities than the full parameter update, which helps to reduce the risk of overfitting and improve the generalization ability of the model. Combined with the joint loss for back propagation optimization, the loss functions of multiple tasks can be considered comprehensively to improve the overall performance. Moreover, the joint loss optimization helps the model to find a balance point between multiple tasks, so that the model can perform higher precision and robustness in the power transmission line defect detection task.

[0130] In summary, the application uses a single fine-tuning multi-modal large model to achieve efficient and unified detection of various defects in the power transmission line, simplifies the system design, reduces the maintenance complexity, and improves the detection efficiency. In addition, by using the cross-modal ability of the VLM, the application can output natural language descriptions and support human-computer interactive question and answer, providing more intelligent and detailed defect analysis and decision support tools for operation and maintenance personnel.

[0131] Embodiment two:

[0132] Please refer to Figure 5 The embodiment of the application provides a power transmission line defect detection device, which comprises a data module 10 and a detection module 20.

[0133] The data module 10 is used to obtain image data of the power transmission line.

[0134] The detection module 20 is used to identify and locate defects in the image data of the power transmission line according to a multi-modal large model, and obtain a defect detection result. The multi-modal large model is obtained by freezing target backbone parameters in an initial multi-modal model, training new low-rank parameters in the initial multi-modal model according to preset data, and optimizing the joint loss for back propagation.

[0135] In one embodiment, the data module 10 comprises an image unit, a preprocessing unit and a division unit. The image unit is a process of obtaining original image data of the power transmission line, and the preprocessing unit and the division unit are processes of preprocessing the original image data to obtain image data.

[0136] The image unit is configured to collect image and video data based on the data collection module and sensors such as high-definition visible light cameras and infrared thermal imagers carried by the unmanned aerial vehicle, according to a preset flight route or inspection plan, so as to obtain original image data of the power transmission line and the surrounding environment. These data cover the original image and continuous video sequence of the power transmission line. In addition, the embodiment also supports accessing data sources from fixed monitoring cameras or other channels.

[0137] The preprocessing unit is configured to perform bounding box labeling on the defect area in the original image data according to a data processing strategy, assign a predefined class label to the image data, and attach a text description to each defect instance through a preset model, so as to obtain comprehensive data suitable for model input after preprocessing and labeling.

[0138] The division unit is configured to divide the comprehensive data into image data including a training data set and a test data set.

[0139] The data processing strategy covers multiple key steps, aiming to optimize the quality and utility of the original image data. This includes: cleaning the data to remove impurities, enhancing the data to improve the model generalization ability; implementing geometric and color transformation, and simulating environmental changes to enrich data diversity; using CutMix / Mixup and other methods to synthesize new data; performing detailed data labeling to clearly define defect location and category; adding natural language description to enhance data semanticity; finally, performing format conversion to meet the needs of different application scenarios. Each step is as follows:

[0140] ① Data cleaning: This step aims to remove low-quality images, including blurred, overexposed, severely occluded or information missing images, to ensure the accuracy and effectiveness of the training data.

[0141] ② Data enhancement: This step is the core. In view of the particularity of power transmission line monitoring, various data enhancement techniques are adopted, such as random flipping, rotation, color jitter, etc., to effectively expand the training sample set, alleviate the problem of data scarcity, and enhance the generalization ability of the model.

[0142] ③ Geometric transformation: Random rotation, cropping, scaling and translation operations are applied to simulate power transmission line images from different angles, increasing the diversity of data.

[0143] ④ Color transformation: Adjust the brightness, contrast and saturation of the image to adapt to image acquisition under different lighting conditions and improve the model's performance under different lighting conditions.

[0144] ⑤ Simulate environmental changes: By adding simulated fog, rain, snow weather effects and light changes, further enrich the training data, so that the model can better cope with complex and variable natural environments.

[0145] ⑥Synthetic Data Generation: This step is optional. Advanced graphics techniques or Generative Adversarial Networks (GANs) are used to generate high-quality synthetic images containing specific power line defects such as conductor breakage, insulator contamination, etc., to supplement the lack of real-world data.

[0146] ⑦CutMix / Mixup Strategy: Advanced image mixing techniques such as Mixup (mixing augmentation) strategy and CutMix (cutting and mixing) strategy are adopted to fuse different images and their labels to generate new training samples, thereby improving the generalization ability and robustness of the model.

[0147] ⑧Detailed Data Annotation: Detailed annotation is performed on the images in the training set, including but not limited to:

[0148] Defect Location Labeling: Bounding boxes are used to accurately label the area where the defect is located.

[0149] Defect Category Assignment: Each defect is assigned a predefined category label, such as "conductor foreign object", "insulator defect", "tower tilt", etc.

[0150] Natural Language Description: A preset model or manual annotation is used to provide detailed text descriptions for each defect instance, such as "crack appears on the edge of the second insulator umbrella skirt". These information can be used for multi-modal training to enhance the understanding ability of the model.

[0151] ⑨Data Format Conversion: The processed data is converted into standard formats required for model training and inference, such as specific size image tensors, corresponding label vectors or text description files, to ensure data consistency and compatibility.

[0152] The preprocessing unit and division unit of the embodiment can accurately locate potential problems in the power transmission line by performing bounding box labeling on the defect area in the original image data, providing a foundation for subsequent analysis and detection. Assigning predefined category labels to image data helps to quickly identify and classify different types of defects, improving data processing efficiency. Adding text descriptions to each defect instance using a preset model increases the semantic information of the data, which helps to better understand the defect situation.

[0153] In one embodiment, the detection module 20 includes a freezing unit and a training unit, a detection unit, and a post-processing unit. The freezing unit and the training unit are the process of building a multi-modal large model, the detection unit is the process of obtaining a preliminary detection result according to the multi-modal large model, and the post-processing unit is the process of post-processing the preliminary detection result to obtain a defect detection result.

[0154] The freezing unit is configured to freeze target backbone parameters θ0 in an initial multi-modal model based on a core processing module, and only open newly added low-rank adaptation parameters Δθ (|Δθ| << |θ0|) in the initial multi-modal model to obtain an initialized initial multi-modal model, i.e., an initialized multi-modal model. The initial multi-modal model adopts a Qwen2.5-VL multi-modal large model (or other advanced multi-modal large model). The model integrates a depth-optimized visual encoder and a high-capacity autoregressive language model through a multi-source data collaborative pre-training mechanism. Through a multi-granularity cross-modal alignment mechanism, dynamic mapping of a visual feature space and a semantic concept space can be realized, the geometric structural integrity of visual representation can be maintained, and context-aware generation of the language mode can be realized. The collaborative enhancement of such dual-modal capabilities, combined with a fine-grained attention routing strategy, can construct a strong generalization capability framework for downstream cross-modal reasoning tasks. Moreover, the newly added low-rank adaptation parameters are low-rank matrix parameters introduced additionally on the basis of the initial multi-modal model for task adaptation.

[0155] The training unit is configured to use a PEFT (Parameter Efficient Fine-Tuning) method to perform reverse propagation iterative optimization on the newly added low-rank adaptation parameters in the initialized multi-modal model according to a training data set in the image data in combination with a joint loss, to obtain a multi-modal large model, i.e., a Qwen2.5-VL model that has been efficiently fine-tuned.

[0156] The PEFT method includes but is not limited to LoRA (Low-Rank Adaptation), AdapterTuning, Visual / Text Prompt Tuning, and BitFit. The following specifically describes the way of training the initialized multi-modal model using each method:

[0157] ① LoRA (Low-Rank Adaptation):

[0158] 1) Parameter freezing stage: directly lock the target weight matrix W0 (such as the Q / K / V projection layer or FFN layer in the Transformer) in the initialized multi-modal model, and keep its parameter value unchanged throughout the process to avoid the gradient propagation overhead brought by full fine-tuning.

[0159] 2) Low-rank adapter implantation: two trainable small matrices are inserted in parallel on both sides of W0: matrix A (dimension r x k): as an input feature encoder, mapping the original input x to a low-dimensional space; matrix B (dimension d x r): as an output feature decoder, mapping the low-dimensional representation back to the original dimension; where the rank r is usually 4-64, which is much smaller than the original dimension d / k, such as d = 4096 and r = 8.

[0160] 3) Incremental computation reconstruction: decompose the update of the original weight matrix ΔW into the product of two low-rank matrices: ΔW = B · A.

[0161] During forward propagation, the input feature x will go through two computation paths simultaneously. One is the original path: x directly generates W0x through the frozen W0. The other is the incremental path: x first goes through dimension reduction by A to get Ax, and then goes through dimension increase by B to get B(Ax).

[0162] The final output is the sum of the results of the two paths: h = (W0 + ΔW)x = W0x + B(Ax).

[0163] 4) Parameter update mechanism: during training, only the following operations are performed: keep the gradient mask of W0 Compute the gradient of A and B Update A and B parameters by the chain rule η is the learning rate).

[0164] With this strategy, the number of training parameters is significantly reduced from "d x k" to "r(d + k)". Specifically, when "d = 4096" and "k = 4096", if full fine-tuning is performed, about 16.7M parameters need to be updated. However, if the LoRA method of the embodiment "r = 8" is used, only about "8 x (4096 + 4096) ≈ 65K" parameters need to be adjusted, and the parameter reduction ratio is as high as 99.6%.

[0165] This process essentially approximates the update direction of the high-dimensional parameter space through a low-dimensional manifold, while preserving the pre-training knowledge, and captures the specific patterns of downstream tasks with a small number of trainable parameters, achieving a balance between computational resources and model performance.

[0166] ②Adapter Tuning: By cleverly embedding small and trainable "adapter" modules between the layers of the initialized multi-modal model, the core parameters of the original model can be frozen, and only these adapter modules are trained, thereby achieving efficient fine-tuning.

[0167] ③(Visual)Prompt Tuning: At the input end of the initialized multi-modal model, trainable visual prompts (Visual Prompts) or text prompts (Text Prompts) are introduced. These prompts act as guides, leading the model to focus on the core features of a specific task or domain without touching the complex structure or parameters inside the model.

[0168] ④BitFit: This strategy focuses on the bias parameters of the model. By fine-tuning them, the performance can be optimized without interfering with the main structure of the model.

[0169] The above manner significantly reduces the dependence of the model on large-scale and high-quality labeled data (benefiting from the large amount of knowledge accumulated in pre-training, only a small number of parameters need to be fine-tuned to adapt to a specific task), effectively solving the problem of data scarcity and high labeling cost in the power transmission field. In addition, by significantly reducing the demand for gradient calculation, optimizer state storage and other needs, this method significantly reduces the computing resources (such as GPU memory and computing power) and time cost required for model training, making it more economical and efficient to use advanced large models for domain-specific adaptation.

[0170] The joint loss is calculated according to the target detection loss, the classification loss and the multi-modal contrast loss, and is specifically:

[0171] According to the prediction box coordinates output by the visual encoder in the initial multi-modal model, combined with the real box label, the boundary box regression error is quantified by the Generalized Intersection over Union (GIoU) loss, and the target detection loss is established. Specifically, in the task involving defect positioning, the Focal Loss strategy is adopted to effectively solve the challenge of uneven class distribution, and loss functions such as Generalized Intersection over Union (GIoU), Distance Intersection over Union (DIoU) and Complete Intersection over Union (CIoU) are introduced to focus on fine-tuning the regression accuracy of the boundary box.

[0172] According to the class probability distribution output by the model classification head in the initial multi-modal model, the classification error is quantified by the Cross-Entropy Loss, and the classification loss is obtained.

[0173] According to the image-text feature vectors output by the model cross-modal interaction layer in the initial multi-modal model, the cosine similarity of the image and the text is calculated, and the multi-modal contrast loss is established combined with the preset temperature coefficient. Specifically, if the training data set covers images and their corresponding text descriptions, the contrast learning loss (such as CLIP Loss) is used, which aims to pull closer the image-text pairs that match each other in the feature space, while pushing away the image-text pairs that do not match, thereby significantly enhancing the model's understanding and association ability of cross-modal information.

[0174] According to the importance or sample number of different defect types, the target detection loss, the classification loss and the multi-modal contrast loss are weighted to obtain the joint loss.

[0175] For the application of the embodiments of the present application, please refer to Figure 2 , Figure 2is a high-efficiency fine-tuning process schematic diagram provided by the embodiment of the present application, which shows the process of efficiently fine-tuning the pre-trained Qwen2.5-VL model. It mainly includes selecting a pre-trained model, preparing a domain dataset, selecting and implementing a parameter efficient fine-tuning (PEFT) method, defining a domain-specific loss function, model training and evaluation.

[0176] In the freezing unit and training unit of the present embodiment, since only the newly added low-rank parameters are trained, the required computing resources and storage space are greatly reduced, and the newly added low-rank adaptive parameters enable the model to be fine-tuned for specific tasks without changing the overall structure of the pre-trained backbone network. This flexibility enables the model to better adapt to new datasets and tasks while maintaining the effective use of pre-trained knowledge. Since most of the model parameters are frozen, only the newly added low-rank parameters are optimized, which helps to maintain the stability of the training. During backpropagation, the model is more likely to converge to a better solution because most of the weights have been optimized through pre-training;

[0177] Moreover, the joint loss combines target detection, classification, and multi-modal contrast tasks, enabling the model to master multiple skills simultaneously and enhancing its generalizability and practicality. The GIoU loss can accurately evaluate the gap between the predicted box and the true box, improving the accuracy of target positioning. The classification loss can effectively handle class imbalance using cross-entropy, ensuring more accurate class probability distribution. The multi-modal contrast loss aligns image and text features through cosine similarity and temperature coefficient, enhancing the understanding and association of cross-modal information and improving multi-modal learning effectiveness.

[0178] Compared to traditional image processing techniques and early machine learning models with limited generalization, the present application relies on the powerful general visual understanding and cross-modal correlation capabilities of advanced multi-modal large models such as Qwen2.5-VL. Through parameter efficient fine-tuning (such as LoRA) and domain-specific data augmentation and loss function design, the present application successfully transfers and optimizes this general capability to the transmission line defect identification task, maintaining the model's broad applicability while enhancing its sensitivity to domain-specific features such as defect morphology and complex background differences. This approach overcomes the adaptability challenges of general-purpose models in the power transmission field, enabling accurate identification of diverse and subtle defects in complex backgrounds, significantly improving detection accuracy, enhancing robustness to lighting, angle, and weather changes, and greatly reducing the rate of missed and false detections.

[0179] The detection unit is configured to identify and locate the transmission line defects in the test dataset based on the core processing module and the multi-modal large model, to obtain a preliminary detection result; wherein the test dataset is derived from image data.

[0180] The preliminary detection result includes a plurality of key information, including:

[0181] Defect location: Provides precise bounding box coordinates for each detected defect, clearly identifying the specific location of the defect in the image;

[0182] Defect category: Accurately classifies each defect through a classification label (e.g., "wire strand break");

[0183] Confidence score: Assigns a confidence score to each detection result, reflecting the model's level of certainty about the detection;

[0184] Natural language description: This description is an optional setting. According to the actual needs of users, the model can generate detailed text descriptions about the detected defects, providing a more intuitive interpretation.

[0185] The post-processing unit is used to eliminate repeated detection boxes for the same defect in the preliminary detection results and filter out detection parts with confidence scores below a certain threshold through a non-maximum suppression algorithm (NMS) based on the post-processing and result generation module, obtaining the defect detection results. It is worth noting that in specific scenarios, such as using multi-model integration processing or analyzing video sequences, the module will further perform a result fusion step to integrate the detection data of each model or frame to generate the final defect detection results.

[0186] The post-processing unit is also used to associate and map the detected defects and the geographic location data or tower number of the transmission line in the detection results through multi-dimensional data fusion technology combined with UAV GPS data or line account information. On this basis, the processed defect information is neatly organized into structured inspection reports and defect lists in a clear and orderly manner, including but not limited to tables or lists, and detailedly containing defect pictures, precise locations, type classification, severity assessment (if conditions permit), and discovery time, etc. Key information. It is worth mentioning that this geographic information association and mapping is an optional enhancement function that can be flexibly applied according to actual needs.

[0187] The post-processing unit is also used to display inspection reports and defect lists on the human-computer interaction interface of desktop applications or Web applications based on the user interaction and display module. The specific presentation method is as follows: The detected defect boxes and corresponding labels are displayed on the original image in an intuitive and superimposed manner, and the geographic location of the defect is accurately marked on the map. In addition, users can flexibly query and filter results according to time, line, defect type, and other multiple conditions. Further, with the advanced interaction capabilities of the VLM (Visual Language Model), users can ask questions in natural language, such as "Please enlarge and display the detailed situation of this insulator breakage" or "Are there any other bird nests on this line?" In addition, this embodiment also supports functions such as user-defined detection parameters and user permission management.

[0188] Thus, the embodiment can respond to the operation instruction of the user in real time, quickly retrieve and display the visual defect detection result, detailed report and system state information.

[0189] To apply the embodiment of the present application, please refer to Figures 3-4 ;

[0190] Figure 3 is the multi-modal large model power transmission line defect detection system architecture provided by the embodiment of the present application, which shows the main component modules of the system of the embodiment and their mutual relationship, and the system mainly includes a data acquisition module (101), a data preprocessing module (102), a core processing module (103), a post-processing and result generation module (104), and a user interaction and display module (105); the data flow starts from the acquisition stage, goes through preprocessing refinement, and then is input to the multi-modal large model for in-depth analysis, finally, after the carefully designed post-processing step, the system generates the defect detection result and displays it to the user through the user-friendly interface. And, Figure 3 The data acquisition module (101) to the user interaction and display module (105) shown in the embodiment are completely consistent with the modules used in the actual data processing process of the embodiment;

[0191] The data acquisition module (101) is configured to acquire raw data from the power transmission line site.

[0192] The data preprocessing module (102) is configured to clean, enhance, label and format convert the raw data, and prepare for model training and inference.

[0193] The core processing module (103) is configured to receive the preprocessed image and possible text prompts; use the multi-modal large model to identify and locate the power transmission line defects in the image, and output the preliminary detection result.

[0194] The post-processing and result generation module (104) is configured to optimize the preliminary detection result output by the model, and generate the final report.

[0195] The user interaction and display module (105) is configured to provide a human-computer interaction interface to facilitate the user to upload data, start detection, view results, manage reports, etc.

[0196] Figure 4 is the multi-modal large model power transmission line defect detection method flowchart provided by the embodiment of the present application, which shows the process of the embodiment from data acquisition to defect detection, and then to defect information display, specifically:

[0197] Step S301: Data acquisition and preprocessing. Start the data acquisition module (101) to capture image or video data of the power transmission line. The captured data is passed to the data preprocessing module (102) for cleaning, enhancement (only in the training phase), resizing, and other preprocessing operations to ensure that the image data meets the detection requirements.

[0198] Step S302: Load fine-tuned model. Load the Qwen2.5-VL model parameters that have been efficiently fine-tuned from the core processing module (103) to prepare for the subsequent inference process.

[0199] Step S303: Model inference. Input the preprocessed image data into the loaded fine-tuned model. The model performs forward propagation and uses its learned expertise to conduct in-depth analysis of the image.

[0200] Step S304: Preliminary detection result output. The model outputs the preliminary detection results including potential defect locations (in the form of bounding boxes), class labels, and confidence scores.

[0201] According to requirements, corresponding natural language descriptions can also be output to provide more intuitive interpretation.

[0202] Step S305: Result post-processing. Pass the preliminary detection results to the post-processing and result generation module (104). Perform non-maximum suppression (NMS), confidence filtering, and other operations to further optimize the detection results and improve accuracy.

[0203] Step S306: Information correlation and report generation, which is an optional operation. According to actual requirements, correlate the optimized defect results with geographic location information or line asset information. Generate structured inspection reports for subsequent analysis and processing.

[0204] Step S307: Result display and user interaction. Send the final defect detection results and reports to the user interaction and display module (105). Intuitively display defect information on the user interface and provide query, analysis, and interaction functions to meet user needs.

[0205] The post-processing unit of the present embodiment eliminates duplicate detection boxes for the same defect through the non-maximum suppression algorithm, avoiding multiple overlapping detection boxes in the output results, making the detection results more clear and reliable. By setting a confidence threshold and filtering out detection results below the threshold, low-confidence false detection results can be excluded, thereby improving overall detection performance.

[0206] And, by multi-dimensional data fusion technology, the detection result is closely combined with geographic position data to form a structured inspection report; such integration not only provides detailed information of the defect, but also explicitly indicates the specific position of the defect on the power transmission line, so that the data is more complete and intuitive. In addition, the structured inspection report is easy to understand and analyze, which helps the operation and maintenance personnel quickly grasp the health status of the power transmission line.

[0207] Overall, the embodiment has the following beneficial effects:

[0208] The application accurately identifies and locates defects in the power transmission line in the image data through a multi-modal large model, avoiding the false detection and missed detection problems that may exist in traditional methods. For the multi-modal large model, freezing the target backbone parameters in the initial multi-modal model helps to maintain the stable performance obtained in the previous learning task. These backbone parameters usually contain a large amount of general features and knowledge, which are basic and important for the specific task of power transmission line defect detection. At the same time, by training the newly added low-rank parameters, the model can adaptively adjust to new image data without damaging the original stable structure, and learn more specific defect features, thereby improving the detection accuracy. In addition, low-rank parameter updating has lower computational complexity and fewer parameter quantities than full parameter updating, which helps to reduce the risk of overfitting and improve the generalization ability of the model. Combined with joint loss backpropagation optimization, the loss functions of multiple tasks can be considered comprehensively to improve the overall performance. Moreover, joint loss optimization helps the model to find a balance point between multiple tasks, so that the model can perform higher precision and robustness in the power transmission line defect detection task.

[0209] In summary, the application uses a single fine-tuning multi-modal large model to realize efficient and unified detection of various defects in the power transmission line, simplifies system design, reduces maintenance complexity, and improves detection efficiency. In addition, by using the cross-modal ability of the VLM, the application can output natural language descriptions and support human-computer interactive question and answer, providing more intelligent and detailed defect analysis and decision support tools for operation and maintenance personnel.

[0210] Embodiment three:

[0211] The embodiment of the application provides a computer readable storage medium, which comprises a stored computer program, wherein the computer readable storage medium controls the device where the computer readable storage medium is located to execute the power transmission line defect detection method when the computer program runs.

[0212] The power transmission line defect detection method can be stored in a computer readable storage medium if it is implemented in the form of a software function unit and used as an independent product. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware. The computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method embodiments can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, an executable file, or some intermediate form. The computer readable medium can include any entity or device capable of carrying the computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0213] The above is the preferred embodiment of the present application. It should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which are also considered within the scope of protection of the present application.

Claims

1. A method for detecting defects in power transmission lines, characterized in that, include: Acquire image data of power transmission lines; The multimodal large model is used to identify and locate transmission line defects in the image data to obtain defect detection results. The multimodal large model is obtained by freezing the target backbone parameters in the initial multimodal model, training the newly added low-rank parameters in the initial multimodal model according to preset data, and performing backpropagation optimization in combination with joint loss.

2. The method for detecting defects in transmission lines as described in claim 1, characterized in that, To acquire image data of transmission lines, specifically: Obtain the original image data of the transmission line; In the original image data, the defect areas are labeled with bounding boxes, predefined category labels are assigned to the image data, and text descriptions are added to each defect instance through a preset model to obtain comprehensive data; The comprehensive data is divided into image data that includes a training dataset and a test dataset.

3. The method for detecting defects in transmission lines as described in claim 2, characterized in that, The large multimodal model is obtained by freezing the target backbone parameters in the initial multimodal model, training the newly added low-rank parameters in the initial multimodal model based on preset data, and then performing backpropagation optimization using joint loss. Specifically: Freeze the target backbone parameters in the initial multimodal model, and only open the newly added low-rank adaptation parameters in the initial multimodal model to obtain the initial multimodal model; Based on the training dataset, the newly added low-rank adaptation parameters in the initial multimodal model are backpropagated and iteratively optimized using the joint loss to obtain the large multimodal model.

4. The method for detecting defects in transmission lines as described in claim 1, characterized in that, The joint loss is calculated based on the target detection loss, classification loss, and multimodal contrast loss, specifically as follows: Based on the predicted bounding box coordinates output by the visual encoder in the initial multimodal model, combined with the ground truth bounding box annotations, the bounding box regression error is quantified by GIoU loss to establish the target detection loss; Based on the class probability distribution output by the model classification head in the initial multimodal model, the classification error is quantified by cross-entropy loss to obtain the classification loss; Based on the image and text feature vectors output by the cross-modal interaction layer of the initial multimodal model, the cosine similarity between the image and the text is calculated, and a multimodal contrast loss is established in combination with a preset temperature coefficient. The joint loss is obtained by weighting the target detection loss, the classification loss, and the multimodal contrast loss.

5. The method for detecting defects in transmission lines as described in claim 1, characterized in that, Based on the multimodal large model, the defects in the transmission lines in the image data are identified and located to obtain the defect detection results, specifically: Based on the multimodal large model, the defects in the transmission lines in the image data are identified and located to obtain preliminary detection results; The nonmaximum suppression algorithm is used to eliminate duplicate detection boxes for the same defect in the preliminary detection results, and the defect detection results with a confidence level higher than a set threshold are selected.

6. A method for detecting defects in transmission lines as described in any one of claims 1-5, characterized in that, After identifying and locating transmission line defects in the image data based on a multimodal large model to obtain defect detection results, the method further includes: By using multi-dimensional data fusion technology, the defect detection results and the geographical location data of the transmission line are correlated and mapped to obtain a structured inspection report; The inspection report is displayed on the human-computer interaction interface.

7. A transmission line defect detection device, characterized in that, Includes a data module and a detection module; The data module is used to acquire image data of the transmission line; The detection module is used to identify and locate transmission line defects in the image data based on a multimodal large model, and obtain defect detection results. The multimodal large model is obtained by freezing the target backbone parameters in the initial multimodal model, training the newly added low-rank parameters in the initial multimodal model based on preset data, and performing backpropagation optimization in combination with joint loss.

8. The transmission line defect detection device as described in claim 7, characterized in that, The data module includes an image unit, a preprocessing unit, and a segmentation unit; The imaging unit is used to acquire the original image data of the transmission line. The preprocessing unit is used to annotate the defect areas with bounding boxes in the original image data, assign predefined category labels to the image data, and add text descriptions to each defect instance through a preset model to obtain comprehensive data. The partitioning unit is used to divide the comprehensive data into image data including a training dataset and a test dataset.

9. A transmission line defect detection device as described in claim 8, characterized in that, The detection module includes a freezing unit and a training unit; The freezing unit is used to freeze the target backbone parameters in the initial multimodal model and only open the newly added low-rank adaptation parameters in the initial multimodal model to obtain the initial multimodal model. The training unit is used to perform backpropagation iterative optimization on the newly added low-rank adaptation parameters in the initial multimodal model based on the training dataset and the joint loss, so as to obtain the large multimodal model.

10. A storage medium, characterized in that, The storage medium stores a computer program, which is called and executed by a computer to implement a method for detecting defects in transmission lines as described in any one of claims 1 to 6.

Images