Insulator defect detection method based on machine vision and large language model

Abstract

The invention discloses an insulator defect detection method based on machine vision and a large language model, and the method comprises the steps: S1, collecting a power grid insulator image, and carrying out the preprocessing of the image; s2, performing data enhancement on the preprocessed power grid insulator image, marking insulator defects, and generating a training set and a test set; s3, inputting the training set and the test set into an improved YOLOv4 model, and training the improved YOLOv4 model to obtain a trained improved YOLOv4 model; and S4, insulator defect detection is carried out through the trained improved YOLOv4 model in combination with a large language model. According to the method, the YOLOv4 model is improved, a large language model is introduced for fault prediction, and excellent performance is shown in insulator defect detection. Compared with the prior art, the method has the remarkable advantages in the aspects of detection precision, robustness, real-time performance, cost control and the like, and the efficiency and reliability of insulator detection in an electric power system are remarkably improved.

Description

Technical Field

[0001] The invention belongs to the field of insulator detection of power systems, and in particular relates to an insulator defect detection method based on machine vision and a large language model. Background Art

[0002] The power system is an important infrastructure for the development of modern social economy, and it is crucial to ensure its safe and stable operation. As a key component in the power system, insulators play the role of electrical insulation between isolating conductors and towers in high-voltage transmission lines, and their working status is directly related to the safe operation of the power system. However, insulators will suffer from surface contamination, aging, cracks and other defects due to environmental factors (such as wind, rain, salt spray, pollution, etc.) and load changes in the power system during long-term use. If these defects are not discovered and repaired in time, they may cause insulator failure, leading to power line failures and even large-scale power outages. Therefore, early detection and preventive maintenance of insulator defects are of great significance to the safety and stability of the power system.

[0003] At present, the technology of insulator defect detection has gradually developed from traditional manual inspection to automation and intelligence, but the existing detection methods still have many shortcomings. First, the manual detection method is inefficient, subjective, and costly, and manual inspectors face huge safety risks when working at high voltage and high altitude. In addition, since the detection relies on manual naked eye observation, it is difficult to detect defects such as fine cracks and pollution on the surface of the insulator. Secondly, although drone inspection can improve the detection efficiency to a certain extent, drones rely on optical cameras and can only identify surface defects, and the accuracy of the detection results is greatly affected when the light conditions are poor. Infrared thermal imaging detection is mainly used to detect local overheating or discharge phenomena of insulators, but its detection effect is limited for surface defects (such as cracks and dirt). In addition, although partial discharge detection is effective in detecting electrical problems inside insulators, its equipment is complex and difficult to be widely used in a large range of transmission lines.

[0004] With the rapid development of artificial intelligence technology, machine vision technology has gradually been widely used in the field of industrial inspection. Through image acquisition and processing technology, machine vision can acquire and analyze the surface image of the insulator in real time, thereby realizing the automatic identification of defects. In particular, the introduction of deep learning algorithms, such as the YOLO (You Only Look Once) series of algorithms, has greatly improved the efficiency and accuracy of target detection. YOLOV4 can achieve efficient target detection in complex environments, and its real-time and detection accuracy advantages are significant. However, the robustness of the traditional YOLOV4 model under complex backgrounds and changing lighting conditions still needs to be improved, especially when dealing with subtle defects such as tiny cracks and contamination on the surface of insulators, its detection accuracy still needs to be improved.

[0005] On the other hand, the health status of insulators is not only related to the current surface defects, but also closely related to their historical operation data, maintenance records and environmental factors in the area. Traditional detection methods often only focus on surface defects and ignore the potential value of these data. Large Language Model (LLM), as a breakthrough technology in the field of natural language processing in recent years, has the ability to process large-scale unstructured data, text analysis and prediction. By introducing a large language model, it is possible to intelligently analyze the historical maintenance records, operation and maintenance data and environmental factors of insulators, and combine the results of machine vision inspection to provide more comprehensive insulator fault prediction and maintenance recommendations. This intelligent analysis method can help the operation and maintenance personnel of the power system better understand the health status of insulators, warn of potential problems in advance, thereby achieving preventive maintenance and reducing the possibility of failures.

[0006] Based on this background, the present invention proposes an insulator defect detection technology based on improved YOLOV4 and a large language model. By improving the YOLOV4 model, it can better adapt to the detection needs of insulators, especially when dealing with complex backgrounds, light changes and minor defect identification, to improve the accuracy and stability of detection. At the same time, the historical data and environmental factors of the insulator are intelligently analyzed in combination with the large language model to form a comprehensive detection solution of multimodal data fusion. This solution can not only detect the surface defects of the insulator in real time, but also analyze the historical data of the insulator through a large language model, predict possible future failures, and provide intelligent decision support for operation and maintenance personnel. Summary of the invention

[0007] In view of the above-mentioned deficiencies in the prior art, the insulator defect detection method based on machine vision and a large language model provided by the present invention solves the problems of low efficiency and insufficient detection accuracy existing in the prior insulator detection technology.

[0008] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: an insulator defect detection method based on machine vision and large language model, comprising the following steps:

[0009] S1, collecting grid insulator images and preprocessing them;

[0010] S2. Perform data enhancement on the preprocessed grid insulator images, mark insulator defects, and generate training sets and test sets;

[0011] S3, inputting the training set and the test set into the improved YOLOv4 model, training the improved YOLOv4 model, and obtaining a trained improved YOLOv4 model;

[0012] S4. Insulator defect detection is performed by combining the trained improved YOLOv4 model with a large language model.

[0013] Further: In said S1, the method for preprocessing the grid insulator image is specifically as follows:

[0014] Perform image rotation, flipping and brightness enhancement operations on the power grid insulator image.

[0015] Further: In S2, the method for performing data enhancement on the preprocessed grid insulator image is specifically:

[0016] Mosaic data enhancement is performed on the preprocessed power grid insulator image to increase the diversity of the power grid insulator image data.

[0017] The beneficial effect of the above further scheme is: the main purpose of the present invention to perform Mosaic data enhancement is to further improve the performance of the model by increasing the diversity of data, enhancing the robustness of the model and improving the batch normalization layer.

[0018] Further: in said S3, the improved YOLOv4 model includes an input layer, a backbone network, a neck network and an output layer connected in sequence, and the neck network includes an SPP module, an FPNet module and a PANet module;

[0019] Wherein, the input layer is used to receive the grid insulator image and pre-process the grid insulator image;

[0020] The backbone network is used to extract feature information from the input grid insulator image;

[0021] The neck network is used to extract features from the output feature map of the backbone network. The SPP module improves the receptive field of the model through pooling operations of different scales, thereby improving the ability to capture details and targets. The FPNet module and the PANet module enhance the transmission and sharing of features through feature fusion technology, making the insulator defect detection results output by the output layer more accurate.

[0022] The output layer is used to output the insulator defect detection results, including the target category, location and confidence information.

[0023] Further: In S3, the backbone network construction method of the improved YOLOv4 model is:

[0024] The constructed MobileNetV2 network is used to replace the CSPDarkNet53 backbone network in the YOLOv4 model. The MobileNetV2 network includes the first to fifth CSP modules connected in sequence.

[0025] The beneficial effect of the above further scheme is: the lightweight MobileNetV2 network is used to replace the CSPDarkNet53 backbone network in the YOLOv4 model, realizing the lightweight design of the model. MobileNetV2 effectively reduces the computational complexity, the number of model parameters and the amount of computation through the deep separable convolution technology, while maintaining good feature extraction capabilities for subtle defects in insulator images. This improvement enables the model to meet the real-time detection requirements of insulator defects in a resource-constrained environment, especially when dealing with small defects such as cracks and contamination, it can maintain a high detection accuracy, thereby ensuring the reliability and efficiency of insulator monitoring in the power system.

[0026] Further: in said S3, the SPP module of the improved YOLOv4 model includes a CBL structure, and a parallel first maximum pooling layer, a second maximum pooling layer and an average pooling layer;

[0027] Among them, the input of the CBL structure is used as the input of the SPP module, and the output of the CBL structure is connected to the input of the first maximum pooling layer, the input of the second maximum pooling layer and the input of the average pooling layer respectively. The output features of the CBL structure, the output of the first maximum pooling layer, the output of the second maximum pooling layer and the output of the average pooling layer are fused as the output of the SPP module.

[0028] The beneficial effect of the above further scheme is: the present invention optimizes the SPP module based on the structure of YOLOv4. In the 9×9 pooling operation, average pooling is used instead of 9×9 maximum pooling to retain more detailed information related to insulator defects. Average pooling can better extract the key features of the target area and avoid the interference of too much background information, thereby improving the accuracy of insulator defect detection, especially when dealing with small targets and complex backgrounds. Through this optimization, the improved SPP module is more suitable for insulator defect detection tasks in complex backgrounds, effectively reducing the false detection or missed detection problems that may be caused when the background information is similar to the defect characteristics.

[0029] Further: in said S3, the structures of the first to fifth CSP modules are the same, and all include a first point-by-point convolution, a depth-wise separable convolution, a second point-by-point convolution and a CBAM submodule;

[0030] Among them, the first point-by-point convolution, the depth-wise separable convolution, the second point-by-point convolution and the CBAM sub-module are connected in sequence, the output of the CBAM sub-module is connected to the output of the second point-by-point convolution and the input of the element multiplication, and the output of the element multiplication is connected to the input of the first point-by-point convolution and the input of the element addition.

[0031] The beneficial effect of the above further scheme is: the present invention introduces a CBAM submodule into the network and applies it to the first to fifth CSP modules of MobileNetV2, so that the network can express the characteristics of insulator defects more accurately. The CBAM submodule can dynamically adjust the attention distribution of the network through the combination of spatial attention and channel attention, ensuring that the network can focus on the insulator defect area when processing complex backgrounds and filter out irrelevant background information.

[0032] Furthermore: the CBAM submodule dynamically adjusts the attention distribution of the MobileNetV2 network by combining an improved spatial attention mechanism and a channel attention mechanism;

[0033] Among them, the improved channel attention mechanism M C The specific expression of (F) is:

[0034]

[0035] Where σ(·) is the Sigmoid function, F avg is the one-dimensional tensor after the average pooling operation, F max is the one-dimensional tensor after the maximum pooling operation, is a one-dimensional convolution with a convolution kernel of k;

[0036] Improved spatial attention mechanism M S The specific expression of (F) is:

[0037] M S (F) = σ(f(f concat (F avg ; F max )))

[0038] Where, f(·) is the convolution calculation; f concat (·) is the splicing in dimension.

[0039] The beneficial effect of the above further scheme is: the present invention replaces the traditional fully connected network with the CBAM sub-module to achieve the improvement of the correlation of the local area of ​​the network. The CBAM sub-module can dynamically adjust the attention of the network to ensure that the information of the defective area of ​​the insulator is fully utilized, thereby improving the accuracy and robustness of the detection.

[0040] Further: S4 includes the following sub-steps:

[0041] S41, inputting the grid insulator image to be detected into the trained improved YOLOv4 model to obtain a defect report of the insulator defect detection, including the type, location and severity of the defect;

[0042] S42. Analyze defect reports through a large language model combined with historical data to generate a health report, including prediction information and maintenance recommendations for future insulator failures.

[0043] The beneficial effects of the present invention are as follows: the present invention provides an insulator defect detection method based on machine vision and a large language model. Compared with the prior art, the present invention is based on the improved YOLOv4 model and the large language model applied to insulator defect detection, which has significant advantages and beneficial effects, which are specifically manifested in the following aspects:

[0044] (1) Significant improvement in detection accuracy:

[0045] Most of the existing insulator defect detection methods rely on traditional image processing technology or are based on simple deep learning algorithms. These methods are prone to false detection or missed detection when dealing with insulator detection in complex backgrounds, especially when facing small targets (such as fine cracks or dirt). The detection effect is poor. The present invention improves the YOLOv4 model, introduces a lightweight MobileNetV2 backbone network, and optimizes the SPP module and feature extraction mechanism. Experimental data show that the detection accuracy of the present invention is improved by about 10% compared with the traditional YOLOv4 model, especially in the task of small target detection, the performance improvement is more obvious. This makes the detection of insulator defects more accurate, especially when dealing with complex environments and multiple targets, and can effectively reduce the probability of false detection and missed detection.

[0046] (2) Improved the robustness and generalization ability of the model:

[0047] By introducing Mosaic data enhancement technology, the present invention significantly improves the robustness and generalization ability of the model. Experiments show that after Mosaic data enhancement, the model can still maintain a high detection accuracy when facing insulator images with different angles, scales and lighting conditions. Compared with the unenhanced data set, the performance of the model of the present invention on the test set is improved by about 15%, which effectively solves the shortcomings of traditional methods in dealing with images with different perspectives and complex backgrounds. In addition, the addition of CBAM (convolutional block attention mechanism) further enhances the network's ability to capture key features, making it more stable when dealing with complex environmental interference.

[0048] (3) Intelligent fault prediction and maintenance suggestions:

[0049] Traditional insulator detection methods usually rely only on current detection data and cannot fully utilize historical data or environmental information for long-term equipment health status assessment and fault prediction. By introducing the Large Language Model (LLM), the present invention can effectively integrate historical maintenance records, operating status data, and environmental monitoring data to provide personalized fault prediction and maintenance recommendations. Compared with existing single detection methods, the present invention can detect potential problems earlier, provide accurate fault warnings, enable operation and maintenance personnel to take action in advance, avoid serious failures, and significantly reduce operation and maintenance costs.

[0050] (4) Reduce equipment maintenance costs:

[0051] Through fault prediction and personalized maintenance suggestions, the present invention reduces unnecessary maintenance and equipment replacement, thereby reducing overall operating and maintenance costs. Traditional insulator detection technology usually requires regular manual inspections, which is not only time-consuming and labor-intensive, but may also lead to premature replacement of equipment. The present invention, through a large language model and an intelligent early warning system, can provide scientific maintenance cycle suggestions based on the operating conditions of the insulators, avoiding cost waste caused by excessive maintenance and human misjudgment. According to the test results of actual applications, the maintenance cost of insulators will be reduced after adopting the detection system of the present invention.

[0052] (5) Real-time and lightweight design:

[0053] The present invention significantly reduces the computational complexity of the model by introducing a lightweight MobileNetV2 to replace the original CSPDarkNet53 backbone network. Compared with the traditional YOLOv4 model, the network of the present invention can achieve faster reasoning and detection in an environment with limited computing resources, and is suitable for embedded devices and real-time detection needs. Experiments show that under the premise of ensuring detection accuracy, the reasoning speed of the model is increased by 40%, which greatly improves the real-time detection capability of the system and meets the real-time requirements for online detection of insulators.

[0054] (6) Multimodal data fusion and intelligent analysis:

[0055] Traditional insulator inspection is usually limited to the analysis of image data, while ignoring the value of other related data. This invention combines the large language model (LLM) to perform fusion analysis of multimodal data, including insulator historical data, sensor monitoring data, and operating environment information. Through this innovation, the system can not only detect current defects, but also comprehensively evaluate the overall health status of the insulator, providing more valuable information support for decision-making.

[0056] In summary, the present invention has shown superior performance in insulator defect detection by improving the YOLOv4 model and introducing a large language model for intelligent data analysis and fault prediction. Compared with the prior art, the present invention has significant advantages in detection accuracy, robustness, real-time performance and cost control, and significantly improves the efficiency and reliability of insulator detection in power systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 The flowchart of the insulator defect detection method based on machine vision and large language model.

[0058] Figure 2 Schematic diagram of the YOLOv4 model structure.

[0059] Figure 3 Schematic diagram of the improved SPP module structure.

[0060] Figure 4 Schematic diagram of the improved CBAM submodule structure.

[0061] Figure 5 Schematic diagram of insulator defect detection results.

[0062] Figure 6 To improve the loss function of the YOLOv4 model during training.

[0063] Figure 7 This is a schematic diagram of the inspection report for insulator damage.

[0064] Figure 8 This is a schematic diagram of the inspection report for insulator wear.

[0065] Fig. 9 This is a schematic diagram of the inspection report for a normal insulator. DETAILED DESCRIPTION

[0066] The specific implementation modes of the present invention are described below to facilitate those skilled in the art to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0067] like Figure 1 As shown, in one embodiment of the present invention, the insulator defect detection method based on machine vision and large language model includes the following steps:

[0068] S1, collecting grid insulator images and preprocessing them;

[0069] S2. Perform data enhancement on the preprocessed grid insulator images, mark insulator defects, and generate training sets and test sets;

[0070] S3, inputting the training set and the test set into the improved YOLOv4 model, training the improved YOLOv4 model, and obtaining a trained improved YOLOv4 model;

[0071] S4. Insulator defect detection is performed by combining the trained improved YOLOv4 model with a large language model.

[0072] The technical problem to be solved by the present invention is the shortcomings of existing insulator defect detection technology in terms of efficiency, accuracy and intelligence. The current detection methods have problems such as low detection efficiency, strong subjectivity, limited detection accuracy, lack of multi-dimensional data analysis, and lack of real-time and intelligence. Specifically, manual inspections rely on the experience and naked eye observation of inspectors, which are highly subjective and inefficient; although drone inspections and infrared thermal imaging can be automated, they can only identify surface defects or temperature anomalies, and it is difficult to handle complex environments and defects such as tiny cracks. In addition, existing detection technologies can usually only make judgments based on current data, lack the ability to analyze historical data and environmental factors, and it is difficult to achieve long-term health assessment and fault prediction of insulators.

[0073] In order to solve the above problems, the present invention improves the accuracy and robustness of insulator defect detection by improving YOLOv4. The improved YOLOv4 model is optimized in terms of feature extraction, feature fusion and target detection, and can better adapt to complex light conditions and background changes, especially the recognition ability of defects such as tiny cracks, contamination and aging on the surface of insulators is greatly improved. Through multi-scale feature extraction and data enhancement technology, the improved YOLOv4 can maintain efficient and accurate detection performance in real-time processing. The model can process a large number of high-resolution images, and can still effectively detect insulator defects in various harsh environments (such as strong light, shadow occlusion), thereby realizing automated and real-time insulator detection.

[0074] In addition, the present invention introduces a large language model (LLM) to analyze the historical operation and maintenance data, maintenance records and environmental conditions (such as humidity, temperature, pollution level, etc.) of the insulator, and combines the real-time detection results to provide intelligent fault prediction of multimodal data fusion. Through the analysis of the large language model, the system can not only judge the current state of the insulator based on the current image detection results, but also predict the potential failure risk of the insulator in the future based on historical data and environmental factors. This intelligent data analysis and prediction method can help the operation and maintenance personnel of the power system take preventive maintenance measures in advance and reduce the risk of accidents caused by insulator failures.

[0075] In summary, the present invention significantly improves the automation, accuracy and intelligence level of insulator defect detection by combining improved YOLOv4 and large language model technology, and solves the problems of low efficiency, limited detection range and lack of intelligent analysis in existing detection methods. Through multimodal data fusion, the present invention can realize real-time detection and fault prediction of insulators, and provide support for intelligent operation and maintenance of power systems, which has broad application prospects and significant technical advantages.

[0076] In S1, the method for preprocessing the grid insulator image is specifically as follows:

[0077] Perform image rotation, flipping and brightness enhancement operations on the power grid insulator image.

[0078] In this embodiment, the present invention uses Python Imaging Library (PIL) to preprocess the image, and the preprocessing includes operations such as image rotation, flipping, brightness enhancement, etc. These preprocessing steps are intended to improve the quality of the image so that it is more suitable for the needs of the deep learning algorithm when it is input into the model.

[0079] In S2, the method for performing data enhancement on the preprocessed grid insulator image is specifically as follows:

[0080] Mosaic data enhancement is performed on the preprocessed power grid insulator image to increase the diversity of the power grid insulator image data.

[0081] In this embodiment, the main purpose of the present invention to perform Mosaic data enhancement is to further improve the performance of the model by increasing the diversity of data, enhancing the robustness of the model, and improving the batch normalization layer.

[0082] like Figure 2 As shown, in S3, the improved YOLOv4 model includes an input layer, a backbone network, a neck network and an output layer connected in sequence, and the neck network includes an SPP module (spatial pyramid pooling module), an FPNet module (feature pyramid network module) and a PANet module (path aggregation network module);

[0083] Wherein, the input layer is used to receive the grid insulator image and pre-process the grid insulator image;

[0084] The backbone network is used to extract feature information from the input grid insulator image;

[0085] In this embodiment, optimization of the backbone network is crucial to improving detection efficiency and accuracy.

[0086] The neck network is used to extract features from the output feature map of the backbone network. It enhances the multi-scale detection capability of the network through pooling and feature fusion, especially in feature expression at different resolutions and small target detection.

[0087] Among them, the SPP module improves the receptive field of the model through pooling operations of different scales, thereby improving the ability to capture details and targets. The FPNet module and the PANet module enhance the transmission and sharing of features through feature fusion technology, making the insulator defect detection results output by the output layer more accurate;

[0088] The output layer is used to output the insulator defect detection results, including the target category, location and confidence information.

[0089] In S3, the backbone network construction method of the improved YOLOv4 model is:

[0090] The constructed MobileNetV2 network is used to replace the CSPDarkNet53 backbone network in the YOLOv4 model. The MobileNetV2 network includes the first to fifth CSP modules connected in sequence.

[0091] In actual insulator defect detection applications, due to limited computing resources, especially in scenarios with low computing power such as embedded devices or drones, the model is required to have a lightweight design to ensure efficient operation under limited hardware conditions.

[0092] Based on this, this embodiment uses a lightweight MobileNetV2 network to replace the CSPDarkNet53 backbone network in the YOLOv4 model, realizing the lightweight design of the model. MobileNetV2 effectively reduces the computational complexity and the number of model parameters and computation through deep separable convolution technology, while maintaining good feature extraction capabilities for subtle defects in insulator images. This improvement enables the model to meet the real-time detection requirements of insulator defects in a resource-constrained environment, especially when dealing with small defects such as cracks and contamination, it can maintain a high detection accuracy, thereby ensuring the reliability and efficiency of insulator monitoring in the power system.

[0093] The MobileNetV2 network effectively reduces the computational cost of the model by filtering the input channels using filters, then performing channel-by-channel convolution, and then combining all outputs with a 1×1 convolution kernel. Specifically, for a dimension of D F ×D F ×M input data, the size of the convolution kernel is D K ×D K, the number of output channels is N. Compared with conventional convolution, the computational cost of the depthwise separable convolution of the MobileNetV2 network is significantly reduced. The ratio of its computational cost to the computational cost of conventional convolution is shown in the following formula:

[0094]

[0095] Where C1 and C2 represent the computational cost of depthwise separable convolution and conventional convolution, respectively, and D F , D K and M are both input dimensions. It can be seen that the depthwise separable convolution divides the traditional convolution operation into two simple steps: channel-by-channel convolution and point-by-point convolution, which significantly reduces the computational complexity.

[0096] In addition, the MobileNetV2 network of the present invention includes first to fifth CSP modules connected in sequence. This lightweight design improves the computing efficiency, so that the model has higher speed and less resource occupation when processing complex tasks. At the same time, the introduction of the MobileNetV2 network not only brings significant advantages in computing overhead, but also ensures the network's ability to extract detailed features when performing insulator defect detection, especially the accuracy of small target detection is significantly improved. Based on this, the MobileNetV2 network can run efficiently in various embedded devices or other resource-constrained environments, ensuring the real-time and reliability of the model in insulator defect detection. While maintaining a high detection accuracy, this optimized design effectively reduces the operating cost of the model, making it more suitable for actual industrial detection scenarios.

[0097] like Figure 3 As shown, in S3, the SPP module of the improved YOLOv4 model includes a CBL structure, and a parallel first maximum pooling layer, a second maximum pooling layer and an average pooling layer;

[0098] Among them, the input of the CBL structure is used as the input of the SPP module, and the output of the CBL structure is connected to the input of the first maximum pooling layer, the input of the second maximum pooling layer and the input of the average pooling layer respectively. The output features of the CBL structure, the output of the first maximum pooling layer, the output of the second maximum pooling layer and the output of the average pooling layer are fused as the output of the SPP module.

[0099] In this embodiment, in the YOLOv4 network framework, the SPP module includes four maximum pooling operations of different scales, namely 1×1, 5×5, 9×9 and 13×13. The main function of maximum pooling is to reduce the amount of calculation and increase the training speed of the network by reducing the feature information. In addition, the pooling operation can effectively reduce the feature dimension and help prevent the network from overfitting. However, in insulator defect detection, if the background information is complex and the background is similar to the insulator defect characteristics, the use of maximum pooling may result in the retention of unnecessary background information while losing the key details of the insulator defect, thereby causing missed detection or false detection problems.

[0100] To address this problem, the present invention optimizes the SPP module based on the structure of YOLOv4. In the 9×9 pooling operation, average pooling is used instead of 9×9 maximum pooling to retain more detailed information related to insulator defects. Average pooling can better extract the key features of the target area and avoid the interference of too much background information, thereby improving the accuracy of insulator defect detection, especially when dealing with small targets and complex backgrounds. Through this optimization, the improved SPP module is more suitable for insulator defect detection tasks in complex backgrounds, and effectively reduces the problem of false detection or missed detection that may be caused when the background information is similar to the defect characteristics. The improved SPP structure is as follows: Figure 3 shown.

[0101] like Figure 4 As shown, in S3, the structures of the first to fifth CSP modules are the same, and all include a first point-by-point convolution, a depth-wise separable convolution, a second point-by-point convolution and a CBAM submodule;

[0102] Among them, the first point-by-point convolution, the depth-wise separable convolution, the second point-by-point convolution and the CBAM sub-module are connected in sequence, the output of the CBAM sub-module is connected to the output of the second point-by-point convolution and the input of the element multiplication, and the output of the element multiplication is connected to the input of the first point-by-point convolution and the input of the element addition.

[0103] When performing insulator defect detection, the background information of power transmission towers is complex, especially some background parts have similar features to the insulator defect area, which makes it easy for the network to misjudge the background information as a defect. Therefore, during the detection process, more attention needs to be paid to the insulator defect area and less attention needs to be paid to irrelevant background information. However, simply adjusting the network structure cannot completely solve this problem, so it is particularly important to introduce an attention mechanism to improve the feature expression ability.

[0104] Based on this, the present invention introduces a CBAM (convolutional block attention mechanism) submodule into the network and applies it to the first to fifth CSP modules of MobileNetV2, so that the network can express the characteristics of insulator defects more accurately. The CBAM submodule can dynamically adjust the attention distribution of the network through the combination of spatial attention and channel attention, ensuring that the network can focus on the insulator defect area when processing complex backgrounds and filter out irrelevant background information.

[0105] The CBAM submodule dynamically adjusts the attention distribution of the MobileNetV2 network by combining an improved spatial attention mechanism and a channel attention mechanism.

[0106] In this embodiment, considering the complexity of the power insulator image, the network needs to effectively handle the mutual interference between the background and the insulator features. Therefore, the present invention uses the CBAM submodule to replace the traditional fully connected network to improve the correlation of the local area of ​​the network. C The specific expression of (F) is:

[0107]

[0108] Where σ(·) is the Sigmoid function, F avg is the one-dimensional tensor after the average pooling operation, F max is the one-dimensional tensor after the maximum pooling operation, is a one-dimensional convolution with a convolution kernel of k;

[0109] In addition, the improved channel attention mechanism also uses the logarithmic operation of the number of channels to improve the feature expression of specific defect areas:

[0110]

[0111] In the formula, C is the odd channel number, which ensures that the insulator defects can be effectively extracted and focused at different resolutions, |·| odd represents an odd value, and k is the number of convolution kernels. In this way, the CBAM submodule can dynamically adjust the network's attention to ensure that the information of the insulator defect area is fully utilized, thereby improving the accuracy and robustness of detection.

[0112] Improved spatial attention mechanism M S The specific expression of (F) is:

[0113] M S (F) = σ(f(f concat (F avg ; F max )))

[0114] Where, f(·) is the convolution calculation; f concat (·) is the splicing in dimension.

[0115] In view of the structural characteristics of the MobileNetV2 network, the present invention introduces a CBAM submodule after the second point-by-point (PW) convolution operation of the first to fifth CSP modules, such as Figure 5 As shown. By adding the CBAM submodule after the second PW convolution, the network can effectively enhance the selective attention to features, especially highlighting the key features of the insulator defect area in a complex background. The role of the inverse residual module is to reduce the amount of calculation through efficient convolution operations while maintaining good feature extraction capabilities. However, when processing insulator images with complex backgrounds and unclear defect features, relying solely on the original convolution operation may not be able to fully extract key defect information. After introducing the CBAM submodule, the module first uses the channel attention mechanism (Channel Attention Module) to assign different weights to different channels according to the importance in the feature map, so that the network can more centrally process the relevant features of the insulator defects. Next, the module further enhances the network's ability to express features in the spatial dimension through the spatial attention mechanism (Spatial Attention Module) to ensure accurate positioning of insulator defects.

[0116] The S4 comprises the following sub-steps:

[0117] S41, inputting the grid insulator image to be detected into the trained improved YOLOv4 model to obtain a defect report of the insulator defect detection, including the type, location and severity of the defect;

[0118] S42. Analyze defect reports through a large language model combined with historical data to generate a health report, including prediction information and maintenance recommendations for future insulator failures.

[0119] The historical data includes historical operation and maintenance data, maintenance records and environmental conditions of the insulators.

[0120] In this embodiment, the large language model can integrate multiple data sources, including historical operating data, maintenance records, environmental information, and sensor detection data of insulators, to perform multi-dimensional data analysis. This data fusion not only enables the system to comprehensively evaluate the health status of insulators, but also makes more accurate fault predictions and maintenance decisions based on multi-dimensional data. In addition, the large language model uses data-driven intelligent fault prediction technology to automatically extract patterns from historical data, identify potential failure modes, and thus predict possible future failures. Compared with traditional experience- and rule-based prediction methods, this data-based analysis method greatly improves the accuracy of predictions and can help operation and maintenance personnel take maintenance measures in advance to avoid equipment failures.

[0121] Another innovation of the big language model is that it can provide personalized maintenance suggestions. According to the different environmental conditions of each insulator, the model can provide the best customized maintenance plan for each insulator. For example, considering factors such as the climate and pollution conditions in the area where the insulator is located, the model can make more accurate maintenance suggestions for equipment in a specific environment, thereby effectively optimizing resource allocation. This personalized maintenance can significantly reduce unnecessary repairs or replacements and improve economic benefits.

[0122] At the same time, the large language model of the present invention has an intelligent fault warning function. The model can not only predict future faults, but also identify abnormal conditions in current operation based on the comparison of real-time data and historical data, and issue warnings in advance. This real-time warning system can effectively reduce the impact of faults on the power system and improve the safety and stability of the system. Another important innovation is the adaptive learning ability of the large language model. As new data is continuously input, the model can automatically optimize and update the prediction algorithm, thereby maintaining efficient fault prediction and maintenance suggestions under different environmental and equipment conditions. This adaptive ability ensures that the model can continue to play a role in the long-term operation process, avoiding the decline in prediction accuracy caused by equipment aging or environmental changes. In addition, the large language model also has innovative advantages in processing unstructured data (such as maintenance records and inspection reports). The model can automatically extract important information related to insulator health from these unstructured texts, make up for the shortcomings of traditional structured data, and further improve the comprehensiveness and accuracy of fault prediction. The long-term and short-term memory capabilities of the large language model also further enhance the flexibility of fault prediction, enabling the model to capture the aging trend of insulators in long-term use, and respond quickly to short-term sudden environmental changes or equipment failures.

[0123] This invention not only improves the accuracy of insulator defect detection, but also enhances the scientificity and efficiency of equipment management through the application of large language models in intelligent data analysis and fault prediction. These innovative contents enable the system to provide more accurate health assessment and fault warning in complex power systems, thereby greatly improving the operational safety of power systems and the efficiency of equipment management.

[0124] The invention also provides experimental data to verify the effect of the invention.

[0125] like Figure 6 The figure shows the loss function of the improved YOLOv4 during training. The loss function measures the gap between the predicted value and the true value. The smaller the loss function, the better the robustness of the model, and the model is less affected by data changes. As can be seen in the figure, as the number of training rounds increases, the loss function decreases and eventually stabilizes and converges.

[0126] As shown in Tables 1 and 2, the numerical differences of the accuracy and mAP value indicators are compared. From the table, we can see that in terms of accuracy comparison, the improved model has improved by 1.6% in glass sheet dirtiness, 2.2% in polymer sheet dirtiness, 5.6% in polymer sheet defects, and 3.1% in insulator flashover. The accuracy of glass sheet defects has decreased by 1.9%. The overall mAP50 has increased by 1.2%. In summary, the overall performance is improved, and it can be considered that the improvement of the improved YOLOv4 model is effective.

[0127] Table 1 Comparison of detection accuracy values ​​of different models

[0128]

[0129] Table 2 Comparison of mAP50 results

[0130]

[0131] Figure 7-Figure 9 A diagnosis report of different defects of the insulator is given. Compared with the traditional method, the present invention can give a diagnosis report independently, which is not available in other methods.

[0132] The beneficial effects of the present invention are as follows: the present invention provides an insulator defect detection method based on machine vision and a large language model. Compared with the prior art, the present invention is based on the improved YOLOv4 model and the large language model applied to insulator defect detection, which has significant advantages and beneficial effects, which are specifically manifested in the following aspects:

[0133] (1) Significant improvement in detection accuracy:

[0134] Most of the existing insulator defect detection methods rely on traditional image processing technology or are based on simple deep learning algorithms. These methods are prone to false detection or missed detection when dealing with insulator detection in complex backgrounds, especially when facing small targets (such as fine cracks or dirt). The detection effect is poor. The present invention improves the YOLOv4 model, introduces a lightweight MobileNetV2 backbone network, and optimizes the SPP module and feature extraction mechanism. Experimental data show that the detection accuracy of the present invention is improved by about 10% compared with the traditional YOLOv4 model, especially in the task of small target detection, the performance improvement is more obvious. This makes the detection of insulator defects more accurate, especially when dealing with complex environments and multiple targets, and can effectively reduce the probability of false detection and missed detection.

[0135] (2) Improved the robustness and generalization ability of the model:

[0136] By introducing Mosaic data enhancement technology, the present invention significantly improves the robustness and generalization ability of the model. Experiments show that after Mosaic data enhancement, the model can still maintain a high detection accuracy when facing insulator images with different angles, scales and lighting conditions. Compared with the unenhanced data set, the performance of the model of the present invention on the test set is improved by about 15%, which effectively solves the shortcomings of traditional methods in dealing with images with different perspectives and complex backgrounds. In addition, the addition of CBAM (convolutional block attention mechanism) further enhances the network's ability to capture key features, making it more stable when dealing with complex environmental interference.

[0137] (3) Intelligent fault prediction and maintenance suggestions:

[0138] Traditional insulator detection methods usually rely only on current detection data and cannot fully utilize historical data or environmental information for long-term equipment health status assessment and fault prediction. By introducing the Large Language Model (LLM), the present invention can effectively integrate historical maintenance records, operating status data, and environmental monitoring data to provide personalized fault prediction and maintenance recommendations. Compared with existing single detection methods, the present invention can detect potential problems earlier, provide accurate fault warnings, enable operation and maintenance personnel to take action in advance, avoid serious failures, and significantly reduce operation and maintenance costs.

[0139] (4) Reduce equipment maintenance costs:

[0140] Through fault prediction and personalized maintenance suggestions, the present invention reduces unnecessary maintenance and equipment replacement, thereby reducing overall operating and maintenance costs. Traditional insulator detection technology usually requires regular manual inspections, which is not only time-consuming and labor-intensive, but may also lead to premature replacement of equipment. The present invention, through a large language model and an intelligent early warning system, can provide scientific maintenance cycle suggestions based on the operating conditions of the insulators, avoiding cost waste caused by excessive maintenance and human misjudgment. According to the test results of actual applications, the maintenance cost of insulators will be reduced after adopting the detection system of the present invention.

[0141] (5) Real-time and lightweight design:

[0142] The present invention significantly reduces the computational complexity of the model by introducing a lightweight MobileNetV2 to replace the original CSPDarkNet53 backbone network. Compared with the traditional YOLOv4 model, the network of the present invention can achieve faster reasoning and detection in an environment with limited computing resources, and is suitable for embedded devices and real-time detection needs. Experiments show that under the premise of ensuring detection accuracy, the reasoning speed of the model is increased by 40%, which greatly improves the real-time detection capability of the system and meets the real-time requirements for online detection of insulators.

[0143] (6) Multimodal data fusion and intelligent analysis:

[0144] Traditional insulator inspection is usually limited to the analysis of image data, while ignoring the value of other related data. This invention combines the large language model (LLM) to perform fusion analysis of multimodal data, including insulator historical data, sensor monitoring data, and operating environment information. Through this innovation, the system can not only detect current defects, but also comprehensively evaluate the overall health status of the insulator, providing more valuable information support for decision-making.

[0145] In summary, the present invention has shown superior performance in insulator defect detection by improving the YOLOv4 model and introducing a large language model for intelligent data analysis and fault prediction. Compared with the prior art, the present invention has significant advantages in detection accuracy, robustness, real-time performance and cost control, and significantly improves the efficiency and reliability of insulator detection in power systems.

[0146] In the description of the present invention, it is necessary to understand that the orientation or positional relationship indicated by the terms "center", "thickness", "upper", "lower", "horizontal", "top", "bottom", "inner", "outer", "radial", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", and "third" are used only for descriptive purposes, and cannot be understood as indicating or implying the relative importance or the number of implicitly specified technical features. Therefore, the features defined by "first", "second", and "third" may explicitly or implicitly include one or more of the features.

Claims

1. An insulator defect detection method based on machine vision and large language model, characterized in that: The following steps are involved: S1, collecting grid insulator images and preprocessing them; S2. Perform data enhancement on the preprocessed grid insulator images, mark insulator defects, and generate training sets and test sets; S3, inputting the training set and the test set into the improved YOLOv4 model, training the improved YOLOv4 model, and obtaining a trained improved YOLOv4 model; S4. Insulator defect detection is performed by combining the trained improved YOLOv4 model with a large language model.

2. The insulator defect detection method based on machine vision and large language model according to claim 1 is characterized in that: In S1, the method for preprocessing the grid insulator image is specifically as follows: Perform image rotation, flipping and brightness enhancement operations on the power grid insulator image.

3. The insulator defect detection method based on machine vision and large language model according to claim 1 is characterized in that: In S2, the method for performing data enhancement on the preprocessed grid insulator image is specifically as follows: Mosaic data enhancement is performed on the preprocessed power grid insulator image to increase the diversity of the power grid insulator image data.

4. The insulator defect detection method based on machine vision and large language model according to claim 1 is characterized in that: In S3, the improved YOLOv4 model includes an input layer, a backbone network, a neck network and an output layer connected in sequence, and the neck network includes an SPP module, an FPNet module and a PANet module; Wherein, the input layer is used to receive the grid insulator image and pre-process the grid insulator image; The backbone network is used to extract feature information from the input grid insulator image; The neck network is used to extract features from the output feature map of the backbone network. The SPP module improves the receptive field of the model through pooling operations of different scales, thereby improving the ability to capture details and targets. The FPNet module and the PANet module enhance the transmission and sharing of features through feature fusion technology, making the insulator defect detection results output by the output layer more accurate. The output layer is used to output the insulator defect detection results, including the target category, location and confidence information.

5. The insulator defect detection method based on machine vision and large language model according to claim 4 is characterized in that: In S3, the backbone network construction method of the improved YOLOv4 model is: The constructed MobileNetV2 network is used to replace the CSPDarkNet53 backbone network in the YOLOv4 model. The MobileNetV2 network includes the first to fifth CSP modules connected in sequence.

6. The insulator defect detection method based on machine vision and large language model according to claim 4 is characterized in that: In S3, the SPP module of the improved YOLOv4 model includes a CBL structure, and a parallel first maximum pooling layer, a second maximum pooling layer and an average pooling layer; Among them, the input of the CBL structure is used as the input of the SPP module, and the output of the CBL structure is connected to the input of the first maximum pooling layer, the input of the second maximum pooling layer and the input of the average pooling layer respectively. The output features of the CBL structure, the output of the first maximum pooling layer, the output of the second maximum pooling layer and the output of the average pooling layer are fused as the output of the SPP module.

7. The insulator defect detection method based on machine vision and large language model according to claim 5, characterized in that: In S3, the first to fifth CSP modules have the same structure, and all include a first point-by-point convolution, a depth-wise separable convolution, a second point-by-point convolution, and a CBAM submodule; Among them, the first point-by-point convolution, the depth-wise separable convolution, the second point-by-point convolution and the CBAM sub-module are connected in sequence, the output of the CBAM sub-module is connected to the output of the second point-by-point convolution and the input of the element multiplication, and the output of the element multiplication is connected to the input of the first point-by-point convolution and the input of the element addition.

8. The insulator defect detection method based on machine vision and large language model according to claim 7, characterized in that: The CBAM submodule dynamically adjusts the attention distribution of the MobileNetV2 network by combining an improved spatial attention mechanism and a channel attention mechanism. Among them, the improved channel attention mechanism M C The specific expression of (F) is: Where σ(·) is the Sigmoid function, F avg is the one-dimensional tensor after the average pooling operation, F max is the one-dimensional tensor after the maximum pooling operation, is a one-dimensional convolution with a convolution kernel of k; Improved spatial attention mechanism M S The specific expression of (F) is: M S (F)=σ(f(f concat (F avg ;F max ))) Where, f(·) is the convolution calculation; f concat (·) is the splicing in dimension.

9. The insulator defect detection method based on machine vision and large language model according to claim 1, characterized in that: The S4 comprises the following sub-steps: S41, inputting the grid insulator image to be detected into the trained improved YOLOv4 model to obtain a defect report of the insulator defect detection, including the type, location and severity of the defect; S42. Analyze defect reports through a large language model combined with historical data to generate a health report, including prediction information and maintenance recommendations for future insulator failures.

Images