Method for detecting defects of power equipment and related products

Abstract

The invention relates to the field of data processing, in particular to a method for defect detection of power equipment and a related product, and the method comprises the steps: obtaining defect image data and text data of the power equipment, preprocessing the image data and the text data, and constructing a power equipment association knowledge graph; the visual detection features and the knowledge graph features are fused through a cross attention mechanism, and multi-modal joint features are generated; performing fine adjustment on the LLaVA model based on the multi-modal joint features to obtain a defect detection model; acquiring a real-time image, and inputting the real-time image into the defect detection model to complete defect detection of the power equipment; through the multi-modal data fusion technology, the visual data, the text data and the associated knowledge graph of the power equipment are fully utilized, and the defects of a traditional single data source detection method in a complex scene are overcome.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method for detecting defects in electric power equipment and related products. Background Art

[0002] As the scale of power systems continues to expand, the maintenance and management of power equipment faces unprecedented challenges. Due to the variety of equipment types and complex working environments, power equipment is prone to various forms of failures and defects during long-term operation, such as aging of electrical components, wear of mechanical structures, insulation damage, etc. These defects may lead to a decline in equipment performance and even affect the safety and stability of the entire power system. Therefore, timely and accurate defect detection of power equipment is a key link in ensuring the efficient operation of the power system.

[0003] At present, the traditional method of power equipment defect detection mainly relies on manual inspection and regular maintenance, which is not only inefficient but also highly dependent on the experience of operation and maintenance personnel. At the same time, with the increasing complexity of power systems, manual detection faces problems such as huge data volume, low information processing efficiency, and low detection accuracy, which makes it difficult to meet the needs of modern power systems. Summary of the invention

[0004] The technical problem to be solved by the present invention is that the existing power equipment defect detection method has insufficient detection accuracy when facing complex power scenarios, especially when the equipment failure types are diverse and the environment is complex, the effect of the defect detection model cannot meet the actual needs. The purpose is to provide a method and related products for power equipment defect detection, which realizes accurate detection and classification of power equipment defects and significantly improves the accuracy, robustness and interpretability of defect detection.

[0005] The present invention is achieved through the following technical solutions: A method for detecting defects in electric power equipment, comprising: Data acquisition and preprocessing: Obtain defect image data and text data of power equipment, and preprocess the image data and text data; Knowledge graph construction: Extract related entities and relationship information of power equipment defects from text data to build a knowledge graph related to power equipment; Feature fusion: Extract visual detection features from image data and extract knowledge graph features from the knowledge graph associated with power equipment; fuse visual detection features with knowledge graph features through a cross-attention mechanism to generate multimodal joint features; Model training: Build a pre-trained model, initialize the LoRA parameters of the LLaVA model based on the parameters of the pre-trained model, and fine-tune the LLaVA model based on multi-modal joint features to obtain a defect detection model; Defect detection: Acquire real-time images, input defect detection models, and complete power equipment defect detection.

[0006] Furthermore, after completing the model training, it also includes: Model quantization and inference acceleration: Quantize the fine-tuned model and load it into TensorRT for inference acceleration.

[0007] Specifically, the data collection and preprocessing methods include: Obtain power equipment defect image data and power equipment text data in the vertical domain scenario of power equipment; Clean the text data of power equipment to remove irrelevant or substandard text content; the text data includes equipment specifications, operating parameters, historical maintenance records, defect cases, expert knowledge and industry standards; Extracting related entities and relationship information related to power equipment defects from the cleaned text data; Clean the defect image data of power equipment and remove images of substandard quality; Annotate the power equipment image data to generate image-text pairs. The annotation processing content includes text description annotation and target box annotation. The annotated image data is normalized and the pixels are scaled to within a preset range.

[0008] Specifically, the method of constructing the knowledge graph includes: Generate RDF triples of the power equipment related knowledge graph using the extracted related entities and relationship information; With power equipment, defect types and detection methods as the main nodes, the structural framework of the power equipment related knowledge graph is constructed; Based on the structural framework of the knowledge graph, the knowledge graph is obtained using SPARQL query expansion.

[0009] Specifically, the feature fusion methods include: Visual feature extraction of defect image data of power equipment is performed through CLIP-ViT-L / 14 to generate image feature representation; Use the Vicuna model to extract text semantic features from power equipment text data and generate text feature representation; Using the entity and relationship information in the power equipment related knowledge graph, the related data of the power equipment is encoded to generate the knowledge graph feature representation; Use the cross-attention mechanism to fuse image feature representation, text feature representation, and knowledge graph feature representation to form a multimodal joint feature representation that includes image, text, and structured knowledge information; The multimodal joint feature representation is processed through a fully connected layer to generate feature representation for power equipment defect detection and classification.

[0010] Specifically, the model training method includes: Use the feature representation after feature fusion to pre-train a LoRA model A; Initialize the LoRA parameters of the LLaVA model through the LoRA parameters of the LoRA model A; The LoRA parameters of model A are injected into the Vicuna module of the language model of the LLaVA model, and the LoRA parameters of model A are injected into the CLIP-ViT-L / 14 module of the visual model of the LLaVA model; Based on the multimodal joint features, the LoRA module in the LLaVA model is fine-tuned based on the injected LoRA parameters; The fine-tuned LoRA module is fused with the main network of the LLaVA model to obtain the defect detection model.

[0011] Optionally, the expression for fine-tuning the LoRA module in the LLaVA model is: ,in, is the pre-trained weight matrix, and is a low-order matrix, is the weight update matrix generated by LoRA fine-tuning, is the field of real numbers, is the input dimension, is the output dimension, is the rank of the low-rank matrix and ; set up , after adding LoRA fine-tuning, it becomes ,in, is the output feature representation, is the input feature representation; Determine the optimization goals for fine-tuning the LoRA module: ,in, is the full model parameter, and For input data and output data, is the current data batch, Model parameters fine-tuned for LoRA, is the conditional probability distribution based on the full set of model parameters.

[0012] Specifically, model quantization and inference acceleration methods include: Convert the trained power equipment defect detection model into ONNX format; Perform INT8 quantization on the ONNX format model, and convert the model weights and activation values ​​from FP32 format to INT8 format through linear mapping; When performing convolutional layer operations, the calculated INT32-bit activation value is converted to INT8 format through requantization to be input to the next layer. The loop is executed until the inference process reaches the last layer, and the activation value is dequantized back to FP32 format; The quantized ONNX model is loaded through TensorRT, layer-by-layer optimization is performed, and inference is accelerated on the GPU.

[0013] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for detecting defects in electric power equipment as described above is implemented.

[0014] A computer program product comprises a computer program / instruction, which implements the method for detecting defects in electric power equipment as described above when the computer program / instruction is executed by a processor.

[0015] Compared with the prior art, the present invention has the following advantages and beneficial effects: The present invention acquires defect image data and text data of power equipment through data collection and preprocessing, and cleans and annotates the data; then, extracts related entities and relationship information from the text data to construct a related knowledge graph of the power equipment; then, adopts a cross-attention mechanism to fuse visual features with knowledge graph features to generate a multimodal joint feature representation; based on this joint feature, the defect detection model is trained through the LoRA fine-tuning method of the pre-trained model, and finally an efficient defect detection model is formed.

[0016] The present invention uses multimodal data fusion technology to make full use of the visual data, text data and related knowledge graphs of power equipment, solving the shortcomings of traditional single data source detection methods in complex scenarios. Feature fusion through the cross-attention mechanism can capture more related information, thereby significantly improving the accuracy and robustness of defect detection. The method also uses LoRA fine-tuning technology, so that the model can be efficiently fine-tuned on the specific data of power equipment while retaining the advantages of pre-training, further improving the model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings illustrate exemplary embodiments of the present invention and, together with the description thereof, are used to explain the principles of the present invention. These drawings are included to provide a further understanding of the present invention, and the accompanying drawings are included in and constitute a part of this specification and do not constitute a limitation of the embodiments of the present invention.

[0018] Figure 1It is a flow chart of a method for detecting defects in electric power equipment according to the present invention. DETAILED DESCRIPTION

[0019] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and implementation methods. It is understood that the specific implementation methods described herein are only used to explain the relevant content, rather than to limit the present invention.

[0020] It should also be noted that, for the convenience of description, only the parts related to the present invention are shown in the drawings.

[0021] In the absence of conflict, the embodiments and features of the embodiments of the present invention may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0022] In recent years, defect detection methods based on image recognition and deep learning technology have been gradually applied to the power industry, detecting and locating defects by analyzing defect image data and related text data of power equipment. However, these methods mainly focus on single-modal data processing and fail to effectively utilize the structured associated information of power equipment. Therefore, when dealing with complex power scenarios, the accuracy and reliability of detection results are still limited.

[0023] In order to improve the accuracy of defect detection, knowledge graph technology has gradually attracted attention. As a structured semantic information expression method, knowledge graph can effectively associate multi-source information of power equipment and realize semantic understanding and intelligent analysis of data. However, the existing knowledge graph construction method usually relies on a single data source, such as text data, and lacks deep integration of visual information such as images, making the application effect in complex scenarios unsatisfactory.

[0024] In addition, with the rapid development of artificial intelligence big models, big models have shown strong reasoning and analysis capabilities in the fields of natural language processing and image recognition. However, the application of big models in the field of power equipment defect detection still faces challenges, mainly in how to make full use of the complementarity of multimodal data and integrate the proprietary knowledge of power equipment to achieve efficient detection and accurate identification of equipment defects.

[0025] Embodiment 1, based on the above situation, as Figure 1 As shown, this embodiment provides a method for detecting defects in electric power equipment, including: Data acquisition and preprocessing: Obtain defect image data and text data of power equipment, and preprocess the image data and text data.

[0026] Power equipment defect image data usually comes from on-site photography of the equipment or monitoring equipment, mainly including the appearance of the power equipment and possible damage or defects. Text data includes equipment operating parameters, historical maintenance records, defect cases, expert knowledge and industry standards. These data usually come from the equipment management system or maintenance records.

[0027] The collected image and text data are preprocessed to remove substandard images or irrelevant or redundant text content. Image data needs to be cleaned, annotated, and normalized (for example, scaling the image size or adjusting the pixel value range), and text data needs to be cleaned and normalized to ensure its quality for subsequent analysis.

[0028] Knowledge graph construction: Extract related entities and relationship information of power equipment defects from text data to construct a knowledge graph related to power equipment.

[0029] The knowledge graph construction mainly extracts entities related to power equipment defects (such as equipment model, defect type, maintenance method, etc.) and their relationships (such as the relationship between a certain defect type and a specific equipment component) from text data. By extracting this information and constructing a knowledge graph related to power equipment, the system can obtain structured knowledge about the complex relationship between equipment and defects. The constructed knowledge graph is usually represented by RDF (Resource Description Framework) triples, that is, the association between various equipment and defects is described by "entity-relationship-entity" triples.

[0030] Feature fusion: Extract visual detection features from image data and extract knowledge graph features from the knowledge graph associated with power equipment; fuse visual detection features with knowledge graph features through a cross-attention mechanism to generate multimodal joint features.

[0031] After extracting features through deep learning models, the cross-attention mechanism is used to fuse the visual features of the image and the structural features of the knowledge graph. The cross-attention mechanism is a model based on the attention mechanism, which allows the model to establish connections between different modalities (images and texts) and weightedly combine different types of data through the self-attention mechanism to produce a comprehensive multi-modal joint feature representation. This joint feature representation can contain both visual and structured knowledge information.

[0032] Model training: Build a pre-trained model, initialize the LoRA parameters of the LLaVA model based on the parameters of the pre-trained model, and fine-tune the LLaVA model based on multimodal joint features to obtain a defect detection model.

[0033] LLaVA (Large Language and Vision Alignment) is a large model architecture that integrates language and visual information, while LoRA (Low-Rank Adaptation) is a fine-tuning strategy that can efficiently fine-tune the pre-trained model through a low-rank adaptation matrix without modifying most of the model's parameters. Based on multimodal joint features, the LoRA method is used to fine-tune the LLaVA model to further optimize the model's ability in power equipment defect detection.

[0034] Defect detection: Acquire real-time images, input defect detection models, and complete power equipment defect detection.

[0035] The system acquires real-time images of power equipment and inputs them into a trained defect detection model. The model analyzes the image and its multimodal features (visual information and knowledge graph information) to determine whether the equipment has defects and give the corresponding defect type and location.

[0036] In addition, after completing the model training, in order to improve the model's reasoning efficiency and real-time response capabilities in practical applications, model quantization and reasoning acceleration are performed.

[0037] The fine-tuned model is quantized and loaded into TensorRT for inference acceleration.

[0038] Model quantization is a technology for optimizing trained models. Its core purpose is to convert the model's weights and activation values ​​from high-precision (such as floating point) formats to low-precision (such as integer) formats, thereby reducing the model's storage space and computing overhead. TensorRT is an efficient deep learning inference engine launched by NVIDIA. It is specially optimized for GPU hardware, which can accelerate the model inference process and improve inference efficiency. TensorRT not only supports model quantization, but also can perform hierarchical optimization of the model, such as further simplifying the calculation process through pruning, fusing, etc.

[0039] In the second embodiment, the method for data collection and preprocessing includes: Obtain power equipment defect image data and power equipment text data in the vertical domain scenario of power equipment; obtain power equipment defect image data through the power equipment monitoring system or manual photography. Image data usually includes photos of the appearance of the equipment, which can clearly show the defects of the equipment, such as corrosion, cracks, and breakage. Text data mainly comes from equipment maintenance records, failure cases, operating parameters, etc., including equipment specifications, historical maintenance records, defect cases, expert knowledge, and industry standards.

[0040] Clean the text data of power equipment to remove irrelevant or substandard text content; the goal of cleaning is to remove irrelevant, repeated, inconsistently formatted or substandard text content to ensure data quality. Irrelevant text content includes documents or redundant information that are not related to equipment defects, and substandard text includes grammatical errors, unclear logic, etc.

[0041] Extract related entities and relationship information related to power equipment defects from the cleaned text data; related entities refer to important elements related to power equipment defects that appear in the text, such as equipment type, defect type, detection method, maintenance measures, etc. Relationship information is data that describes the relationship between these entities, such as a certain type of equipment is prone to a certain defect, or a certain defect requires a specific maintenance method.

[0042] Data cleaning is performed on the defective image data of power equipment to remove images of unqualified quality; images of unqualified quality may include blurred images, underexposed or overexposed images, damaged images or images containing irrelevant backgrounds, etc.

[0043] The image data of power equipment is annotated to generate image-text pairs. The annotation processing includes text description annotation and target box annotation. Text description annotation refers to the detailed text description of the defects in the image, such as "cracks in the equipment shell" or "broken cables". Target box annotation is to mark the specific location of the defect by drawing a rectangular box in the image, which helps the subsequent target detection model to identify the defect area.

[0044] Normalize the labeled image data and scale the pixels to a preset range (usually between 0 and 1). This can eliminate the effects of factors such as image size and brightness on model training, allowing the model to maintain consistent performance under different image conditions.

[0045] In the third embodiment, the method for constructing a knowledge graph includes: The extracted associated entities and relationship information are used to generate RDF triples of the associated knowledge graph of power equipment; RDF (Resource Description Framework) is a standard model for representing knowledge graphs, which represents entities and their relationships in the form of triples (subject-predicate-object). Through the combination of these triples, various attributes, defect types, fault conditions, maintenance methods and other information of power equipment can be effectively stored in the graph in a structured form.

[0046] With power equipment, defect types and detection methods as the main nodes, the structural framework of the power equipment associated knowledge graph is constructed; the power equipment node includes information such as the types, models, and components of various power equipment. The defect type node includes different types of defects (such as cracks, corrosion, overheating, etc.) and their occurrence on different equipment. The detection method node includes detection technologies, methods and tools for different defect types (such as infrared imaging, ultrasonic detection, etc.).

[0047] Based on the structural framework of the knowledge graph, the knowledge graph is obtained using SPARQL query extension. SPARQL is a query language used to query RDF format data. SPARQL query extension is used to deeply mine the constructed power equipment related knowledge graph to obtain more knowledge extensions.

[0048] Embodiment 4: Feature fusion effectively fuses data from different modalities (images, texts, and knowledge graphs) to form a multimodal joint feature representation, thereby improving the accuracy and robustness of the defect detection model. The method includes: CLIP-ViT-L / 14 is used to extract visual features from defective image data of power equipment and generate image feature representations. CLIP (Contrastive Language-Image Pre-training) is a large-scale pre-trained visual-language model that can learn to semantically associate images with text by jointly training images and text. ViT (Vision Transformer) is a visual model based on the Transformer architecture that is suitable for processing image data. After processing the image with CLIP-ViT-L / 14, the visual feature representation of the image can be extracted, which contains important information in the image, such as the type, location, and appearance of the equipment defect.

[0049] The Vicuna model is used to extract text semantic features from power equipment text data and generate text feature representations; Vicuna is a large-scale language model specifically designed for understanding and generating text. By extracting the semantic features of text through this model, the description of equipment defects, detection methods, and related standards can be converted into text feature representations that can be used for model training.

[0050] Using the entity and relationship information in the power equipment related knowledge graph, the related data of the power equipment is encoded to generate the knowledge graph feature representation; The cross-attention mechanism is used to fuse the image feature representation, text feature representation, and knowledge graph feature representation to form a multimodal joint feature representation that contains image, text, and structured knowledge information; the cross-attention mechanism allows image features, text features, and knowledge graph features to complement each other. For example, defect information in an image can be combined with descriptive information in the text, and the understanding of equipment defects can be further enhanced through the association relationship in the knowledge graph. In this way, the information of images, text, and structured knowledge can influence and interweave each other, resulting in a richer multimodal joint feature representation.

[0051] The multimodal joint feature representation is processed through a fully connected layer to generate a feature representation for power equipment defect detection and classification. The fully connected layer is a neural network layer that performs a weighted sum of the input features and outputs a final feature representation. Through the fully connected layer, the system can convert the fused multimodal information into a feature representation suitable for power equipment defect detection.

[0052] In the fifth embodiment, the purpose of model training is to effectively train and fine-tune the deep learning model through multimodal data after feature fusion, and finally generate an efficient power equipment defect detection model. The model training method includes: Use the feature representation after feature fusion to pre-train a LoRA model A; Initialize the LoRA parameters of the LLaVA model through the LoRA parameters of the LoRA model A; use the multimodal feature representation obtained by fusion of image data, text data and knowledge graph to pre-train a LoRA model A. LoRA (Low-Rank Adaptation) is a method for efficient fine-tuning of pre-trained models. It optimizes certain parameters of the model by introducing a low-rank matrix without completely updating all the weights of the model. Therefore, the LoRA model A is pre-trained using the fused feature representation. The model will learn the relevant features of power equipment defects through these multimodal data and form preliminary defect detection capabilities.

[0053] LLaVA (Large Language and Vision Alignment) is a multimodal large model architecture that can process both image and text data. By migrating the parameters learned in the LoRA model A to the LLaVA model, a better initial state can be provided for the LLaVA model, thereby accelerating the subsequent fine-tuning process and helping the LLaVA model better adapt to the task of power equipment defect detection.

[0054] The LoRA parameters of model A are injected into the language model Vicuna module of the LLaVA model, and the LoRA parameters of model A are injected into the visual model CLIP-ViT-L / 14 module of the LLaVA model; the Vicuna model is responsible for processing text data and extracting semantic features of text; the CLIP-ViT-L / 14 model is responsible for extracting visual features from the image data of power equipment.

[0055] Based on the multimodal joint features, the LoRA module in the LLaVA model is fine-tuned based on the injected LoRA parameters; fine-tuning refers to optimizing the pre-trained model using data from a specific field.

[0056] The fine-tuned LoRA module is integrated with the main network of the LLaVA model to obtain the defect detection model. The LoRA module will provide additional adaptability to the model, allowing the LLaVA model to fully utilize the advantages of the low-rank matrix, quickly adjust and efficiently perform power equipment defect detection tasks.

[0057] The mathematical formula and optimization objectives for fine-tuning the LoRA module in the LLaVA model are as follows: The expression for fine-tuning the LoRA module in the LLaVA model is: ,in, For pre-trained weight matrices, freeze weight parameters during fine-tuning and do not update gradients; and It is a low-order matrix responsible for gradient update during fine-tuning; is the weight update matrix generated by LoRA fine-tuning, is the field of real numbers, is the input dimension, is the output dimension, is the rank of the low-rank matrix and ; LoRA updates the weight matrix by introducing two low-rank matrices to generate a new weight update matrix.

[0058] set up , after adding LoRA fine-tuning, it becomes ,in, is the output feature representation, is the input feature representation; the input feature is transformed through the original pre-trained weight matrix to generate the output feature representation. The term is a modification of the original feature representation, which enables the model to be optimized based on the power equipment defect data.

[0059] The goal of fine-tuning is to maximize the model's prediction probability for input data and output data. The specific optimization goal is to maximize the log-likelihood function, that is, to determine the optimization goal for fine-tuning the LoRA module: ,in, is the full model parameter, and For input data and output data, is the current data batch, Model parameters fine-tuned for LoRA, is the conditional probability distribution based on the full set of model parameters.

[0060] Embodiment 6: Model quantization and reasoning acceleration are steps to improve the reasoning efficiency and response speed of the power equipment defect detection model in practical applications. By quantizing and optimizing the trained model, the computational burden of the model can be significantly reduced, the reasoning process can be accelerated, and the consumption of hardware resources can be reduced. The model quantization and reasoning acceleration method includes: In order to facilitate the deployment of the model on different hardware platforms, the trained power equipment defect detection model is converted into ONNX format; ONNX is an open deep learning model exchange format that allows trained models to interoperate between multiple frameworks and platforms.

[0061] The ONNX format model is quantized to INT8, and the model weights and activation values ​​are converted from FP32 format to INT8 format through linear mapping; quantization is the conversion of high-precision floating-point numbers (FP32 format) to low-precision integers (INT8 format), thereby reducing the storage requirements and computing overhead of the model.

[0062] When performing convolutional layer operations, the calculated INT32-bit activation value is converted to INT8 format through requantization to be input to the next layer, and the loop is executed until the inference process reaches the last layer, and the activation value is dequantized back to FP32 format; the activation value will be repeatedly quantized and dequantized between the calculations of each layer to ensure that the accuracy of the calculation will not be significantly reduced due to quantization. After multiple dequantization processes, the final output activation value will be converted back to FP32 format.

[0063] The quantized ONNX model is loaded through TensorRT, layer-by-layer optimization is performed, and inference is accelerated on the GPU.

[0064] Embodiment 7. A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the method for detecting defects in electric power equipment as described above is implemented.

[0065] The processor implements the steps of the above-mentioned antenna interface unit testing method when executed.

[0066] Without loss of generality, computer readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer readable instruction data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state storage technology, CD-ROM, DVD or other optical storage, cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above. The above-mentioned system memory and mass storage devices can be collectively referred to as memory.

[0067] A computer program product comprises a computer program / instruction, which implements the method for detecting defects in electric power equipment as described above when the computer program / instruction is executed by a processor.

[0068] A computer program product includes a computer program or set of instructions for performing specific tasks or implementing specific functions. These programs or instructions are designed to be executed by a processor to implement a series of predefined steps or operations. The program product may be stored in various forms of computer storage media, such as memory, hard disk, solid-state drive, optical disk or other forms of digital storage devices. It may exist in the form of compiled binary code or in the form of scripts or bytecodes that can be executed by an interpreter. The program product uses carefully designed algorithms and logical instructions to enable the processor to process data in a specific order and manner to complete various functions such as data analysis, user interaction, device control, etc.

[0069] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment / method or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments / methods or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments / methods or examples described in this specification and the features of the different embodiments / methods or examples, unless they are contradictory.

[0070] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0071] It should be understood by those skilled in the art that the above embodiments are only for the purpose of clearly illustrating the present invention, and are not intended to limit the scope of the present invention. For those skilled in the art, other changes or modifications may be made based on the above invention, and these changes or modifications are still within the scope of the present invention.

Claims

1. A method for detecting defects in electric power equipment, characterized in that: include: Data acquisition and preprocessing: Obtain defect image data and text data of power equipment, and preprocess the image data and text data; Knowledge graph construction: Extract related entities and relationship information of power equipment defects from text data to build a knowledge graph related to power equipment; Feature fusion: extract visual detection features of image data and extract knowledge graph features of the knowledge graph associated with power equipment; The visual detection features are fused with the knowledge graph features through the cross-attention mechanism to generate multimodal joint features; Model training: Build a pre-trained model, initialize the LoRA parameters of the LLaVA model based on the parameters of the pre-trained model, and fine-tune the LLaVA model based on multi-modal joint features to obtain a defect detection model; Defect detection: Acquire real-time images, input defect detection models, and complete power equipment defect detection.

2. A method for detecting defects in electric power equipment according to claim 1, characterized in that: After completing the model training, it also includes: Model quantization and inference acceleration: Quantize the fine-tuned model and load it into TensorRT for inference acceleration.

3. A method for detecting defects in electric power equipment according to claim 1, characterized in that: Data collection and preprocessing methods include: Obtain power equipment defect image data and power equipment text data in the vertical domain scenario of power equipment; Clean the text data of power equipment to remove irrelevant or substandard text content; the text data includes equipment specifications, operating parameters, historical maintenance records, defect cases, expert knowledge and industry standards; Extracting related entities and relationship information related to power equipment defects from the cleaned text data; Clean the defect image data of power equipment and remove images of substandard quality; Annotate the power equipment image data to generate image-text pairs. The annotation processing content includes text description annotation and target box annotation. The annotated image data is normalized and the pixels are scaled to within a preset range.

4. A method for detecting defects in electric power equipment according to claim 1, characterized in that: The methods for constructing knowledge graphs include: Generate RDF triples of the power equipment related knowledge graph using the extracted related entities and relationship information; With power equipment, defect types and detection methods as the main nodes, the structural framework of the power equipment related knowledge graph is constructed; Based on the structural framework of the knowledge graph, the knowledge graph is obtained using SPARQL query expansion.

5. A method for detecting defects in electric power equipment according to claim 1, characterized in that: Feature fusion methods include: Visual feature extraction of defect image data of power equipment is performed through CLIP-ViT-L / 14 to generate image feature representation; Use the Vicuna model to extract text semantic features from power equipment text data and generate text feature representation; Using the entity and relationship information in the power equipment related knowledge graph, the related data of the power equipment is encoded to generate the knowledge graph feature representation; Use the cross-attention mechanism to fuse image feature representation, text feature representation, and knowledge graph feature representation to form a multimodal joint feature representation that includes image, text, and structured knowledge information; The multimodal joint feature representation is processed through a fully connected layer to generate feature representation for power equipment defect detection and classification.

6. A method for detecting defects in electric power equipment according to claim 1, characterized in that: Model training methods include: Use the feature representation after feature fusion to pre-train a LoRA model A; Initialize the LoRA parameters of the LLaVA model through the LoRA parameters of the LoRA model A; The LoRA parameters of model A are injected into the Vicuna module of the language model of the LLaVA model, and the LoRA parameters of model A are injected into the CLIP-ViT-L / 14 module of the visual model of the LLaVA model; Based on the multimodal joint features, the LoRA module in the LLaVA model is fine-tuned based on the injected LoRA parameters; The fine-tuned LoRA module is fused with the main network of the LLaVA model to obtain the defect detection model.

7. A method for detecting defects in electric power equipment according to claim 6, characterized in that: The expression for fine-tuning the LoRA module in the LLaVA model is: ,in, is the pre-trained weight matrix, and is a low-order matrix, is the weight update matrix generated by LoRA fine-tuning, is the field of real numbers, is the input dimension, is the output dimension, is the rank of the low-rank matrix and ; set up , after adding LoRA fine-tuning, it becomes ,in, is the output feature representation, is the input feature representation; Determine the optimization goals for fine-tuning the LoRA module: ,in, is the full model parameter, and For input data and output data, is the current data batch, Model parameters fine-tuned for LoRA, is the conditional probability distribution based on the full set of model parameters.

8. A method for detecting defects in electric power equipment according to claim 2, characterized in that: Model quantization and inference acceleration methods include: Convert the trained power equipment defect detection model into ONNX format; Perform INT8 quantization on the ONNX format model, and convert the model weights and activation values ​​from FP32 format to INT8 format through linear mapping; When performing convolutional layer operations, the calculated INT32-bit activation value is converted to INT8 format through requantization to be input to the next layer. The loop is executed until the inference process reaches the last layer, and the activation value is dequantized back to FP32 format; The quantized ONNX model is loaded through TensorRT, layer-by-layer optimization is performed, and inference is accelerated on the GPU.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, a method for detecting defects in electric power equipment as described in any one of claims 1 to 8 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, a method for detecting defects in electric power equipment as described in any one of claims 1 to 8 is implemented.

Images