A power defect image detection method based on a picture-text question and answer multi-modal model

Abstract

The present application relates to a kind of power defect image detection method based on picture question and answer multi-modal model, compared with prior art, solve the scene complex in power defect image detection, defect type is many and small, the defect of low defect of multiple, disorder, mixed, interaction of deep learning target detection model.This application includes the following steps: the acquisition of multi-modal data set;Power defect knowledge data set acquisition;The construction of picture question and answer multi-modal model;Picture question and answer multi-modal model training;Power defect image detection result is obtained.The present application creates multi-modal power defect field Chinese data set, increases the picture knowledge of model power defect field;Fusion LoRA and Q-Former method fine-tuning model, reduce the model parameter quantity, improve the extraction ability of model to power defect visual and semantic features, align visual and semantic information.

Description

Technical Field

[0001] This invention relates to the field of power defect analysis technology, specifically a power defect image detection method based on a text-image question-and-answer multimodal model. Background Technology

[0002] Currently, power transmission, distribution, and substation mainly rely on regular manual inspections to detect defects in the power sector. This not only poses potential threats but also imposes a huge workload on inspection personnel and wastes a lot of resources.

[0003] With the popularization and development of artificial intelligence technology, deep learning-based object detection models have been introduced into power defect detection tasks and have achieved certain results in this field. However, current power defect detection algorithms require a large amount of manual defect labeling, while small-scale detection models suffer from problems such as being numerous, disorganized, having low interactivity, and being difficult to improve, resulting in time-consuming and labor-intensive power defect detection and hindering further development. These problems have always existed and have become more pronounced with the development of deep learning and object detection algorithms. In recent years, the continuous development of large-scale models has brought about rapid changes in the fields of human-computer interaction and intelligence. Large-scale models based on natural language processing technology, such as GPT-4, are beginning to show their potential with their powerful human-computer interaction capabilities and graph-text question answering capabilities.

[0004] Power defect detection is a crucial task and should be further developed by fully utilizing large-scale models. Therefore, how to propose a multimodal large-scale model for power defect image-text question answering, which can assist power practitioners in identifying power scenarios, capturing power defects, and understanding defect knowledge through image-text interaction, has become an urgent technical problem to be solved. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of power defect image detection, such as complex scenes, numerous defect types, and many small target detection models that are chaotic, messy, and have low interactivity. This invention provides a power defect image detection method based on a multimodal image-text question-answering model to solve the above problems.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] A power defect image detection method based on a text-image question-answering multimodal model includes the following steps:

[0008] 11) Acquisition of multimodal datasets: Acquire multimodal datasets and perform preprocessing;

[0009] 12) Acquisition of power defect knowledge dataset: Acquire power defect knowledge dataset and perform preprocessing;

[0010] 13) Construction of a text-image question-answering multimodal model: A text-image question-answering multimodal model is constructed based on a visual model and a language dialogue model;

[0011] 14) Training of the text-image question answering multimodal model: The text-image question answering model is trained based on the low-rank adaptation LoRA method and Q-Former. Among them, the LoRA method is a method for training large models with low resources, and Q-Former is a transformer framework for interactive training of multimodal models.

[0012] 15) Obtaining power defect image detection results: Through multimodal model image-text question answering test, the image-text question answering multimodal model answers the identified power scene, defect, and defect knowledge based on the input image and question prompt.

[0013] The acquisition of the multimodal dataset includes the following steps:

[0014] 21) By manually setting the question format, i.e., designing the prompt, it can be phrased as "What kind of scene is this picture? Please describe this picture."

[0015] 22) Obtain the image address (image) using a Python script;

[0016] 23) Use the trained Yolov8 model to classify the images and obtain the scene categories of the images, such as power transmission scenes;

[0017] 24) Use the target detection model trained in the power transmission scenario to perform defect detection, obtain the detection result category of each image, including normal and defective results; add coherent words to the image scene category and defect detection result category information to form a descriptive sentence and create a label;

[0018] 25) Organize the above information and construct a multimodal dataset for an image: {image, prompt, label}.

[0019] The acquisition of the power defect knowledge dataset includes the following steps:

[0020] 31) Obtain knowledge text about defect types in power scenarios;

[0021] 32) Obtain a power knowledge dataset, which includes five types: general power knowledge, transmission defect knowledge, distribution defect knowledge, substation defect knowledge, and safety supervision defect knowledge;

[0022] 33) Using Chat-GPT and the developed automated processing scripts, the question-and-answer semantic data is processed, redundant information is removed, incoherent sentences are modified and the order is sorted, and the data is organized according to categories to form a power defect knowledge dataset.

[0023] The construction of the text-image question-answering multimodal model includes the following steps:

[0024] 41) Select a pre-trained vision model, Vision Transformer, to extract visual features from power images;

[0025] 42) Select the well-trained language dialogue model ChatGLM-6B as the base model for dialogue, and extract the semantic features of the text and the text to answer the question;

[0026] 43) The LoRA module is added to the self-attention layer of the Vision Transformer visual model and the self-attention layer of the ChatGLM-6B language model respectively to train image and semantic knowledge related to electricity; the range of LoRA modules added to the visual model and the language model is set according to the actual number of self-attention layers, and the parameter LoRA_rank is normally set to 12.

[0027] 44) Use Q-Former to connect the visual model Vision Transformer with the language dialogue model ChatGLM-6B;

[0028] The Q-Former is configured to contain two Transformer modules, namely an image Transformer and a text Transformer, which share a self-attention layer.

[0029] The image Transformer interacts with the visual model with frozen parameters to extract image features output by the visual model. The text Transformer acts as both an encoder and a decoder. Learnable parameters are introduced into the image Transformer, which interacts with the text through a self-attention layer and sets a cross-attention layer to interact with image information.

[0030] The training of the text-image question-answering multimodal model includes the following steps:

[0031] 51) Initialize the multimodal model base network using the pre-trained weights W0 of the Vision Transformer visual model and the ChatGLM-6B language dialogue model; add the LoRA module to the self-attention layers of Vision Transformer and ChatGLM-6B, adding layers 0-27, with LoRA_rank set to 12; the formula is as follows:

[0032] For the pre-trained weight matrix W0∈R d×kLet d and k be the length and width of the weight matrix, respectively. The parameter update ΔW is represented by a low-rank decomposition, i.e.:

[0033] W0 + ΔW = W0 + DC (1)

[0034] Where C∈R r×k , D∈R d×r ,

[0035] C and D are low-rank matrices set for training, and r << min(d,k), where r is the low-rank set during the experiment.

[0036] During training, the parameter W0 is frozen, and only the parameters in C and D are trained. h is the parameter for training the complete set of parameters. For h = W0x, the forward propagation process becomes:

[0037] h = W0x + ΔWx = W0x + DCx; (2)

[0038] 52) Q-Former is used as the main training network. A set of learnable parameter vectors query is set. Each power defect image corresponds to a set of queries. The query obtains the image feature vector obtained by the Vision Transformer visual model through cross-attention in Q-Former.

[0039] The overall training process consists of three tasks: power defect image-text comparison learning, power defect image-based text generation, and power defect image-text matching. The specific steps are as follows:

[0040] 521) For image-text comparison learning, the text side obtains the text side CLS token through ChatGLM-6B. The text side CLS token contains a vector of global semantic features. For each query of an electric image in each multimodal dataset, calculate its cosine similarity with the text side CLS token, and take the query with the largest cosine similarity and the text feature CLS token.

[0041] This process learns to align image representations and text representations to maximize their mutual information. The resulting query is used in the image-based text generation process. Here, the query and text CLS token are training positive samples, and all other correspondences are negative samples. The cosine similarity calculation formula is defined as (3):

[0042] csim=(A·B) / (||A||||B||) (3)

[0043] Where A and B are two vectors, ||A|| and ||B|| are the magnitudes of A and B respectively, and A·B represents the inner product of vectors A and B, which can be expressed as formula (4):

[0044]

[0045] Where n represents the dimension of the two vectors, A i With B i It is each element;

[0046] When calculating the inner product, the two vectors are first multiplied element by element, and then all the products are added together.

[0047] The loss function used in the training process is shown in formula (5):

[0048]

[0049] Where τ is the temperature hyperparameter, which is a scalar, and k + and k - The numerator represents positive and negative samples, and q represents the ratio of positive samples to k. + The cosine similarity calculation; the denominator represents: the cosine similarity calculation of q with positive samples plus the cosine similarity calculation with all negative samples, the summation sign is from i=0 to K, a total of K+1 terms;

[0050] 522) For image-based text generation, the query is first used to extract the visual information with the highest similarity required to generate the text through the above image-text comparison learning. This information is then passed to the text CLS token during training through the self-attention shared in Q-Former. The text CLS token is then input into ChatGLM-6B to generate the text. In this process, a multimodal causal self-attention mask is used to control the query-text interaction. The whole process enables Q-Former to learn to generate text given the input image.

[0051] 523) For image-text matching, this is a binary classification task in which the model is required to predict whether the image-text pair is positive or negative, i.e., a match or a non-match.

[0052] In the Q-Former transformer architecture, a bidirectional self-attention mask is used to make all queries and text mutually attention; each query and text CLS token is put into a binary classification head to obtain the logit, which is the log odds of a sample belonging to the positive class, and is used to represent the confidence level of the model in that the sample belongs to the positive class. Finally, the average value is obtained as the final output matching score.

[0053] The sample pair with the highest matching score is selected as the positive sample, and the rest are negative samples; this process learns fine-grained alignment between image and text representations, which is more refined than image-text comparison learning.

[0054] 524) Perform the training process for the third objective in steps 521) to 523), where image-text comparison learning and image-text matching are used to achieve image classification, image retrieval and visual question answering to understand the task, and image-based text generation is used to achieve image description generation tasks.

[0055] 53) Set the learning rate to 0.001, batch_size to 1, and the number of training epochs to 600;

[0056] 54) Add the multimodal dataset and the power defect knowledge dataset to the text-to-image question answering multimodal model to enable training of the text-to-image question answering multimodal model:

[0057] 541) In the multimodal dataset, the images are fed into the Vision Transformer visual model to extract visual features, the question prompt and the answer label are fed into the ChatGLM-6B language model to extract semantic features, and then the visual features and semantic features are fed into the Q-Former framework for comparative learning.

[0058] 542) For the power defect knowledge dataset, when the input question prompt and answer label are fed into ChatGLM-6B, power defect images are also randomly sampled and input into the Vision Transformer. The extracted semantic features and visual features are then fed into the Q-Former framework for learning.

[0059] Obtaining the power defect image detection results includes the following steps:

[0060] 61) Input the test power defect image into the trained image-text question-answering multimodal model, and input the question prompt, assuming it is "What kind of power scene is this picture? Please describe it".

[0061] 62) The image-text question-answering multimodal model first uses Vision Transformer to extract image features, and uses ChatGLM-6B to extract semantic features of the question prompt. The image features and semantic features are concatenated together to obtain image-semantic joint features, which are then input into the language model. Through comparison and matching, the text with the highest cosine similarity to the image-semantic joint features is found. The image-text question-answering multimodal model outputs the text result, which is the power scene recognition text result, defect detection text result, and defect knowledge question-answering text response related to the input image and prompt.

[0062] Beneficial effects

[0063] This invention discloses a power defect image detection method based on a multimodal image-text question-answering model. By creating a multimodal Chinese dataset in the field of power defects, the model gains more image-text knowledge in the power defect domain. The method also integrates LoRA and Q-Former methods to fine-tune the model, reduce the number of model parameters, and improve the model's ability to extract visual and semantic features of power defects, thus aligning visual and semantic information.

[0064] Compared with existing target detection technologies, this invention has the advantages of stronger interactivity, clearer graphic and text responses, and greater room for improvement. Furthermore, in multiple aspects such as identifying power scenarios, defect detection, and defect knowledge Q&A, it has higher Q&A accuracy in the field of power defects compared with existing multimodal large models, thus improving the efficiency of power defect detection. Attached Figure Description

[0065] Figure 1 This is a sequence diagram of the method of the present invention.

[0066] Figure 2 This is a flowchart of the present invention;

[0067] Figure 3 This is a sequence diagram of the method for creating the multimodal dataset in this invention;

[0068] Figure 4 This is a schematic diagram of the LoRA method in this invention;

[0069] Figure 5 This is a test result diagram of the method described in this invention. Detailed Implementation

[0070] To provide a better understanding of the structural features and effects achieved by the present invention, a detailed description is provided below, accompanied by preferred embodiments and accompanying drawings:

[0071] like Figure 1 and Figure 2 As shown, the power defect image detection method based on a text-image question-and-answer multimodal model of the present invention includes the following steps:

[0072] The first step is to obtain the multimodal dataset: obtain the multimodal dataset and preprocess it.

[0073] (1) By setting the question prompt by human intervention, the prompt is designed as "What kind of scene is this picture? Please describe this picture".

[0074] (2) Obtain the image address using a Python script.

[0075] (3) Use the trained Yolov8 model to classify the images to obtain the scene categories of the images, such as power transmission scenes.

[0076] (4) Use the target detection model trained in the power transmission scenario to detect defects and obtain the detection result category of each image, including normal and defective results; add coherent words to the scene category and defect detection result category information of the image to form a descriptive sentence and create a label.

[0077] (5) After organizing the above information, a multimodal dataset of an image {image, prompt, label} was successfully constructed.

[0078] like Figure 3 As shown, here, the prompt is designed by manually setting the question format, which is "What kind of scene is this picture? Please describe this picture." The image address (image) is obtained through a Python script. In practical applications, a trained Yolov8 model is used to classify the image, obtaining the image category, such as a power transmission scene. A target detection model trained in the power transmission scene is used to detect defects, obtaining the detection result category for each image, including normal and defective results, such as "composite insulator damaged," "bird spikes normal," "vibration hammer rusted," "bird nest foreign object," etc. Then, the scene classification results and target detection results are combined into a sentence to obtain the required image description information label, such as "This is a power transmission scene picture, depicting composite insulator damage, bird spikes normal, vibration hammer rusted, etc." By organizing this information, a multimodal dataset {image, prompt, label} for an image is successfully constructed.

[0079] The second step is to obtain the power defect knowledge dataset: obtain the power defect knowledge dataset and perform preprocessing.

[0080] (1) Obtain knowledge text of important defect types in power scenarios.

[0081] (2) Obtain power knowledge datasets, which include five types: general power knowledge, transmission defect knowledge, distribution defect knowledge, substation defect knowledge and safety supervision defect knowledge.

[0082] (3) Using Chat-GPT and the developed automated processing scripts, the question-and-answer semantic data is processed, redundant information is deleted, incoherent sentences are modified and the order is sorted, and the data is organized according to categories to form a power defect knowledge dataset.

[0083] In practical applications, a series of power defect detection rulebooks have been collected, detailing various types of power defects, including common issues such as insulator damage, vibration damper corrosion, and bird nests on power poles. Knowledge texts on important defect types in power scenarios have been extracted. General power knowledge, including information on power equipment, power scenarios, and power safety, was searched online. Information was initially screened based on its authenticity and reasonableness, resulting in a comprehensive general power knowledge text. The dataset mainly covers five types: general power knowledge, transmission defect knowledge, distribution defect knowledge, substation defect knowledge, and safety supervision defect knowledge. This knowledge helps power professionals better understand the types and characteristics of defects. Chat-GPT and a developed automated processing script were used to process the question-and-answer semantic data, removing redundant information, correcting incoherent sentences, reordering the data, and organizing it according to categories.

[0084] The third step is to construct a multimodal image-text question-answering model: based on a visual model and a language dialogue model, a multimodal image-text question-answering model is constructed.

[0085] Since vision and language are two different modes of data, ordinary models struggle to achieve mutual understanding and interaction. Therefore, this invention employs the Vision Transformer visual model and the ChatGLM-6B language dialogue model to extract visual and language information respectively, process them into the same spatial vector, and then feed them into Q-Former for comparative learning. This process enables the fusion of heterogeneous data, aligning vision with semantics.

[0086] (1) A pre-trained vision model, Vision Transformer, is selected to extract visual features from the power images. These visual features are in vector form. The pre-trained Vision Transformer possesses a wealth of common image knowledge from the real world.

[0087] (2) A pre-trained language dialogue model, ChatGLM-6B, was selected as the base model for dialogue, extracting semantic features from text and responding to question prompts. ChatGLM-6B has functions for extracting semantic features, which can extract semantic features from text in vector form. In addition, a pre-trained ChatGLM-6B can realize text-based question-and-answer dialogues on many real-world knowledge topics.

[0088] (3) The LoRA module was added to the self-attention layer of the Vision Transformer visual model and the self-attention layer of the ChatGLM-6B language model to train image and semantic knowledge related to electricity. The range of LoRA modules added to the visual and language models was set according to the actual number of self-attention layers, and the scalar parameter LoRA_rank was normally set to 12. The range of added layers affects the training depth and the fitting effect of new knowledge. If the number of layers is small, the training depth is shallow and the integration effect of new knowledge is low. Conversely, if the training depth is large, the integration effect of new knowledge is high, but the model is prone to overfitting, which will destroy the dialogue effect of the original model.

[0089] (4) Use Q-Former to connect the visual model Vision Transformer with the language dialogue model ChatGLM-6B;

[0090] The Q-Former is configured to contain two Transformer modules, namely an image Transformer and a text Transformer, which share a self-attention layer.

[0091] The image Transformer interacts with the visual model with frozen parameters to extract image features output by the visual model. The text Transformer acts as both an encoder and a decoder. Learnable parameters are introduced into the image Transformer, which interacts with the text through a self-attention layer and sets a cross-attention layer to interact with image information.

[0092] For image modeling, a pre-trained Vision Transformer (ViT) was chosen for image feature extraction. ViT, through a continuous multi-layered attention mechanism, captures image information with local and global context and generates a set of image embedding vectors. Q-Former focuses on extracting the most informative visual representation of the text content. A series of learnable parameters are set to learn the relationship between visual and semantic features, and the goal of aligning visual and semantic features is achieved by training a contrastive loss function between visual feature vectors and semantic vectors. The features output by Q-Former are input into the language model to ensure the model has dynamic adaptability and strong learning capabilities. As a large language model, ChatGLM-6B receives information from Q-Former, including image embedding vectors containing semantic information and embedding vectors of the prompt text. ChatGLM-6B outputs a textual answer containing visual information, and then extracts a semantic embedding vector from this textual answer. This semantic embedding vector is then fed back into the Q-Former part, and a contrastive loss between the visual feature vector and the semantic embedding vector is calculated to further align visual and semantic features.

[0093] The fourth step is the training of the image-text question-answering multimodal model: training of the image-text question-answering model for power defect image detection based on the low-rank adaptive LoRA method and Q-Former. Among them, the LoRA method is a method for training large models with low resources, and Q-Former is a transformer framework for interactive training of multimodal models.

[0094] Because existing visual and language dialogue models have a large number of parameters, it is difficult for ordinary R&D personnel to train them with high configurations and full parameters. This invention uses the LoRA method to freeze the main parameters of the model and train only the low-rank matrix, which greatly reduces the number of training parameters and improves the feasibility of training. In addition, in the comparative training of Q-Former, due to the complexity and diversity of power defect images, it is very difficult to achieve fine-grained alignment between images and text. Therefore, the Q-Former training process of this invention includes three tasks: power defect image-text comparison learning, power defect image-based text generation, and power defect image-text matching. The first two tasks are for image classification, image retrieval, and visual question answering to understand the task, while the last task is for generating image descriptions. The three tasks work together to improve the fine-grained alignment of power defect images and text, and achieve more accurate power defect image-text responses.

[0095] (1) Initialize the multimodal model base network using the pre-trained weights W0 of the Vision Transformer visual model and the ChatGLM-6B language dialogue model; add the LoRA module to the self-attention layers of Vision Transformer and ChatGLM-6B, adding layers 0-27, with LoRA_rank set to 12; the principle of LoRA is as follows. Figure 4 As shown, its formula is expressed as follows:

[0096] For the pre-trained weight matrix W0∈R d×k Let d and k be the length and width of the weight matrix, respectively. The parameter update ΔW is represented by a low-rank decomposition, i.e.:

[0097] W0 + ΔW = W0 + DC (1)

[0098] Where C∈R r×k , D∈R d×r C and D are low-rank matrices set for training, and r << min(d,k), where r is the low-rank set during the experiment.

[0099] During training, the parameter W0 is frozen, and only the parameters in C and D are trained. h is the parameter for training the complete set of parameters. For h = W0x, the forward propagation process becomes:

[0100] h=W0x+ΔWx=W0x+DCx. (2)

[0101] (2) Q-Former is used as the main training network. A set of learnable parameter vectors query is set. Each power defect image corresponds to a set of queries. The query obtains the image feature vector obtained by the Vision Transformer visual model through cross-attention in Q-Former.

[0102] The overall training process consists of three tasks: power defect image-text comparison learning, power defect image-based text generation, and power defect image-text matching. The specific steps are as follows:

[0103] A1) For image-text comparison learning, the text side obtains the text side CLS token through ChatGLM-6B. The text side CLS token contains a vector of global semantic features. For each query of an electric image in each multimodal dataset, calculate its cosine similarity with the text side CLS token, and take the query with the largest cosine similarity and the text feature CLS token.

[0104] This process learns to align image representations and text representations to maximize their mutual information. The resulting query is used in the image-based text generation process. Here, the query and text CLS token are training positive samples, and all other correspondences are negative samples. The cosine similarity calculation formula is defined as (3):

[0105] csim=(A·B) / (||A||||B||) (3)

[0106] Where A and B are two vectors, ||A|| and ||B|| are the magnitudes of A and B respectively, and A·B represents the inner product of vectors A and B, which can be expressed as formula (4):

[0107]

[0108] Where n represents the dimension of the two vectors, A i With B i It is each element;

[0109] When calculating the inner product, the two vectors are first multiplied element by element, and then all the products are added together.

[0110] The loss function used in the training process is shown in formula (5):

[0111]

[0112] Where τ is the temperature hyperparameter, which is a scalar, and k + And k- represents positive and negative samples, the numerator represents: q for positive samples k + The cosine similarity is calculated as follows: the denominator represents the cosine similarity of q with positive samples plus the cosine similarity with all negative samples. The summation sign ranges from i=0 to K, for a total of K+1 terms.

[0113] The core idea of ​​the above process is as follows: For any pair of samples (image features and semantic features), calculate their distance in the feature space, and then adjust the model parameters according to their labels (similar or dissimilar). For similar sample pairs, minimize their distance in the feature space, which can be measured using Euclidean distance and cosine similarity; this paper chooses cosine similarity. For dissimilar sample pairs, maximize their distance in the feature space. The training objective is to learn visual semantic feature representations by maximizing the distance between similar samples (images and text correspondences) and minimizing the distance between dissimilar samples (images and text not corresponding).

[0114] A2) For image-based text generation, the query is first used to extract the visual information with the highest similarity required to generate the text through the image-text comparison learning mentioned above. This information is then passed to the text CLS token during training through the self-attention shared in Q-Former. The text CLS token is then input into ChatGLM-6B to generate the text. In this process, a multimodal causal self-attention mask is used to control the query-text interaction. The entire process enables Q-Former to learn to generate text given the input image.

[0115] A3) For image-text matching, this is a binary classification task in which the model is required to predict whether the image-text pair is positive or negative, i.e., a match or a non-match.

[0116] In the Q-Former transformer architecture, a bidirectional self-attention mask is used to make all queries and text mutually attention; each query and text CLS token is put into a binary classification head to obtain the logit, which is the log odds of a sample belonging to the positive class, and is used to represent the confidence level of the model in that the sample belongs to the positive class. Finally, the average value is obtained as the final output matching score.

[0117] The sample pair with the highest matching score is selected as the positive sample, and the rest are negative samples; this process learns fine-grained alignment between image and text representations, which is more refined than image-text comparison learning.

[0118] A4) Perform the training process for the three objectives in steps A1)-A3), where image-text comparison learning and image-text matching are used to achieve image classification, image retrieval, and visual question answering to understand the task, and image-based text generation is used to achieve image description generation tasks.

[0119] (3) Set the learning rate to 0.001, the batch size to 1, and the number of training rounds to 600.

[0120] (4) Add the multimodal dataset and the power defect knowledge dataset to the text-image question-answering multimodal model to enable training of the text-image question-answering multimodal model:

[0121] In the B1) multimodal dataset, images are fed into the Vision Transformer visual model to extract visual features, while the question prompt and answer label are fed into the ChatGLM-6B language model to extract semantic features. Then, the visual and semantic features are fed into the Q-Former framework for comparative learning.

[0122] B2) For the power defect knowledge dataset, the design mainly focuses on the defect knowledge question prompt and answer label, and is unrelated to the image content. However, in order to ensure that the input format of the entire training set is the same and that Q-Former can be trained normally, when the prompt and answer label are input into ChatGLM-6B, power defect images are also randomly sampled and input into the Vision Transformer. Then, the extracted semantic features and visual features are fed into the Q-Former framework for learning.

[0123] The fifth step is to obtain the power defect image detection results: through a multimodal model image-text question-and-answer test, the multimodal model answers the identified power scene, defect, and defect knowledge based on the input image and the question prompt.

[0124] (1) Input the test power defect image into the trained image-text question-answering multimodal model and input the question prompt, assuming it is "What kind of power scene is this picture? Please describe it".

[0125] (2) The image-text question answering multimodal model first uses Vision Transformer to extract image features and ChatGLM-6B to extract semantic features of the question prompt. The image features and semantic features are connected together to obtain image-semantic joint features, which are then input into the language model. Through comparison and matching, the text with the highest cosine similarity to the image-semantic joint features is found. The image-text question answering multimodal model outputs the text result, which is the power scene recognition text result, defect detection text result, and defect knowledge question answering text response related to the input image and prompt.

[0126] like Figure 5 As shown, this is the test result diagram of the final model. It is the final graphic test result diagram of the multimodal model. The user inputs a picture of a power defect scene and asks a question. The model effectively responds with power-related content, including scene recognition, defect detection, and defect knowledge Q&A.

[0127] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A power defect image detection method based on a picture-text question and answer multi-modal model, characterized in that, Includes the following steps: 11) Acquisition of multimodal datasets: Acquire multimodal datasets and perform preprocessing; 12) Acquisition of power defect knowledge dataset: Acquire power defect knowledge dataset and preprocess it; 13) Construction of a text-image question-answering multimodal model: A text-image question-answering multimodal model is constructed based on a visual model and a language dialogue model; The construction of the text-image question-answering multimodal model includes the following steps: 131) Select a pre-trained vision model, Vision Transformer, to extract visual features from power images; 132) Select the well-trained language dialogue model ChatGLM-6B as the base model for dialogue, and extract the semantic features of the text and the text to answer the question; 133) The LoRA module is added to the self-attention layer of the Vision Transformer visual model and the self-attention layer of the ChatGLM-6B language model respectively to train image and semantic knowledge related to electricity; the range of LoRA module layers added to the visual model and the language model is set according to the actual number of self-attention layers, and the parameter LoRA_rank is normally set to 12. 134) Using Q-Former to connect the visual model Vision Transformer with the language dialogue model ChatGLM-6B; The Q-Former is configured to contain two Transformer modules, namely an image Transformer and a text Transformer, which share a self-attention layer. The image Transformer interacts with the visual model with frozen parameters to extract image features output by the visual model. The text Transformer acts as both an encoder and a decoder. Learnable parameters are introduced into the image Transformer, which interacts with the text through a self-attention layer and sets a cross-attention layer to interact with image information. 14) Training of the image and text question answering multimodal model: The image and text question answering model is trained based on the low-rank adaptation LoRA method and Q-Former. Among them, the LoRA method is a method for training large models with low resources, and Q-Former is a transformer framework for interactive training of multimodal models. 15) Obtaining power defect image detection results: Through multimodal model image-text question answering test, the image-text question answering multimodal model identifies power scenes, defects, and defect knowledge based on the input image and the prompt.

2. The power defect image detection method based on the graphic-text question and answer multi-modal model according to claim 1, characterized in that, The acquisition of the multimodal dataset includes the following steps: 21) By manually setting the question format, i.e., designing the prompt, it can be phrased as "What kind of scene is this picture? Please describe this picture." 22) Obtain the image address (.image) using a Python script; 23) Use the trained Yolov8 model to classify the images and obtain the scene categories of the images; 24) Use the target detection model trained in the power transmission scenario to perform defect detection, obtain the detection result category of each image, including normal and defective results; add coherent words to the image scene category and defect detection result category information to form a descriptive sentence and create a label; 25) Organize the above information and construct a multimodal dataset for an image: {image, prompt, label}.

3. The power defect image detection method based on the graphic-text question and answer multi-modal model according to claim 1, characterized in that, The acquisition of the power defect knowledge dataset includes the following steps: 31) Obtain knowledge text about defect types in power scenarios; 32) Obtain a power knowledge dataset, which includes five types: general power knowledge, transmission defect knowledge, distribution defect knowledge, substation defect knowledge, and safety supervision defect knowledge; 33) Using Chat-GPT and the developed automated processing scripts, the question-and-answer semantic data is processed, redundant information is removed, incoherent sentences are modified and the order is sorted, and the data is organized according to categories to form a power defect knowledge dataset.

4. The power defect image detection method based on the graphic-text question and answer multi-modal model according to claim 1, characterized in that, The training of the text-image question-answering multimodal model includes the following steps: 41) Utilize the pre-trained weights of the Vision Transformer visual model and the ChatGLM-6B language dialogue model. Initialize the multimodal model's base network; add the LoRA module to the self-attention layers of the Vision Transformer and ChatGLM-6B, adding layers 0-27, with LoRA_rank set to 12; the formula is as follows: For the pre-trained weight matrix d and k are the length and width of the weight matrix, respectively, and the parameter update is represented by a low-rank decomposition. ,Right now: (1) wherein , , C and D are set low-rank matrices for training, and r is a low-rank set during the experiment process; Freezing parameters during training , only train parameters in C and D, h is training full parameters, for , the forward propagation process becomes: ； (2) 42) Q-Former is used as the main training network. A set of learnable parameter vectors query is set. Each power defect image corresponds to a set of queries. The query obtains the image feature vector obtained by the visual model VisionTransformer through cross-attention in Q-Former. The overall training process consists of three tasks: power defect image-text comparison learning, power defect image-based text generation, and power defect image-text matching. The specific steps are as follows: 421) For image-text comparison learning, the text side obtains the text side CLS token through ChatGLM-6B. The text side CLS token contains a vector of global semantic features. For each query of an electric image in each multimodal dataset, calculate its cosine similarity with the text side CLS token, and take the query with the largest cosine similarity and the text feature CLS token. This process learns to align image representations and text representations to maximize their mutual information. The resulting query is used in the image-based text generation process. Here, the query and text CLS token are training positive samples, and all other correspondences are negative samples. The cosine similarity calculation formula is defined as (3): （3） Where A and B are two vectors, and ||A|| and ||B|| are the magnitudes of A and B, respectively. Let the inner product of vectors A and B be expressed as formula (4): （4） where n represents the dimension of the two vectors, with is each element; When calculating the inner product, the two vectors are first multiplied element by element, and then all the products are added together. The loss function formula used in the training process (5): （5） in, It's a temperature hyperparameter, which is a scalar. and The numerator represents positive and negative samples, and q represents the number of positive samples. The cosine similarity calculation; the denominator represents: the cosine similarity calculation of q with positive samples plus the cosine similarity calculation with all negative samples, the summation sign is from i=0 to K, a total of K+1 terms; 422) For image-based text generation, the query is first used to extract the visual information with the highest similarity required to generate the text through the image-text comparison learning mentioned above. This information is then passed to the text CLS token during training through the self-attention shared in Q-Former. The text CLS token is then input into ChatGLM-6B to generate the text. In this process, a multimodal causal self-attention mask is used to control the query-text interaction. The entire process enables Q-Former to learn to generate text given the input image. 423) For image-text matching, this is a binary classification task in which the model is required to predict whether the image-text pair is positive or negative, i.e., a match or a non-match. In the Q-Former transformer architecture, a bidirectional self-attention mask is used to make all queries and text mutually attention; each query and text CLS token is put into a binary classification head to obtain the logit, which is the log odds of a sample belonging to the positive class, and is used to represent the confidence level of the model in that the sample belongs to the positive class. Finally, the average value is obtained as the final output matching score. The sample pair with the highest matching score is selected as the positive sample, and the rest are negative samples; this process learns fine-grained alignment between image and text representations, which is more refined than image-text comparison learning. 424) Perform the training process for the third objective in steps 421) to 423), where image-text comparison learning and image-text matching are used to achieve image classification, image retrieval and visual question answering to understand the task, and image-based text generation is used to achieve image description generation tasks; 43) Set the learning rate to 0.001, batch_size to 1, and the number of training epochs to 600; 44) Add the multimodal dataset and the power defect knowledge dataset to the text-to-image question answering multimodal model to enable training of the text-to-image question answering multimodal model: 441) In the multimodal dataset, the images are fed into the Vision Transformer visual model to extract visual features, the question prompt and the answer label are fed into the ChatGLM-6B language model to extract semantic features, and then the visual features and semantic features are fed into the Q-Former framework for comparative learning. 442) For the power defect knowledge dataset, when the input question prompt and answer label are fed into ChatGLM-6B, power defect images are also randomly sampled and input into the Vision Transformer. The extracted semantic features and visual features are then fed into the Q-Former framework for learning.

5. The power defect image detection method based on a text-image question-and-answer multimodal model according to claim 1, characterized in that, Obtaining the power defect image detection results includes the following steps: 51) Input the test power defect image into the trained image-text question-answering multimodal model, and input the question prompt; 52) The image-text question-answering multimodal model first uses Vision Transformer to extract image features, and uses ChatGLM-6B to extract semantic features of the question prompt. The image features and semantic features are concatenated together to obtain image-semantic joint features, which are then input into the language model. Through comparison and matching, the text with the highest cosine similarity to the image-semantic joint features is found. The image-text question-answering multimodal model outputs text results, namely, power scene recognition text results, defect detection text results, and defect knowledge question-answering text responses related to the input image and prompt.

Images