A method and system for detecting defects in a power transmission line

Abstract

The present application belongs to the technical field of power transmission line detection, and provides a power transmission line defect detection method and system, comprising: acquiring a power transmission line inspection image and a power transmission line instruction following data set; obtaining a generated description text based on the acquired power transmission line instruction following data set, combining the acquired power transmission line inspection image to form an image-text pair containing multimodal large language knowledge; obtaining an expert description text based on the acquired power transmission line inspection image, matching the generated description text and the expert description text to obtain an image-text pair containing field expert knowledge; constructing a joint knowledge power transmission line image-text data set according to the image-text pair containing multimodal large language knowledge and the image-text pair containing field expert knowledge; and performing defect detection of the power transmission line according to the constructed joint knowledge power transmission line image-text data set and a defect detector.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of power transmission line detection, and particularly relates to a power transmission line defect detection method and system. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] Stable operation of the power transmission line is the guarantee of the safety of the power system and the quality of power supply. The power transmission line is exposed to the harsh outdoor environment for a long time, and is prone to damage defects (such as rust, deformation, shedding and displacement); at the same time, affected by human and animal activities, non-component defects (such as bird nests and foreign matter) formed by external interference factors are also prone to occur in the power transmission line; these defects pose a serious threat to the safety of the power system. With the construction of the power system and the increase of the erection distance of the power transmission line, the traditional manual inspection method has been unable to undertake the tedious inspection work. The rapid development of unmanned aerial vehicle technology makes the unmanned aerial vehicle-based power transmission line inspection a new mainstream inspection mode; through the camera carried by the unmanned aerial vehicle to shoot the power transmission line inspection image, the collected image is processed and analyzed by using the defect detector based on deep learning to find out the defects therein, which effectively improves the intelligent level of the inspection process and reduces the burden of the inspection operation. However, due to the imaging distance and the shooting angle, the defects in the image collected by the unmanned aerial vehicle are usually accompanied by the problems of small size, partial occlusion and insufficient defect-related visual information in the complex background, which makes it difficult for the defect detector to learn sufficient knowledge for accurate defect recognition.

[0004] In the power inspection process, the inspection personnel describes the text (expert description text) as the file name of each inspection image with defects to help the subsequent data labeling personnel to accurately label the defect position; these texts contain rich defect-related semantic information, which can be used to alleviate the problem of insufficient defect-related visual information in the image. However, there is no research to use these texts as a supervision signal to train the defect detector. In recent years, visual-linguistic pre-training technology provides a way to make full use of these texts. By visual-linguistic pre-training, the information in these texts is introduced into the defect detector, and there are still the following three problems:

[0005] (1) The proportion of defects in the image is too small, and a large amount of redundant background information seriously interferes with the learning of the visual-linguistic alignment between the expert description text and the original inspection image. Therefore, the expert description text and the original inspection image cannot be directly used for visual-linguistic pre-training.

[0006] (2) In addition to defects, there are a large number of normal components in the inspection image that are not mentioned in the expert description text, and these components can provide additional visual clues for the model to capture the key features of the defects. Therefore, learning only the visual-linguistic alignment of the expert description text and the defect site is less efficient in utilizing the original inspection image.

[0007] (3) The existing mainstream visual-linguistic pre-training technology and open-source pre-training model are mainly based on image-level supervision, lacking region-word-level alignment and understanding ability. The expert description text often involves the defect site and its surrounding components, so the existing visual-linguistic pre-training technology can only get a suboptimal representation when processing the expert description text, which is not conducive to learning fine-grained visual-linguistic alignment. SUMMARY

[0008] To solve the above problems, the present application provides a power transmission line defect detection method and system, which introduces description text containing knowledge of different sources into the detector using a multi-modal large language model to provide additional semantic information, and pre-trains the backbone of the detector using a global-local image representation learning algorithm based on attention to better learn global and local visual-linguistic alignment, thereby improving the accuracy of power transmission line defect detection.

[0009] According to some embodiments, the first aspect of the present application provides a power transmission line defect detection method, which adopts the following technical solution:

[0010] A power transmission line defect detection method, comprising:

[0011] Obtaining a power transmission line inspection image and a power transmission line instruction following dataset;

[0012] Based on the obtained power transmission line instruction following dataset, generating a description text, and combining the obtained power transmission line inspection image to form an image-text pair containing multi-modal large language knowledge;

[0013] Based on the obtained power transmission line inspection image, obtaining an expert description text, matching the generated description text and the expert description text, and obtaining an image-text pair containing field expert knowledge;

[0014] According to the image-text pair containing multi-modal large language knowledge and the image-text pair containing field expert knowledge, a joint knowledge power transmission line image-text dataset is constructed;

[0015] According to the constructed joint knowledge power transmission line image-text dataset and the defect detector, the defect detection of the power transmission line is performed.

[0016] As a further technical limitation, on the basis of the constructed joint knowledge power line image-text data set, a global-local image representation learning algorithm based on attention is introduced to pre-train the visual base model, and the knowledge in the image-text data set is introduced into the visual base model to complete the pre-training of the visual base model.

[0017] Further, the defect detector is used for defect detection of the power line, that is, the text encoder is removed from the pre-trained visual base model, and the image encoder is retained as the backbone network of the defect detector. A feature pyramid network, a region proposal network and a bounding box head are added at the output end of the backbone network to construct the defect detector, wherein the feature pyramid network provides multi-level features for defect detection, and the region proposal network and the bounding box head are used to analyze the multi-level features and locate and classify defects in the input image. The defect detector is fine-tuned through the original inspection image with bounding box annotation, the knowledge obtained by pre-training is transferred to the defect detection task, and the final defect detector is obtained.

[0018] As a further technical limitation, high-quality images are screened from the power line instruction following data set, and the screened images are inferred using the tuned InternVL to obtain generated description texts; according to the obtained power line inspection images and their corresponding generated description texts, image-text pairs containing multi-modal large language model knowledge are obtained.

[0019] Further, the ChatGLM is used to perform semantic matching on the generated description texts corresponding to all screened high-quality images containing defects and the expert description texts provided by the inspection personnel for the original inspection images to which the defects belong, and according to the defect images and the expert description texts matched with the generated description texts corresponding to the defect images, image-text pairs containing domain expert knowledge are obtained.

[0020] As a further technical limitation, in the process of constructing the joint knowledge of the power line image-text dataset according to the image-text pair containing the multimodal large language knowledge and the image-text pair containing the domain expert knowledge, in order to enhance the supervision signal, the category label carried by each power line image is filled into the pre-defined text template to obtain the description text based on the pre-defined template, and the obtained description text and the power line image form the image-text pair based on the pre-defined template; each image in the image part of the pre-training dataset is traversed to view the category of the image-text pair formed by the image, if the image forms the image-text pair containing the domain expert knowledge, the image-text pair containing the domain expert knowledge is used as a sample of the pre-training dataset, otherwise, one of the image-text pair containing the multimodal large language model knowledge and the image-text pair based on the pre-defined template formed by the image is randomly selected as a sample of the pre-training dataset, and the pre-training dataset fusing the multimodal large language model and the domain expert knowledge is constructed, that is, the joint knowledge of the power line image-text dataset is obtained.

[0021] According to some embodiments, the second aspect of the present application provides a power line defect detection system, which adopts the following technical solution:

[0022] A power line defect detection system comprises:

[0023] An acquisition module configured to acquire power line inspection images and a power line instruction following dataset;

[0024] A construction module configured to obtain generated description text based on the acquired power line instruction following dataset, and combine the acquired power line inspection images to form an image-text pair containing multimodal large language knowledge; obtain expert description text based on the acquired power line inspection images, match the generated description text and the expert description text, and obtain an image-text pair containing domain expert knowledge; and construct a joint knowledge of a power line image-text dataset according to the image-text pair containing the multimodal large language knowledge and the image-text pair containing the domain expert knowledge;

[0025] A detection module configured to perform power line defect detection according to the constructed joint knowledge of the power line image-text dataset and a defect detector.

[0026] According to some embodiments, the third aspect of the present application provides a computer readable storage medium, which adopts the following technical solution:

[0027] A computer readable storage medium having stored thereon a program which, when executed by a processor, implements the steps of the power transmission line defect detection method according to the first aspect of the present application.

[0028] According to some embodiments, the fourth aspect of the present application provides an electronic device, which adopts the technical scheme as follows:

[0029] An electronic device includes a memory, a processor, and a program stored in the memory and running on the processor, and the processor implements the steps of the power transmission line defect detection method according to the first aspect of the present application when executing the program.

[0030] According to some embodiments, the fifth aspect of the present application provides a computer program product, which adopts the technical scheme as follows:

[0031] A computer program product includes software code, and the program in the software code implements the steps of the power transmission line defect detection method according to the first aspect of the present application.

[0032] Compared with the prior art, the present application has the following beneficial effects:

[0033] The present application utilizes multi-modal large language model and visual-language pre-training technology, introduces text containing knowledge description of different sources to provide additional semantic information for the detector, and introduces attention-based global-local image representation learning algorithm to pre-train the detector backbone to better learn global and local visual-language alignment, thereby improving the accuracy of power transmission line defect detection.

[0034] The present application cooperates with the attention-based global-local image representation learning algorithm to achieve the goal of training a high-performance image encoder jointly based on two sources of knowledge. The pre-trained image encoder can effectively alleviate the problem of insufficient defect-related visual information in the original inspection image, and the fine-tuned power transmission line defect detector has higher precision, which has important research significance for intelligent inspection of power transmission lines. BRIEF DESCRIPTION OF DRAWINGS

[0035] The drawings constituting a part of this embodiment are used to provide a further understanding of the embodiment, and the schematic embodiment and its description are used to explain the embodiment, and do not constitute an improper limitation on the embodiment.

[0036] Figure 1 A flowchart of a power transmission line defect detection method in the first embodiment of the present application;

[0037] Figure 2 A detailed step schematic diagram of the power transmission line defect detection method in the first embodiment of the present application;

[0038] Figure 3 The overall training process in the embodiment one of the present application is shown in the schematic diagram;

[0039] Figure 4 The method of introducing the generated description text and the expert description text in the embodiment one of the present application is shown in the schematic diagram;

[0040] Figure 5 The attention-based global-local image representation learning algorithm in the embodiment one of the present application is shown in the schematic diagram;

[0041] Figure 6 The effect diagram of the transmission line defect detection in the embodiment one of the present application is shown in the schematic diagram, Figure 6 (b) in the schematic diagram, Figure 6 (a) is Figure 6 the local schematic diagram of (b);

[0042] Figure 7 The structural block diagram of a transmission line defect detection system in the embodiment two of the present application is shown in the schematic diagram. DETAILED DESCRIPTION

[0043] The present application will be further described below in conjunction with the accompanying drawings and embodiments.

[0044] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application pertains.

[0045] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, there is a presence of a feature, step, operation, device, component, and / or combinations thereof.

[0046] In the present application, the terms such as "upper", "lower", "left", "right", "front", "back", "vertical", "horizontal", "side", "bottom", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, which is only a relationship word determined for the convenience of describing the structural relationship of the components or elements of the present application, and cannot be understood as a limitation on the present application.

[0047] In the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0048] Embodiment one

[0049] The embodiment one of the present application introduces a power transmission line defect detection method.

[0050] As shown in a power transmission line defect detection method shown in Figure 1 , comprising:

[0051] Obtain a power transmission line inspection image and a power transmission line instruction following data set;

[0052] Based on the obtained power transmission line instruction following data set, generate a description text, and combine the obtained power transmission line inspection image to form an image-text pair containing multi-modal large language knowledge;

[0053] Based on the obtained power transmission line inspection image, obtain an expert description text, match the generated description text and the expert description text, and obtain an image-text pair containing domain expert knowledge;

[0054] According to the image-text pair containing multi-modal large language knowledge and the image-text pair containing domain expert knowledge, a joint knowledge power transmission line image-text data set is constructed;

[0055] According to the constructed joint knowledge power transmission line image-text data set and the defect detector, the defect of the power transmission line is detected.

[0056] This embodiment combines multi-modal large models and joint knowledge to detect defects of power transmission lines, and the specific steps are as shown in Figure 2 , specifically:

[0057] Step S1: label and crop the original power transmission line inspection image, and use the large language model ChatGLM to batch generate question and answer pairs, construct a power transmission line instruction following data set, and contain multi-modal dialogue data related to various power transmission line components and defects, which covers various power transmission line defect detection related concepts;

[0058] Step S2: On the power transmission line instruction following data set, use QLoRA technology to optimize the high-performance multi-modal large language model InternVL, so that it learns the visual-linguistic alignment of various power transmission line defect detection related concepts;

[0059] Step S3: Select high-quality images from the power transmission line instruction following data set, and use the optimized InternVL to infer all selected images to obtain a generated description text, and each image and the corresponding generated description text form an image-text pair containing multi-modal large language model knowledge;

[0060] Step S4: using ChatGLM to perform semantic matching between the generated description text corresponding to the image containing the defect screened out and the expert description text provided by the inspection personnel for the original inspection image to which the defect belongs, and the defect image and the expert description text correctly matched with the generated description text constitute the image-text pair containing the domain expert knowledge;

[0061] Step S5: using the image-text pair containing the knowledge of the multi-modal large language model and the image-text pair containing the domain expert knowledge to construct a power line image-text dataset with joint knowledge;

[0062] Step S6: on the joint knowledge power line image-text dataset, introduce a global-local image representation learning algorithm based on attention to pre-train the visual base model Chinese-CLIP, and introduce the rich knowledge in the dataset into Chinese-CLIP;

[0063] Step S7: use the image encoder in the pre-trained Chinese-CLIP as the backbone network to construct a defect detector, and use a small amount of labeled defect detection data to fine-tune to obtain the final defect detector.

[0064] The embodiment overcomes the problem of insufficient visual information related to defects in power line inspection images from the perspective of multi-modal learning, and the overall training process is as shown in Figure 3 In the original power line inspection data, the expert description text provided by the inspection personnel only describes a small part of the defect-containing area in the high-resolution inspection image, while the image often contains a large number of components and other complex backgrounds unrelated to the expert description text. Therefore, the image-text pair directly constituted by the original inspection data is not suitable for directly learning visual-linguistic alignment, otherwise it will be adversely affected by a large amount of interfering background information in the image, resulting in poor learning effect, and the utilization rate of image information is also very low.

[0065] To better learn visual-linguistic alignment and make full use of the information contained in the inspection image, first, the defect area and other normal components in the image need to be cropped to obtain single-instance images (for all defects and components) that are more conducive to visual-linguistic alignment and context-enhanced instance images (only for defects, because the expert description text often involves defects and their surrounding components). Then, considering that only a small part of the cropped images can correspond to the expert description text, the remaining images (normal component images and defect images not described by the expert) lack matching description text, so it is necessary to generate matching description text for them through a multi-modal large language model.

[0066] Although the existing large language models and multi-modal large language models contain knowledge related to power line inspection (derived from open source documents and materials on the Internet), they either do not support image input or cannot access a large number of power line related components or defect images during the training process due to data barriers. This results in better responses from large language models and multi-modal large language models when only inputting power line inspection related query text; while inputting power line inspection related query text and images at the same time, multi-modal large language models often make incorrect responses, which indicates that existing multi-modal large language models cannot better align visual-linguistic information in the power line scene. Therefore, with the help of large language models, the above single-instance images and context-enhanced instance images can be used to construct a dedicated instruction-following dataset for the power line scene to fine-tune the multi-modal large language model to help it learn visual-linguistic alignment in the power line scene and generate higher quality description text for images.

[0067] In step S1, the original power line inspection images are labeled and cropped, and the large language model ChatGLM is used to generate a large number of question and answer pairs, and a power line instruction-following dataset is constructed, which contains multi-modal dialogue data related to various power line components and defects, and covers various concepts related to power line defect detection, including:

[0068] First, the components and defects in the original power line inspection images are labeled and the labeled boxes are cropped to obtain a large number of single-instance images with class labels (images mainly containing a single component or defect) and context-enhanced instance images with class labels and defect position coordinates (images containing defect and context information such as component), which constitute the image part of the power line instruction-following dataset.

[0069] Secondly, for single-instance images, first prompt the large language model ChatGLM to generate a set of question and answer pairs about the category, such as: “Please generate 30 sets of single-turn question and answer pairs about the appearance, color, position or function of the shielding ring in the power line”, and then select the parts that are helpful for learning concepts related to power line defect detection from the generated question and answer pairs as seed question and answer pairs. Then use the context learning ability of the large language model to generate coarse-grained paired question and answer pairs for each single-instance image belonging to the category based on the seed question and answer pairs.

[0070] Then, for the context-enhanced instance image, first, the basic description about each context-enhanced instance image is obtained according to the manually designed description template and by using the category label and defect position coordinates, such as: "The picture is a damaged grading ring on the power transmission line, located at the lower left corner of the image, and the area of the region accounts for a moderate proportion in the picture"; and then the ChatGLM is prompted to generate a question-answer pair with coarse-grained matching of the image.

[0071] Finally, the question-answer pairs with single-instance image coarse matching and the question-answer pairs with context-enhanced instance image coarse-grained matching constitute the text part of the power transmission line instruction following dataset, which together with the image part of the power transmission line instruction following dataset constitutes the power transmission line instruction following dataset, which covers visual and language instances of various power transmission line defect detection related concepts, and thus can help the multimodal large language model learn the visual-language alignment in the power transmission line scene.

[0072] In the embodiment, considering that the expert description text is Chinese and the existing device conditions and data scale are difficult to support end-to-end optimization of the multimodal large language model, it is necessary to use a multimodal large language model supporting Chinese input and a model optimization technology more efficient than end-to-end fine-tuning. Therefore, in the step S2, the QLoRA technology is used to fine-tune the high-performance multimodal large language model InternVL on the power transmission line instruction following dataset, so that it learns the visual-language alignment of various power transmission line defect detection related concepts, specifically including:

[0073] First, the high-performance multimodal large language model InternVL that performs well in Chinese visual-language tasks is selected for optimization in the power transmission line scene, and considering the memory conditions of the available device, the open source InternVL2-2B weight initialization model parameters are selected.

[0074] Then, the InternVL model is efficiently optimized on the constructed power transmission line instruction following dataset using the QLoRA technology, which effectively avoids the problem of requiring a large amount of computing power and consuming a large amount of time for traditional end-to-end optimization under limited computing resources.

[0075] In this embodiment, considering that although a large number of components and defect images are obtained by cropping in step S1, there are many low-quality images among them and there is a certain degree of class distribution imbalance between the component and defect classes. In order to improve the quality of text generation and the subsequent visual-language pre-training effect, it is necessary to filter the data. Using the tuned multi-modal large language model to infer on the high-quality images obtained by filtering can integrate the knowledge related to the seen components or defects in the multi-modal large language model into the generated text.

[0076] Therefore, in the step S3, high-quality images are filtered from the power transmission line instruction following data set, and the tuned InternVL is used to infer on all the filtered images to obtain generated description texts, and each image and the corresponding generated description text constitutes an image-text pair containing multi-modal large language model knowledge, specifically including:

[0077] First, high-quality defect and component images are filtered from the image part of the power transmission line instruction following data set, and these images constitute the image part of the pre-training data set required for visual-language pre-training.

[0078] Then, in order to ensure the diversity of the generated results, a variety of manually designed instructions, such as “describe the component in the image and determine whether it has a defect”, are used to prompt the tuned InternVL to infer on all the filtered images to generate a piece of fine-grained matching generated description text for each image, which contains the power field knowledge provided by the tuned InternVL (including the power field knowledge provided by ChatGLM and the power field knowledge contained in InternVL itself), and each image and the corresponding generated description text constitutes an image-text pair containing multi-modal large language model knowledge, and the process is as shown in Figure 4

[0079] In this embodiment, considering that the original expert description text contains some information irrelevant to the content of the defect image, such as “500kV Mengji I line #0091 tower right conductor small side left side string iron tower end 3rd insulator explosion” in the description of voltage level, line name and tower number in “500kV Mengji I line #0091 tower”, which cannot form a direct correspondence with the visual information in the image, so it will interfere with visual-language pre-training. In addition, sometimes an image contains multiple defects, and the expert description text only mentions one of them, so the defect images obtained by cropping cannot be directly paired with the expert description text corresponding to the original image.

[0080] ​In the step S4, the ChatGLM is used to perform semantic matching between the generated description text corresponding to all the images containing defects and the expert description text provided by the inspection personnel for the original inspection image to which the defect belongs. The defect image and the expert description text correctly matched with the generated description text constitute the image-text pair containing domain expert knowledge, specifically including:

[0081] Firstly, the expert description text provided by the inspection personnel for the original inspection image is preprocessed to delete redundant content such as voltage level, line name and tower number, so as to avoid interference of these contents on the subsequent pre-training effect.

[0082] Then, for each image containing defects, the ChatGLM is required to judge the semantic matching degree between the generated description text corresponding to the image and the expert description text corresponding to the original inspection image to which the defect belongs through the instruction “Please judge the matching degree between the following two texts and give a matching degree score between 0 and 10”, and give a score between 0 and 10. When the score is greater than or equal to the threshold score 6, it is considered that the image is correctly matched with the expert description text, and the two constitute the image-text pair containing domain expert knowledge. The process is as shown in Figure 4

[0083] The two image-text pairs containing different source knowledge obtained by the above steps S3 and S4 are used to construct a joint knowledge multi-modal data set for visual-linguistic pre-training. At the same time, considering that the supervision signal provided by the two types of text is limited in strength, and the class label retained when the image is cropped can be used to enhance the supervision signal in the pre-training process. Therefore, in the step S5, the joint knowledge transmission line image-text data set is constructed using the image-text pair containing multi-modal large language model knowledge and the image-text pair containing domain expert knowledge, specifically including:

[0084] Firstly, in order to enhance the supervision signal, the class label carried by each image is filled into the pre-defined text template to obtain the description text based on the pre-defined template, such as “There is a normal grading ring in the image”. The text and the image together constitute the pre-defined template image-text pair.

[0085] ​Then, each image in the pre-training data set image part is traversed to view the image-text pair category composed of the image, if it participates in the composition of the image-text pair containing the domain expert knowledge, the image-text pair containing the domain expert knowledge is used as a sample of the pre-training data set, otherwise, one of the image-text pairs containing the multimodal large language model knowledge and the image-text pairs based on the pre-defined template is randomly selected as a sample of the pre-training data set, and the pre-training data set constructed by fusing the knowledge from the multimodal large language model and the domain expert is called the power line image-text data set with joint knowledge.

[0086] In the embodiment, considering that visual-language pre-training from scratch needs to use data of the order of 100 million and consume a large amount of computing power, in order to avoid training the model from scratch, the embodiment adopts the idea of transfer learning, and migrates the existing open source pre-training model to the joint knowledge power line image-text data set constructed in step S5 for pre-training for the power line scene. On the other hand, the existing mainstream visual-language pre-training technology and open source pre-training model are mainly based on image-level supervision, although they have global image-text level alignment capability, they lack regional-word level alignment and understanding capability. Expert description text often involves defect positions and their surrounding components, such as: "500kV Linui I line #0146 tower middle conductor small side string bowl head hanging plate and connecting plate connecting bolt missing pin". In addition to mentioning the "bolt missing pin" defect, it also mentions the connecting fittings "bowl head hanging plate" and "connecting plate" on both sides. Therefore, in order to better learn the more fine-grained regional-word level alignment, the existing visual language pre-training algorithm needs to be improved to help the model learn local representation while learning global representation of image and text.

[0087] In the step S6, on the joint knowledge power line image-text data set, a global-local image representation learning algorithm based on attention is introduced to pre-train the visual base model Chinese-CLIP, and the rich knowledge in the data set is introduced into Chinese-CLIP, which specifically includes:

[0088] Firstly, the open source Chinese-CLIP ViT-B-16 pre-training weight is used to initialize the model to improve the pre-training efficiency, so as to avoid the problem of consuming a large amount of computing power cost for pre-training the model from scratch. On this basis, the Chinese-CLIP model is pre-trained using a global-local image representation learning algorithm based on attention, as shown in FIG. 2. Figure 5 For each image-text pair, the image encoder in Chinese-CLIP encodes the image therein into global image representation and local image representation, and the formula is as follows:

[0089] ; (1)

[0090] wherein, denotes the image encoder of Chinese-CLIP, and denote the global image representation learning function and the local image representation learning function, respectively, that convert the image encoder output into the multi-modal feature space, is the input image, and the image global representation is a d -dimensional vector, and the image local representation contains m -dimensional vectors (corresponding to d image patches split from the image, respectively), Similarly, the text encoder in Chinese-CLIP encodes the text in the image-text pair into global text representation and local text representation, the formula is: m

[0091] ; (2)

[0092] wherein, denotes the text encoder of Chinese-CLIP, and denote the global text representation learning function and the local text representation learning function, respectively, that convert the text encoder output into the multi-modal feature space, is the input text, and the text global representation is a d -dimensional vector, and the text local representation contains w -dimensional vectors (corresponding to d tokens split from the text, respectively). w

[0093] Secondly, in order to make Chinese-CLIP learn the image global representation fully aligned with the text on the joint knowledge transmission line image-text dataset, while avoiding damaging the original pre-trained representation, the Chinese-CLIP is pre-trained using the image-text contrastive learning objective, and the loss calculation formula is:

[0094] (3)

[0095] (4)

[0096] (5)

[0097] wherein, and​​ the image global representation of the i-th image in the batch of training samples, i the image global representation of the i-th image in the batch of training samples, j the image global representation of the i-th image in the batch of training samples, the image global representation of the i-th image in the batch of training samples, the image global representation of the i-th image in the batch of training samples, i the text global representation of the i-th text in the batch of training samples, j the text global representation of the i-th text in the batch of training samples, b the batch size, τ 1 the temperature coefficient, the image-text global contrastive loss from image to text, the image-text global contrastive loss from image to text, the global contrastive loss.

[0098] Again, in order to make Chinese-CLIP learn the image local representation fully aligned with the text on the joint knowledge power line image-text dataset, Chinese-CLIP is pre-trained using an attention-based region-wordpiece contrastive learning objective, specifically, by calculating the dot product similarity between each image block feature vector and text wordpiece feature vector, the formula is:

[0099] (6)

[0100] wherein, denotes the dot product operation, is a similarity matrix, is the element in the i-th row and the j-th column of , indicating the similarity score between the i-th image block feature vector and the j-th text wordpiece feature vector, and for each column in i , the attention matrix j can be obtained by standardization, wherein the calculation formula of the attention weight is:

[0101] (7)

[0102] wherein, τ 2 the temperature coefficient, is the element in the i-th row and the j-th column of , and the context-aware image representation can be obtained by weighted summation of the image block feature vectors in i , and the calculation formula is: k (8)

[0103] (8)

[0104] wherein, is​​​​ the first image block feature vector in the j context-aware image representation based on the first token, i and then comparing the context-aware image representation based on each token with the text local representation to help the Chinese-CLIP capture finer-grained visual-linguistic alignment, and the loss calculation formula is:

[0105] (9)

[0106] (10)

[0107] (11)

[0108] wherein, the context-aware image representation based on the first token, j and are the feature vectors corresponding to the first token and the second token, respectively, i j and are the image-text local contrast losses from image to text and from text to image, respectively, is the local contrast loss.

[0109] Finally, the total loss of pre-training is the weighted sum of the global contrast loss and the local contrast loss, and the calculation formula is:

[0110] (12)

[0111] wherein, is the total loss of pre-training, and are the weights of the global contrast loss and the local contrast loss, respectively.

[0112] The method uses the Chinese-CLIP model pre-trained in the above step S6 to construct and fine-tune the final defect detector, achieving the effect of transferring the knowledge in the multi-modal large language model and the expert description text to the defect detection. In the step S7, the image encoder in the pre-trained Chinese-CLIP is used as the backbone network to construct the defect detector, and a small amount of labeled defect detection data is used to fine-tune the final defect detector, which specifically includes:

[0113] ​​​​Firstly, remove the text encoder from the pre-trained Chinese-CLIP model while retaining the image encoder as the backbone network of the defect detector, and add a simple feature pyramid network, a region proposal network and a bounding box head at the output end to construct a complete defect detector, wherein the simple feature pyramid network provides multi-level features for defect detection, and the region proposal network and the bounding box head are used to analyze the multi-level features and locate and classify the defects in the input image.

[0114] Then, the entire defect detector is fine-tuned with small-scale defect detection data, i.e., original inspection images with bounding box annotations, to finally transfer the knowledge obtained by pre-training to the defect detection task and obtain the final defect detector.

[0115] The detection effect of the method of the present application in the defect detection task of the power transmission line is as shown in (a) and (b) of Figure 6 The present application fully utilizes the visual and language information contained in the original power transmission line inspection data by using a multi-modal large language model, introduces a joint knowledge power transmission line image-text data set constructed by the generated description text provided by the multi-modal large language model and the expert description text provided by the inspection personnel to train a detector backbone network containing rich knowledge, thereby effectively alleviating the problem of insufficient defect-related visual information in the image; in addition, in view of the problem that the existing visual-language pre-training method is difficult to learn the fine-grained visual-language alignment in the power transmission line scene, the present application introduces an attention-based global-local image representation learning algorithm to pre-train the Chinese-CLIP model for the power transmission line scene, which helps to achieve finer visual-language alignment to better capture the knowledge contained in the text and has better transfer effect in the downstream detection task. It can be seen that the method of the present application can jointly utilize visual and language modalities to obtain more rich information, help the power transmission line defect detection model to overcome the problem of insufficient defect-related visual information in the original inspection image, thereby effectively improving the detection accuracy of the power transmission line defects.

[0116] The embodiment introduces a description text containing knowledge of different sources to provide additional semantic information for the detector, and pre-trains the detector backbone based on a global-local image representation learning algorithm based on attention to better learn global and local visual-linguistic alignment, so as to improve the precision of transmission line defect detection; cooperate with the global-local image representation learning algorithm based on attention, and achieve the goal of training a high-performance image encoder based on the joint knowledge of two sources. The image encoder obtained by pre-training can effectively alleviate the problem of insufficient defect-related visual information of the original inspection image, and the transmission line defect detector obtained by fine-tuning has higher precision, which has important research significance for intelligent inspection of transmission lines.

[0117] Embodiment two

[0118] The embodiment two of the present application introduces a transmission line defect detection method system.

[0119] As shown in a transmission line defect detection system shown in Figure 7 , comprising:

[0120] An acquisition module configured to acquire a transmission line inspection image and a transmission line instruction following data set;

[0121] A construction module configured to obtain a generated description text based on the acquired transmission line instruction following data set, combine the acquired transmission line inspection image to form an image-text pair containing multimodal large language knowledge; obtain an expert description text based on the acquired transmission line inspection image, match the generated description text and the expert description text, and obtain an image-text pair containing domain expert knowledge; and construct a joint knowledge transmission line image-text data set according to the image-text pair containing multimodal large language knowledge and the image-text pair containing domain expert knowledge;

[0122] A detection module configured to perform defect detection of a transmission line according to the constructed joint knowledge transmission line image-text data set and a defect detector.

[0123] The detailed steps are the same as the transmission line defect detection method provided in embodiment one, and will not be repeated here.

[0124] Embodiment three

[0125] The embodiment three of the present application provides a computer readable storage medium.

[0126] A computer readable storage medium having a program stored thereon, the program being executed by a processor to implement the steps in the transmission line defect detection method of the embodiment one of the present application.

[0127] The detailed steps are the same as the power transmission line defect detection method provided in Embodiment One, and are not described here again.

[0128] Embodiment Four

[0129] Embodiment Four of the present application provides an electronic device.

[0130] An electronic device includes a memory, a processor, and a program stored in the memory and running on the processor, and the processor implements the steps in the power transmission line defect detection method as described in Embodiment One of the present application when executing the program.

[0131] The detailed steps are the same as the power transmission line defect detection method provided in Embodiment One, and are not described here again.

[0132] Embodiment Five

[0133] Embodiment Five of the present application provides a computer program product.

[0134] A computer program product includes software code, and the program in the software code implements the steps in the power transmission line defect detection method as described in Embodiment One of the present application.

[0135] The detailed steps are the same as the power transmission line defect detection method provided in Embodiment One, and are not described here again.

[0136] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0137] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce the functions described in the flowcharts and / or block diagrams. Figure 1 one flow or multiple flows and / or blocks Figure 1means for performing the function specified by the block or blocks.

[0138] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow Figure 1 one or more flows and / or blocks Figure 1 means for performing the function specified by the block or blocks.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 one or more flows and / or blocks Figure 1 means for performing the function specified by the block or blocks.

[0140] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments without departing from the spirit and scope of the application. Therefore, it should be understood that the appended claims are intended to cover all such modifications and variations as falling within the scope of the application.

[0141] It is apparent that a large number of changes and modifications can be made to the application without departing from the spirit thereof. Accordingly, whatever comes within the scope of the claims and their equivalents are intended to fall into the scope of the application.

[0142] The above description is only preferred embodiments of the present application, and is not intended to limit the present application. The present application can be variously changed and modified by those skilled in the art without departing from the spirit and scope of the present application. Therefore, any modification, equivalent replacement, improvement, and the like made by those skilled in the art should be included in the scope of the present application.

Claims

1. A method of transmission line defect detection, characterized by, The application relates to a method for constructing a power transmission line image-text data set based on joint knowledge. The method comprises the following steps: acquiring a power transmission line inspection image and a power transmission line instruction following data set; generating a description text based on the acquired power transmission line instruction following data set, combining the acquired power transmission line inspection image to form an image-text pair containing multimodal large language knowledge; obtaining an expert description text based on the acquired power transmission line inspection image, matching the generated description text and the expert description text to obtain an image-text pair containing field expert knowledge; constructing a joint knowledge power transmission line image-text data set according to the image-text pair containing multimodal large language knowledge and the image-text pair containing field expert knowledge; performing defect detection on the power transmission line according to the constructed joint knowledge power transmission line image-text data set and a defect detector; 2. A method of detecting defects in a power transmission line as claimed in claim 1, wherein, in the process of constructing the joint knowledge power transmission line image-text data set according to the image-text pair containing multimodal large language knowledge and the image-text pair containing field expert knowledge, in order to enhance the supervision signal, the category label carried by each power transmission line image is filled into a predefined text template to obtain a description text based on the predefined template, and the obtained description text and the power transmission line image form a predefined template image-text pair; each image in the image part of the pre-training data set is traversed, and the category of the image-text pair formed by the image is viewed; if the image forms an image-text pair containing field expert knowledge, the image-text pair containing field expert knowledge is used as a sample of the pre-training data set; otherwise, one of the image-text pair containing multimodal large language model knowledge and the predefined template image-text pair formed by the image is randomly selected as a sample of the pre-training data set; a pre-training data set fusing multimodal large language model and field expert knowledge is constructed, that is, a joint knowledge power transmission line image-text data set is obtained.

3. A method of detecting defects in a power transmission line as claimed in claim 2, wherein, On the basis of the constructed joint knowledge power transmission line image-text data set, a global-local image representation learning algorithm based on attention is introduced to pre-train a visual basic model, the knowledge in the image-text data set is introduced into the visual basic model, and the pre-training of the visual basic model is completed. The defect detector is used for defect detection on the power transmission line, that is, the text encoder in the pre-trained visual basic model is removed, the image encoder is retained as the backbone network of the defect detector, a feature pyramid network, a region proposal network and a bounding box head are added at the output end of the backbone network to construct the defect detector, wherein the feature pyramid network provides multi-level features for defect detection, and the region proposal network and the bounding box head are used for analyzing the multi-level features and positioning and classifying defects in the input image; the defect detector is fine-tuned through the original inspection image with bounding box annotation, the knowledge obtained through pre-training is transferred to the defect detection task, and a final defect detector is obtained.

4. A method of detecting defects in a power transmission line as defined in claim 1, characterized in that, Filter high-quality images from the power transmission line instruction following data set, use the optimized InternVL to infer the filtered images, and obtain generated description text; obtain image-text pairs containing multimodal large language model knowledge according to the obtained power transmission line inspection images and their corresponding generated description text.

5. A method of detecting defects in a power transmission line as claimed in claim 4, wherein, Perform semantic matching on the generated description text corresponding to all filtered high-quality images containing defects and the expert description text provided by the inspection personnel for the original inspection image of the defect through ChatGLM, and obtain image-text pairs containing domain expert knowledge according to the defect image and the expert description text matched with the generated description text corresponding to the defect image.

6. A power line defect detection system characterized by, It comprises: An acquisition module configured to acquire power transmission line inspection images and a power transmission line instruction following data set; A construction module configured to obtain generated description text based on the acquired power transmission line instruction following data set, combine the acquired power transmission line inspection images to form image-text pairs containing multimodal large language knowledge, obtain expert description text based on the acquired power transmission line inspection images, match the generated description text and the expert description text, and obtain image-text pairs containing domain expert knowledge; and construct a joint knowledge power transmission line image-text data set according to the image-text pairs containing multimodal large language knowledge and the image-text pairs containing domain expert knowledge; A detection module configured to perform defect detection on the power transmission line according to the constructed joint knowledge power transmission line image-text data set and a defect detector. In the process of constructing the joint knowledge power transmission line image-text data set according to the image-text pairs containing multimodal large language knowledge and the image-text pairs containing domain expert knowledge, to enhance the supervision signal, the class label carried by each power transmission line image is filled into a predefined text template to obtain description text based on the predefined template, and the obtained description text and the power transmission line image form a predefined template image-text pair; each image in the image part of the pre-training data set is traversed to view the category of the image-text pair formed by the image, if the image forms an image-text pair containing domain expert knowledge, the image-text pair containing domain expert knowledge is used as a sample of the pre-training data set, otherwise, one of the image-text pairs containing multimodal large language model knowledge and the predefined template image-text pair formed by the image is randomly selected as a sample of the pre-training data set, and a pre-training data set fused from multimodal large language model and domain expert knowledge is constructed, i.e., a joint knowledge power transmission line image-text data set is obtained.

7. A computer readable storage medium having stored thereon a computer program, characterized in that The program is executed by the processor to realize the steps of the power transmission line defect detection method in any one of claims 1-5.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the program to realize the steps of the power transmission line defect detection method in any one of claims 1-5.

9. A computer program product comprising software code, characterized in that, The programs in the software code perform the steps of the power line defect detection method as claimed in any one of claims 1-5.

Images