Power transmission line defect identification method and system based on semantic prompt multi-mode fusion

Abstract

The invention provides a semantic prompt multi-mode fusion-based power transmission line defect identification method and system. The method comprises the steps of obtaining real-time image data transmitted when unmanned aerial vehicle inspection shooting is performed on a power transmission line to be identified, and real-time labeling information corresponding to the real-time image data; performing semantic extraction on the real-time annotation information to obtain corresponding real-time semantic extraction information; converting the real-time semantic extraction information into a real-time information embedding vector by adopting an embedding vector model; and inputting the real-time image data and the real-time information embedding vector into a pre-constructed optimal line defect identification model, wherein the output of the optimal line defect identification model is a power transmission line defect identification result. According to the method and the system, the semantic prompt and multi-modal fusion technology are utilized, the line defect identification is performed based on the image polled by the unmanned aerial vehicle through the established optimal line defect identification model, and the accuracy and the efficiency of power transmission line defect identification are improved.

Description

Technical Field

[0001] This invention relates to the field of overhead transmission line operation and maintenance technology, and more specifically, to a method and system for transmission line defect identification based on semantic prompting multimodal fusion. Background Technology

[0002] With the continuous expansion of power transmission lines, traditional manual inspection methods are no longer sufficient to meet the increasing maintenance demands, and intelligent drone inspections are becoming increasingly common. Using deep learning technology to perform target detection on massive amounts of inspection images and identify power transmission line defects is crucial for saving manpower and improving inspection efficiency.

[0003] Existing image recognition algorithms based on deep learning technology can be broadly classified into two categories: Two-Stage object recognition algorithms and One-Stage algorithms. Two-Stage algorithms first generate candidate regions using a Region Proposal Network (RPN) and then classify these regions using a convolutional neural network (CNN). Faster R-CNN is a typical example of this type. One-Stage algorithms, on the other hand, do not require the generation of candidate regions; they directly transform the object detection problem into a regression problem. The YOLO series of algorithms is a representative of this type. The differences between these two methods lead to performance variations. Two-Stage methods perform better in classification accuracy and localization precision, while One-Stage methods have an advantage in processing speed.

[0004] However, the above methods are not ideal for identifying images from drone inspections of power transmission lines. This is because there are many types of power transmission line towers, which vary in different regions and line grades, resulting in differences in equipment shape and defect presentation. For example, differences in regional landscapes and background environments lead to significant variations in the effectiveness of defect identification along power transmission lines; furthermore, the different bird-proofing facilities used on different sections of power transmission lines can also cause a decrease in algorithm recognition rates due to variations in defect presentation. Summary of the Invention

[0005] To address the issue of unsatisfactory accuracy and efficiency of existing deep learning-based target recognition algorithms for UAV inspection images of power transmission lines, this invention provides a method and system for power transmission line defect identification based on semantic prompting and multimodal fusion.

[0006] According to one aspect of the present invention, the present invention provides a method for identifying transmission line defects based on semantic prompting multimodal fusion, the method comprising:

[0007] Acquire real-time image data transmitted during drone inspection and photography of the power transmission line to be identified, as well as real-time annotation information corresponding to the real-time image data;

[0008] Semantic extraction is performed on the real-time annotation information to obtain the corresponding real-time semantic extraction information;

[0009] The real-time semantic extraction information is transformed into a real-time information embedding vector using an embedding vector model.

[0010] The real-time image data and the real-time information are embedded into a vector and input into a pre-constructed optimal line defect identification model. The output of the optimal line defect identification model is the transmission line defect identification result.

[0011] According to another aspect of the present invention, the present invention provides a transmission line defect identification system based on semantic prompting multimodal fusion, the system comprising:

[0012] According to one aspect of the present invention, a transmission line defect identification system based on semantic prompting multimodal fusion is provided, the system comprising:

[0013] The data acquisition module is used to acquire real-time image data transmitted during drone inspection and photography of the power transmission line to be identified, as well as real-time annotation information corresponding to the real-time image data.

[0014] The semantic extraction module is used to extract semantic information from the real-time annotation information and obtain corresponding real-time semantic extraction information.

[0015] The embedding vector module is used to convert the real-time semantic extraction information into a real-time information embedding vector using an embedding vector model.

[0016] The defect identification module is used to embed the real-time image data and the real-time information into a pre-constructed optimal line defect identification model. The output of the optimal line defect identification model is the transmission line defect identification result.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, the storage medium storing a computer program that, when executed by a processor, implements the methods described in any of the above aspects of the present invention.

[0018] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the method described in any of the preceding aspects of the present invention.

[0019] The present invention discloses a method and system for identifying transmission line defects based on semantic prompting and multimodal fusion. The method includes acquiring real-time image data transmitted during UAV inspection of the transmission line to be identified, and real-time annotation information corresponding to the real-time image data; performing semantic extraction on the real-time annotation information to obtain corresponding real-time semantic extraction information; converting the real-time semantic extraction information into real-time information embedding vectors using an embedding vector model; and inputting the real-time image data and the real-time information embedding vectors into a pre-constructed optimal line defect identification model, the output of which is the transmission line defect identification result. This method and system utilize semantic prompting and multimodal fusion technology, and through the established optimal line defect identification model, identify line defects based on UAV inspection images, improving the accuracy and efficiency of transmission line defect identification and making it more suitable for practical application scenarios. Attached Figure Description

[0020] Exemplary embodiments of the present invention can be more fully understood by referring to the following figures:

[0021] Figure 1 This is a flowchart of a transmission line defect identification method based on semantic prompting multimodal fusion according to a preferred embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram of the structure of the optimal line defect identification model according to a preferred embodiment of the present invention;

[0023] Figure 3 This is a schematic diagram of the structure of a transmission line defect identification system based on semantic prompting multimodal fusion according to a preferred embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of the structure of an electronic device according to a preferred embodiment of the present invention. Detailed Implementation

[0025] Exemplary embodiments of the invention will now be described with reference to the accompanying drawings. However, the invention may be embodied in many different forms and is not limited to the embodiments described herein. These embodiments are provided to fully and completely disclose the invention and to fully convey its scope to those skilled in the art. The terminology used in the exemplary embodiments illustrated in the drawings is not intended to limit the invention. In the drawings, the same units / elements are referred to by the same reference numerals.

[0026] Unless otherwise stated, the terms used herein (including technical terms) have their common meaning as understood by one of ordinary skill in the art. Furthermore, it is understood that terms defined in commonly used dictionaries should be understood to have a meaning consistent with the context of their relevant field, and not to be interpreted as having an idealized or overly formal meaning.

[0027] Exemplary methods

[0028] Figure 1 This is a flowchart of a transmission line defect identification method based on semantic prompting multimodal fusion according to a preferred embodiment of the present invention. Figure 1 As shown, the transmission line defect identification method based on semantic prompting multimodal fusion described in this preferred embodiment starts from step 101.

[0029] In step 101, real-time image data transmitted during drone inspection and photography of the power transmission line to be identified, as well as real-time annotation information corresponding to the real-time image data, are acquired.

[0030] In existing technologies, the naming convention for transmission line defect images follows the format of "Voltage Level_Line Name_Pole Number (Phase)_Description Information_Inspection Date". The transmission line voltage level is a fixed value, with common voltage levels including 110kV, 220kV, 330kV, 500kV, 750kV, 1100kV, and DC ±400kV, ±500kV, ±660kV, and ±800kV. The line name corresponds to the transmission line in the relevant region; for example, #278 of the Tuha line corresponds to the shooting location in Xinjiang, and #0459 of the Jiangcheng line corresponds to the shooting location in Hunan. The description information includes the pole type and the shooting method. The types of poles and towers include 11 types: single-circuit straight-line "wine glass" towers for AC lines, single-circuit straight-line "cathead" towers for AC lines, single-circuit straight-line towers for DC lines, single-circuit tension towers for DC lines, double-circuit straight-line towers for AC lines, double-circuit tension towers for AC lines, double-circuit straight-line towers for DC lines, transposition towers, compact towers, single-circuit tension "tough guy" towers for AC lines, and guyed towers. The number of images captured for each type of pole and tower varies. Taking the UAV inspection and shooting rules for double-circuit tension towers for AC lines as an example, a total of 54 shooting points at 15 locations are captured, as shown in Table 1.

[0031] Table 1

[0032]

[0033]

[0034]

[0035]

[0036]

[0037]

[0038]

[0039]

[0040]

[0041]

[0042]

[0043]

[0044]

[0045]

[0046] As can be seen from Table 1, for a given type of tower, the inspection location and shooting point in the real-time annotation information of the image can be easily confirmed by using the drone inspection and shooting rules.

[0047] Preferably, before acquiring the image data transmitted during UAV inspection of the transmission line to be identified, and the corresponding annotation information of the image data, the process further includes establishing an optimal line defect identification model, wherein:

[0048] An initial line defect identification model was established based on the VIT model;

[0049] Acquire historical image data and the corresponding historical annotation information and defect types;

[0050] According to the custom semantic extraction rules, the historical annotation information is semantically extracted to obtain the corresponding historical semantic extraction information;

[0051] The historical semantic extraction information is transformed into historical information embedding vectors using an embedding vector model;

[0052] A task dataset is constructed based on the historical image data and its corresponding historical information embedding vectors and defect types.

[0053] The initial line defect identification model is trained, tested, and validated based on the data in the task dataset to generate the optimal line defect identification model.

[0054] ViT (Vision Transformer) has demonstrated strong potential and application value in power transmission line defect identification. This technology utilizes deep learning algorithms to efficiently process and analyze images of power transmission lines, thereby achieving accurate identification of various defects. The core idea of ​​Transformer lies in effectively modeling the dependency relationship between any two points in the input sequence as a fixed-length computational path through a self-attention mechanism, thus eliminating the dependence on a specific order of operations found in traditional RNNs and CNNs. This innovative design enables Transformer to exhibit excellent scalability when processing long sequences, as the dependency relationship between any two points in the sequence can be directly calculated without passing information layer by layer. The flexible computation mode of the Transformer architecture allows the model to handle even longer sequences while maintaining low computational complexity.

[0055] In this preferred embodiment, the initial line defect identification model based on the VIT model is first pre-trained on a large dataset (such as ImageNet) to learn a general representation of image data. Then, it is fine-tuned on a corresponding task dataset to adapt to specific task requirements. Specifically, the task dataset includes historical image data acquired by UAV inspections of transmission lines, as well as historical information embedding vectors and defect types obtained based on historical annotation information corresponding to the historical image data. By inputting the historical image data and historical information embedding vectors into the initial line defect identification model for training, defect type prediction results are obtained. These prediction results are then compared with actual defect types to test and verify the initial line defect identification model, thereby obtaining the optimal initial line defect identification model.

[0056] In step 102, semantic extraction is performed on the real-time annotation information to obtain the corresponding real-time semantic extraction information.

[0057] Preferably, semantic extraction is performed on the real-time annotation information to obtain corresponding real-time semantic extraction information, including:

[0058] The real-time annotation information is decomposed into a dictionary to obtain corresponding semantic tags, which include voltage level, line name and shooting location;

[0059] The inspection area name of the drone is determined based on the route name;

[0060] The voltage level, inspection area name, and shooting location are used as real-time semantic extraction information corresponding to the real-time image data.

[0061] As described above, dictionary decomposition of real-time annotation information can yield corresponding semantic tags. The line name in the semantic tags can be used to identify the region where the line is located. For example, the region where the line is located can be identified as Xinjiang based on the Tuha Line 278#. In addition, the voltage level and shooting location included in the semantic tags can be used to obtain complete real-time semantic extraction information, thereby accurately locating the transmission line.

[0062] Figure 2 This is a schematic diagram of the structure of the optimal line defect identification model according to a preferred embodiment of the present invention. Figure 2 As shown, the optimal line defect identification model of this invention is similar to the conventional VIT model in that it segments the images taken by UAVs during transmission line inspections to generate a series of small images. Each image block is then flattened and embedded using a linear mapping region block. After introducing position encoding, a series of ordered vector sequences are formed. Next, the embedded image sequences are input into an encoder. Through the encoder, VIT can capture the long-distance dependencies between different parts of the image, thereby achieving a deep understanding and analysis of the global information of the image. However, unlike the conventional VIT model, this preferred embodiment first uses a dictionary based on the annotation information "±1100kV Jiquan Line #3367 Tower Pole II Insulator Conductor End Hanging Point Connecting Plate". The obtained line name "Jiquan Line #3367" was decomposed to determine that the image shooting area was Xinjiang. Then, the voltage level "±1100kV" and the shooting location "insulator conductor end hanging point" obtained from the annotation information were combined to generate semantic prompt information including three pieces of information: voltage level, province information, and shooting location. After generating an information embedding vector through the embedding vector model, the information embedding vector was embedded into the ViT model. This allows the encoder to not only complete image recognition and thus identify transmission line defects, but also integrate the information embedding vector generated by semantic extraction information. Furthermore, when outputting the type of transmission line defect, it can output the area and accurate location of the defect, thereby improving the accuracy and efficiency of transmission line defect identification and making it more suitable for practical application scenarios.

[0063] In step 103, the real-time semantic extraction information is transformed into a real-time information embedding vector using an embedding vector model.

[0064] Preferably, the real-time semantic extraction information is transformed into a real-time information embedding vector using an embedding vector model, wherein the embedding vector model is the Word2Vec model.

[0065] In this preferred embodiment, there are no restrictions on the embedding vector model that converts the semantic extracted information into an information embedding vector. In addition to the Word2Vec model, GloVe, FastText, ELMo, and BERT models are all acceptable.

[0066] Furthermore, the method of obtaining the corresponding historical semantic extraction information based on the real-time annotation information and transforming the historical semantic extraction information into a historical information embedding vector using an embedding vector model is the same as the method of obtaining the real-time semantic extraction information and the real-time information embedding vector, and will not be described again here.

[0067] In step 104, the real-time image data and the real-time information are embedded into a vector and input into a pre-constructed optimal line defect identification model. The output of the optimal line defect identification model is the transmission line defect identification result.

[0068] The preferred embodiment of the transmission line defect identification method based on semantic prompting and multimodal fusion utilizes semantic prompting and multimodal fusion technology. Through the establishment of an optimal line defect identification model, it identifies line defects based on images from UAV inspections, improving the accuracy and efficiency of transmission line defect identification and making it more suitable for practical application scenarios.

[0069] Exemplary System

[0070] Figure 3 This is a schematic diagram of the structure of a transmission line defect identification system based on semantic prompting multimodal fusion according to a preferred embodiment of the present invention. Figure 3 As shown, the transmission line defect identification system 300 based on semantic prompting multimodal fusion according to this preferred embodiment includes:

[0071] The data acquisition module 301 is used to acquire real-time image data transmitted during drone inspection and photography of the power transmission line to be identified, as well as real-time annotation information corresponding to the real-time image data.

[0072] The semantic extraction module 302 is used to extract semantic information from the real-time annotation information and obtain corresponding real-time semantic extraction information.

[0073] Embedded vector module 303 is used to convert the real-time semantic extraction information into a real-time information embedded vector using an embedded vector model;

[0074] The defect identification module 304 is used to embed the real-time image data and the real-time information into a pre-constructed optimal line defect identification model. The output of the optimal line defect identification model is the transmission line defect identification result.

[0075] Preferably, the system further includes a model building module for building an optimal line defect identification model, wherein:

[0076] An initial line defect identification model was established based on the VIT model;

[0077] Acquire historical image data and the corresponding historical annotation information and defect types;

[0078] According to the custom semantic extraction rules, the historical annotation information is semantically extracted to obtain the corresponding historical semantic extraction information;

[0079] The historical semantic extraction information is transformed into historical information embedding vectors using an embedding vector model;

[0080] A task dataset is constructed based on the historical image data and its corresponding historical information embedding vectors and defect types.

[0081] The initial line defect identification model is trained, tested, and validated based on the data in the task dataset to generate the optimal line defect identification model.

[0082] Preferably, the semantic extraction module performs semantic extraction on the real-time annotation information to obtain corresponding real-time semantic extraction information, including:

[0083] The real-time annotation information is decomposed into a dictionary to obtain corresponding semantic tags, which include voltage level, line name and shooting location;

[0084] The inspection area name of the drone is determined based on the route name;

[0085] The voltage level, inspection area name, and shooting location are used as real-time semantic extraction information corresponding to the real-time image data.

[0086] Preferably, the embedding vector module uses an embedding vector model to convert the real-time semantic extraction information into a real-time information embedding vector, wherein the embedding vector model is the Word2Vec model.

[0087] In summary, the transmission line defect identification system based on semantic prompting multimodal fusion described in this invention extracts the semantic information of the real-time annotation information corresponding to the real-time image data transmitted during UAV inspection of the transmission line to be identified, converts it into an information embedding vector, and inputs it together with the real-time image data into the constructed optimal line defect identification model to determine the transmission line defect identification result. The steps are the same as those of the transmission line defect identification method based on semantic prompting multimodal fusion, and the technical effects achieved are also the same. Therefore, they will not be repeated here.

[0088] Exemplary electronic devices

[0089] Figure 4 This is a schematic diagram of the structure of an electronic device according to a preferred embodiment of the present invention. Figure 4As shown, the electronic device includes one or more processors 401 and memory 402.

[0090] The processor 401 may be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0091] The memory 402 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 401 may execute the program instructions to implement the semantic-cue multimodal fusion-based transmission line defect identification method of the various embodiments disclosed above, and / or other desired functions. In one example, the electronic device may also include an input device 403 and an output device 404, these components being interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0092] In addition, the input device 403 may also include, for example, a keyboard, a mouse, etc.

[0093] The output device 404 can output various information to the outside. The output device 404 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0094] Of course, for the sake of simplicity, Figure 4 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0095] Exemplary computer program products and computer-readable storage media

[0096] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the semantically prompted multimodal fusion-based transmission line defect identification method according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0097] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0098] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the semantically prompted multimodal fusion-based transmission line defect identification method according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0099] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0100] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0101] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0102] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0103] The apparatus and methods of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0104] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps are decomposable and / or recombinable. Such decomposition and / or recombination should be considered equivalent to the present disclosure. The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0105] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A power transmission line defect identification method based on semantic prompt multi-modal fusion, characterized in that, The method comprises: acquiring real-time image data transmitted during unmanned aerial vehicle inspection and photographing of a power transmission line to be identified, and real-time labeling information corresponding to the real-time image data; performing semantic extraction on the real-time labeling information to acquire corresponding real-time semantic extraction information; using an embedding vector model to convert the real-time semantic extraction information into real-time information embedding vectors; inputting the real-time image data and the real-time information embedding vectors into a pre-constructed optimal line defect identification model, and the output of the optimal line defect identification model is a power transmission line defect identification result.

2. The method of claim 1, wherein, Before acquiring image data transmitted during unmanned aerial vehicle inspection and photographing of a power transmission line to be identified, and labeling information corresponding to the image data, an optimal line defect identification model is established, wherein: an initial line defect identification model is established based on a VIT model; acquiring historical image data, historical labeling information corresponding to the historical image data, and defect types; performing semantic extraction on the historical labeling information according to a self-defined semantic extraction rule to acquire corresponding historical semantic extraction information; using an embedding vector model to convert the historical semantic extraction information into historical information embedding vectors; constructing a task data set according to the historical image data, the corresponding historical information embedding vectors, and the defect types; training, testing, and verifying the initial line defect identification model based on data in the task data set to generate an optimal line defect identification model.

3. The method of claim 1, wherein, The semantic extraction on the real-time labeling information to acquire corresponding real-time semantic extraction information comprises: performing dictionary decomposition on the real-time labeling information to acquire corresponding semantic labels, wherein the semantic labels comprise voltage levels, line names, and photographing points; determining a patrol area name of the unmanned aerial vehicle according to the line name; using the voltage levels, the patrol area name, and the photographing points as the real-time semantic extraction information corresponding to the real-time image data.

4. The method of claim 1, wherein, The embedding vector model is a Word2Vec model.

5. A power transmission line defect identification system based on semantic prompt multi-modal fusion, characterized in that, The system comprises: a data acquisition module configured to acquire real-time image data transmitted during unmanned aerial vehicle inspection and photographing of a power transmission line to be identified, and real-time labeling information corresponding to the real-time image data; a semantic extraction module configured to perform semantic extraction on the real-time labeling information to acquire corresponding real-time semantic extraction information; an embedding vector module configured to use an embedding vector model to convert the real-time semantic extraction information into real-time information embedding vectors; a defect identification module configured to input the real-time image data and the real-time information embedding vectors into a pre-constructed optimal line defect identification model, and the output of the optimal line defect identification model is a power transmission line defect identification result.

6. The system of claim 5, wherein, The system further comprises a model establishment module configured to establish an optimal line defect identification model, wherein: an initial line defect identification model is established based on a VIT model; acquiring historical image data, historical labeling information corresponding to the historical image data, and defect types; According to the self-defined semantic extraction rule, the historical annotation information is subjected to semantic extraction to obtain corresponding historical semantic extraction information; The historical semantic extraction information is converted into historical information embedding vectors by using an embedding vector model; A task data set is constructed according to the historical image data, the corresponding historical information embedding vectors and the defect types; The initial line defect recognition model is trained, tested and verified based on the data in the task data set to generate an optimal line defect recognition model.

7. The system of claim 5, wherein, The semantic extraction module extracts semantics from the real-time annotation information to obtain corresponding real-time semantic extraction information, including: The real-time annotation information is subjected to dictionary decomposition to obtain corresponding semantic labels, including voltage levels, line names and shooting points; The line name is used to determine the inspection area name of the unmanned aerial vehicle; The voltage level, the inspection area name and the shooting point are used as the real-time semantic extraction information corresponding to the real-time image data.

8. The system of claim 5, wherein, The embedding vector module converts the real-time semantic extraction information into real-time information embedding vectors by using an embedding vector model, wherein the embedding vector model is a Word2Vec model.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by a processor to implement the steps of the method of any one of claims 1 to 4.

10. An electronic device, comprising: Comprise: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the steps of the method of any one of claims 1 to 4.

Images