Power transmission line operation and maintenance method and device based on multi-modal feature fusion and computer readable storage medium

Abstract

The invention belongs to the technical field of specific computer models based on power transmission line detection, and particularly relates to a power transmission line operation and maintenance method and device based on multi-modal feature fusion and a computer readable storage medium. The method comprises the steps of obtaining a hidden danger image in a power transmission channel scene and power transmission channel dimension information, constructing an image-text sample set, modeling based on an adaptive visual feature branch, an adaptive text feature branch and a cross-modal fusion module, using the cross-modal fusion module to associate the visual feature branch with the text feature branch, and obtaining the hidden danger image in the power transmission channel scene. And performing modal feature alignment and gradient updating calculation by using KL divergence and a Wasserstein distance to obtain the power transmission line operation and maintenance method based on multi-modal feature fusion. According to the method, the problems that the difficulty of accurately extracting key features from the image is increased and the detection accuracy and efficiency are relatively low because the image possibly contains a large amount of background information and interference factors are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of specific computer models based on transmission line detection, and more specifically, to a transmission line operation and maintenance method, device and computer-readable storage medium based on multimodal feature fusion. Background Art

[0002] With the continuous deepening of my country's power grid construction, the contradiction between the demand for transmission line construction and the high cost of manual operation and maintenance is gradually increasing. Transmission visualization equipment can effectively alleviate the contradiction between the demand for power grid construction and the cost of manual operation and maintenance. It has the advantages of low cost, simple operation, and no environmental impact. In the process of visual shooting and inspection of transmission lines, a series of technical and practical challenges are faced. First of all, the environment of transmission lines is complex and changeable, and may exist under various geographical and climatic conditions, which requires the inspection system to have a high degree of adaptability and robustness.

[0003] Chinese patent document CN117808826A discloses a directional image segmentation method for remote sensing images: 1) using the pre-trained language model BERT and the visual model Swin-Transformer to obtain language features and initial visual features; 2) using the intra-scale feature interaction module to model fine-grained local visual features; 3) using cross-layer multi-scale attention to fuse and interact visual features of different granularities; 4) extracting angle information based on features after sufficient interaction between scales, and decoding features using an adaptive rotation dynamic convolution method that dynamically resamples convolution kernel weights according to the angle to obtain a predicted segmentation mask; 5) proposing a new directional remote sensing image segmentation dataset annotated using a semi-automatic method.

[0004] In summary, how to provide an accurate and efficient method for identifying hidden dangers in transmission channels during the operation and maintenance inspection of transmission lines is an urgent problem to be solved by technicians in this field. The detection of these hidden dangers requires the inspection system to have accurate image recognition and analysis capabilities. However, due to the low frequency of occurrence of some types of hidden dangers, and the fact that they may be affected by factors such as angle, light, and occlusion in actual shooting, it is very difficult to collect relevant samples. This directly affects the training effect of deep learning algorithms, because deep learning models usually require a large number of labeled samples for effective training. In addition, the image data of transmission line inspections is characterized by high dimensionality and high complexity, and the images may contain a large amount of background information and interference factors, which increases the difficulty of accurately extracting key features from the images. In order to improve the accuracy and efficiency of detection, it is necessary to develop more advanced image processing and feature extraction algorithms, as well as more sophisticated classification systems, for better distinction and identification. Summary of the invention

[0005] The present invention aims to overcome at least one of the above-mentioned defects of the prior art, and provide a transmission line operation and maintenance method based on multimodal feature fusion to identify different types of hidden dangers.

[0006] The present invention adopts the idea of ​​a multimodal fusion model based on transfer learning and domain adaptation to solve the problem. It combines different types of data, such as surveillance images, historical warning fields, channel scene information, weather season information, voltage levels, etc., to provide richer environmental information, which helps to improve the accuracy of hidden danger detection. It also uses the method of transfer learning to migrate the model pre-trained on a large-scale data set to a specific power transmission line inspection task, reducing the reliance on a large number of labeled samples. At the same time, through domain adaptation technology, the model can better adapt to different power transmission line environments.

[0007] The detailed technical scheme of the present invention is as follows: A transmission line operation and maintenance method based on multimodal feature fusion, the method comprising: A transmission line operation and maintenance method based on multimodal feature fusion, characterized by: S1: Obtain hidden danger images in the transmission channel scenario, collect text information corresponding to the hidden danger images, construct image-text sample pairs based on the images and corresponding information, and save the samples in dictionary format; Furthermore, the text information includes but is not limited to weather information, scene information, season information and voltage level information.

[0008] S2: Construct a multimodal transmission line inspection model, including an adaptive visual feature extraction branch, an adaptive text feature extraction branch, a first feature selection layer, a cross-modal fusion module, a second feature selection layer, a pooling layer, and a classifier; The adaptive visual feature extraction branch extracts visual features, and the adaptive text feature extraction branch extracts text features; The first feature selection layer is used to select appropriate visual features and text features, and to combine the two features; The cross-modal fusion module fuses the visual feature and the text feature vector to obtain a fused feature map; The second feature selection layer is used to select key features of the feature layer; The pooling layer is used to thin out features, filter out invalid features, and retain key features; The classifier classifies the target based on the fully connected layer; S3: Training of a multimodal transmission line inspection model, including training of an adaptive visual feature extraction branch, an adaptive text feature extraction branch, and a cross-modal fusion module; S4: Update the parameters of the multimodal transmission line inspection model, coordinate the input of image-text data, use KL divergence and Wasserstein distance to align modal features, perform gradient backpropagation and parameter update, and gradually align the modes based on the distance metric to achieve the best modal adaptation combination; S5: The multimodal transmission line inspection model obtained in the above steps is used to detect hidden dangers in the transmission channel scenario.

[0009] Furthermore, the adaptive visual feature extraction branch uses an adaptive visual feature extraction branch improved based on Swin Transformer: First, the image is divided into multiple small blocks, and a hierarchical Transformer structure is used to capture local and global visual information; Input each small block of the image into multiple stacked core component Blocks, including LayerNorm, MLP, Window Attention, and Shifted Window Attention; Then, the output features of the previous Block module are fused with the output features of the next Block module by means of feature stacking and concatenation. Finally, the fusion results are passed through the encoder to obtain the visual features of each image.

[0010] Furthermore, the adaptive text feature extraction branch uses an adaptive text feature extraction branch based on BERT improvement: The BERT model is changed to a four-way feature adaptive interaction network, where the “four-way” refers to the context, grammar, semantics, and domain features of the text; The context adaptation adjusts the sensitivity to context information according to the context environment of different texts through an adaptive interaction module; The grammar adaptation refers to the model identifying and emphasizing key components in the grammatical structure; The semantic adaptation refers to understanding the meaning of a sentence by analyzing the semantic relationship between words; The domain adaptation refers to the model recognizing and adapting to the terms and expressions of the preset domain.

[0011] Furthermore, before inputting the adaptive text feature extraction branch based on BERT improvement, it also includes: Sentence preprocessing is performed, including but not limited to word segmentation and conversion into word vectors. At the same time, in order to capture the four-way features, additional feature representations are introduced, including but not limited to the embedding of domain-specific vocabulary and the representation of grammatical structure. Ultimately, the output of the text branch is a rich representation of each text feature that contains context, grammar, semantics, and domain information. In downstream tasks, these features can be used for classification, entity recognition, sentiment analysis, and other tasks. The traditional BERT is a deep bidirectional structure that can combine the contextual information on the left and right sides to understand the meaning of words when processing text. The improved structure is expanded from bidirectional to four-directional, which means that when capturing text features, it is not limited to the front and back directions, but comprehensively analyzes text information from four directions or more. The four-way feature adaptive interaction can automatically adjust the weights and interaction methods of features in each direction according to the characteristics and semantic relationships of the text. For example, for some words with complex semantic relationships or polysemy, the four-way structure can comprehensively judge their exact meaning from the features in different directions, making the feature extraction more comprehensive and accurate.

[0012] Furthermore, the cross-modal fusion module uses a cross-modal fusion module based on Self-Attention to learn the correlation between visual features and text features through the Self-Attention layer, including: For the visual feature vector and the text feature vector, the dot product between them is calculated and then normalized to the attention weight through the softmax function; For visual features, each feature element is multiplied by the corresponding attention weight, and then summed to obtain the weighted visual feature; similarly, for text features, each feature element is multiplied by the corresponding attention weight, and then summed to obtain the weighted text features; Finally, the weighted features of the two modalities are stacked to obtain a fused feature map.

[0013] This can highlight features that are useful for modal fusion and suppress less relevant features. This process not only retains the feature information of the original modality, but also enhances the expressiveness of the features through cross-modal association learning. Through this deep multimodal feature fusion, the algorithm of the present invention can more comprehensively understand the complex scenarios of transmission lines and improve the accuracy and robustness of hidden danger identification. This fusion strategy is not only innovative in theory, but also shows significant performance advantages in practical applications, providing strong technical support for the intelligent operation and maintenance of transmission lines.

[0014] Furthermore, before inputting the adaptive text feature extraction branch based on BERT improvement, it also includes: Sentence preprocessing is performed, including but not limited to word segmentation and conversion into word vectors. At the same time, in order to capture the four-way features, additional feature representations are introduced, including but not limited to the embedding of domain-specific vocabulary and the representation of grammatical structure. Furthermore, the S3 specifically includes: S31: Freeze the adaptive visual feature extraction branch and the cross-modal fusion module, and train the adaptive text feature extraction branch separately; S32: Refreeze the adaptive text feature extraction branch and the cross-modal fusion module, and train the adaptive visual feature extraction branch separately; S33: Refreeze the adaptive visual feature extraction branch and the adaptive text feature extraction branch, and train the cross-modal fusion module separately; S34: Completely unfreeze the visual feature branch, text feature branch and cross-modal fusion module, and use image-text pairs for comprehensive fine-tuning.

[0015] Furthermore, the S4 specifically includes: S41: Coordinate input of image-text data, inputting the image data into the visual branch, and inputting the text data into the text branch; S42: Modality feature alignment using KL divergence and Wasserstein distance: (1) The KL divergence formula is as follows: (1); In formula (1), T(cls) represents the category prediction probability value output by the visual feature branch, S(cls) represents the category prediction probability value output by the text feature branch, and N refers to the number of categories; (2) The Wasserstein distance formula is as follows: (2); In formula (2), is the joint distribution of all features The marginal distribution of the joint distribution of features includes P and Q. It is the distance measure between two points x and y in X in the feature space; (3) For and Increase the corresponding weight coefficients respectively and add them as the loss function to enhance the effect of modal alignment; S43: Perform gradient backpropagation and parameter update to find the best modality adaptation combination.

[0016] In another aspect of the present invention, there is also provided an apparatus comprising: at least one processor; and A memory storing instructions, which, when executed by the at least one processor, enables the at least one processor to execute the transmission line operation and maintenance method based on multimodal feature fusion as described above.

[0017] In another aspect of the present invention, a machine-readable storage medium is provided, which stores executable instructions, and when the instructions are executed, the machine executes the transmission line operation and maintenance method based on multimodal feature fusion as described above.

[0018] Compared with the prior art, the present invention has the following beneficial effects: The present invention designs a transmission line operation and maintenance method, device and computer-readable storage medium based on multimodal feature fusion. By constructing an image-text sample set, the core of the modeling includes an improved adaptive visual feature branch based on Swin Transformer, an improved adaptive text feature branch based on Bert and a cross-modal fusion module based on Self-Attention, and uses KL divergence and Wasserstein distance for modal feature alignment, which can effectively and accurately extract key features from images, and does not require a large number of labeled samples, while improving the accuracy and efficiency of hidden danger identification during transmission channel inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic flow chart of the method of the present invention.

[0020] Figure 2 It is a schematic diagram of the structure of the multi-modal transmission line inspection model described in this embodiment. DETAILED DESCRIPTION

[0021] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0022] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0023] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0024] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0025] Example 1 Ginseng Figure 1 This embodiment provides a transmission line operation and maintenance method based on multimodal feature fusion, the method comprising: S1: Obtain the hidden danger image in the transmission channel scene through the transmission visualization equipment, collect the text information corresponding to the hidden danger image, build the image-text sample pair by combining the image and the corresponding information, and save the sample in dictionary format; The text information includes but is not limited to weather information, scene information, season information and voltage level information.

[0026] S2: Construct a multimodal power transmission line inspection model; the model adopts an innovative deep learning architecture that combines the advantages of the Swin Transformer and BERT models and introduces a cross-modal fusion strategy based on the Self-Attention mechanism. Furthermore, the present invention adaptively adjusts these network structures to make them more suitable for tasks in the power transmission field. For the Swin Transformer model and the Bert model, features at different levels are cross-layered to obtain richer feature representations.

[0027] like Figure 2 As shown, the multimodal transmission line inspection model includes an adaptive visual feature extraction branch, an adaptive text feature extraction branch, a first feature selection layer, a cross-modal fusion module, a second feature selection layer, a pooling layer and a classifier; The adaptive visual feature extraction branch extracts visual features, and the adaptive text feature extraction branch extracts text features; The first feature selection layer selects visual features and text feature vectors and then concatenates them; The cross-modal fusion module fuses the spliced ​​visual features and text feature vectors to obtain a fused feature map; The second feature selection layer is used to extract key features of the fused feature map; The pooling layer is used to thin out the key features, filter out invalid features of the fused feature map, and retain the key features; The classifier performs final classification based on the fully connected layer.

[0028] Preferably, the adaptive visual feature extraction branch uses an adaptive visual feature extraction branch based on Swin Transformer improvement, such as Figure 2 As shown: First, the image is divided into multiple patches and a hierarchical Transformer structure is used to capture local and global visual information. Each small block of the image is input into multiple stacked core component blocks, including LayerNorm, MLP (Multi-layer Perceptron), Window Attention and Shifted Window Attention. These blocks are stacked to achieve layer-by-layer feature extraction and conversion; Then, the output features of the previous Block module are fused with the output features of the next Block module by means of feature stacking and concatenation. These feature maps can capture fine-grained information in the image, such as object edges, textures, and shapes, providing a basis for subsequent feature fusion. Finally, the fusion results are passed through the encoder to obtain the visual features of each image.

[0029] Preferably, the adaptive text feature extraction branch uses an adaptive text feature extraction branch improved based on BERT (Bidirectional Encoder Representations from Transformers), such as Figure 2 As shown: The BERT model is changed to a four-way feature adaptive interaction network, where the “four-way” refers to the context, grammar, semantics, and domain features of the text; Context adaptation adjusts the sensitivity to context information according to the context of different texts through the adaptive interaction module; Syntactic adaptation means that the BERT model recognizes and emphasizes key components in grammatical structures, such as subject, predicate, and object; Semantic adaptation refers to understanding the meaning of a sentence by analyzing the semantic relationship between words; Domain adaptation refers to the BERT model recognizing and adapting to the terms and expressions of the preset domain.

[0030] Preferably, the cross-modal fusion module uses a cross-modal fusion module based on Self-Attention to learn the correlation between visual features and text features through a Self-Attention layer, including: For the visual feature vector and the text feature vector, calculate the dot product between them, and then normalize them into attention weights through the softmax function. For example, the visual features include visual feature vectors A~H, and the text features include text feature vectors N~Z. Calculate the dot product of the visual feature vectors A~H and the text feature vectors N~Z respectively, and get the dot product of any two visual feature vectors and the text feature vector and their attention weights; For visual features, each feature element is multiplied by the corresponding attention weight, and then summed to obtain the weighted visual feature; similarly, for text features, each feature element is multiplied by the corresponding attention weight, and then summed to obtain the weighted text features; Finally, the weighted features of the two modalities are stacked to obtain a fused feature map, thereby solving the problem of usually requiring a large number of labeled samples for effective training.

[0031] Preferably, before inputting the adaptive text feature extraction branch improved based on BERT, it also includes: Sentence preprocessing is performed, including but not limited to word segmentation and conversion into word vectors. At the same time, in order to capture four-way features, additional feature representations are introduced, including but not limited to the embedding of domain-specific vocabulary and the representation of grammatical structure.

[0032] S3: Training of a multimodal transmission line inspection model, including training of an adaptive visual feature extraction branch, an adaptive text feature extraction branch, and a cross-modal fusion module.

[0033] Preferably, the S3 specifically includes: S31: Freeze the adaptive visual feature extraction branch and the cross-modal fusion module, and train the adaptive text feature extraction branch separately; S32: Refreeze the adaptive text feature extraction branch and the cross-modal fusion module, and train the adaptive visual feature extraction branch separately; S33: Refreeze the adaptive visual feature extraction branch and the adaptive text feature extraction branch, and train the cross-modal fusion module separately; S34: Completely unfreeze the visual feature branch, text feature branch and cross-modal fusion module, and use image-text pairs for comprehensive fine-tuning.

[0034] S4: Update the parameters of the multimodal transmission line inspection model, coordinate the input of image-text data, use KL divergence and Wasserstein distance to align modal features, perform gradient backpropagation and parameter update, and gradually align the modes based on the distance metric to achieve the best modal adaptation combination.

[0035] The S4 includes the following detailed steps: S41: Coordinate input of image-text data, inputting the image data into the visual branch, and inputting the text data into the text branch; S42: Modality feature alignment using KL divergence and Wasserstein distance: (1) The KL divergence formula is as follows: (1); In formula (1), T(cls) represents the category prediction probability value output by the visual feature branch, S(cls) represents the category prediction probability value output by the text feature branch, and N refers to the number of categories; (2) The Wasserstein distance formula is as follows: (2); In formula (2), is the joint distribution of all features (this feature includes text features, image features and fusion feature maps) The marginal distribution of the joint distribution of features includes P and Q. It is the distance measure between two points x and y in feature space X.

[0036] (3 for and The corresponding weight coefficients are increased respectively and added as the loss function to enhance the effect of modal alignment.

[0037] S43: Perform gradient backpropagation and parameter update to find the best modality adaptation combination.

[0038] S5: The multimodal transmission line inspection model obtained in the above steps is used to detect hidden dangers in the transmission channel scenario.

[0039] Preferably, the visual input of this embodiment is a hidden danger map in the transmission channel scenario, and the text input is the description of the hidden danger map, for example: 20230715; Shanghai Ge Nan Line; Today's temperature is 17-24 degrees, and the typhoon orange warning continues to be issued, and the heavy rain yellow warning is issued; Urban + Road; The probability of crane hidden danger is 75%. Taking the above image-text as input, the model processing results are compared as shown in Table 1: Table 1 Comparison results

[0040] In summary, the multimodal power transmission line inspection model of the present invention has significantly improved performance compared to the previous pure vision-text model. The accuracy of the new model has increased from 76% to 80%. This means that the model is more accurate in predicting the correct samples and reduces the number of misclassifications. The recall rate has increased from 82% to 85%. This shows that the new model does a better job of identifying all relevant samples and reduces missed detections.

[0041] Multimodal models can understand data more comprehensively by combining visual and textual information. This fusion of information can provide richer context and help the model better capture the complex characteristics of the data. Since multimodal models can process and learn multiple types of data, they may show better generalization ability when faced with diverse data sets and different application scenarios. In some cases, a single modality may be affected by noise or occlusion. Multimodal models can improve resistance to these interferences through complementary information from different modalities, thereby maintaining performance in complex environments.

[0042] Example 2 This embodiment provides a device for implementing a transmission line operation and maintenance method based on multi-modal feature fusion, the device comprising: at least one processor; and A memory storing instructions, which, when executed by the at least one processor, enables the at least one processor to execute the transmission line operation and maintenance method based on multimodal feature fusion as described above.

[0043] In this embodiment, the device includes but is not limited to: personal computers, server computers, workstations, desktop computers, laptop computers, notebook computers, mobile computing devices, smart phones, tablet computers, cellular phones, personal digital assistants (PDAs), handheld devices, messaging devices, wearable computing devices, consumer electronic devices, etc.

[0044] Example 3 This embodiment also provides a computer-readable storage medium storing executable instructions, which, when executed, enable the machine to execute the transmission line operation and maintenance method based on multimodal feature fusion as described above.

[0045] Specifically, a system or device equipped with a readable storage medium can be provided, on which software program codes that implement the functions of any of the above-mentioned embodiments are stored, and a computer or processor of the system or device can read and execute instructions stored in the readable storage medium.

[0046] In this case, the program code itself read from the machine-readable medium can realize the function of any one of the above embodiments, and thus the machine-readable code and the machine-readable storage medium storing the machine-readable code constitute part of this specification.

[0047] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Optionally, the program code may be downloaded from a server computer or a cloud via a communication network.

[0048] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0049] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0050] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0051] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0052] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A transmission line operation and maintenance method based on multimodal feature fusion, characterized in that: The following steps are involved: S1: Obtain hidden danger images in the transmission channel scenario, collect text information corresponding to the hidden danger images, construct image-text sample pairs based on the images and corresponding information, and save the samples in dictionary format; S2: Construct a multimodal transmission line inspection model, including an adaptive visual feature extraction branch, an adaptive text feature extraction branch, a first feature selection layer, a cross-modal fusion module, a second feature selection layer, a pooling layer, and a classifier; The adaptive visual feature extraction branch extracts visual features, and the adaptive text feature extraction branch extracts text features; The first feature selection layer is used to select appropriate visual features and text features, and to combine the two features; The cross-modal fusion module fuses the visual feature and the text feature vector to obtain a fused feature map; The second feature selection layer is used to select key features of the feature layer; The pooling layer is used to thin out features, filter out invalid features, and retain key features; The classifier classifies the target based on the fully connected layer; S3: Training of a multimodal transmission line inspection model, including training of an adaptive visual feature extraction branch, an adaptive text feature extraction branch, and a cross-modal fusion module; S4: Update the parameters of the multimodal transmission line inspection model, coordinate the input of image-text data, use KL divergence and Wasserstein distance to align modal features, and perform gradient backpropagation and parameter update; S5: The multimodal transmission line inspection model obtained in the above steps is used to detect hidden dangers in the transmission channel scenario.

2. A transmission line operation and maintenance method based on multimodal feature fusion according to claim 1, characterized in that: The adaptive visual feature extraction branch uses an adaptive visual feature extraction branch based on the Swin Transformer improvement: First, the image is divided into multiple small blocks, and a hierarchical Transformer structure is used to capture local and global visual information; Input each small block of the image into multiple stacked core component Blocks, including LayerNorm, MLP, Window Attention, and Shifted Window Attention; Then, the output features of the previous Block module are fused with the output features of the next Block module by means of feature stacking and concatenation. Finally, the fusion results are passed through the encoder to obtain the visual features of each image.

3. A transmission line operation and maintenance method based on multimodal feature fusion according to claim 1 or 2, characterized in that: The adaptive text feature extraction branch uses an adaptive text feature extraction branch based on BERT improvement: The BERT model is changed to a four-way feature adaptive interaction network, where the "four-way" refers to the context, grammar, semantics, and domain features of the text; The context adaptation adjusts the sensitivity to context information according to the context environment of different texts through an adaptive interaction module; The grammar adaptation refers to the model identifying and emphasizing key components in the grammatical structure; The semantic adaptation refers to understanding the meaning of a sentence by analyzing the semantic relationship between words; The domain adaptation refers to the model recognizing and adapting to the terms and expressions of the preset domain.

4. A transmission line operation and maintenance method based on multimodal feature fusion according to claim 3, characterized in that: The cross-modal fusion module uses a cross-modal fusion module based on Self-Attention to learn the correlation between visual features and text features through the Self-Attention layer, including: For the visual feature vector and the text feature vector, the dot product between them is calculated and then normalized to the attention weight through the softmax function; For visual features, each feature element is multiplied by the corresponding attention weight, and then summed to obtain the weighted visual feature; similarly, for text features, each feature element is multiplied by the corresponding attention weight, and then summed to obtain the weighted text features; Finally, the weighted features of the two modalities are stacked to obtain a fused feature map.

5. A transmission line operation and maintenance method based on multimodal feature fusion according to claim 3, characterized in that: Before entering the adaptive text feature extraction branch based on BERT improvement, it also includes: Sentence preprocessing is performed, including but not limited to word segmentation and conversion into word vectors. At the same time, in order to capture four-way features, additional feature representations are introduced, including but not limited to the embedding of domain-specific vocabulary and the representation of grammatical structure.

6. A transmission line operation and maintenance method based on multimodal feature fusion according to claim 4, characterized in that: The S3 specifically includes: S31: Freeze the adaptive visual feature extraction branch and the cross-modal fusion module, and train the adaptive text feature extraction branch separately; S32: Refreeze the adaptive text feature extraction branch and the cross-modal fusion module, and train the adaptive visual feature extraction branch separately; S33: Refreeze the adaptive visual feature extraction branch and the adaptive text feature extraction branch, and train the cross-modal fusion module separately; S34: Completely unfreeze the visual feature branch, text feature branch and cross-modal fusion module, and use image-text pairs for comprehensive fine-tuning.

7. A transmission line operation and maintenance method based on multimodal feature fusion according to claim 6, characterized in that: The S4 specifically includes: S41: Coordinate input of image-text data, inputting the image data into the visual branch, and inputting the text data into the text branch; S42: Modality feature alignment using KL divergence and Wasserstein distance: (1) The KL divergence formula is as follows: In formula (1), T(cls) represents the category prediction probability value output by the visual feature branch, S(cls) represents the category prediction probability value output by the text feature branch, and N refers to the number of categories. (2) The Wasserstein distance formula is as follows: W(P,Q)=inf γ∈π(P,Q) ∫d(x,y)d γ (x,y) (2); In formula (2), π(P,Q) is the set of all joint distributions γ of the features, the marginal distribution of the joint distribution of the features includes P and Q, and d(x,y) is the distance measure between two points x and y in X in the feature space; (3) For L kl (T, S) and W(P, Q) increase the corresponding weight coefficients respectively and add them as the loss function to enhance the effect of modality alignment; S43: Perform gradient backpropagation and parameter update to find the best modality adaptation combination.

8. A transmission line operation and maintenance device based on multimodal feature fusion, characterized in that: The device comprises: processor; a memory having stored thereon a computer program executable on the processor; Wherein, when the computer program is executed by the processor, the steps of the transmission line operation and maintenance method based on multimodal feature fusion as described in any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images