Power line detection method and system based on multi-modal fusion and related device

Abstract

A kind of power line detection method, system and related device based on multi-modal fusion, method includes obtaining visible light RGB image data and thermal infrared TIR image data, input pre-established multi-modal power line detection network PLNet, different types of features are extracted using detail branch and context branch;Through multi-scale cross-modal context fusion method to fuse shallow multi-modal context information;Through feature decoupling and fusion, the similarity between different modal features is evaluated, shared and different features are separated, and feature fusion is carried out according to the complementarity between different modal features, to enhance the discriminant ability of features;And using context perception fusion and extracting dynamic capture power line features and the dependence between surrounding pixels, fuse deep multi-modal context information;Feature decoding is carried out, and the power line detection result after multi-modal context information supplements detail branch information is obtained.The present application can improve the accuracy and robustness of power line prediction.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and intelligent detection technology, specifically relating to a power line detection method, system and related devices based on multimodal fusion. Background Technology

[0002] With rapid economic development and the continuous expansion of power systems, regular inspection and maintenance of transmission equipment has become a crucial link in ensuring the safety and reliability of power systems. However, traditional inspection methods heavily rely on manual on-site operations, which are not only inefficient but also pose significant safety risks. In recent years, with the rapid development of drone and aircraft technology, their application in power line inspection has significantly improved inspection efficiency and frequency, effectively reducing the risk of equipment damage caused by external factors and ensuring the stable operation of power lines. Although drones have demonstrated significant advantages in power line inspection, power lines are still difficult to reliably detect in complex environments such as low light or severe weather, making them prone to collisions, which can lead to line damage or even large-scale power outages, causing significant economic losses. Therefore, ensuring that drone navigation systems possess efficient and accurate power line detection capabilities is a prerequisite for their safe execution of inspection tasks.

[0003] Power line detection is essentially a dense prediction task, aiming to accurately locate power line regions at the pixel level and obtain spatial location information of transmission lines. Inspired by infrastructures such as fully convolutional networks and U-Net, a large number of deep learning-based power line detection methods have emerged in recent years, significantly promoting the application of drones in smart grid inspection. However, facing increasingly complex real-world scenarios, relying solely on visible light (RGB) sensors is no longer sufficient to meet the high-precision requirements of power line detection. On the one hand, power lines have low contrast in RGB images, and their elongated shapes often blend into the background, making them difficult to identify. On the other hand, in structurally complex areas such as urban environments, power lines are easily confused with linear structures such as building edges and road outlines. This is mainly due to the lack of sufficient contextual information in the RGB modality, making it difficult for the model to accurately distinguish power lines from other background lines at the semantic level. To alleviate these problems, researchers have gradually introduced multimodal perception technologies, fusing additional information sources such as thermal infrared (TIR) ​​or depth sensors to enhance the expressive power of the scene. Among them, thermal infrared sensors, due to their insensitivity to light and strong ability to extract target thermal radiation features, exhibit high robustness in complex environments and have been widely used in multimodal segmentation tasks.

[0004] However, most existing multimodal fusion power line detection methods employ simple information fusion strategies and have not fully explored the complementarity between multimodal features. Mainstream methods typically treat power line detection as a semantic segmentation or saliency detection task, extracting deep semantic features through image downsampling and recovering detailed information during the upsampling stage using skip connections. But power lines differ from regular objects; they possess a unique geometric characteristic of being "long and thin": their length typically spans the entire image, while their width is only a few pixels. This shape makes power lines easily submerged by background noise during downsampling, making it difficult to preserve their overall structure, especially against backgrounds with numerous similar linear structures, such as road edges or building outlines, where the model is highly prone to false detections. Summary of the Invention

[0005] The purpose of this invention is to address the problems in the prior art by providing a power line detection method, system, and related apparatus based on multimodal fusion, which effectively improves the accuracy and robustness of power line prediction.

[0006] To achieve the above objectives, the present invention provides the following technical solution: Firstly, a power line detection method based on multimodal fusion is provided, comprising: Acquire visible light RGB image data and thermal infrared (TIR) ​​image data, and input them into a pre-established multimodal electric line detection network (PLNet). The PLNet uses detail branches and context branches to extract different types of features. For the extracted features of different types, a multi-scale cross-modal context fusion method is used to fuse shallow multimodal context information; at the same time, feature decoupling and fusion are used to evaluate the similarity between different modal features, separate shared and differential features, and perform feature fusion based on the complementarity between different modal features to enhance the discriminative ability of features; and context-aware fusion and extraction of the dynamic capture of the dependence between electric field line features and surrounding pixels are used to fuse deep multimodal context information. The features, after fusing shallow multimodal context information, enhanced discrimination capability, and fused deep multimodal context information, are decoded to obtain the power line detection results after supplementing the detailed branch information of the multimodal context information.

[0007] As a preferred embodiment, the multimodal power line detection network PLNet uses a convolutional neural network (CNN) as the backbone network in the detail branch to remove downsampling operations; and in the context branch of the multimodal power line detection network PLNet, it uses a visual transformer (ViT) as the backbone network to perform context feature inference by leveraging the global modeling capability of the self-attention mechanism.

[0008] As a preferred embodiment, in the step of fusing shallow multimodal context information by using a multi-scale cross-modal context fusion method for the extracted different types of features, the features of visible light RGB image data and thermal infrared TIR image data are respectively processed by hollow spatial pyramid pooling to capture multi-scale single-modal context features, and then the features of different modalities but the same scale are mapped to a common feature space.

[0009] As a preferred embodiment, the steps of evaluating the similarity between different modal features through feature decoupling and fusion, separating shared and differential features, and performing feature fusion based on the complementarity between different modal features to enhance the discriminative ability of features include two stages: In the first stage, for the visible light RGB image data branch, the visible light RGB context features are concatenated with the thermal infrared (TIR) ​​context features to calculate their shared features, denoted as... :

[0010] In the formula, Conv(*) represents a convolutional block with a 3×3 convolutional layer; Cat (*;*) indicates a cascading operation; the symbol Hadamard product is represented by subtracting shared features from an all-one matrix. By measuring the differences in corresponding spatial locations, complementary information weights can be obtained. The calculation expression is as follows:

[0011] In the formula, Softmax (*) denotes the normalization function. I Represents a matrix consisting entirely of 1s; in In this context, a higher value indicates stronger contextual complementarity in the same region across different modalities, and vice versa; complementary information weights are utilized. and Calculate complementary features of TIR images Complementary features of RGB images :

[0012]

[0013] In the formula, the symbol ⊙ represents element-wise multiplication; In the second stage, through calculation and Cosine similarity is used to measure the difference information of corresponding spatial locations, and the corresponding context similarity weights are obtained. The expression is as follows:

[0014] In the formula, Cos (*) refers to cosine similarity calculation; in In this context, a higher value indicates stronger contextual similarity in the same region across different modalities, and vice versa; this yields the contextual similarity weight. Subsequently, features are shared across modal contexts. and Calculated in the following way:

[0015]

[0016] By fusing complementary contextual features and shared contextual features, enhanced contextual features are obtained, i.e. and :

[0017]

[0018] In the formula, This indicates a 3×3 convolutional layer followed by a BN layer and a ReLU activation function; resulting in enhanced contextual features at four scales. and Next, the layers are first concatenated along the channel dimension, and then 1×1 convolutions are used to fuse multi-scale contextual information. The mathematical expression is as follows:

[0019]

[0020] Constructing cross-modal attention operations to enable the interaction and combination of information from different modalities, including processing data of size 10 ... The input features are flattened as ,in N = H * W Then, a vector of query Q, key K, and value V is generated through linear projection; A global attention interaction is performed between the key K and value V vectors of the current modality and the query Q vectors of other modalities to capture long-distance dependencies between contextual features of different modalities, as expressed mathematically below:

[0021]

[0022]

[0023]

[0024] In the formula, the symbol Represents matrix multiplication; Indicates matrix transpose; ), ), This represents three linear projection layers, with parameters for each layer as follows: , and ; Will and Input subsequent context-aware fusion to supplement the contextual information of detailed branches;

[0025] Multimodal context features Perform contextual attention fusion to enhance the interaction of contextual features at different levels.

[0026] As a preferred embodiment, the context attention fusion introduces vertical and horizontal compressed attention mechanisms to capture global contextual dependencies in the vertical and horizontal directions, respectively; it simultaneously captures important local information and global contextual information through global average pooling (GAP) and global max pooling (GMP) operations; the mathematical expression for the context attention fusion is as follows:

[0027]

[0028] In the formula, and This represents 1×1 and 3×3 convolutional layers with parameters γ and α, followed by a BN layer and a ReLU activation function; Cat(*,*) represents cascaded operations along the channels; AvP(*) and MaxP(*) represent global average pooling and global max pooling operations, respectively; by introducing horizontal and vertical attention mechanisms, correlations between different locations in the feature map are captured; for the visible light RGB image data branch, axially enhanced contextual features are used. and The calculation expression is as follows:

[0029]

[0030] Axially enhanced RGB and TIR features, i.e. and Obtained through the following methods:

[0031]

[0032] In the formula, and These represent the mean calculations along the vertical and horizontal directions, respectively; Att(*) indicates multi-head attention operation; axial enhancement feature. and Interacting with contextual information from the previous layer enables effective fusion of contextual information from different levels and modalities; multimodal contextual features Calculated in the following way:

[0033] Will and Input context-aware fusion to supplement contextual information for detailed branches; multimodal contextual features Input-following context attention fusion to enhance the interaction of contextual features at different levels.

[0034] As a preferred embodiment, the steps of employing context-aware fusion and extracting the dynamic capture of the dependence between electric field line features and surrounding pixels, and fusing deep multimodal contextual information, include: context-aware feature fusion extracts electric field line features using dynamic serpentine convolution, adaptively adjusting the convolution kernel shape, and introducing multimodal contextual information to supplement detail branches in context-aware feature fusion, thereby establishing a contextual association between the electric field lines and the surrounding environment; given axially enhanced RGB features and TIR features, i.e. and and the detailed branch features at the current level, i.e. and The global attention map is calculated, and the global context information is modeled as a weighted average of feature space locations. The global context information is aggregated to each location of the feature through element-wise addition. For the visible light RGB image data branch, the mathematical expression is:

[0035]

[0036] In the formula, , and Indicates a parameter { },{ }and{ The algorithm consists of 1×1 and 3×3 convolutional layers, followed by BN layers and ReLU activation functions; subsequently, dynamic serpentine convolutions are used to adaptively extract electric field features; and then, these features are combined with contextual features. Interactively establish the contextual relationship between the extracted electric field line features and different modal features; Features that supplement contextual information Calculated according to the following expression:

[0037] In the formula, DSConv (*) indicates dynamic serpentine convolution. This indicates element-wise multiplication.

[0038] As a preferred embodiment, the step of decoding the features after fusing shallow multimodal context information, enhancing discriminative ability, and fusing deep multimodal context information to obtain the power line detection result after supplementing the multimodal context information with detailed branch information includes: Given the outputs from the visible light RGB image data branch and the thermal infrared TIR image data branch and They are concatenated and fused together using 1×1 convolutional blocks to obtain Sum the outputs to get { |i=0,1,2} is used to progressively restore details for the final prediction. ; The space size is 1 / 4 of the original input data, and the mathematical expression is as follows:

[0039]

[0040]

[0041]

[0042] In the formula, {DBR(*; )} and {DBR(*; )} indicates that it has parameters. and A 3×3 deconvolutional layer, followed by a BN layer and a ReLU activation function; SegHead represents the probability map used for prediction. 1×1 convolution; output features that fuse multi-scale cross-modal context and contextual attention. The prediction of the auxiliary supervision probability map is based on three segment heads. The mathematical expression is as follows:

[0043] The loss function of the multimodal power line detection network PLNet consists of two parts: a main loss function and an auxiliary loss function.

[0044] In the formula, , and There are three hyperparameters used to balance the weights among the losses; the final probability plot. Depend on Supervision, auxiliary probability diagrams are generated by , and Supervision: Since the special shape of the power line target occupies only a small portion of all pixels, the proposed network uses a combination of binary cross-entropy and Dice loss as the loss function, as shown in the following mathematical expression:

[0045]

[0046]

[0047] In the formula, It is a hyperparameter used to balance the weights of the binary cross-entropy and the Dice loss, in the context of... This represents the predicted probability of the i-th pixel. This represents the corresponding true label, and N represents the total number of input pixels. These are model parameters.

[0048] Secondly, a power line detection system based on multimodal fusion is provided, comprising: The feature extraction module is used to acquire visible light RGB image data and thermal infrared TIR image data, and input them into the pre-established multimodal electric line detection network PLNet. The multimodal electric line detection network PLNet uses detail branches and context branches to extract different types of features. The feature processing module is used to fuse shallow multimodal context information for different types of extracted features through a multi-scale cross-modal context fusion method; at the same time, it evaluates the similarity between different modal features through feature decoupling and fusion, separates shared and differential features, and performs feature fusion based on the complementarity between different modal features to enhance the discriminative ability of features; and it uses context-aware fusion to extract the dependency relationship between dynamically captured electric field line features and surrounding pixels, and fuses deep multimodal context information. The feature decoding module is used to decode the features after fusing shallow multimodal context information, enhancing the discrimination ability, and fusing deep multimodal context information, to obtain the power line detection results after supplementing the detailed branch information of the multimodal context information.

[0049] Thirdly, an electronic device is provided, comprising: A memory for storing at least one instruction; and a processor for executing the instructions stored in the memory to implement the power line detection method based on multimodal fusion.

[0050] Fourthly, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being executed by a processor in an electronic device to implement the power line detection method based on multimodal fusion.

[0051] Compared with the prior art, the present invention has at least the following beneficial effects: This invention utilizes a multimodal fusion-based power line detection method. Visible light RGB image data and thermal infrared (TIR) ​​image data are input into a multimodal power line detection network (PLNet). The PLNet employs detail branches and context branches to extract different types of features. In the detail branch, downsampling is removed to maintain high resolution and prevent the disappearance of transmission line features or their being drowned out by irrelevant background noise. In the context branch, contextual feature inference is performed while maintaining low resolution. Subsequently, by coupling detailed power line features with the multimodal global representation, a deep relationship between power line features and surrounding pixels is established, effectively filtering irrelevant noise. This process ensures that key features are preserved while minimizing background interference, significantly reducing false identification of similar objects, thereby improving the accuracy and robustness of power line prediction. Specifically, for the extracted features of different types, a multi-scale cross-modal context fusion method is used to fuse shallow multimodal context information. At the same time, feature decoupling and fusion are used to evaluate the similarity between different modal features, separate shared and differential features, and perform feature fusion based on the complementarity between different modal features to enhance the discriminative ability of features, thereby enhancing the integrity and distinguishability of context information. Context-aware fusion is used to extract the dynamic capture of the dependency relationship between electric field line features and surrounding pixels, fuse deep multimodal context information, enhance the understanding of image spatial and structural features, improve the ability to recognize details, maintain a global perspective, improve the accuracy and integrity of multimodal context information, enhance the distinguishability and robustness of electric field line features, guide the adaptive extraction of electric field line features, and effectively distinguish target features from similar backgrounds. This invention ultimately decodes the features after fusing shallow multimodal context information, enhancing discriminative ability, and fusing deep multimodal context information to obtain power line detection results after multimodal context information supplements detailed branch information. By integrating multimodal context features with original detailed branch features, not only is the context information supplemented, but the feature representation is also enhanced. This enables the model to capture more complex relationships between power lines and the surrounding environment, thereby improving the overall performance of power line detection. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0053] Figure 1 A schematic diagram of the multimodal power line detection network (PLNet) structure according to an embodiment of the present invention; Figure 2 A schematic diagram of the architecture of the multi-scale cross-modal context fusion module MCCFM according to an embodiment of the present invention; Figure 3 A schematic diagram illustrating the design principle of feature decoupling and fusion in this embodiment of the invention; Figure 4 A schematic diagram of the architecture of the context-aware fusion module according to an embodiment of the present invention; Figure 5 A schematic diagram of the architecture of the context-aware feature extraction module CAFEM proposed in this invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, those skilled in the art can obtain other embodiments without creative effort.

[0055] This invention proposes a power line detection method based on multimodal fusion. First, it proposes a multimodal power line detection network PLNet with a multi-branch network architecture to achieve the extraction and fusion of detailed features and contextual features. Figure 1 The detailed structure of the proposed PLNet is shown. In the detail branch, feature processing maintains high resolution, leveraging the powerful detail extraction capabilities of convolutional neural networks to capture geometric and appearance information. In the context branch, the powerful long-range dependency modeling capabilities of global attention networks (Transformers) are utilized to extract rich contextual information. Meanwhile, as... Figure 2As shown, an innovative multi-scale cross-modal context fusion module (MCCFM) is designed to integrate shallow multimodal context information. The MCCFM first inputs RGB and TIR features into two shared-parameter dilated spatial pyramid pooling (ASPP) modules to capture multi-scale context features. By proposing a feature decoupling and fusion block (DFB), cosine similarity is used to evaluate the similarity between features of different modalities, separating shared and differential features, and effectively fusing them based on modal complementarity, thereby enhancing the integrity and discriminability of context information. A non-local cross-attention mechanism is used to extract rich cross-modal context information and aggregate different modal features along the channel dimension. Finally, a context attention fusion module (CAFM) is proposed to further integrate and infer deep multimodal context information. This approach leverages contextual information to enhance understanding of image spatial and structural features. Global average pooling and global max pooling operations simultaneously capture important local and overall contextual information, improving detail recognition while maintaining a global perspective and enhancing the accuracy and completeness of multimodal contextual information. A context-aware feature extraction module (CAFEM) is introduced through PLNet to further enhance the discriminativeness and robustness of electric field line features. CAFEM models multimodal contextual information to guide adaptive extraction of electric field line features, dynamically capturing the complex dependencies between electric field line features and surrounding pixels, effectively distinguishing target features from similar backgrounds. The specific implementation steps include the following: S1. Acquire visible light RGB image data and thermal infrared TIR image data, and input them into the pre-established multimodal power line detection network PLNet. The multimodal power line detection network PLNet uses detail branches and context branches to extract different types of features. S2. For the extracted features of different types, shallow multimodal context information is fused through multi-scale cross-modal context fusion method; at the same time, the similarity between different modal features is evaluated through feature decoupling and fusion, shared and differential features are separated, and feature fusion is performed according to the complementarity between different modal features to enhance the discriminative ability of features; and context-aware fusion and dynamic capture of the dependency relationship between electric field line features and surrounding pixels are used to fuse deep multimodal context information. S3. Decode the features after fusing shallow multimodal context information, enhancing discrimination ability, and fusing deep multimodal context information to obtain the power line detection result after supplementing the detailed branch information of the multimodal context information.

[0056] In one possible implementation, the multimodal power line detection network PLNet described in step S1 uses a convolutional neural network (CNN) as the backbone network in the detail branch, removing downsampling operations to maintain high resolution; in the context branch of the multimodal power line detection network PLNet, a visual transformer (ViT) is used as the backbone network, utilizing the global modeling capability of the self-attention mechanism for contextual feature inference while maintaining low resolution. The multimodal power line detection network PLNet of this embodiment adopts a multi-branch network architecture, integrating CNN and Transformer as single-modal feature extractors to capture RGB and thermal infrared features at different levels. Simultaneously, downsampling operations are eliminated in the detail branch design to retain rich detail information. Figure 1 The STEM module in the detail branch contains two convolutional blocks, each downsampled by 1 / 2. Each detail block is a CNN block with a residual structure. Notably, no downsampling operation is applied in the detail branch except for the STEM module, maintaining the feature resolution at 1 / 4 of the original input to fully preserve spatial and geometric information. To capture rich contextual information, a visual transformer is used as the backbone network of the context branch. Specifically, the output features of the first detail block are downsampled by 1 / 2 and used as the input of the context branch. Thus, four levels of detail features (i.e., ...) are obtained. and ) and four levels of features with rich contextual information (i.e. and By utilizing a multi-branch architecture, PLNet can preserve key geometric and appearance information in the detail branches while extracting rich global contextual information from the context branches. This enables the network to achieve more accurate power line detection in complex scenes.

[0057] In one possible implementation, step S2, when fusing shallow multimodal context information using a multi-scale cross-modal context fusion method for different types of extracted features, involves passing the features of visible light RGB image data and thermal infrared (TIR) ​​image data through dilated spatial pyramid pooling to capture multi-scale single-modal context features, and then mapping features of different modalities but the same scale to a common feature space. PLNet introduces a novel multi-scale cross-modal context fusion module to effectively fuse shallow context information. The multi-scale cross-modal context fusion module (MCCFM) uses dilated convolutional pyramids to capture rich contextual cues and utilizes decoupling and fusion blocks to enhance the discriminative ability of features at the same scale. Subsequently, enhanced features at different scales are concatenated and cross-modal attention interactions are performed to capture rich multimodal context information. Specifically, the output features of the context branch... and First, input two shared-parameter hollow spatial pyramid pooling (ASPP) modules to capture multi-scale unimodal contextual features (i.e., and The ASPP module maps features from different modalities but of the same scale to a common feature space. Each ASPP module contains four parallel branches, including a 1×1 convolutional branch and three dilated convolutional branches with dilation rates of {1, 3, 5}. Each branch is followed by a batch normalization (BN) layer and a ReLU activation function. Mathematically, this process can be represented as:

[0058]

[0059] In one possible implementation, embodiments of the present invention propose a novel decoupling and fusion module (DFM) designed to effectively utilize shared and complementary information between different modalities, thereby enhancing feature discrimination capabilities. Specifically, as... Figure 3 As shown, the Decoupling and Fusion Module (DFM) proposed in this embodiment of the invention performs step S2, which involves evaluating the similarity between different modal features through feature decoupling and fusion, separating shared and differential features, and fusing features based on the complementarity between different modal features to enhance the discriminative ability of the features. This includes two stages: In the first stage, for the visible light RGB image data branch, the visible light RGB context features are concatenated with the thermal infrared (TIR) ​​context features to calculate their shared features, denoted as... :

[0060] In the formula, Conv(*) represents a convolutional block with a 3×3 convolutional layer; Cat (*;*) indicates a cascading operation; the symbol Hadamard product is represented by subtracting shared features from an all-one matrix. By measuring the differences in corresponding spatial locations, complementary information weights can be obtained. The calculation expression is as follows:

[0061] In the formula, Softmax (*) denotes the normalization function. I Represents a matrix consisting entirely of 1s; in In this context, a higher value indicates stronger contextual complementarity in the same region across different modalities, and vice versa; complementary information weights are utilized. and Calculate complementary features of TIR images Complementary features of RGB images :

[0062]

[0063] In the formula, the symbol ⊙ represents element-wise multiplication; In the second stage, through calculation and Cosine similarity is used to measure the difference information of corresponding spatial locations, and the corresponding context similarity weights are obtained. The expression is as follows:

[0064] In the formula, Cos (*) refers to cosine similarity calculation; in In this context, a higher value indicates stronger contextual similarity in the same region across different modalities, and vice versa; this yields the contextual similarity weight. Subsequently, features are shared across modal contexts. and Calculated in the following way:

[0065]

[0066] By fusing complementary contextual features and shared contextual features, enhanced contextual features are obtained, i.e. and :

[0067]

[0068] In the formula, This indicates a 3×3 convolutional layer followed by a BN layer and a ReLU activation function; resulting in enhanced contextual features at four scales. and Next, the layers are first concatenated along the channel dimension, and then 1×1 convolutions are used to fuse multi-scale contextual information. The mathematical expression is as follows:

[0069]

[0070] Constructing cross-modal attention operations to enable the interaction and combination of information from different modalities, including processing data of size 10 ... The input features are flattened as ,in N = H * W Then, a vector of query Q, key K, and value V is generated through linear projection; A global attention interaction is performed between the key K and value V vectors of the current modality and the query Q vectors of other modalities to capture long-distance dependencies between contextual features of different modalities, as expressed mathematically below:

[0071]

[0072]

[0073]

[0074] In the formula, the symbol Represents matrix multiplication; Indicates matrix transpose; ), ), This represents three linear projection layers, with parameters for each layer as follows: , and ; Will and Input subsequent context-aware fusion to supplement the contextual information of detailed branches;

[0075] Multimodal context features Perform contextual attention fusion to enhance the interaction of contextual features at different levels.

[0076] To further integrate and reason about deep multimodal contextual information, this invention proposes a contextual attention fusion module. This module introduces vertical and horizontal compressed attention mechanisms to capture global contextual dependencies in the vertical and horizontal directions, respectively. This method effectively extracts and utilizes contextual information, enhancing the understanding of image spatial and structural features. Furthermore, it simultaneously captures important local and global contextual information through global average pooling (GAP) and global max pooling (GMP) operations. This dual-pooling strategy not only improves the ability to recognize details but also maintains a global perspective, improving the accuracy and completeness of multimodal contextual information. Through this method, the contextual attention fusion module effectively integrates deep information from different modalities, significantly improving the understanding of complex scenes.

[0077] The mathematical expression for context attention fusion is as follows:

[0078]

[0079] In the formula, and This represents 1×1 and 3×3 convolutional layers with parameters γ and α, followed by BN layers and ReLU activation functions; Cat(*,*) represents cascaded operations along channels; AvP(*) and MaxP(*) represent global average pooling and global max pooling operations, respectively; subsequently, horizontal and vertical attention mechanisms are introduced to capture the correlation between different locations in the feature map; for the visible light RGB image data branch, axially enhanced contextual features are used. and The calculation expression is as follows:

[0080]

[0081] Axially enhanced RGB and TIR features, i.e. and Obtained through the following methods:

[0082]

[0083] In the formula, and These represent the mean calculations along the vertical and horizontal directions, respectively; Att(*) indicates multi-head attention operation; axial enhancement feature. and Interacting with contextual information from the previous layer enables effective fusion of contextual information from different levels and modalities; multimodal contextual features Calculated in the following way:

[0084] Will and Input context-aware fusion to supplement contextual information for detailed branches; multimodal contextual features Input-following context attention fusion to enhance the interaction of contextual features at different levels.

[0085] In one possible implementation, to better capture power line features and effectively interact with multimodal context information, a context-aware feature extraction module (CAFEM) is introduced into the proposed PLNet. For example... Figure 5As shown, CAFEM utilizes dynamic serpentine convolution to extract electric field line features, which can adaptively adjust the shape of the convolution kernel, providing greater flexibility and allowing the kernel energy to focus on the unique geometric features of the target. Simultaneously, the detail branch maintains high resolution, significantly preserving the semantic information of the electric field lines. However, performing feature inference while maintaining high resolution severely impairs the ability to capture long-range contextual information. The lack of rich contextual information leads to misclassification of segmented targets (such as electric field lines) from other similar objects. To address this issue, multimodal contextual information is introduced into CAFEM to supplement the detail branch, thereby establishing a contextual association between the electric field lines and their surrounding environment. By integrating multimodal contextual features with the original detail branch features, not only is the contextual information supplemented, but the feature representation is also enhanced. This enables the model to capture more complex relationships between electric field lines and their surrounding environment, thereby improving the overall performance of electric field line detection. Specifically, in step S2 of this embodiment, context-aware fusion and extraction are used to dynamically capture the dependencies between electric field line features and surrounding pixels. When fusing deep multimodal contextual information, such as... Figure 4 As shown, given the axially enhanced RGB and TIR features, i.e. and and the detailed branch features at the current level, i.e. and The global attention map is calculated, and the global context information is modeled as a weighted average of feature space locations. The global context information is aggregated to each location of the feature through element-wise addition. For the visible light RGB image data branch, the mathematical expression is:

[0086]

[0087] In the formula, , and Indicates a parameter { },{ }and{ The algorithm consists of 1×1 and 3×3 convolutional layers, followed by BN layers and ReLU activation functions; subsequently, dynamic serpentine convolutions are used to adaptively extract electric field features; and then, these features are combined with contextual features. Interactively establish the contextual relationship between the extracted electric field line features and different modal features; Features that supplement contextual information Calculated according to the following expression:

[0088] In the formula, DSConv (*) indicates dynamic serpentine convolution. This indicates element-wise multiplication.

[0089] In one possible implementation, step S3 includes: Given the final detail block output from the visible light RGB image data branch and the thermal infrared TIR image data branch. and They are concatenated and fused together using 1×1 convolutional blocks to obtain Summing the outputs of the first three detail blocks yields { |i=0,1,2} is used to progressively restore details for the final prediction. ; The space size is 1 / 4 of the original input data, and the mathematical expression is as follows:

[0090]

[0091]

[0092]

[0093] In the formula, {DBR(*; )} and {DBR(*; )} indicates that it has parameters. and A 3×3 deconvolutional layer, followed by a BN layer and a ReLU activation function; SegHead represents the probability map used for prediction. 1×1 convolution; in addition, output features of the Multi-Scale Cross-Modal Context Fusion Module (MCCFM) and the Context Attention Fusion Module (CAFM). The prediction of the auxiliary supervision probability map is based on three segment heads. To ensure the reliability of low-resolution branch feature extraction:

[0094] The loss function of the multimodal power line detection network PLNet consists of two parts: the main loss function and the auxiliary loss function;

[0095] In the formula, , and These are three hyperparameters used to balance the weights among the losses; , and Empirically, the values ​​were set to 0.2, 0.3, and 0.4. Final probability plot. Depend on Supervision, auxiliary probability diagrams are generated by , and Supervision: Since the special shape of the power line target occupies only a small portion of all pixels, the proposed network uses a combination of binary cross-entropy and Dice loss as the loss function, as shown in the following mathematical expression:

[0096]

[0097]

[0098] In the formula, It is a hyperparameter used to balance the weights of the binary cross-entropy and the Dice loss, in this context, This represents the predicted probability of the i-th pixel. This represents the corresponding true label, and N represents the total number of input pixels. These are model parameters.

[0099] The multimodal fusion-based power line detection method proposed in the above embodiments of the present invention was experimentally evaluated using publicly available RGB-TIR power line detection datasets. The VITLD and GTPD datasets were selected for evaluation. VITLD used a DJI Phantom 4 Pro Plus v2.0 drone equipped with a FLIR Vue Pro R infrared camera to capture 400 pairs of RGB-TIR images of high-altitude power line scenes at a resolution of 1920×1080. The RGB and infrared images were aligned and cropped to the same resolution. Furthermore, to evaluate the robustness of PLNet under different conditions, the VITLD dataset used Python libraries to simulate various scenarios of visible light images, such as daytime, nighttime, fog, and snow conditions. The VITLD dataset assumes that infrared images are highly robust to weather and illumination changes; therefore, only the visible light images were enhanced.

[0100] The GTPD dataset, developed in collaboration with the Turkish Electricity Transmission Company, utilizes aerial footage captured from aircraft. Through rigorous selection, 400 infrared images and an equal number of visible light images were extracted, containing scenes with and without power lines. These images were then resized to a uniform resolution of 512×512. The video was recorded in 21 different regions across multiple seasons to capture a wide range of environmental conditions. Given variations in background, temperature, weather patterns, and lighting, the dataset includes many challenging scenes, including instances where power lines are almost indistinguishable under low-contrast conditions. The primary goal of this dataset is to provide diverse samples for power line detection tasks in complex and dynamic environments. In this context, the dataset is divided into training, validation, and test sets in an 8:1:1 ratio, with 320 images designated for training and the remaining 80 images allocated to the validation and test sets. The ratio of images with and without power lines in the dataset is 1:1. This arrangement aligns with academic standards for evaluating the performance and robustness of power line detection algorithms.

[0101] Different methods are evaluated using F-score, precision, recall, and intersection-over-union (IoU), and their calculation expressions are as follows:

[0102]

[0103]

[0104]

[0105] in, Set to 1, These represent true positives, false positives, and false negatives for each category, respectively, and N is the total number of categories, including the background.

[0106] The network proposed in this embodiment of the invention is implemented on an NVIDIA GTX4090 Ti GPU (24GB RAM) using PyTorch 1.7.1 + CUDA 12.5. The entire model is trained using the Adam optimizer. The initial learning rate is set to 0.001, decaying by a factor of 0.5 every 20 epochs. The batch size is fixed at 5, and the weight decay is set to 5e-4. The entire network is trained for approximately 200 epochs. The dataset is divided into training, validation, and test sets at a ratio of 70%, 10%, and 20%, respectively. Furthermore, data augmentation strategies such as random scaling, flipping, cropping, brightness adjustment, contrast adjustment, and saturation adjustment are used to collaboratively increase the diversity of training samples.

[0107] To investigate the individual contributions of various key components in PLNet to power line detection, this embodiment of the invention conducted comprehensive ablation experiments on the VITLD dataset. Specifically, CAFEM, MCCFM, and CAFM were removed from the complete architecture to establish a baseline model, denoted as "B". Notably, in this baseline model, cross-modal fusion features were obtained by element-wise addition.

[0108] To evaluate the effectiveness of the proposed CAFEM, it was integrated into the baseline model, resulting in the new model "B+CAFEM", as shown in Table 1. By comparing the results of "B" and "B+CAFEM", it was observed that incorporating rich cross-modal contextual information during feature extraction significantly improved power line detection performance and reduced pixel misclassification errors.

[0109] Table 1: Quantitative results of ablation studies on each key component in the proposed model.

[0110] Next, to verify the effectiveness of MCCFM, it was integrated into the baseline model "B" and the enhanced model "B+CAFEM", resulting in "B+MCCFM" and "B+CAFEM+MCCFM" respectively. Comparison of the results among "B", "B+MCCFM", and "B+CAFEM+MCCFM" clearly demonstrates that the proposed MCCFM significantly improves the performance of RGB-TIR power line detection. This improvement is attributed to MCCFM's utilization of complementary information across four scales to enhance the representation of different modal features, thereby promoting the comprehensive exploration and utilization of shallow cross-modal contextual information.

[0111] Finally, CAFM is integrated into "B+CAFEM+MCCFM" to enhance deep contextual information modeling capabilities. The introduction of CAFM significantly improves power line detection performance. This enhancement stems from CAFM's ability to explicitly extract contextual information along different axes and fully interact with features from the previous stage. This process enhances the discriminativeness and completeness of the multimodal contextual features in the current stage. Ultimately, the proposed PLNet achieves the best performance.

[0112] As shown in Table 2, the effectiveness of each component in CAFEM was comprehensively evaluated through rigorous experiments using four variant models. The complete CAFEM module is denoted as CAFEM-V1 in Table 2. To evaluate the effectiveness of context information modeling, the original context modeling module was removed from CAFEM-V1, and context information was incorporated through element-wise addition, resulting in CAFEM-V2. To verify the indispensable role of integrating context information into detail branches, CAFEM was modified to remove context information, retaining only the adaptive serpentine convolution structure, ultimately resulting in CAFEM-V3. In the CAFEM_V4 model, the dynamic serpentine convolution (DSConv) module was removed to specifically evaluate the advantages of adaptive extraction of electric field features.

[0113] Table 2: Quantitative results of ablation studies of key components in CAFEM

[0114] Comparing the four variant models reveals that employing a multi-branch network architecture, integrating contextual information into the detail branches, and effectively modeling the context enhances the correlation between power lines and surrounding pixels. This approach significantly improves the discriminative power of power line features and reduces misclassification of surrounding pixels. Furthermore, results from the CAFEM-V4 model demonstrate that DSConv effectively extracts features from slender, narrow power lines, considering their position and orientation within the image. This adaptive method achieves more accurate and comprehensive power line feature localization and extraction.

[0115] To evaluate the effectiveness of each component in MCCFM, a full ablation experiment was conducted on four variant modules, as shown in Table 3. Checkmarks in Table 3 indicate that the module is included in MCCFM. Note that the complete MCCFM module is represented as MCCFM-V1. To investigate the effectiveness of the cross-modal attention mechanism, the cross-modal attention module was removed from the original MCCFM, and contextual information from different modalities was fused element-wise, resulting in MCCFM-V2. To explore the effectiveness of DFB, DFB was removed from the original CAFEM, and the output of the ASPP module was concatenated and input into the cross-modal attention module, resulting in MCCFM-V3. In MCCFM-V4, both DFB and the cross-modal attention module were removed.

[0116] The results in Table 3 for MCCFM-V1, MCCFM-V2, MCCFM-V3, and MCCFM-V4 demonstrate that using a cross-modal attention module can effectively model and fuse contextual information from different modalities. The results for models MCCFM-V1 and MCCFM-V3 show that the proposed DFB improves the robustness of current-scale feature representations by utilizing complementary and specific information between modalities. The results for models MCCFM-V1 and MCCFM-V4 in Table 3 prove that jointly using DFB and a cross-modal attention module can significantly improve power line detection performance. The experimental results for the four variant models in Table 3 show that integrating MCCFM into PLNet, utilizing ASPP to extract multi-scale contextual information, and leveraging specific and complementary information between different modalities significantly improves the discriminative ability of multimodal fusion contextual information, thereby enhancing power line detection performance.

[0117] Table 3: Quantitative results of ablation studies of key components in MCCFM

[0118] The proposed MCCFM and CAFM are compared with several mainstream multimodal fusion methods to verify their superiority in cross-modal feature fusion. These fusion methods are designed for RGB-TIR power line detection, semantic segmentation, and saliency detection, including "Summation", "Concatenation", "EF" (early fusion), "EF+SE", "FuseNet", "CWF" (channel-weighted fusion), "AF" (attention-based fusion), "AMFuse", "CMF" (fusion module based on spatial gate strategy and channel-weighted fusion strategy), "MFM", and "HFM". As shown in Table 4, the baseline model, denoted as "B", is obtained by removing MCCFM and CAFM from the complete PLNet. The proposed MCCFM and CAFM achieve the best quantitative results among these mainstream fusion modules. MCCFM enhances the model's ability to capture and integrate details and global information in images by extracting features at different scales, thereby improving spatial consistency. This is crucial for maintaining the continuity of power line detection results. CAFM further strengthens the model by integrating contextual information from deep features, providing a more comprehensive global feature representation. These two proposed modules significantly enhance the discriminative power of cross-modal context fusion features, enabling the model to better understand the overall scene and context.

[0119] Table 4: Quantitative Results of Different Cross-Modal Feature Fusion Methods

[0120] In environments such as low light, darkness, rain, snow, and fog, thermal images can more clearly delineate heated objects. This capability enables the network to accurately highlight targets and suppress background noise. By introducing thermal modes, power line detection becomes more accurate under extreme conditions. To demonstrate the role of thermal modes in power line detection, ablation experiments were conducted, and the results are shown in Table 6. Comparing the performance with and without RGB and TIR modes clearly shows that adding an infrared branch to the single-modal model significantly improves recall and IoU.

[0121] Table 6: Ablation experimental results under different modal configurations (%)

[0122] To fully validate the effectiveness and superiority of the proposed network, this invention compares PLNet with various state-of-the-art methods on the VITLD and GTPD datasets. These methods include RGB semantic segmentation models, RGB-TIR power line detection models, and RGB-TIR semantic segmentation models.

[0123] To verify the generalization ability of the proposed model under different weather conditions, various scenarios, including daytime, nighttime, fog, and heavy snow, were simulated using the VITLD dataset via OpenCV. The overall quantitative results and results under each weather condition are shown in Table 7. Clearly, the quantitative results obtained from the RGB-TIR power line detection method adapted from the semantic segmentation model are suboptimal compared to models specifically designed for RGB-TIR power line detection. This may be because these models do not fully consider the unique morphological features and thermal data characteristics of power lines. The proposed network significantly outperforms other methods, achieving the highest IoU (57.68%) among all state-of-the-art models. These results demonstrate that the proposed network achieves more complete power line segmentation under various weather conditions. This success is attributed to the multi-branch network structure adopted by PLNet. In the context branch, the proposed MCCFM and CAFM fully utilize the complementarity of cross-modal context; while in the detail branch, CAFEM improves the accuracy and completeness of power line detection by incorporating cross-modal context information into the overall inference process.

[0124] Table 7: Performance comparison of different models on the VITLD dataset (%)

[0125] The experimental results of PLNet on the GTPD dataset are shown in Table 8. For comparison, the numerical results of state-of-the-art methods are included in the table. On the GTPD dataset, the proposed model achieves a significant 63.2% IoU, clearly outperforming all other methods. Specifically, PLNet ranks first in IoU, outperforming the second-place method by 2.1%. This highlights the strong generalization ability of the proposed network on a wide range of datasets covering diverse scenes. For well-defined and unoccluded power line targets, all mentioned methods demonstrate good performance, but PLNet stands out due to its higher accuracy. However, when faced with complex scenes involving occlusion or obfuscated background objects, other power line detection methods and multimodal semantic segmentation methods, besides PLNet, often struggle to accurately locate power lines. This is mainly attributed to the carefully designed multi-branch strategy, which significantly enhances the contextual interaction between pixels while ensuring information integrity, crucial for accurate pixel classification.

[0126] Table 8: Performance Comparison of Different Models on the GTPD Dataset (%)

[0127] Another embodiment of the present invention also proposes a power line detection system based on multimodal fusion, comprising: The feature extraction module is used to acquire visible light RGB image data and thermal infrared TIR image data, and input them into the pre-established multimodal electric line detection network PLNet. The multimodal electric line detection network PLNet uses detail branches and context branches to extract different types of features. The feature processing module is used to fuse shallow multimodal context information for different types of extracted features through a multi-scale cross-modal context fusion method; at the same time, it evaluates the similarity between different modal features through feature decoupling and fusion, separates shared and differential features, and performs feature fusion based on the complementarity between different modal features to enhance the discriminative ability of features; and it uses context-aware fusion to extract the dependency relationship between dynamically captured electric field line features and surrounding pixels, and fuses deep multimodal context information. The feature decoding module is used to decode the features after fusing shallow multimodal context information, enhancing the discrimination ability, and fusing deep multimodal context information, to obtain the power line detection results after supplementing the detailed branch information of the multimodal context information.

[0128] Another embodiment of the present invention also provides an electronic device comprising: A memory for storing at least one instruction; and a processor for executing the instructions stored in the memory to implement the power line detection method based on multimodal fusion.

[0129] Another embodiment of the present invention provides a computer-readable storage medium storing at least one instruction, which is executed by a processor in an electronic device to implement the power line detection method based on multimodal fusion.

[0130] The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals. For ease of explanation, the above content only shows the parts related to the embodiments of the present invention; for specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. This computer-readable storage medium is non-transitory and can be stored in storage devices formed by various electronic devices, enabling the execution process described in the method of the embodiments of the present invention.

[0131] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0132] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0133] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0134] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A power line detection method based on multimodal fusion, characterized in that, include: Acquire visible light RGB image data and thermal infrared (TIR) ​​image data, and input them into a pre-established multimodal electric line detection network (PLNet). The PLNet uses detail branches and context branches to extract different types of features. For the extracted features of different types, shallow multimodal context information is fused through a multi-scale cross-modal context fusion method; at the same time, the similarity between different modal features is evaluated through feature decoupling and fusion, shared and differential features are separated, and feature fusion is performed based on the complementarity between different modal features to enhance the discriminative ability of features; It also employs context-aware fusion and extraction of dynamic capture of the dependencies between electric field line features and surrounding pixels, fusing deep multimodal contextual information; The features, after fusing shallow multimodal context information, enhanced discrimination capability, and fused deep multimodal context information, are decoded to obtain the power line detection results after supplementing the detailed branch information of the multimodal context information.

2. The power line detection method based on multimodal fusion according to claim 1, characterized in that, In the detail branch, the multimodal power line detection network PLNet uses a convolutional neural network (CNN) as the backbone network and removes the downsampling operation; in the context branch, the visual transformer (ViT) is used as the backbone network, and the global modeling capability of the self-attention mechanism is used for context feature inference.

3. The power line detection method based on multimodal fusion according to claim 1, characterized in that, In the step of fusing shallow multimodal context information by using a multi-scale cross-modal context fusion method for different types of extracted features, the features of visible light RGB image data and thermal infrared TIR image data are respectively processed by hollow spatial pyramid pooling to capture multi-scale single-modal context features, and then the features of different modalities but the same scale are mapped to a common feature space.

4. The power line detection method based on multimodal fusion according to claim 1, characterized in that, The steps of evaluating the similarity between different modal features through feature decoupling and fusion, separating shared and differential features, and performing feature fusion based on the complementarity between different modal features to enhance the discriminative ability of features include two stages: In the first stage, for the visible light RGB image data branch, the visible light RGB context features are concatenated with the thermal infrared (TIR) ​​context features to calculate their shared features, denoted as... : In the formula, Conv(*) represents a convolutional block with a 3×3 convolutional layer; Cat (*;*) indicates a cascading operation; the symbol Represents the Hadamard product; by subtracting shared features from an all-one matrix. By measuring the differences in corresponding spatial locations, complementary information weights can be obtained. The calculation expression is as follows: In the formula, Softmax (*) denotes the normalization function. I Represents a matrix consisting entirely of 1s; in In this context, a higher value indicates stronger contextual complementarity in the same region across different modalities, and vice versa; complementary information weights are utilized. and Calculate complementary features of TIR images Complementary features of RGB images : In the formula, the symbol ⊙ represents element-wise multiplication; In the second stage, through calculation and Cosine similarity is used to measure the difference information of corresponding spatial locations, and the corresponding context similarity weights are obtained. The expression is as follows: In the formula, Cos (*) refers to cosine similarity calculation; in In this context, a higher value indicates a stronger contextual similarity in the same region across different modalities, and vice versa. Obtain context similarity weights Subsequently, features are shared across modal contexts. and Calculated in the following way: By fusing complementary contextual features and shared contextual features, enhanced contextual features are obtained, i.e. and : In the formula, This indicates a 3×3 convolutional layer followed by a BN layer and a ReLU activation function; Obtain enhanced contextual features at four scales and Next, the layers are first concatenated along the channel dimension, and then 1×1 convolutions are used to fuse multi-scale contextual information. The mathematical expression is as follows: Constructing cross-modal attention operations to enable the interaction and combination of information from different modalities, including processing data of size 10 ... The input features are flattened as ,in N = H * W Then, a vector of query Q, key K, and value V is generated through linear projection; A global attention interaction is performed between the key K and value V vectors of the current modality and the query Q vectors of other modalities to capture long-distance dependencies between contextual features of different modalities, as expressed mathematically below: In the formula, the symbol Represents matrix multiplication; Indicates matrix transpose; ), ), This represents three linear projection layers, with parameters for each layer as follows: , and ; Will and Input subsequent context-aware fusion to supplement the contextual information of detailed branches; Multimodal context features Perform contextual attention fusion to enhance the interaction of contextual features at different levels.

5. The power line detection method based on multimodal fusion according to claim 4, characterized in that, The context attention fusion introduces vertical and horizontal compressed attention mechanisms to capture global context dependencies in the vertical and horizontal directions, respectively; it simultaneously captures important local information and global context information through global average pooling (GAP) and global max pooling (GMP) operations; the mathematical expression for the context attention fusion is as follows: In the formula, and This represents 1×1 and 3×3 convolutional layers with parameters γ and α, followed by BN layers and ReLU activation functions; Cat(*,*) represents cascaded operations along channels; AvP(*) and MaxP(*) represent global average pooling and global max pooling operations, respectively; by introducing horizontal and vertical attention mechanisms, the correlation between different locations in the feature map is captured; For the visible light RGB image data branch, axially enhanced contextual features and The calculation expression is as follows: Axially enhanced RGB and TIR features, i.e. and Obtained through the following methods: In the formula, and These represent the mean calculations along the vertical and horizontal directions, respectively; Att(*) indicates multi-head attention operation; axial enhancement feature. and Interacting with contextual information from the previous layer enables effective fusion of contextual information from different levels and modalities; multimodal contextual features Calculated in the following way: Will and Input context-aware fusion to supplement contextual information for detailed branches; multimodal contextual features Input-following context attention fusion to enhance the interaction of contextual features at different levels.

6. The power line detection method based on multimodal fusion according to claim 5, characterized in that, The steps of employing context-aware fusion and extracting the dynamic capture of the dependency relationship between electric field line features and surrounding pixels, and fusing deep multimodal contextual information, include: context-aware feature fusion extracts electric field line features using dynamic serpentine convolution, adaptively adjusting the convolution kernel shape, and introducing multimodal contextual information to supplement detail branches in context-aware feature fusion, thereby establishing a contextual association between the electric field lines and the surrounding environment; given axially enhanced RGB features and TIR features, i.e. and and the detailed branch features at the current level, i.e. and The global attention map is calculated, and the global context information is modeled as a weighted average of feature space locations. The global context information is aggregated to each location of the feature through element-wise addition. For the visible light RGB image data branch, the mathematical expression is: In the formula, , and Indicates a parameter { },{ }and{ The algorithm consists of 1×1 and 3×3 convolutional layers, followed by BN layers and ReLU activation functions; subsequently, dynamic serpentine convolutions are used to adaptively extract electric field features; and then, these features are combined with contextual features. Interactively establish the contextual relationship between the extracted electric field line features and different modal features; Features that supplement contextual information Calculated according to the following expression: In the formula, DSConv (*) indicates dynamic serpentine convolution. This indicates element-wise multiplication.

7. The power line detection method based on multimodal fusion according to claim 6, characterized in that, The step of decoding the features after fusing shallow multimodal context information, enhancing discriminative ability, and fusing deep multimodal context information to obtain the power line detection result after supplementing the multimodal context information with detailed branch information includes: Given the outputs from the visible light RGB image data branch and the thermal infrared TIR image data branch and They are concatenated and fused together using 1×1 convolutional blocks to obtain Sum the outputs to get { |i=0,1,2} is used to progressively restore details for the final prediction. ; The space size is 1 / 4 of the original input data, and the mathematical expression is as follows: In the formula, {DBR(*; )} and {DBR(*; )} indicates that it has parameters. and A 3×3 deconvolutional layer, followed by a BN layer and a ReLU activation function; SegHead represents the probability map used for prediction. 1×1 convolution; output features that fuse multi-scale cross-modal context and contextual attention. The prediction of the auxiliary supervision probability map is based on three segment heads. The mathematical expression is as follows: The loss function of the multimodal power line detection network PLNet consists of two parts: a main loss function and an auxiliary loss function. In the formula, , and There are three hyperparameters used to balance the weights among the losses; the final probability plot. Depend on Supervision, auxiliary probability diagrams are generated by , and Supervision: Since the special shape of the power line target occupies only a small portion of all pixels, the proposed network uses a combination of binary cross-entropy and Dice loss as the loss function, as shown in the following mathematical expression: In the formula, It is a hyperparameter used to balance the weights of the binary cross-entropy and the Dice loss, in the context of... This represents the predicted probability of the i-th pixel. This represents the corresponding true label, and N represents the total number of input pixels. These are model parameters.

8. A power line detection system based on multimodal fusion, characterized in that, include: The feature extraction module is used to acquire visible light RGB image data and thermal infrared TIR image data, and input them into the pre-established multimodal electric line detection network PLNet. The multimodal electric line detection network PLNet uses detail branches and context branches to extract different types of features. The feature processing module is used to fuse shallow multimodal context information for different types of extracted features through a multi-scale cross-modal context fusion method; at the same time, it evaluates the similarity between different modal features through feature decoupling and fusion, separates shared and differential features, and performs feature fusion based on the complementarity between different modal features to enhance the discriminative ability of features; and it uses context-aware fusion to extract the dependency relationship between dynamically captured electric field line features and surrounding pixels, and fuses deep multimodal context information. The feature decoding module is used to decode the features after fusing shallow multimodal context information, enhancing the discrimination ability, and fusing deep multimodal context information, to obtain the power line detection results after supplementing the detailed branch information of the multimodal context information.

9. An electronic device, characterized in that, include: Memory, storing at least one instruction; and The processor executes instructions stored in the memory to implement the power line detection method based on multimodal fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is executed by a processor in an electronic device to implement the power line detection method based on multimodal fusion as described in any one of claims 1 to 7.

Images