Cross-modal fusion transmission line bird detection method and device

Abstract

The invention relates to a cross-modal fusion transmission line bird detection method and device. Firstly, a dual-light image is collected through a visible light camera and an infrared camera; thirdly, a transmission line tower bird detection model VIF-YOLO is provided for bird detection, YOLOv11 is used as a basic framework in the method, meanwhile, a backbone network is designed to be of a double-branch structure, and features of visible light and infrared images are extracted respectively; secondly, designing two lightweight plug-and-play modules, namely a bimodal Mamba fusion DMF (Dimethyl Formamide) module and a multi-scale feature enhancement MFE (Multiscale Feature Enhancement) module; the DMF utilizes the global modeling capability of Mamba to realize deep fusion among different modals while keeping linear complexity; the MFE passes through multiple groups of one-dimensional convolution and a multi-dimensional collaborative attention mechanism MCA; thirdly, deploying the model to an edge terminal for real-time detection; and finally, providing a time difference CIOU dynamic start-stop strategy to realize dynamic bird repelling. Through example verification, the method provided by the invention has relatively high accuracy and real-time performance on bird detection, and can realize dynamic bird repelling.

Description

Technical Field

[0001] This invention relates to the field of power transmission line fault diagnosis technology, and in particular to a cross-modal fusion method and equipment for bird detection in power transmission lines. Background Technology

[0002] With the development of urbanization, the contradiction between the widespread construction of transmission lines and bird activities has become increasingly prominent. The number of transmission line trips caused by bird activities is second only to lightning strikes and external damage, and it is showing an increasing trend year by year. The main types are as follows: (1) Bird droppings: When birds defecate near the tower, the droppings form a conductive channel, causing the air gap of the tower to break down, or the droppings adhere to the insulators and cause surface flashover, resulting in the tripping of the overhead transmission line. (2) Bird pecking: Birds peck at the skirts or sheaths of composite insulators, causing damage to the composite insulators and endangering the safe operation of the line. (3) Bird nest material: When birds build nests on the tower, the material in the nest falls between the lines, which reduces or short-circuits the air gap, resulting in the tripping of the overhead transmission line. (4) Bird body short circuit: Birds' bodies come into contact with the air gap between phases or between phase and ground of the overhead transmission line, causing air breakdown and causing the overhead transmission line to trip. Early bird control efforts relied primarily on manual inspections and traditional non-identifying bird deterrents. Manual inspections were inefficient and unable to achieve real-time bird control; while traditional bird deterrents used noise and lasers, employing a single, fixed method that birds easily adapted to. To address these issues, flexible and adaptable bird control solutions must be developed, with the key being the ability to detect birds in real time. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a cross-modal fusion transmission line bird detection method and equipment, which has high accuracy and real-time performance in bird detection and can achieve dynamic bird deterrence.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: a cross-modal fusion method for bird detection in transmission lines, comprising the following steps:

[0005] Step 1: Simultaneously acquire visible light and infrared image data using a dual-modal sensor, and align the two through calibration; input the processed data into the backbone network to extract features from the visible light and infrared images respectively;

[0006] Step 2: Utilize Mamba's efficient global modeling capabilities to design a dual modal Mamba fusion (DMF) module to embed into the backbone network; this DMF module includes a feature compression and enhancement (FCE) module, a multi-path Mamba fusion (MPMF) module, and a lightweight attention-gated multilayer perceptron (LAG) module;

[0007] Step 3: Design a multi-scale feature enhancement (MFE) module, which includes pinwheel-shaped convolution (PConv), three sets of one-dimensional convolutions of different sizes, and a multi-dimensional collaborative attention (MCA) module.

[0008] Step 4: The backbone network is connected to a neck structure, through which further feature extraction is performed;

[0009] Step 5: Input the features extracted from the neck structure into the head structure for target detection;

[0010] Step 6: Deploy the trained model to the edge terminal. Perform quantization operations on the PyTorch model trained on the GPU and deploy it to the edge device for image inference and detection.

[0011] Step 7: Start the bird deterrent device using the time-difference CIOU dynamic start-stop strategy.

[0012] In a preferred embodiment, step 1 specifically includes: collecting experimental data multiple times at different times and locations, deleting duplicate and corrupted data, and collecting a total of 11,109 frames each of visible light and infrared images. Scene enhancement is used to add varying degrees of rain, fog, and strong sunlight conditions to some of the original data to construct the VIB2025 dataset. All datasets are divided into training, validation, and test sets in an 8:1:1 ratio. The dataset is implemented using the VIF-YOLO architecture, which includes a backbone network combining single-modal feature extraction and dual-modal feature fusion, a neck structure, and a head structure. The backbone network is redesigned from a single-branch structure to a dual-branch structure to extract visible light and infrared image features respectively.

[0013] In a preferred embodiment, in the feature compression and enhancement module FCE, given visible light image features and infrared image features Image features are segmented into four small blocks using a slicing operation, each considered a local region of the image. Slicing increases the proportion of the target feature within its respective region, making it more prominent and helping to capture finer texture, shape, and edge information. Next, pointwise convolution is used to perform a non-linear transformation of the channel dimension on each local block, enhancing feature discriminativity and enabling cross-channel information exchange. Finally, a depthwise convolution is used for feature integration and channel dimensionality reduction to generate the final feature. and The expression is:

[0014]

[0015] in, This indicates a slice operation. This represents pointwise convolution. This indicates a depthwise convolution operation.

[0016] In a preferred embodiment, the multipath Mamba fusion module (MPMF) first calculates the temporary fusion features of the three paths respectively. , and :

[0017]

[0018] here , Cat(·) represents element-wise addition, subtraction, and channel concatenation operations, respectively; SA(·) represents spatial attention operation, and ECA(·) represents efficient channel attention operation; element-wise addition achieves smooth feature fusion by adding pixels one by one, which is suitable for regions with high intermodal consistency; while element-wise subtraction highlights complementary information between modalities by calculating differences. After the fusion of element-wise addition and subtraction, a spatial attention (SA) module is used to focus on the target features; efficient channel attention (ECA) operation and deep convolution are used to reduce channel redundancy;

[0019] Next , and Furthermore, the input to the 2D Selective Scan (SS2D) module is fused through element-wise addition, and residual branches are introduced to obtain features. This is used to capture long-range spatial dependencies; it is represented as:

[0020]

[0021] In the formula, SS2D(·) represents the SS2D operation.

[0022] In a preferred embodiment, in the lightweight attention-gated multilayer perceptron module (LAG), it is assumed that the characteristics of the input LAG module are as follows: First, the SA module is used to enhance the modeling capability of spatial information, and then pointwise convolution is used to implement fully connected operations.

[0023]

[0024] Immediately afterwards The feature is projected onto two branches. One branch uses a depthwise separable convolution with a residual branch for feature extraction, and the GeLu activation function is used to control the flow of feature information at each layer. Then, the two branches are merged through element-wise multiplication, and a pointwise convolution is used to refine the global features and enhance the feature representation capability. Finally, the extracted features are added to the input features through a residual branch.

[0025]

[0026] In the formula This represents element-wise multiplication. represents the activation function, and Linear(·) represents the linear projection.

[0027] In a preferred embodiment, the features of the input MFE module are described in step S3. First, features are extracted using PConv:

[0028]

[0029] In the formula, PConv(·) represents the PConv operation;

[0030] Next, three sets of horizontal and vertical one-dimensional convolutional kernels with kernel sizes of 3, 5, and 7 are used to capture detailed information of small targets at multiple scales, as shown below:

[0031]

[0032] Then, MCA is used to extract multi-dimensional features from features at different scales;

[0033] After extracting multi-dimensional features, the features at three different scales and the residual branches are merged through a concatenation operation. Finally, a 1×1 convolution is used to effectively integrate the features at different scales, and the number of channels in the feature map is adjusted, as shown below:

[0034]

[0035] In the formula, MCA(·) represents the multidimensional collaborative attention mechanism.

[0036] In a preferred embodiment, in step S4, the neck structure connects the middle part of the backbone network and the head structure, and through upsampling and lateral connection, feature maps of different scales are fused to enhance the model's ability to detect targets of different sizes.

[0037] In a preferred embodiment, in step S5, the 20×20 detection head originally used for large target detection is replaced with a 160×160 miniature detection head (Tiny Head).

[0038] In a preferred embodiment, step S7: The principle of the time-difference CIOU dynamic start-stop strategy is as follows: Assuming the first frame prediction box is A, and the second frame prediction box B is obtained after an interval of 3 seconds, where... Find the smallest bounding box C that can simultaneously enclose A and B, where a and b represent the center points of the three boxes A and B respectively. Then calculate CIOU:

[0039]

[0040] When the difference CIOU is greater than 0.5, it is determined that birds are perched on the transmission line tower, and the bird deterrent device is activated to drive them away.

[0041] The present invention also provides a cross-modal fusion transmission line bird detection device, comprising:

[0042] The processor, memory, and bus, wherein the memory stores machine-readable instructions executed by the processor;

[0043] When the device is running, the processor communicates with the memory via a bus, and when the machine-readable instructions are executed by the processor, a cross-modal fusion transmission line bird detection method as described above is performed.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. Most existing bird fault detection technologies are based on single-modal detection using visible light and infrared images. However, visible light images are easily affected by weather conditions and cannot meet the requirements for all-weather detection; infrared images suffer from low resolution and blurred details, resulting in poor bird identification. Compared to single-modal detection, target detection based on multimodal fusion can provide richer feature information and achieve information complementarity between different modalities.

[0046] 2. To fully integrate information from visible light and infrared images and compensate for the limitations of a single modality, early deep learning-based fusion methods typically employed Convolutional Neural Networks (CNNs). However, CNN-based fusion networks are limited by their finite receptive fields, making it difficult to capture global contextual information. With the rise of Transformers, several fusion methods combining the advantages of CNNs and Transformers have been proposed, addressing the problem of CNNs' difficulty in extracting global features. However, the Transformer's self-attention mechanism has quadratic complexity, resulting in high computational costs and making it unsuitable for deployment on edge devices. This method designs an efficient DMF module that, through multi-path fusion design and Mamba's long-range modeling capabilities, adaptively fuses features from different modalities with lower computational complexity.

[0047] 3. To address the issue of the easy loss of small bird features in deep networks, a multi-scale feature enhancement (MFE) module was designed to strengthen the extraction of edge features of small bird targets and enhance the model's ability to identify small targets in complex environments.

[0048] 4. Design a time-difference CIOU dynamic start-stop strategy. By predicting the CIOU of the target bounding box from two frames of images at a certain time interval, it can determine whether birds have been perched on the power transmission line tower for an extended period of time. This effectively solves the problem of wasting resources caused by prolonged operation of bird deterrents, avoids birds developing adaptation, and can greatly reduce computational complexity. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating a preferred embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of the overall structure of VIF-YOLO according to a preferred embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the DMF module structure according to a preferred embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram of the MPMF module structure according to a preferred embodiment of the present invention;

[0053] Figure 5 This is a schematic diagram of the LAG module structure according to a preferred embodiment of the present invention;

[0054] Figure 6 This is a schematic diagram of the MFE module structure according to a preferred embodiment of the present invention;

[0055] Figure 7 This is a schematic diagram of the PConv module structure according to a preferred embodiment of the present invention;

[0056] Figure 8 This is a schematic diagram of the MCA module structure according to a preferred embodiment of the present invention; Figure 9 This is a schematic diagram of an edge terminal platform according to a preferred embodiment of the present invention. Detailed Implementation

[0057] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0058] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0059] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations according to this application; as used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise; furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0060] refer to Figure 1-8 In this embodiment, a dual-spectral bird dataset in the scenario of a power transmission line tower was collected and created using a visible light + thermal imaging binocular sensor for verification.

[0061] In this embodiment, the bird detection system needs to operate under extreme weather conditions. Therefore, this invention simulates different levels of rainy, foggy, and high-exposure weather conditions by adjusting image brightness, contrast, color, and adding noise. Furthermore, the bird detection task is inevitably affected by the activities of line maintenance personnel; therefore, the dataset of this invention includes labels for both birds and humans.

[0062] This embodiment proposes a novel cross-modal fusion method for bird detection on transmission lines. The specific implementation process includes the following:

[0063] Experimental data was collected multiple times at different times and locations. After removing duplicate and corrupted data, a total of 11,109 frames of visible light and infrared images were collected. Scene enhancement was then used to add varying degrees of rain, fog, and strong sunlight conditions to some of the original data to construct the VIB2025 dataset. All datasets were divided into training, validation, and test sets in an 8:1:1 ratio.

[0064] The collected data was used for training on a computer equipped with an Intel(R) Core(TM) i7-14700 CPU, GeForce RTX 4060 GPU, 32GB of system memory, and Ubuntu 22.04.5LTS operating system. The training was based on the PyTorch 2.4.1 framework, accelerated using CUDA 11.8, and implemented with a Python 3.11.11 interpreter. For training the VIF-YOLO model, we used a momentum of 0.937, a weight decay of 0.0005, an initial learning rate of 0.01, and applied a cosine annealing-scheduled SGD optimizer. The model was then deployed to an edge terminal, where a bird deterrent was started using a time-difference CIOU strategy. The specific steps included:

[0065] Step S1: Input the processed data into the backbone network to extract features from visible light and infrared images respectively. The overall architecture of VIF-YOLO is as follows: Figure 2 As shown, the method mainly consists of three parts: a backbone network that integrates single-modal feature extraction and dual-modal feature fusion, neck structure, and head structure. Based on the YOLOv11 framework, this invention redesigns the backbone network from a single-branch structure to a dual-branch structure to fully utilize the complementary information of the two modalities, extracting features from visible light and infrared images respectively. Small target features in birds are more prominent in shallow feature maps, so the focus is on enhancing the extraction of shallow features. Therefore, a Small Backbone structure is adopted, which removes the last CBS layer and C3K2 layer of the original backbone network, enhancing the model's attention to shallow features.

[0066] Step S2: To effectively promote the fusion of visible light and infrared image features, a simple and efficient DMF module was designed and embedded into the backbone network using Mamba's efficient global modeling capabilities (e.g., Figure 3 As shown in the figure, this module mainly consists of three components: Feature Compression and Enhancement Module (FCE), Multipath Mamba Fusion Module (MPMF), and Lightweight Attention Gated Multilayer Perceptron Module (LAG).

[0067] FCE module such as Figure 3 The left side shows the given visible light image features. and infrared image features The image features are divided into four small blocks using a slicing operation, each of which can be considered a local region of the image. Slicing increases the proportion of the target feature within its respective block, making it more prominent and helping to capture finer texture, shape, and edge information. Next, pointwise convolution is used to perform a non-linear transformation of the channel dimension on each local block, enhancing the discriminative power of the features and enabling cross-channel information exchange. Finally, a depthwise convolution is used for feature integration and channel dimensionality reduction to generate the final feature. and The expression is:

[0068]

[0069] in, This indicates a slice operation. This represents pointwise convolution. This indicates a depthwise convolution operation.

[0070] Unlike most single-path fusion methods, this paper innovatively proposes a Mamba-based multi-path fusion architecture (MPMF) to fully explore the synergistic and complementary information among multimodal data. This architecture adaptively optimizes paths and enhances the fusion of information from different modalities. MPMF modules include... Figure 4 As shown, the temporary fusion features of the three paths are first calculated separately. , and :

[0071]

[0072] here , `Cat(·)` and `Cat(·)` represent element-wise addition, subtraction, and channel concatenation operations, respectively; `SA(·)` represents spatial attention, and `ECA(·)` represents efficient channel attention. Element-wise addition achieves smooth feature fusion by adding pixels one by one, suitable for regions with high intermodal consistency; while element-wise subtraction highlights complementary information between modalities through difference calculation, such as the difference between visible light texture information and the thermal radiation difference in infrared images. The branch after element-wise addition and subtraction fusion is then processed by a spatial attention module `SA` to focus on the target features; channel concatenation can retain all the information of the original modality, but may lead to channel information redundancy, so efficient channel attention operation `ECA` and depthwise convolution are used to reduce channel redundancy.

[0073] Next , and The input to the SS2D module is further fused through element-wise addition, and residual branches are introduced to obtain features. This is used to capture long-range spatial dependencies. It is represented as:

[0074]

[0075] In the formula, SS2D(·) represents the SS2D operation. This architecture constructs a dynamic feature fusion space through three temporary interaction paths, which is ultimately input into the SS2D module to enhance global modeling capabilities.

[0076] The core advantage of the Multi-layer Perceptron (MLP) lies in its ability to learn complex nonlinear relationships between inputs and outputs. The Gated Multi-layer Perceptron (GMLP) introduces a gating mechanism into the MLP, enhancing its feature selection capabilities and computational efficiency while maintaining its simplicity. Based on this, a LAG module was designed, such as... Figure 5 As shown.

[0077] Assume the characteristics of the input LAG module are First, the SA module is used to enhance the modeling capability of spatial information, and then pointwise convolution is used to implement fully connected operations.

[0078]

[0079] Immediately afterwards The projection is applied to two branches. One branch uses a depthwise separable convolution with a residual branch for feature extraction, and the GeLu activation function is used to control the flow of feature information at each layer. Then, the two branches are merged through element-wise multiplication, and a pointwise convolution is used to refine the global features, enhancing feature representation. Finally, the extracted features are added to the input features through a residual branch, more effectively re-injecting gradients.

[0080]

[0081] In the formula This represents element-wise multiplication. represents the activation function, and Linear(·) represents the linear projection.

[0082] Step S3: Small target features based on convolutional neural networks are prone to disappearance, which limits the detection accuracy. To address the problem that bird target features are easily lost in deep networks, a multi-scale feature enhancement (MFE) module was designed to strengthen the extraction of bird target edge features and enhance the model's ability to identify targets in complex environments.

[0083] like Figure 6 As shown, the MFE module mainly includes a windmill-shaped convolutional module (PConv), three sets of one-dimensional convolutions of different sizes, and a multi-dimensional collaborative attention module (MCA). The features input to the MFE module... First, features are extracted using PConv:

[0084]

[0085] In the formula, PConv(·) represents the PConv operation. For example... Figure 7As shown, PConv performs asymmetric padding on the input feature map. For example, the padding parameter P(1,0,0,3) represents the number of pixels padded in the top, bottom, left, and right directions as 1, 0, 0, and 3, respectively. Asymmetric padding generates four parallel, directional one-dimensional convolutional paths. The features extracted from these four paths are concatenated and then passed through a convolutional layer for cross-path information interaction and fusion. Its unique "windmill-shaped" structure, forming a dense center and sparse periphery, provides an effective receptive field that is highly compatible with the physical characteristics of small targets.

[0086] Next, three sets of horizontal and vertical one-dimensional convolutional kernels with kernel sizes of 3, 5, and 7 are used to capture detailed information about small targets at multiple scales, which can be represented as:

[0087]

[0088] Then, MCA is used to extract multi-dimensional features from features at different scales. For example... Figure 8 As shown, MCA processes the input feature maps in parallel across the height, width, and channel dimensions to learn the relationships between features across dimensions. The mechanism first permutes the input feature maps, then applies average pooling and standard deviation pooling (AvPool & StdPool) to compress the features. Next, it generates attention weights through one-dimensional convolution and activation functions, and weights the original feature maps to highlight important features and suppress unimportant information. Finally, the feature maps obtained from the three dimensions are fused to enhance the model's ability to capture key information.

[0089] After extracting multi-dimensional features, the features at three different scales and the residual branches are merged through a concatenation operation. Finally, a 1×1 convolution is used to effectively integrate the features at different scales, and the number of channels in the feature map is adjusted, which can be expressed as:

[0090]

[0091] In the formula, MCA(·) represents the multidimensional collaborative attention mechanism.

[0092] Step S4: Input the features extracted in the above steps. Figure 2 The neck structure further extracts features. The neck, the middle part connecting the backbone network and the detection head, primarily serves to further process and optimize the features extracted by the backbone network. The design of the neck influences the feature fusion and propagation methods, thus significantly impacting detection performance. Through upsampling and lateral connections, feature maps of different scales are effectively fused, enhancing the model's ability to detect targets of varying sizes.

[0093] Step S5: The features extracted from the neck structure are input into the detection head for target detection. To improve the detection efficiency of birds, the original 20×20 detection head used for large target detection is replaced with a 160×160 tiny head specifically for small target detection, allowing the model to focus more on the detailed features of small targets.

[0094] Step S6: Deploy the trained model to the edge terminal. Perform quantization on the PyTorch model trained on the GPU and deploy it to the edge device for image inference and detection. Actual deployment details are as follows: Figure 9 As shown.

[0095] Table 1 shows the detection results of the VIF-YOLO model on edge devices and computers, where FPS is the number of frames detected per second, and AP and R are the average precision and recall rates. The VIF-YOLO model designed in this invention maintains high accuracy and detection speed on both edge devices and computers, meeting the needs of practical applications.

[0096] Table 1. Detection results of the VIF-YOLO model on edge devices and computers.

[0097] Step S7: Activate the bird deterrent device using the time-difference CIOU dynamic start-stop strategy. The principle of the time-difference CIOU dynamic start-stop strategy is as follows: Assuming the first frame prediction box is A, and the second frame prediction box B is obtained after a 3-second interval, where... Find the smallest bounding box C that can simultaneously enclose A and B, where a and b represent the center points of the three boxes A and B respectively. Then calculate CIOU:

[0098]

[0099] When the difference CIOU is greater than 0.5, it can be determined that birds are perched on the transmission line tower, and the bird deterrent device will be activated to drive them away.

[0100] This patent is not limited to the above-described preferred embodiments. Anyone can derive other forms of bird detection methods and systems for cross-modal transmission line towers based on the inspiration of this patent. All equivalent changes and modifications made within the scope of this patent application shall fall within the scope of this patent.

Claims

1. A cross-modal fusion power transmission line bird detection method, characterized in that, The method comprises the following steps: Step 1: Simultaneously collect visible light and infrared image data by using a dual-mode sensor, and realize alignment of the two through calibration; input the processed data into a backbone network to extract features of the visible light and infrared images respectively; Step 2: Design a dual-mode Mamba fusion module DMF embedded in the backbone network by using the efficient global modeling capability of Mamba; the dual-mode Mamba fusion module comprises a feature compression and enhancement module FCE, a multi-path Mamba fusion module MPMF and a lightweight attention gate multi-layer perception module LAG; Step 3: Design a multi-scale feature enhancement MFE module, which comprises a pinwheel convolution PConv, three groups of one-dimensional convolutions with different sizes and a multi-dimensional collaborative attention module MCA; Step 4: The backbone network is connected with a neck structure, and further feature extraction is performed through the neck structure; Step 5: The features extracted by the neck structure are input into a head structure for target detection; Step 6: Deploy the trained model to an edge terminal, perform quantization operation on the Pytorch model trained on the GPU, and deploy it to the edge device for image inference detection; Step 7: Start the bird repeller through the difference time CIOU dynamic start-stop strategy.

2. The method of claim 1, wherein, The step 1 specifically comprises: realizing through a VIF-YOLO architecture, the VIF-YOLO architecture comprising a backbone network integrating single-mode feature extraction and dual-mode feature fusion, a neck structure and a head structure; designing the backbone network from a single-branch structure to a double-branch structure to extract visible light and infrared image features respectively.

3. The method of claim 1, wherein, In the feature compression and enhancement module FCE, given visible light image features and infrared image features are respectively divided into four small blocks by slicing operation, each of which is regarded as a local region of the image; through slicing operation, the proportion of the target feature in the region block where it is located is correspondingly increased, so that its feature is more prominent, which helps to capture more subtle texture, shape and edge information; then point-wise convolution is used to perform nonlinear transformation on the channel dimension of each local block, to enhance the discriminability of the feature and realize cross-channel information interaction; finally, a deep convolution is used for feature integration and channel dimension reduction, to generate feature and , and the expression is: wherein, represents a slice operation, represents a point-wise convolution, represents a depthwise convolution operation.

4. The method of claim 1, wherein, In the multi-path Mamba fusion module MPMF, first, the temporary fusion features of the three paths are calculated respectively , and : Here , and Cat(·) represent element-wise addition, subtraction and channel concatenation operation respectively; SA(·) denotes spatial attention operation, and ECA(·) denotes efficient channel attention operation; element-wise addition realizes feature smooth fusion through pixel-wise addition, which is suitable for regions with high inter-modality consistency; and element-wise subtraction highlights the complementary information between modalities through difference calculation, and the branches fused through element-wise addition and subtraction are focused on target features through a spatial attention module SA; Efficient channel attention ECA operation and deep convolution are used to reduce channel redundancy; Next, we , and further fuse the input modules SS2D by element-wise addition and introduce a residual branch to obtain the feature to capture long-range spatial dependencies; It is expressed as: In the formula, SS2D(·) represents the SS2D operation.

5. The method of claim 1, wherein, In the lightweight attention gate multi-layer perception module LAG, it is assumed that the features input into the LAG module are First, the SA module is used to enhance the modeling capability of spatial information, and then point-wise convolution is used to realize full connection operation. Immediately afterwards The feature is projected onto two branches. One branch uses a depthwise separable convolution with a residual branch for feature extraction, and the GeLu activation function is used to control the flow of feature information at each layer. Then, the two branches are merged through element-wise multiplication, and a pointwise convolution is used to refine the global features and enhance the feature representation capability. Finally, the extracted features are added to the input features through a residual branch. In the formula represents an element multiplication operation, represents an activation function, and Linear(·) represents a linear projection.

6. The method of claim 1, wherein, Features input to the MFE module in step S3 First, features are extracted by PConv: In the formula, PConv(·) represents the PConv operation; Then, three groups of convolution kernel sizes of 3, 5 and 7 horizontal and vertical one-dimensional convolution kernels are used to capture small target detail information from multiple scales, expressed as: Then, the MCA is used to extract multi-dimensional features from different scale features; After extracting the multi-dimensional features, the three different scale features and the residual branch are combined through a splicing operation; finally, 1x1 convolution is used to effectively integrate the features of different scales and adjust the channel number of the feature map, expressed as: In the formula, MCA(·) represents the multi-dimensional collaborative attention mechanism.

7. The method of claim 1, wherein, In step S4, the neck structure is connected to the middle part of the backbone network and the head structure, and different scale feature maps are fused through upsampling and horizontal connection to enhance the detection capability of the model for targets of different sizes.

8. The method of claim 1, wherein, In step S5, the original 20x20 detection head for large target detection is replaced by a 160x160 micro detection head Tiny Head.

9. The method of claim 1, wherein, Step S7: The principle of the difference time CIOU dynamic start-stop strategy is as follows: assuming that the first frame prediction box is A, and the second frame prediction box B is obtained after an interval of 3 seconds, wherein , the smallest box C capable of simultaneously surrounding A and B is found, wherein , a and b respectively represent the center points of the three boxes of A and B, and then CIOU is calculated: When the difference time CIOU is greater than 0.5, it is judged that birds stay on the power line tower, and the bird repeller is started to drive the birds.

10. A cross-modal fusion power line bird detection device, characterized in that, It comprises: a processor, a memory and a bus, the memory storing machine readable instructions executed by the processor; When the device is running, the processor communicates with the memory through a bus, and the machine readable instructions are executed by the processor to perform the method of claim 1-9.

Images