Power transmission equipment detection method based on multispectral image fusion
By using the YOLOv8 network structure and feature fusion module in multispectral image fusion, the infrared and visible images are integrated in the medium term, which solves the problems of image quality reduction and detail loss in the prior art, and achieves higher detection accuracy and robustness.
Patent Information
- Application Number
- CN202411797448.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-09
- Publication Date
- 2025-04-15
AI Technical Summary
Existing multispectral image fusion methods are noise-sensitive in pixel-level fusion, which may lead to image quality degradation, and simple pixel-level fusions cannot fully utilize key information in the image, which may lead to the loss of important details.
The YOLOv8 network structure is adopted to perform mid-term fusion of infrared images and visible light images. By setting up two YOLOv8 backbone networks, one for feature extraction of infrared images and the other for visible light images; at the junction of backbone network and neck network, a feature fusion module is added to fuse infrared features and visible light features, and feature enhancement and suppression is performed through differential feature enhancement fusion module and common feature selection fusion module.
Effectively integrate information from infrared and visible light images to significantly improve the accuracy and reliability of detection. Especially in the case of insufficient light or complex environment, infrared images can maintain effective information and enhance the robustness of the system.
Smart Images
Figure CN119832371A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of image fusion, and in particular relates to a power transmission equipment detection method based on multi-spectral image fusion. Background Art
[0002] In the multispectral image target detection task, the fusion of multispectral images is crucial. Images in different bands can provide rich information. For example, visible light images can provide rich color and detail information, while infrared images are sensitive to temperature changes and can provide the thermal radiation characteristics of the target under insufficient light conditions. Therefore, how to effectively fuse multispectral images to make full use of the characteristics of each band is an important way to improve detection accuracy. Common multispectral image fusion methods include pixel-level fusion, feature-level fusion, and decision-level fusion. Among them, feature-level fusion can be further divided into early fusion, mid-term fusion, and late fusion.
[0003] Pixel-level fusion is a direct fusion method that generates a comprehensive image containing multispectral information by fusing images from different bands at the pixel level. Specifically, this method operates on images of different bands at the pixel level, such as weighted averaging, direct superposition, or fusion using transform domain methods. This fusion method has a simple implementation process and a small amount of calculation. It is suitable for scenes with high real-time requirements and is often used in the early stages of multispectral image fusion. Due to the direct processing of pixel values, the fused image is very sensitive to noise and may result in a decrease in image quality. The distribution of image information in different bands usually varies greatly. Simple pixel-level fusion cannot make full use of the key information in the image and may result in the loss of important details. Summary of the invention
[0004] In view of this, the present invention proposes a power transmission equipment detection method based on multispectral image fusion, performs mid-term fusion of infrared images and visible light images, adopts a YOLOv8 network structure, and sets two YOLOv8 backbone network settings, one for feature extraction of infrared images and the other for extraction of visible light images; at the junction of the backbone network and the neck network, a feature fusion module is added to fuse the infrared features with the visible light features, and the fused features are sent to the neck network for convolution, up-scaling and splicing.
[0005] Preferably, the feature fusion module includes a difference feature enhancement fusion module and a common feature selection fusion module to perform difference feature enhancement and common feature suppression.
[0006] Preferably, the difference feature enhancement fusion module performs the differential feature map F d Perform channel attention and spatial attention mechanisms to extract the difference channel weights w d1 and the difference space weight wd2 , and then perform weighted fusion of the infrared image feature F1 and the visible light image feature F2 to perform feature enhancement.
[0007] Preferably, the difference feature enhancement process of the difference feature enhancement fusion module is:
[0008] F d =F1-F2 (1)
[0009] s1=avgpool(F d ) (3)
[0010] s2=maxpool(F d ) (4)
[0011] z1=mlp(s1) (5)
[0012] z2=mlp(s2) (6)
[0013] w d1 =Sigmoid(z1+z2) (7)
[0014] F d_ =F d × d1 (8)
[0015] w d2 =Sigmoid(conv(F Max ,F Mean )) (9)
[0016] F dm =F1×(1+w d1 × d2 )+F2×(1+w d1 × d2 ) (10)
[0017] Where F1 is the infrared image feature, F2 is the visible light image feature, avgpool and maxpool are the global average pooling and global maximum pooling in the spatial dimension, mlp is a multi-layer perceptron with shared weights, Sigmoid is the Sigmoid activation function, and w d1 is the difference channel weight, w d2 is the difference space weight, conv is a 7×7 or 3×3 convolution, F Max and F Mean Represents the maximum pooling and average pooling results in the channel dimension.
[0018] Preferably, the common feature selection fusion module adaptively selects bimodal channel and spatial information by adding channel attention mechanism and spatial attention mechanism to the common features, and after obtaining the weights, weights the infrared features and complementarily weights the visible light features, and the process is as follows:
[0019] F c =F1+F2 (2)
[0020] s1′=avgpool(F c ) (11)
[0021] s2′=maxpool(F c ) (12)
[0022] z1′=mlp(s1′) (13)
[0023] z2′=mlp(s2′) (14)
[0024] w c1 =Sigmoid(z1′+z2′) (15)
[0025] F c_ =F c × c1 (16)
[0026] w c2 =Sigmoid(conv(F Max ,F Mean )) (17)
[0027] F cm =F1×(w c1 × c2 )+F2×(1-w c1 × c2 ) (18)
[0028] Where F1 is the infrared image feature, F2 is the visible light image feature, avgpool and maxpool are the global average pooling and global maximum pooling in the spatial dimension, mlp is a multi-layer perceptron with shared weights, Sigmoid is the sigmoid activation function, and w c1 is the common channel weight, w c2 is the common spatial weight, conv is a 7×7 or 3×3 convolution, F Max and F Mean Represents the maximum pooling and average pooling results in the channel dimension.
[0029] Compared with the prior art, the power transmission equipment detection method based on multispectral image fusion disclosed in the present invention has at least the following beneficial effects:
[0030] By introducing the feature fusion strategy of infrared and visible light images, we can effectively integrate the information of these two bands. Infrared images provide temperature changes and thermal features, while visible light images contain rich color and shape information. This complementarity enables us to better capture subtle changes in the target and environmental features, thereby significantly improving the accuracy and reliability of detection. In addition, in the case of insufficient lighting or complex environment, infrared images can maintain effective information and enhance the robustness of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to make the purpose, technical solution and beneficial effects of the present invention clearer, the present invention provides the following drawings for illustration:
[0032] Figure 1 This is a network structure diagram of a power transmission equipment detection method based on multi-spectral image fusion according to an embodiment of the present invention;
[0033] Figure 2 This is a structural diagram of a difference feature enhancement fusion module of a power transmission equipment detection method based on multi-spectral image fusion according to an embodiment of the present invention;
[0034] Figure 3 This is a structural diagram of a channel weight extraction module of a power transmission equipment detection method based on multi-spectral image fusion according to an embodiment of the present invention;
[0035] Figure 4 This is a structural diagram of a spatial weight extraction module of a power transmission equipment detection method based on multi-spectral image fusion according to an embodiment of the present invention;
[0036] Figure 5 This is a structural diagram of a common feature selection fusion module of a power transmission equipment detection method based on multi-spectral image fusion according to an embodiment of the present invention. DETAILED DESCRIPTION
[0037] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
[0038] See also Figure 1 The power transmission equipment detection method based on multispectral image fusion of the present invention performs mid-term fusion of infrared images and visible light images, adopts the YOLOv8 network structure, and sets two YOLOv8 backbone network settings, one for feature extraction of infrared images and the other for extraction of visible light images; at the junction of the backbone network and the neck network, a feature fusion module Fusion is added to fuse infrared features with visible light features, and the fused features are sent to the neck network for convolution, up-scaling and splicing.
[0039] Feature-level fusion is more in-depth than pixel-level fusion. It does not operate directly on pixels, but first extracts the key features of images of different bands through the feature extraction module, and then fuses them at the feature level. According to the timing of fusion, feature-level fusion can be divided into early fusion, mid-term fusion and late fusion.
[0040] Early fusion refers to the fusion of infrared image features and visible light image features in the early stage of the neural network. This method mainly fuses the underlying features of the image such as edges, textures, colors, shapes and other local information.
[0041] Mid-term fusion is performed in the middle layer of the feature extraction network. After the images in different bands undergo their own convolution operations, the intermediate feature maps are obtained, and then fused in the middle layer of the network. This method allows the network to extract features from information in different bands separately, and then fuse the extracted features through a certain fusion mechanism to form a new feature expression.
[0042] Late fusion usually occurs in the last few layers of the network, after sufficient feature extraction, the features are fused. This fusion method requires huge computation, almost twice the computation of a single network.
[0043] YOLOv8 is an improved version of YOLOv7, which inherits and optimizes the superior performance of YOLOv7 in the field of object detection. Compared with the previous generation model, YOLOv8 has made many improvements in network architecture, feature extraction and inference speed, so that it can better adapt to the needs of various detection tasks. YOLOv8 mainly consists of the backbone network, the neck network and the detection head.
[0044] Backbone: Responsible for extracting meaningful features from the input image, it is the feature extractor of the entire model. In the initial layer, it can capture simple patterns in the image, such as edges, textures, etc.; as the network layer deepens, it can capture features at different levels of abstraction, thereby providing a rich, hierarchical feature representation for subsequent detection.
[0045] Neck network: acts as a bridge, connecting the Backbone and the Head. It mainly performs feature fusion operations, integrates the features of different levels extracted by the Backbone, and integrates context information. In this way, the model can better understand the contextual relationship of the target in the image, thereby improving the accuracy of detection.
[0046] Detection head: It is the last part of the model and is responsible for generating the output of the model, such as the bounding box of the predicted target and the corresponding confidence score.
[0047] The feature fusion module includes a difference feature enhancement fusion module and a common feature selection fusion module, which enhances the difference features and suppresses the common features. This feature fusion module not only considers the difference information and enhances the difference information, i.e., the difference feature enhancement fusion module, but also considers the commonality of infrared and visible light image features. Through the common features, the infrared and visible light feature maps select the difference parts and suppress the common parts, i.e., the common feature selection fusion module.
[0048] See also Figure 2-Figure 4 The difference feature enhancement fusion module is based on the difference feature map F d Perform channel attention and spatial attention mechanisms to extract the difference channel weights w d1 and the difference space weight w d2 , and then perform weighted fusion of the infrared image feature F1 and the visible light image feature F2 to perform feature enhancement.
[0049] The difference feature enhancement process of the difference feature enhancement fusion module is as follows:
[0050] F d =F1-F2 (1)
[0051] s1=avgpool(F d ) (3)
[0052] s2=maxpool(F d ) (4)
[0053] z1=mlp(s1) (5)
[0054] z2=mlp(s2) (6)
[0055] w d1 =Sigmoid(z1+z2) (7)
[0056] F d_ =F d × d1 (8)
[0057] w d2 =Sigmoid(conv(F Max ,F Mean )) (9)
[0058] F dm =F1×(1+w d1 × d2 )+F2×(1+w d1 × d2 ) (10)
[0059] Where F1 is the infrared image feature, F2 is the visible light image feature, avgpool and maxpool are the global average pooling and global maximum pooling in the spatial dimension, mlp is a multi-layer perceptron with shared weights, Sigmoid is the Sigmoid activation function, and w d1 is the difference channel weight, w d2 is the difference space weight, conv is a 7×7 or 3×3 convolution, F Max and F Mean Represents the maximum pooling and average pooling results in the channel dimension.
[0060] See also Figure 5 The common feature selection fusion module adaptively selects bimodal channel and spatial information by adding channel attention mechanism and spatial attention mechanism to common features. After obtaining the weights, the infrared features are weighted and the visible light features are complementary weighted, which can reduce the introduction of redundant features. The process is as follows:
[0061] F c =F1+F2 (2)
[0062] s1′=avgpool(F c ) (11)
[0063] s2′=maxpool(F c ) (12)
[0064] z1′=mlp(s1′) (13)
[0065] z2′=mlp(s2′) (14)
[0066] w c1 =Sigmoid(z1′+z2′) (15)
[0067] F c_ =F c × c1 (16)
[0068] w c2 =Sigmoid(conv(F Max ,F Mean )) (17)
[0069] F cm =F1×(w c1 × c2 )+F2×(1-w c1 × c2 ) (18)
[0070] Where F1 is the infrared image feature, F2 is the visible light image feature, avgpool and maxpool are the global average pooling and global maximum pooling in the spatial dimension, mlp is a multi-layer perceptron with shared weights, Sigmoid is the Sigmoid activation function, and w c1 is the common channel weight, w c2 is the common spatial weight, conv is a 7×7 or 3×3 convolution, F Max and F Mean Represents the maximum pooling and average pooling results in the channel dimension.
[0071] Experimental results: The commonly used evaluation index in target detection is map (Mean Average Precision), which is the average of the average precision AP (Average Precision) of all classes in map target recognition. AP is the weighted average of the precision under different recall rates, which comprehensively considers the performance of the model under different thresholds. For a category, by changing the predicted intersection over union (IoU) threshold, a series of precision and recall values can be obtained. The curve drawn with recall as the horizontal axis and precision as the vertical axis is called the Precision-Recall curve (PR curve), and AP is the area under the PR curve. map can comprehensively evaluate the comprehensive performance of the target recognition model in multiple categories. The higher the value, the better the overall performance of the model, and its maximum value is 1. This study uses two map indicators, map50 and map50-95, to evaluate the training results.
[0072] mAP50 means that the intersection over union (IoU) threshold used when calculating the mean average precision is 50%; map50-95 is a more rigorous and comprehensive indicator for evaluating model performance, which means that when calculating the mean average precision, mAP is calculated separately at multiple values of IoU threshold from 50% to 95% with a step size of 5% (i.e. IoU = 50%, 55%, ..., 90%, 95%), and then the average of these mAPs is calculated.
[0073] This experiment compares the mAP50 and mAP50-95 of three power transmission equipment detection methods: single infrared image detection, single visible light image detection, and combined detection of infrared image and visible light image. In the experiment, the main detection methods are the wiring heads, ceramic insulators, composite insulators, vertical insulators (insulators used to insulate the ground from the bottom of rheological equipment, etc.), rheological equipment, etc. of the power transmission equipment. The results show that in the early stage of training, the mAP results of the three detection methods are similar, and differences begin to appear after 50 rounds of training. The detection method of infrared and visible light image fusion of the present invention is better than the detection method of single infrared and single visible light image. The mAP50 of the combined detection method proposed by the present invention is improved by about 11% compared with the single visible light detection method, and by about 5% compared with the single infrared detection method; mAP50-95 is improved by about 25% compared with the single visible light detection method, and by about 7% compared with the single infrared detection method. The experiment proves that the power transmission equipment detection based on the combination of infrared image and visible light image features of the present invention is useful.
[0074] In addition to the above embodiments, the present invention may also have other implementation modes. Any technical solution formed by equivalent replacement or equivalent transformation is within the protection scope required by the present invention.
[0075] The present invention is described in detail above, but the specific implementation of the present invention is not limited thereto. Those skilled in the art may make various modifications or alterations without departing from the spirit and scope of the claims of the present application.
Claims
1. A power transmission equipment detection method based on multispectral image fusion, characterized in that: The YOLOv8 network structure is used to perform mid-term fusion of infrared images and visible light images. Two YOLOv8 backbone networks are set up, one for feature extraction of infrared images and the other for extraction of visible light images. A feature fusion module is added at the junction of the backbone network and the neck network to fuse the infrared features with the visible light features, and the fused features are sent to the neck network for convolution, upscaling and splicing.
2. The power transmission equipment detection method based on multispectral image fusion according to claim 1, characterized in that: The feature fusion module includes a difference feature enhancement fusion module and a common feature selection fusion module, which performs difference feature enhancement and common feature suppression.
3. The power transmission equipment detection method based on multispectral image fusion according to claim 2, characterized in that: The difference feature enhancement fusion module is constructed by d Perform channel attention and spatial attention mechanisms to extract the difference channel weight w d1 and the difference space weight w d2 , and then perform weighted fusion of infrared image features F1 and visible light image features F2 to perform feature enhancement.
4. The power transmission equipment detection method based on multispectral image fusion according to claim 3, characterized in that: The difference feature enhancement process of the difference feature enhancement fusion module is as follows: F d =F1-F2 (1) s1=avgpool(F d ) (3) s2=maxpool(F d ) (4) z1=mlp(s1) (5) z2=mlp(s2) (6) w d1 =Sigmoid(z1+z2) (7) F d_ =F d ×w d1 (8) w d2 =Sigmoid(conv(F Max ,F Mean )) (9) F dm =F1×(1+w d1 ×w d2 )+F2×(1+w d1 ×w d2 ) (10) Where F1 is the infrared image feature, F2 is the visible light image feature, avgpool and maxpool are the global average pooling and global maximum pooling in the spatial dimension, mlp is a multi-layer perceptron with shared weights, Sigmoid is the Sigmoid activation function, and w d1 is the difference channel weight, w d2 is the difference space weight, conv is a 7×7 or 3×3 convolution, F Max and F Mean Represents the maximum pooling and average pooling results in the channel dimension.
5. The power transmission equipment detection method based on multispectral image fusion according to claim 2, characterized in that: The common feature selection and fusion module adaptively selects bimodal channel and spatial information by adding channel attention mechanism and spatial attention mechanism to the common features. After obtaining the weights, the infrared features are weighted and the visible light features are complementary weighted. The process is as follows: F c =F1+F2 (2) s1′=avgpool(F c ) (11) s2′=maxpool(F c ) (12) z1′=mlp(s1′) (13) z2′=mlp(s2′) (14) w c1 =Sigmoid(z1′+z2′) (15) F c_ =F c ×w c1 (16) w c2 =Sigmoid(conv(F Max ,F Mean )) (17) F cm =F1×(w c1 ×w c2 )+F2×(1-w c1 ×w c2 ) (18) Where F1 is the infrared image feature, F2 is the visible light image feature, avgpool and maxpool are the global average pooling and global maximum pooling in the spatial dimension, mlp is a multi-layer perceptron with shared weights, Sigmoid is the Sigmoid activation function, and w c1 is the common channel weight, w c2 is the common spatial weight, conv is a 7×7 or 3×3 convolution, F Max and F Mean Represents the maximum pooling and average pooling results in the channel dimension.