Power transmission line forest fire early warning method based on multi-modal image fusion

Abstract

The invention discloses a power transmission line forest fire early warning method based on multi-modal image fusion, and aims to solve the problems that when an existing forest fire early warning method is in butt joint with the actual operation condition of a power transmission line, multi-source information complementation is difficult to consider at the same time, false detection and missing detection are often caused by environmental influence, and the early warning timeliness sometimes cannot meet the real-time requirement of power grid operation. According to the method, a power transmission line forest fire early warning model based on multi-modal image fusion is constructed, and the model comprises preprocessing, multi-scale feature extraction, fire point detection and positioning, risk level definition and early warning level division. And training the model according to the visible light image, the infrared image and the radar image at each moment in the process from the beginning to the end of the mountain fire combustion. The invention belongs to the technical field of power transmission line forest fire early warning.

Description

Technical Field

[0001] This invention relates to the field of early warning technology for wildfires along power transmission lines, specifically to an early warning method for wildfires along power transmission lines based on multimodal image fusion. Background Technology

[0002] With the rapid development of my country's power industry, high-voltage transmission systems have become a crucial infrastructure for ensuring energy security and economic development. Transmission lines in mountainous areas are particularly sensitive to environmental changes due to their wide coverage and complex terrain. In recent years, frequent wildfires have posed significant risks to the safe operation of these transmission lines.

[0003] Existing wildfire monitoring and early warning technologies can be mainly divided into two categories. The first category is single-sensor monitoring methods, which typically involve installing visible light cameras or infrared thermal imagers on key towers of power transmission lines. By pre-setting azimuth, elevation angles, and field of view, the monitoring field of view of each sensor is mapped one-to-one with the coordinates of the power transmission line, achieving precise mapping between the sensor and the transmission line location. This type of method is simple to deploy, but visible light monitoring is easily affected by smoke, obstructions, and insufficient light at night, making it difficult to detect fire sources in a timely manner. Infrared monitoring is extremely sensitive to temperature, but it often produces false detections and missed detections in complex backgrounds, and a single sensor cannot simultaneously handle real-time performance and accuracy under changing environmental conditions, making it difficult to meet the high-reliability early warning requirements for power transmission lines. The second category is fire identification methods based on artificial intelligence, which typically involve mounting camera devices based on deep learning object detection models (such as Faster R-CNN, YOLO, etc.) on key towers along the power transmission line to detect fire points in single-modal images. By precisely calibrating the latitude, longitude, altitude, orientation, and field of view parameters of each sensor and combining this with the digital coordinate information of the transmission line, a one-to-one correspondence between fire detection results and line locations is achieved, ensuring that once a fire source is detected, the specific tower section or line segment can be quickly located. Although these methods improve recognition accuracy, their single-modal input results in insufficient adaptability to changing scenarios, making it difficult to further reduce false alarm and false negative rates. Furthermore, some models run on the cloud or on a single device, failing to fully utilize the synergistic advantages of edge computing and cloud computing. This leads to communication delays and computational bottlenecks in the data acquisition and early warning generation process, resulting in long early warning response times that cannot meet the power grid operation requirements for minute-level responses to sudden wildfires.

[0004] Therefore, existing technologies struggle to achieve a balance between complementary fusion of multi-source information, real-time reasoning capabilities of deep learning object detection models, and system robustness. Summary of the Invention

[0005] To address the problems of existing wildfire early warning methods, which struggle to simultaneously consider the complementary nature of multi-source information when integrated with the actual operating conditions of power transmission lines, and which are often subject to false detections and missed detections due to environmental influences, resulting in early warning timeliness sometimes failing to meet the real-time requirements of power grid operation, this invention proposes a wildfire early warning method for power transmission lines based on multimodal image fusion.

[0006] The technical solution adopted in this invention is:

[0007] It includes the following steps:

[0008] S1. Construct a transmission line wildfire early warning model based on multimodal image fusion. Obtain visible light images, infrared images, and radar images at each moment during the start and end of a wildfire. Input the visible light images, infrared images, and radar images at each moment into the transmission line wildfire early warning model based on multimodal image fusion for training, and obtain the trained transmission line wildfire early warning model based on multimodal image fusion. The specific process is as follows:

[0009] S11. Obtain visible light images, infrared images, and radar images at each moment during the start and end of a wildfire. Preprocess the visible light images, infrared images, and radar images at each moment, and merge the preprocessed visible light images, infrared images, and radar images at all moments into a training dataset.

[0010] S12. Extract the multi-scale features of each visible light image, infrared image and radar image in the training dataset respectively. Fuse the multi-scale features of the visible light image, infrared image and radar image at the same time to obtain the multi-information fusion feature map at the corresponding time. Similarly, obtain the multi-information fusion feature map at each time.

[0011] S13. Use the YOLOv5 network to perform fire detection and localization on the multi-information fusion feature map at each time step to obtain the real fire points on the multi-information fusion feature map at each time step.

[0012] S14. Perform spatiotemporal clustering analysis on the real fire points of all fire point candidate boxes on all multi-information fusion feature maps to obtain a set of fire events. Use a linear weighting method to calculate the risk level of each fire event, classify the warning level according to the risk level, and issue a warning according to the warning level.

[0013] Thus, a well-trained early warning model for wildfires along power transmission lines based on multimodal image fusion was obtained;

[0014] S2. Input the visible light image, infrared image and radar image of the wildfire to be tested into the trained transmission line wildfire early warning model based on multimodal image fusion, output the early warning level, and issue an early warning based on the early warning level.

[0015] Further, in S11, visible light images, infrared images, and radar images are acquired at each moment during the start and end of a wildfire. The visible light images, infrared images, and radar images at each moment are preprocessed, and the preprocessed visible light images, infrared images, and radar images at all moments are merged into a training dataset. The specific process is as follows:

[0016] S111. Acquire visible light images, infrared images, and radar images at each moment from the start to the end of a wildfire. Combine the visible light images, infrared images, and radar images at the same moment into a set of image data to obtain multiple sets of image data.

[0017] S112. Record the geographic location coordinates and meteorological parameters of each set of image data, including temperature, humidity and cloud cover;

[0018] Each set of image data, along with its corresponding geographic location coordinates and meteorological parameters, is combined into a single set of data. Multiple sets of data are then merged into a multimodal dataset, CJData.

[0019] S113. Perform geometric registration and temporal registration on each group of image data in the multimodal dataset CJData, and update the multimodal dataset CJData with each group of registered image data to obtain the registered dataset RegData.

[0020] S114. Manually label the fire source areas in each image of the registered dataset RegData, and update the registered dataset RegData with each labeled image to obtain the labeled dataset LabelMap.

[0021] S115. Adjust the resolution of each image in the labeled dataset LabelMap to 640 pixels wide × 480 pixels high, and update the labeled dataset LabelMap with the adjusted image to obtain the scaled dataset ScaleData.

[0022] S116. Crop each image in the scaled dataset ScaleData, and update the scaled dataset ScaleData with each cropped image to obtain the cropped dataset CropData.

[0023] S117. Perform pixel normalization processing on each image in the cropped dataset CropData, mapping the pixel values ​​to [0,1]. Update the cropped dataset CropData with each image after pixel normalization processing to obtain the dataset NormData.

[0024] S118. Perform data augmentation on each image in the NormData dataset. Generate a maximum of 3–5 data augmented images for each image. Obtain the data augmented images corresponding to all images. Merge all images and their corresponding data augmented images into the training dataset ZQDatasets.

[0025] Further, in S113, geometric registration and temporal registration are performed on each group of image data in the multimodal dataset CJData, and the multimodal dataset CJData is updated with the registered group of image data to obtain the registered dataset RegData. The specific process is as follows:

[0026] For each set of image data in the multimodal dataset CJData, which includes visible light images, infrared images, and radar images, we extract the common points within the visible light images, infrared images, and radar images as matching benchmarks. Based on the matching benchmarks, we use affine transformation to align the infrared images and radar images to the visible light images, thus obtaining the aligned visible light images, infrared images, and radar images for each set of image data.

[0027] The visible light image, infrared image, and radar image after each set of image data is aligned using timestamps. Frame-to-frame matching is performed, and image frames with a time difference greater than a threshold are removed to obtain each set of registered image data. The multimodal dataset CJData is then updated with each set of registered image data to obtain the registered dataset RegData.

[0028] Further, in S116, each image in the scaled dataset ScaleData is cropped, and the scaled dataset ScaleData is updated with each cropped image to obtain the cropped dataset CropData. The specific process is as follows:

[0029] S1161. Adjust the size of each image in the scaled dataset ScaleData;

[0030] S1162. Within each image after resizing, the effective warning range defined by the safety boundary of the transmission line is divided. Based on the effective warning range, the fire point area in the corresponding image is marked using the line geometric model to obtain all images containing the fire point area.

[0031] S1163. Use the UNet model based on semantic segmentation to extract features and classify pixels at the pixel level for each fire point region to obtain the fire point candidate probability for the corresponding fire point region. Based on each fire point candidate probability, obtain the fire point candidate probability map corresponding to each image.

[0032] S1164. Denoise each fire candidate probability map by using threshold binarization, and then fill the holes in the denoised fire candidate probability map by using morphological opening and closing operations to obtain the corresponding new fire candidate probability map. Use connected component analysis to extract each fire candidate connected component in the new fire candidate probability map.

[0033] S1165. Calculate the minimum bounding rectangle of each candidate connected component of the fire point. Based on each minimum bounding rectangle, crop each adjusted image to obtain the corresponding sub-image. Remove the background around the transmission line in the sub-image and keep only the transmission line to obtain the cropped image.

[0034] Similarly, after obtaining all cropped images, merge all cropped images into a cropped dataset CropData.

[0035] Furthermore, the data augmentation methods in S118 include random rotation, horizontal or vertical flipping, noise superposition, brightness or contrast adjustment, and one or more data augmentation methods can be randomly selected during application.

[0036] Further, in step S12, multi-scale features are extracted from each visible light image, infrared image, and radar image in the training dataset. The multi-scale features of the visible light image, infrared image, and radar image at the same time are fused to obtain the multi-information fusion feature map at the corresponding time. Similarly, the multi-information fusion feature map at each time is obtained. The specific process is as follows:

[0037] S121. Add channel attention modules to the ResNet18 network, MobileNetV2 network and EfficientNet-B0 network respectively to obtain the new ResNet18 network, the new MobileNetV2 network and the new EfficientNet-B0 network;

[0038] The new ResNet18 network includes, in sequence, an input layer, a basic convolutional layer, a max pooling layer, a first-stage residual module, a second-stage residual module, a third-stage residual module, a fourth-stage residual module, and an output layer. The channel attention module is added to the first-stage residual module, the second-stage residual module, the third-stage residual module, and the fourth-stage residual module, respectively.

[0039] The new MobileNetV2 network consists of an input layer, an initial convolutional layer, an inverted residual block sequence, and an output layer, with the channel attention module added to the inverted residual block sequence.

[0040] The EfficientNet-B0 new network consists of an input layer, an initial convolutional layer, an MBConv module sequence, and an output layer, with the channel attention module added to the MBConv module sequence.

[0041] We use the new ResNet18 network to extract the low-scale, mid-scale, and high-scale features of each visible light image in the training dataset ZQDatasets to obtain a feature map F_vis;

[0042] The low-level scale features are edge features or texture features, the mid-level scale features are local pattern features, and the high-level scale features are semantic structure features;

[0043] The low-scale, mid-scale, and high-scale features of each infrared image in the training dataset ZQDatasets are extracted using the new MobileNetV2 network to obtain a feature map F_ir;

[0044] The low-scale, mid-scale, and high-scale features of each radar image in the training dataset ZQDatasets are extracted using the new EfficientNet-B0 network to obtain a feature map F_sar.

[0045] Similarly, feature maps F_vis, F_ir, and F_sar are obtained at each time step;

[0046] S122. Input the feature maps F_vis, F_ir, and F_sar at the same time into the Transformer multi-head attention network and output a fused feature map F_fuse. Similarly, obtain the fused feature map F_fuse at each time.

[0047] S123. Using location coding, combine the geographic location coordinates, meteorological parameters and fused feature map F_fuse at the same time to obtain a multi-information fusion feature map MIF_fuse. Similarly, obtain the multi-information fusion feature map MIF_fuse for each time.

[0048] Furthermore, in step S13, the YOLOv5 network is used to perform fire detection and localization on the multi-information fusion feature map at each time step to obtain the real fire points on the multi-information fusion feature map at each time step. The specific process is as follows:

[0049] S131. Input the multi-information fusion feature map MIF_fuse at a certain moment into the YOLOv5 network, and generate N fire point candidate boxes on the multi-information fusion feature map MIF_fuse;

[0050] S132. Obtain the infrared image, visible light image, and radar image corresponding to the multi-information fusion feature map MIF_fuse in S131. For each fire candidate box in the multi-information fusion feature map MIF_fuse, use mathematical statistics to calculate the mean and variance of the temperature pixel values ​​in the region corresponding to each fire candidate box in the infrared image. Extract the flame color histogram features in the region corresponding to each fire candidate box in the visible light image based on flame features. Analyze the scattering intensity distribution and texture direction features in the region corresponding to each fire candidate box in the radar image based on radar scattering texture.

[0051] Similarly, the mean and variance of temperature pixel values ​​in the corresponding region of the corresponding infrared image and the flame color histogram features in the corresponding region of the corresponding visible light image are obtained for each fire candidate box, as well as the scattering intensity distribution and texture direction features in the corresponding region of the corresponding radar image.

[0052] S133. Based on the mean and variance of temperature pixel values ​​in the corresponding region of the corresponding infrared image and the flame color histogram features in the corresponding region of the corresponding visible light image, as well as the scattering intensity distribution and texture direction features in the corresponding region of the corresponding radar image, a new confidence score C_i' of the current fire candidate box is calculated using a multimodal fusion validator. If C_i' ≥ a custom confidence score threshold, it is marked as a real fire point, and the real fire point of the current fire candidate box is obtained. The custom confidence score threshold is 0.6.

[0053] Similarly, the true fire points of all fire point candidate boxes on all multi-information fusion feature maps are obtained.

[0054] Furthermore, in step S14, spatiotemporal clustering analysis is performed on the actual fire points of all fire point candidate boxes on all multi-information fusion feature maps to obtain a set of fire events. A linear weighted method is used to calculate the risk level of each fire event, and warning levels are assigned based on the risk levels. Warnings are then issued according to the warning levels. The specific process is as follows:

[0055] S141. Perform spatiotemporal clustering analysis on all real fire points in the multi-information fusion feature maps obtained in S133, and merge the real fire points with a time interval Δt≤30 minutes and a spatial distance D≤500 meters to obtain a set of fire events.

[0056] S142. The risk level L_risk of each fire event is calculated using a linear weighting method. L_risk = w1×area + w2×maximum temperature + w3×terrain slope + w4×wind speed, where w1, w2, w3, and w4 are the dynamic weight coefficients of the corresponding risk factors, and satisfy the normalization constraint w1+ w2+ w3+ w4= 1.

[0057] S143. Classify the early warning level according to the risk level L_risk, configure the corresponding response measures, and respond to the early warning according to the early warning level:

[0058] When 0 ≤ L_risk < 0.3, the risk level is defined as Level I, and the response measure is to send an SMS / APP alert.

[0059] When 0.3 ≤ L_risk < 0.6, the risk level is Level II, and the response measure is to notify the on-duty personnel for manual verification.

[0060] When L_risk is defined as 0.6 ≤ L_risk < 0.8, the risk level is Level III, and the response measure is to instruct the drone to automatically conduct a follow-up inspection.

[0061] When L_risk ≥ 0.8, the risk level is Level IV, and the response measures are to interrupt the power supply to the transmission line, set up isolation zones, and implement dispatch and control.

[0062] Furthermore, when the following conditions occur, the current weather forecast and power grid operation and maintenance status are loaded using a rule engine. Multi-source information is then parsed using predefined logical rules to optimize response measures. The weather forecast includes wind speed, precipitation probability, and drought index. The power grid operation and maintenance status includes real-time load rate, maintenance plan, and line voltage level, specifically including:

[0063] When the grid load rate is >85%, the interruption of power supply to transmission lines in the Level IV response measures will be replaced by step-down operation + isolation zone designation;

[0064] When the wind force is ≥ level 6, the Level III response measures shall include an instruction to shut down power transmission lines in fire-prone sections along the route;

[0065] When the drought index SPI < -1.5, the Level II trigger threshold will be lowered from 0.3 to 0.2;

[0066] During the maintenance period, the Level I trigger threshold was increased from 0.3 to 0.4.

[0067] Furthermore, the method for acquiring the visible light image, infrared image, and radar image of the wildfire to be tested in S2 includes:

[0068] Based on the comprehensive risk indicators of mountainous areas, towers that are 500 meters apart or cross important sections on the power transmission lines in mountainous areas are designated as key towers, resulting in multiple key towers. A high-definition visible light camera and a thermal infrared imager are installed in the upper middle part of each key tower.

[0069] The comprehensive risk indicators include terrain slope, vegetation density, historical fire frequency, drought index, and line voltage level. The important sections include steep slopes and forest belt intersection areas.

[0070] Visible light cameras, thermal infrared imagers, and radar were used to acquire visible light images, infrared images, and radar images at every moment during the wildfire.

[0071] The beneficial effects of this invention are as follows:

[0072] This invention extracts multi-scale features from preprocessed visible light, infrared, and radar images, fully utilizing the intuitive morphological features of visible light imaging, the temperature anomaly information of infrared thermal imaging, and the scattering characteristics of SAR images. By combining convolutional neural networks, Transformer, and temporal correlation analysis, the multi-scale features of visible light, infrared, and radar images are fused to obtain a multi-information fusion feature map. This map, by integrating multi-modal features and multi-source information, enhances the characterization ability of fire points and the robustness of the model, effectively reducing false alarms and false negatives. The YOLOv5 network is used to detect and locate fire points on the multi-information fusion feature map, obtaining the actual fire points on the map and overcoming the monitoring blind spots of single-modal methods in complex smoke, nighttime, or foggy weather. Spatiotemporal clustering analysis is performed on all actual fire points to calculate risk levels, classify warning levels according to risk levels, and issue warnings based on the warning levels. This improves the accuracy and timeliness of wildfire monitoring and warnings, meets the high reliability requirements of power transmission lines, and achieves high-precision, low-latency early warning of fire points on power transmission lines in mountainous areas. This invention can provide reliable data support for subsequent power grid information analysis in mountainous areas (such as transmission line adjustment and transmission line emergency repair), and provide precise and efficient technical support for the management of power grids and related industries. Attached Figure Description

[0073] Figure 1 This is a flowchart of the processing of the power transmission line wildfire early warning model based on multimodal image fusion according to the present invention;

[0074] Figure 2 This is a risk level diagram;

[0075] Figure 3 This is an actual monitoring image from a single visible light camera in the embodiment;

[0076] Figure 4 This is an actual monitoring image from a single infrared camera in the embodiment;

[0077] Figure 5 This is an actual monitoring image obtained by simply splicing together multiple modes in the embodiment;

[0078] Figure 6This is the actual monitoring diagram of the present invention; Detailed Implementation

[0079] Specific implementation method one: Combining Figures 1-6 This embodiment describes a method for early warning of wildfires along power transmission lines based on multimodal image fusion, which includes the following steps:

[0080] S1. Construct a transmission line wildfire early warning model based on multimodal image fusion. Obtain visible light images, infrared images, and radar images at each moment during the start and end of a wildfire. Input the visible light images, infrared images, and radar images at each moment into the transmission line wildfire early warning model based on multimodal image fusion for training, and obtain the trained transmission line wildfire early warning model based on multimodal image fusion. The specific process is as follows:

[0081] S11. Obtain visible light, infrared, and radar images at each moment during the start and end of a wildfire. Preprocess the visible light, infrared, and radar images at each moment, and merge the preprocessed visible light, infrared, and radar images at all moments into a training dataset. The specific process is as follows:

[0082] S111. Randomly select a mountainous area containing one or more power transmission lines, and obtain visible light images, infrared images, and radar images at each moment during the process from the start to the end of a wildfire in the mountainous area. Take the visible light images, infrared images, and radar images at the same moment as a set of image data to obtain multiple sets of image data.

[0083] S112. Record the geographic location coordinates (Geo) and meteorological parameters for each set of image data, including temperature (T), humidity (H), and cloud cover (C). Combine each set of image data, along with its corresponding geographic location coordinates and meteorological parameters, into a single set of data. Then, merge these multiple sets of data into a multimodal dataset (CJData).

[0084] S113. Perform geometric and temporal registration on each group of image data in the multimodal dataset CJData, and update the multimodal dataset CJData with the registered image data to obtain the registered dataset RegData. The specific process is as follows:

[0085] For each set of image data in the multimodal dataset CJData, including visible light images, infrared images, and radar images, common points within these images are extracted as matching benchmarks. This extraction can be done manually or automatically. Automatic extraction methods include those based on classical feature point detection and description (such as SIFT, SURF, ORB, etc.) or deep learning-based keypoint detection networks (such as SuperPoint, KeypointNet, etc.). The common points include tower bases and power line intersections. Based on the matching benchmarks, affine transformation or perspective transformation methods are used to align the infrared and radar images to the visible light images, resulting in aligned visible light, infrared, and radar images for each set of image data.

[0086] In addition, although image acquisition is synchronized as much as possible, there are still millisecond-level deviations. Therefore, the visible light image, infrared image and radar image after each set of image data is aligned with the timestamp is used to perform frame-to-frame matching. Image frames with a time difference greater than the timestamp's own threshold are removed to obtain each set of image data after registration. The multimodal dataset CJData is updated with each set of registered image data to obtain the registered dataset RegData.

[0087] Although visible light, infrared, and radar images are collected at the same time, due to different imaging principles and communication delays, there may be frame differences of tens to hundreds of milliseconds between infrared and radar images. Therefore, this invention achieves pixel-level fusion between images by using unified spatial reference coordinates for geometric registration of the three types of images. Then, timestamps are used to ensure that the three types of images are truly "data from the same moment" to avoid mismatch and affect the final warning accuracy.

[0088] S114. Domain experts manually label the fire source regions in each image of the registered dataset RegData, and update the registered dataset RegData with each labeled image to obtain the labeled dataset LabelMap.

[0089] S115. Adjust the resolution of each image in the labeled dataset LabelMap to 640 pixels wide × 480 pixels high, and update the labeled dataset LabelMap with the adjusted image to obtain the scaled dataset ScaleData.

[0090] S116. Crop each image in the scaled dataset ScaleData, and update the scaled dataset ScaleData with each cropped image to obtain the cropped dataset CropData. The specific process is as follows:

[0091] S1161. Adjust the size of each image in the scaled dataset ScaleData to 512×512 to ensure that each image conforms to the input size of the UNet model based on semantic segmentation.

[0092] S1162. Within each image after resizing, the effective warning range defined by the safety boundary of the transmission line is divided. Based on the effective warning range, the fire point areas in the corresponding images are marked using the line geometric model. The number of fire point areas is greater than or equal to 0, thus obtaining all images containing fire point areas.

[0093] S1163. Use the UNet model based on semantic segmentation to extract features and classify pixels at the pixel level for each fire point region to obtain the fire point candidate probability of the corresponding fire point region, thereby obtaining the fire point candidate probability map corresponding to each image; the size of the fire point candidate probability map is the same as that of the input image, and the pixel value range is [0,1].

[0094] The semantic segmentation-based UNet model sequentially comprises an encoder path, a bottleneck layer, a decoder path, and an output layer. The encoder path sequentially includes four downsampling stages: The first encoding stage consists of a first 3×3 convolutional layer (3 input channels, 64 output channels), a first ReLU activation layer, a second 3×3 convolutional layer (64 channels), a second ReLU activation layer, and a first 2×2 max-pooling layer (stride 2). The second encoding stage consists of a third 3×3 convolutional layer (64 inputs, 128 outputs), a third ReLU activation layer, a fourth 3×3 convolutional layer (128 channels), a fourth ReLU activation layer, and a second 2×2 max-pooling layer (stride 2). The third encoding stage consists of a fifth 3×3 convolutional layer (128 inputs, 256 outputs), a fifth ReLU activation layer, a sixth 3×3 convolutional layer (256 channels), a sixth ReLU activation layer, and a third 2×2 max-pooling layer (stride 2). The fourth encoding stage consists of the seventh 3×3 convolutional layer (input 256, output 512), the seventh ReLU activation layer, the eighth 3×3 convolutional layer (channels 512), the eighth ReLU activation layer, and the fourth 2×2 max pooling layer (stride 2).

[0095] The bottleneck layers consist of the ninth 3×3 convolutional layer (input 512, output 1024), the ninth ReLU activation layer, the tenth 3×3 convolutional layer (channel 1024), and the tenth ReLU activation layer.

[0096] The decoder path consists of four upsampling stages: The first decoding stage includes a first 2×2 transposed convolutional layer (input 1024, output 512, stride 2), a first skip connection layer (concatenated with the output of the fourth encoding stage), an eleventh 3×3 convolutional layer (input 1024, output 512), an eleventh ReLU activation layer, a twelfth 3×3 convolutional layer (512 channels), and a twelfth ReLU activation layer. The second decoding stage includes a second 2×2 transposed convolutional layer (input 512, output 256, stride 2), a second skip connection layer (concatenated with the output of the third encoding stage), a thirteenth 3×3 convolutional layer (input 512, output 256), a thirteenth ReLU activation layer, a fourteenth 3×3 convolutional layer (256 channels), and a fourteenth ReLU activation layer. The third decoding stage consists of, in sequence, a third 2×2 transposed convolutional layer (input 256, output 128, stride 2), a third skip connection layer (concatenated with the output of the second encoding stage), a fifteenth 3×3 convolutional layer (input 256, output 128), a fifteenth ReLU activation layer, a sixteenth 3×3 convolutional layer (128 channels), and a sixteenth ReLU activation layer. The fourth decoding stage consists of, in sequence, a fourth 2×2 transposed convolutional layer (input 128, output 64, stride 2), a fourth skip connection layer (concatenated with the output of the first encoding stage), a seventeenth 3×3 convolutional layer (input 128, output 64), a seventeenth ReLU activation layer, an eighteenth 3×3 convolutional layer (64 channels), and an eighteenth ReLU activation layer.

[0097] The output layer consists of a first 1×1 convolutional layer (input 64, output 1) and a Sigmoid activation layer.

[0098] S1164. Denoise each candidate fire point probability map by using threshold binarization, then fill holes in the denoised candidate fire point probability map using morphological opening and closing operations to obtain the corresponding new candidate fire point probability map. Extract each candidate fire point connected component from the new candidate fire point probability map using connected component analysis.

[0099] S1165. Calculate the minimum bounding rectangle of each candidate connected component of the fire point. Based on each minimum bounding rectangle, crop each adjusted image to obtain the corresponding sub-image. Remove the background around the transmission line in the sub-image and keep only the transmission line to obtain the cropped image.

[0100] Similarly, after obtaining all cropped images, merge all cropped images into a cropped dataset CropData.

[0101] S117. Perform pixel normalization processing on each image in the cropped dataset CropData, mapping the pixel values ​​to [0,1]. Update the cropped dataset CropData with each image after pixel normalization processing to obtain the dataset NormData.

[0102] S118. Perform data augmentation on each image in the NormData dataset. Generate a maximum of 3–5 data augmented images for each image. Obtain the data augmented images corresponding to all images. Merge all images and their corresponding data augmented images into the training dataset ZQDatasets.

[0103] The data augmentation methods include random rotation (±10°), horizontal or vertical flipping, noise overlay, and brightness or contrast adjustment. One or more data augmentation methods can be randomly selected during application. This invention uses rotation +10° and horizontal flipping sequentially. This invention generates three data-augmented images for each image.

[0104] S12. Extract multi-scale features from each visible light image, infrared image, and radar image in the training dataset. Fuse the multi-scale features of the visible light image, infrared image, and radar image at the same time to obtain the multi-information fusion feature map for the corresponding time. Similarly, obtain the multi-information fusion feature map for each time. The specific process is as follows:

[0105] S121. Add a channel attention module, Squeeze-and-Excitation (SE), to the ResNet18, MobileNetV2, and EfficientNet-B0 networks respectively to enhance the importance of channels related to the fire source and suppress background interference features. This results in new ResNet18, MobileNetV2, and EfficientNet-B0 networks.

[0106] The new ResNet18 network consists of an input layer, a basic convolutional layer, a max pooling layer, a first-stage residual module, a second-stage residual module, a third-stage residual module, a fourth-stage residual module, and an output layer.

[0107] The basic convolutional layers sequentially include a 7×7 convolutional layer (3 input channels, 64 output channels, stride 2), a batch normalization layer, and a ReLU activation layer. The max pooling layer uses a 3×3 pooling kernel (stride 2). The first-stage residual module sequentially includes two residual units, each of which sequentially includes a 3×3 convolutional layer (64 channels), a batch normalization layer, a ReLU activation layer, another 3×3 convolutional layer (64 channels), a batch normalization layer, a channel attention module (r=16), and a residual direct addition operation. The second-stage residual module sequentially includes a third and a fourth residual unit. The third residual unit sequentially includes a 3×3 convolutional layer (128 channels, stride 2), a batch normalization layer, a ReLU activation layer, a 3×3 convolutional layer (128 channels), a batch normalization layer, an SE module (r=16), a 1×1 convolutional shortcut layer (channels 64→128), and a residual addition operation. The fourth residual unit is identical in structure to the third residual unit, except that the convolutional layers have no stride and no convolutional shortcut layers. The third-stage residual module consists of the fifth and sixth residual units. The fifth residual unit consists of a 3×3 convolutional layer (256 channels, stride 2), a batch normalization layer, a ReLU activation layer, another 3×3 convolutional layer (256 channels), a batch normalization layer, an SE module (r=16), a 1×1 convolutional shortcut layer (128→256 channels), and a residual addition operation. The sixth residual unit is identical in structure to the fifth residual unit, except that the convolutional layers have no stride and no convolutional shortcut layers. The fourth-stage residual module consists of the seventh and eighth residual units. The seventh residual unit consists of a 3×3 convolutional layer (512 channels, stride 2), a batch normalization layer, a ReLU activation layer, another 3×3 convolutional layer (512 channels), a batch normalization layer, an SE module (r=16), a 1×1 convolutional shortcut layer (256→512 channels), and a residual addition operation. The eighth residual unit is identical to the seventh residual unit except that the convolutional layers have no stride and no convolutional shortcut layers. The output layer consists of a global average pooling layer and a fully connected classification layer.

[0108] The new MobileNetV2 network consists of an input layer, an initial convolutional layer, an inverse residual block sequence, and an output layer. The initial convolutional layer comprises a 3×3 depthwise separable convolutional layer (3 inputs, 32 outputs, 2 stride), a batch normalization layer, and a ReLU6 activation layer. The inverse residual block sequence consists of seven layers. Layer 1 comprises a dilated 1×1 convolutional layer (32 inputs, 16 outputs), a batch normalization layer, a ReLU6 activation layer, a 3×3 depthwise convolutional layer (1 stride), a batch normalization layer, a ReLU6 activation layer, a projected 1×1 convolutional layer (16 outputs), a batch normalization layer, a SE module (r=4), and a direct residual connection operation (enabled when input and output channels are the same). Layers 2-6 have the same structure as layer 1, except that the number of channels in the SE module is progressively increased to 24 / 32 / 64 / 96 / 160. Layer 7 consists of, in sequence, a dilated 1×1 convolutional layer (input 160, output 960), a batch normalization layer, a ReLU6 activation layer, a 3×3 depthwise convolutional layer (stride 2), a batch normalization layer, a ReLU6 activation layer, a projected 1×1 convolutional layer (output 320), a batch normalization layer, and an SE module (r=4). The output layer consists of, in sequence, a 1×1 convolutional condensation layer (input 320, output 1280), a global average pooling layer, and a fully connected classification layer.

[0109] The EfficientNet-B0 new network consists of an input layer, an initial convolutional layer, a sequence of MBConv modules, and an output layer, with the channel attention module added to the MBConv module sequence. The specific structure of the EfficientNet-B0 new network is consistent with that described in Tan M, Le Q. Efficientnet: Rethinking model scaling for convolutional neural networks[C] / / International conference on machine learning. PMLR, 2019:6105-6114.

[0110] We use the new ResNet18 network to extract low-scale, mid-scale, and high-scale features of each visible light image in the training dataset ZQDatasets, and obtain a feature map F_vis with mixed features.

[0111] The low-level scale features are edge features or texture features, the mid-level scale features are local pattern features, and the high-level scale features are semantic structure features.

[0112] The low-scale, mid-scale, and high-scale features of each infrared image in the training dataset ZQDatasets are extracted using the new MobileNetV2 network to obtain a feature map F_ir with mixed features.

[0113] The low-scale, mid-scale, and high-scale features of each radar image in the training dataset ZQDatasets are extracted using the new EfficientNet-B0 network to obtain a feature map F_sar with mixed features.

[0114] Based on the above feature processing, we can obtain the feature map F_vis, feature map F_ir, and feature map F_sar at each time step.

[0115] S122. Input the feature maps F_vis, F_ir, and F_sar at the same time into the Transformer multi-head attention network, calculate the inter-modal dependencies through self-attention, and output a fused feature map F_fuse. Similarly, obtain the fused feature map F_fuse at each time.

[0116] The Transformer multi-head attention network comprises an input mapping layer, a serialization layer, a position encoding layer, an L-level encoder stack layer, and an output layer. The input mapping layer uses three independent 1×1 convolutional layers to map the number of channels of the visible light feature map F_vis, the infrared feature map F_ir, and the SAR feature map F_sar to a unified dimension d_model (typically 256). The serialization layer expands the three d_model channel feature maps into a sequence of length N = H × W based on their spatial location. The position encoding layer injects spatial location information by element-wise addition of a learnable two-dimensional position encoding matrix to the serialized features. The L-level encoder stacked layers (L=4) consist of a multi-head self-attention sub-layer, a first residual connection and layer normalization, a feedforward neural network sub-layer, and a second residual connection and layer normalization. The multi-head self-attention sub-layer computes H attention heads in parallel (H=8). Each attention head performs Softmax attention computation by querying the projection matrix, key projection matrix, and value projection matrix (dimension d_k=d_model / H=32). The multi-head outputs are concatenated and then processed through the first residual connection and layer normalization. The feedforward neural network sub-layer consists of a fully connected layer 1 (d_model→4·d_model), a ReLU activation layer, and a fully connected layer 2 (4·d_model→d_model). The output is processed through the second residual connection and layer normalization. The output layer reshapes the sequence output of the final encoder into an H×W×d_model 3D feature map and reduces the dimensionality to the target number of channels C_out (typically 128) through a 1×1 convolutional layer.

[0117] The Transformer multi-head attention network uses cross-modal contrastive loss to optimize the modal consistency and feature representation capability of the fused feature map F_fuse. This invention guides the training of the fused feature F_fuse through cross-modal contrastive loss, making the feature vectors of the same region under different modalities closer, improving semantic consistency, and thus improving the discrimination of real fire points.

[0118] S123. Using positional encoding, the geographic coordinates Geo, meteorological parameters, and fused feature map F_fuse at the same time are combined to obtain a multi-information fusion feature map MIF_fuse. Similarly, the multi-information fusion feature map MIF_fuse for each time moment is obtained. This preserves spatial location information, which can be used as input to the fire detection and early warning model to enhance the fire point representation capability. The positional encoding method includes two-dimensional sine and cosine functions.

[0119] S13. Using the YOLOv5 network, fire detection and localization are performed on the multi-information fusion feature map at each time step to obtain the real fire points on the multi-information fusion feature map at each time step. The specific process is as follows:

[0120] S131. Input the multi-information fusion feature map MIF_fuse at a certain moment into the YOLOv5 network, and generate N fire point candidate boxes B = {(x_i, y_i, w_i, h_i)} on the multi-information fusion feature map MIF_fuse, where x_i and y_i represent the center coordinates of the candidate box, w_i and h_i are the width and height of the box, respectively, and output the initial confidence score C_i of each fire point candidate box.

[0121] The YOLOv5 network comprises, in sequence, an Input layer, a Backbone network, a Neck network, and a Head. The Input layer receives an input image with a size of 640×640×3. The Backbone network includes, in sequence, a first convolutional module (6×6 convolution / stride 2 / 64 channels), a second convolutional module (3×3 convolution / stride 2 / 128 channels), a first C3 SimAM module (128 channels), a third convolutional module (3×3 convolution / stride 2 / 256 channels), a second C3 SimAM module (256 channels), a fourth convolutional module (3×3 convolution / stride 2 / 512 channels), a third C3 SimAM module (512 channels), a fifth convolutional module (3×3 convolution / stride 2 / 1024 channels), and a fourth C3 SimAM module (1024 channels). The neck network consists of the following modules arranged sequentially: SPPF (composed of parallel max-pooling layers of 5×5, 9×9, and 13×13), the first GSConv module (512 channels), the first upsampling layer (scaling factor 2), the first Bi-Concat layer (concatenating the current feature with the output of the third C3 SimAM module in the backbone), the first VOVGSCSP module (512 channels), the second GSConv module (256 channels), the second upsampling layer (scaling factor 2), the second Bi-Concat layer (concatenating the current feature with the output of the second C3 SimAM module in the backbone), the second VOVGSCSP module (256 channels), the third GSConv module (256 channels), the third Bi-Concat layer (concatenating the current feature with the output of the first VOVGSCSP module), the third VOVGSCSP module (256 channels), the fourth GSConv module (512 channels), the fourth Bi-Concat layer (concatenating the current feature with the output of the SPPF module), and the fourth VOVGSCSP module (512 channels). The detection head contains three parallel detection branches: branch P3, which applies a 1×1 convolution (3×(5+80) channels) to the output of the third VOVGSCSP module to predict targets at a scale of 80×80; branch P4, which applies a 1×1 convolution (3×(5+80) channels) to the output of the fourth VOVGSCSP module to predict targets at a scale of 40×40; and branch P5, which applies a 1×1 convolution (3×(5+80) channels) to the output of the SPPF module to predict targets at a scale of 20×20. Each branch outputs the bounding box coordinate offset (Δx, Δy, Δw, Δh), confidence score, and 80 class classification probabilities.

[0122] S132. Obtain the infrared image, visible light image, and radar image corresponding to the multi-information fusion feature map MIF_fuse in S131. For each fire candidate box in the MIF_fuse, calculate the mean and variance of the temperature pixel values ​​within the corresponding region of the fire candidate box in the infrared image using mathematical statistics methods. Extract the flame color histogram features within the corresponding region of the fire candidate box in the visible light image based on flame features. The histogram features include R channel proportion, orange color index OCI (pixel proportion in the hue range of 5°~25°), and histogram entropy. This invention uses R channel proportion. Analyze the scattering intensity distribution and texture direction features within the corresponding region of the fire candidate box in the radar image based on radar scattering texture analysis.

[0123] Similarly, the mean and variance of temperature pixel values ​​in the corresponding region of the corresponding infrared image, the flame color histogram features in the corresponding region of the corresponding visible light image, and the scattering intensity distribution and texture direction features in the corresponding region of the corresponding radar image are obtained for each fire candidate box.

[0124] S133. Based on the mean and variance of temperature pixel values ​​in the corresponding region of the corresponding infrared image, the flame color histogram features in the corresponding region of the corresponding visible light image, and the scattering intensity distribution and texture direction features in the corresponding region of the corresponding radar image, a new confidence score C_i' of the fire point candidate box is calculated using the multimodal fusion validator MMV (i.e., secondary verification). If C_i' ≥ a custom confidence score threshold T_conf, it is marked as a true fire point TPoint, thus obtaining the true fire point TPoint of the fire point candidate box. The custom confidence score threshold T_conf is flexibly set according to the tolerance of the wildfire early warning system for missed and false alarms, typically taking a value of 0.6 (i.e., 60%), and can be dynamically adjusted within the range of 0.5 to 0.7 according to the field environment and algorithm performance.

[0125] Similarly, the true fire points TPoint of all fire point candidate boxes on all multi-information fusion feature maps are obtained.

[0126] S14. Perform spatiotemporal clustering analysis on the actual fire points of all fire point candidate boxes on all multi-information fusion feature maps to obtain a set of fire events. Calculate the risk level of each fire event using a linear weighted method, classify warning levels according to the risk level, and issue warnings based on the warning levels. The specific process is as follows:

[0127] S141. Perform spatiotemporal clustering analysis on the real fire points TPoint of all fire point candidate boxes on all the multi-information fusion feature maps obtained in S133. Merge the real fire points with a time interval Δt ≤ 30 minutes and a spatial distance D ≤ 500 meters to obtain the fire event set E = {e1, e2, …}. The spatiotemporal clustering analysis includes the ST-DBSCAN method.

[0128] S142. Calculate e for each fire event using a linear weighting method. i The risk level L_risk is calculated as follows: L_risk = w1×area + w2×maximum temperature + w3×terrain slope + w4×wind speed, where w1, w2, w3, and w4 are the dynamic weight coefficients of the corresponding risk factors, and satisfy the normalization constraint w1+ w2+ w3+ w4= 1.

[0129] S143. Classify the early warning level according to the risk level L_risk, configure the corresponding response measures, and respond to the early warning according to the early warning level:

[0130] When 0 ≤ L_risk < 0.3, the risk level is defined as Level I, and the response measure is to send an SMS / APP alert.

[0131] When 0.3 ≤ L_risk < 0.6, the risk level is Level II, and the response measure is to notify the on-duty personnel for manual verification.

[0132] When L_risk is defined as 0.6 ≤ L_risk < 0.8, the risk level is Level III, and the response measure is to instruct the drone to automatically conduct a follow-up inspection.

[0133] When L_risk ≥ 0.8, the risk level is Level IV, and the response measures are to interrupt the power supply to the transmission line, set up isolation zones, and implement dispatch and control.

[0134] When the following situations occur, the rule engine loads the current weather forecast and power grid operation and maintenance status, and dynamically optimizes response measures by parsing multi-source information through predefined logical rules, specifically including:

[0135] When the grid load rate is >85%, the Level IV response measure of "interrupting power supply to transmission lines" will be replaced with "reduced voltage operation + isolation zone designation";

[0136] When the wind force is ≥ level 6, the instruction to "shut down power transmission lines in fire-prone sections along the route" should be added to the Level III response measures.

[0137] When the drought index SPI < -1.5, the Level II trigger threshold will be lowered from 0.3 to 0.2;

[0138] During the maintenance period, the Level I trigger threshold will be increased from 0.3 to 0.4;

[0139] The aforementioned optimization mechanism achieves real-time adaptation of response strategies based on the dynamic risk avoidance requirements of the "Emergency Plan for Extreme Weather in Power Grid".

[0140] The rule engine is a core decision-making component in SCADA or early warning systems, commonly including Drools and OpenRules. The weather forecast includes wind speed, precipitation probability, and drought index. The power grid operation and maintenance status includes real-time load rate, maintenance plan, and line voltage level.

[0141] S144. After the fire is dealt with, a field verification dataset V is collected by a manual inspection team or a drone. The field verification dataset V includes the coordinates of the fire point, the actual fire level, real-time image data, location and area, and the warning hit rate and false alarm rate. The warning hit rate = number of correct warnings / number of actual fire points, and the false alarm rate = number of incorrect warnings / total number of warning events.

[0142] S145. Feed back the field verification dataset V to S11-S14 to continuously optimize the early warning method of the present invention.

[0143] This step sets up intelligent early warning and multi-level response, performs dynamic threshold adjustment based on detection results, generates real-time early warning, and transmits fire risk information to the monitoring center and related execution units or systems through GIS visualization and multi-channel linkage push.

[0144] S2. In practical application, based on the comprehensive risk indicators of mountainous areas, towers on power transmission lines that are spaced 500 meters apart or cross important sections are designated as critical towers, resulting in multiple critical towers. A high-definition visible light camera (VisCAM) and a thermal infrared imager (IRCAM) are installed in the upper middle part of each critical tower. The comprehensive risk indicators include terrain slope, vegetation density, historical fire frequency, drought index, and line voltage level. The important sections include steep slopes and areas where forest belts intersect.

[0145] Based on visible light cameras, thermal infrared imagers, and radar, visible light images (VisData), infrared images (IRData), and radar images (SAR) are acquired at each moment during the occurrence of the wildfire under test. These images are then input into a pre-trained power transmission line wildfire early warning model based on multimodal image fusion. The model outputs the early warning level and issues an early warning based on the level.

[0146] This invention acquires visible light images, infrared images, and radar images at every moment when a wildfire occurs using a visible light camera, a thermal infrared imager, and radar. This invention monitors and provides timely warnings of wildfires in mountainous areas in this way, thereby achieving comprehensive monitoring.

[0147] Example

[0148] Taking a typical scenario of monitoring wildfires along power transmission lines as an example, the performance of the method of this invention is compared with existing single visible light image early warning methods, existing single infrared image early warning methods, and existing multimodal image stitching early warning methods on the same test set. The test set contains 1000 frames of multimodal image sequences and 120 real fire points. The results are shown in the table below:

[0149]

[0150] The environmental robustness score is composed of 1–5 points (5 being the best), comprehensively evaluating the stability against false alarms or missed alarms under conditions of smoke, low light at night, and complex terrain. The multimodal image stitching early warning method refers to directly stitching feature maps from visible light, infrared, and radar images in the channel dimension, without cross-modal attention or secondary verification mechanisms. The detection results of the above four methods are as follows: Figures 3-6 As shown in the figure, white represents towers and transmission lines, black represents the background, red represents fire points, and blue boxes represent detected fire points. As can be seen from the figure, the method of the present invention can detect the location of fire points more precisely and accurately.

[0151] Example 2

[0152] Pseudocode for multimodal image fusion in S12:

[0153] def multimodal_fusion_example(I_vis, I_ir, I_sar):

[0154] # Mean-variance normalization

[0155] vis_norm = normalize(I_vis)

[0156] ir_norm = normalize(I_ir)

[0157] sar_norm = normalize(I_sar)

[0158] # Feature Extraction

[0159] F_vis = E_vis(vis_norm)

[0160] F_ir = E_ir(ir_norm)

[0161] F_sar = E_sar(sar_norm)

[0162] # Multi-head attention fusion

[0163] MIF_fuse = transformer_attention([F_vis, F_ir, F_sar])

[0164] # Fire detection

[0165] B = D(MIF_fuse)

[0166] return B

[0167] S14 Dynamic Threshold Warning Pseudocode

[0168] def dynamic_threshold_warning(candidates, history_stats):

[0169] # Calculate sliding window statistics

[0170] stats = sliding_window_stats(history_stats)

[0171] T_area, T_temp = adjust_thresholds(stats)

[0172] alerts = []

[0173] For region in candidates:

[0174] area, temp = calc_region_stats(region)

[0175] if area > T_area and temp > T_temp:

[0176] alert = send_alert(region, area, temp)

[0177] alerts.append(alert)

[0178] return alerts

[0179] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for early warning of wildfires along power transmission lines based on multimodal image fusion, characterized in that, It includes the following steps: S1. Construct a transmission line wildfire early warning model based on multimodal image fusion. Obtain visible light images, infrared images, and radar images at each moment during the start and end of a wildfire. Input the visible light images, infrared images, and radar images at each moment into the transmission line wildfire early warning model based on multimodal image fusion for training, and obtain the trained transmission line wildfire early warning model based on multimodal image fusion. The specific process is as follows: S11. Obtain visible light images, infrared images, and radar images at each moment during the start and end of a wildfire. Preprocess the visible light images, infrared images, and radar images at each moment, and merge the preprocessed visible light images, infrared images, and radar images at all moments into a training dataset. S12. Extract the multi-scale features of each visible light image, infrared image and radar image in the training dataset respectively. Fuse the multi-scale features of the visible light image, infrared image and radar image at the same time to obtain the multi-information fusion feature map at the corresponding time. Similarly, obtain the multi-information fusion feature map at each time. S13. Use the YOLOv5 network to perform fire detection and localization on the multi-information fusion feature map at each time step to obtain the real fire points on the multi-information fusion feature map at each time step. S14. Perform spatiotemporal clustering analysis on the real fire points of all fire point candidate boxes on all multi-information fusion feature maps to obtain a set of fire events. Use a linear weighting method to calculate the risk level of each fire event, classify the warning level according to the risk level, and issue a warning according to the warning level. Thus, a well-trained early warning model for wildfires along power transmission lines based on multimodal image fusion was obtained; S2. Input the visible light image, infrared image and radar image of the wildfire to be tested into the trained transmission line wildfire early warning model based on multimodal image fusion, output the early warning level, and issue an early warning based on the early warning level.

2. The method for early warning of wildfires on power transmission lines based on multimodal image fusion according to claim 1, characterized in that: In step S11, visible light images, infrared images, and radar images are acquired at each moment during the start and end of a wildfire. These images are then preprocessed, and the preprocessed images from all moments are merged into a training dataset. The specific process is as follows: S111. Acquire visible light images, infrared images, and radar images at each moment from the start to the end of a wildfire. Combine the visible light images, infrared images, and radar images at the same moment into a set of image data to obtain multiple sets of image data. S112. Record the geographic location coordinates and meteorological parameters of each set of image data, including temperature, humidity and cloud cover; Each set of image data, along with its corresponding geographic location coordinates and meteorological parameters, is combined into a single set of data. Multiple sets of data are then merged into a multimodal dataset, CJData. S113. Perform geometric registration and temporal registration on each group of image data in the multimodal dataset CJData, and update the multimodal dataset CJData with each group of registered image data to obtain the registered dataset RegData. S114. Manually label the fire source areas in each image of the registered dataset RegData, and update the registered dataset RegData with each labeled image to obtain the labeled dataset LabelMap. S115. Adjust the resolution of each image in the labeled dataset LabelMap to 640 pixels wide × 480 pixels high, and update the labeled dataset LabelMap with the adjusted image to obtain the scaled dataset ScaleData. S116. Crop each image in the scaled dataset ScaleData, and update the scaled dataset ScaleData with each cropped image to obtain the cropped dataset CropData. S117. Perform pixel normalization processing on each image in the cropped dataset CropData, mapping the pixel values ​​to [0,1]. Update the cropped dataset CropData with each image after pixel normalization processing to obtain the dataset NormData. S118. Perform data augmentation on each image in the NormData dataset. Generate a maximum of 3–5 data augmented images for each image. Obtain the data augmented images corresponding to all images. Merge all images and their corresponding data augmented images into the training dataset ZQDatasets.

3. The method for early warning of wildfires on power transmission lines based on multimodal image fusion according to claim 1, characterized in that: In step S113, geometric and temporal registration are performed on each group of image data in the multimodal dataset CJData. The multimodal dataset CJData is then updated with the registered group of image data to obtain the registered dataset RegData. The specific process is as follows: For each set of image data in the multimodal dataset CJData, which includes visible light images, infrared images, and radar images, we extract the common points within the visible light images, infrared images, and radar images as matching benchmarks. Based on the matching benchmarks, we use affine transformation to align the infrared images and radar images to the visible light images, thus obtaining the aligned visible light images, infrared images, and radar images for each set of image data. The visible light image, infrared image, and radar image after each set of image data is aligned using timestamps. Frame-to-frame matching is performed, and image frames with a time difference greater than a threshold are removed to obtain each set of registered image data. The multimodal dataset CJData is then updated with each set of registered image data to obtain the registered dataset RegData.

4. The method for early warning of wildfires on power transmission lines based on multimodal image fusion according to claim 1, characterized in that: In step S116, each image in the scaled dataset ScaleData is cropped, and the scaled dataset ScaleData is updated with each cropped image to obtain the cropped dataset CropData. The specific process is as follows: S1161. Adjust the size of each image in the scaled dataset ScaleData; S1162. Within each image after resizing, the effective warning range defined by the safety boundary of the transmission line is divided. Based on the effective warning range, the fire point area in the corresponding image is marked using the line geometric model to obtain all images containing the fire point area. S1163. Use the UNet model based on semantic segmentation to extract features and classify pixels at the pixel level for each fire point region to obtain the fire point candidate probability for the corresponding fire point region. Based on each fire point candidate probability, obtain the fire point candidate probability map corresponding to each image. S1164. Denoise each fire candidate probability map by using threshold binarization, and then fill the holes in the denoised fire candidate probability map by using morphological opening and closing operations to obtain the corresponding new fire candidate probability map. Use connected component analysis to extract each fire candidate connected component in the new fire candidate probability map. S1165. Calculate the minimum bounding rectangle of each candidate connected component of the fire point. Based on each minimum bounding rectangle, crop each adjusted image to obtain the corresponding sub-image. Remove the background around the transmission line in the sub-image and keep only the transmission line to obtain the cropped image. Similarly, after obtaining all cropped images, merge all cropped images into a cropped dataset CropData.

5. The method for early warning of wildfires on power transmission lines based on multimodal image fusion according to claim 1, characterized in that: The data augmentation methods in S118 include random rotation, horizontal or vertical flipping, noise superposition, brightness or contrast adjustment, and one or more data augmentation methods can be randomly selected during application.

6. The method for early warning of wildfires on power transmission lines based on multimodal image fusion according to claim 1, characterized in that: In step S12, multi-scale features are extracted from each visible light image, infrared image, and radar image in the training dataset. The multi-scale features of the visible light image, infrared image, and radar image at the same time are then fused to obtain the multi-information fusion feature map for that time. Similarly, the multi-information fusion feature map for each time is obtained. The specific process is as follows: S121. Add channel attention modules to the ResNet18 network, MobileNetV2 network and EfficientNet-B0 network respectively to obtain the new ResNet18 network, the new MobileNetV2 network and the new EfficientNet-B0 network; The new ResNet18 network includes, in sequence, an input layer, a basic convolutional layer, a max pooling layer, a first-stage residual module, a second-stage residual module, a third-stage residual module, a fourth-stage residual module, and an output layer. The channel attention module is added to the first-stage residual module, the second-stage residual module, the third-stage residual module, and the fourth-stage residual module, respectively. The new MobileNetV2 network consists of an input layer, an initial convolutional layer, an inverted residual block sequence, and an output layer, with the channel attention module added to the inverted residual block sequence. The EfficientNet-B0 new network consists of an input layer, an initial convolutional layer, an MBConv module sequence, and an output layer, with the channel attention module added to the MBConv module sequence. We use the new ResNet18 network to extract the low-scale, mid-scale, and high-scale features of each visible light image in the training dataset ZQDatasets to obtain a feature map F_vis; The low-level scale features are edge features or texture features, the mid-level scale features are local pattern features, and the high-level scale features are semantic structure features; The low-scale, mid-scale, and high-scale features of each infrared image in the training dataset ZQDatasets are extracted using the new MobileNetV2 network to obtain a feature map F_ir; The low-scale, mid-scale, and high-scale features of each radar image in the training dataset ZQDatasets are extracted using the new EfficientNet-B0 network to obtain a feature map F_sar. Similarly, feature maps F_vis, F_ir, and F_sar are obtained at each time step; S122. Input the feature maps F_vis, F_ir, and F_sar at the same time into the Transformer multi-head attention network and output a fused feature map F_fuse. Similarly, obtain the fused feature map F_fuse at each time. S123. Using location coding, combine the geographic location coordinates, meteorological parameters and fused feature map F_fuse at the same time to obtain a multi-information fusion feature map MIF_fuse. Similarly, obtain the multi-information fusion feature map MIF_fuse for each time.

7. The method for early warning of wildfires on power transmission lines based on multimodal image fusion according to claim 1, characterized in that: In step S13, the YOLOv5 network is used to perform fire detection and localization on the multi-information fusion feature map at each time step to obtain the real fire points on the multi-information fusion feature map at each time step. The specific process is as follows: S131. Input the multi-information fusion feature map MIF_fuse at a certain moment into the YOLOv5 network, and generate N fire point candidate boxes on the multi-information fusion feature map MIF_fuse; S132. Obtain the infrared image, visible light image, and radar image corresponding to the multi-information fusion feature map MIF_fuse in S131. For each fire candidate box in the multi-information fusion feature map MIF_fuse, use mathematical statistics to calculate the mean and variance of the temperature pixel values ​​in the region corresponding to each fire candidate box in the infrared image. Extract the flame color histogram features in the region corresponding to each fire candidate box in the visible light image based on flame features. Analyze the scattering intensity distribution and texture direction features in the region corresponding to each fire candidate box in the radar image based on radar scattering texture. Similarly, the mean and variance of temperature pixel values ​​in the corresponding region of the corresponding infrared image and the flame color histogram features in the corresponding region of the corresponding visible light image are obtained for each fire candidate box, as well as the scattering intensity distribution and texture direction features in the corresponding region of the corresponding radar image. S133. Based on the mean and variance of temperature pixel values ​​in the corresponding region of the corresponding infrared image and the flame color histogram features in the corresponding region of the corresponding visible light image, as well as the scattering intensity distribution and texture direction features in the corresponding region of the corresponding radar image, a new confidence score C_i' of the current fire candidate box is calculated using a multimodal fusion validator. If C_i' ≥ a custom confidence score threshold, it is marked as a real fire point, and the real fire point of the current fire candidate box is obtained. The custom confidence score threshold is 0.

6. Similarly, the true fire points of all fire point candidate boxes on all multi-information fusion feature maps are obtained.

8. The method for early warning of wildfires on power transmission lines based on multimodal image fusion according to claim 1, characterized in that: In step S14, spatiotemporal clustering analysis is performed on the actual fire points of all fire point candidate boxes on all multi-information fusion feature maps to obtain a set of fire events. A linear weighted method is used to calculate the risk level of each fire event, and warning levels are assigned based on the risk levels. Warnings are then issued based on the warning levels. The specific process is as follows: S141. Perform spatiotemporal clustering analysis on all real fire points in the multi-information fusion feature maps obtained in S133, and merge the real fire points with a time interval Δt≤30 minutes and a spatial distance D≤500 meters to obtain a set of fire events. S142. The risk level L_risk of each fire event is calculated using a linear weighting method. L_risk = w1×area + w2×maximum temperature + w3×terrain slope + w4×wind speed, where w1, w2, w3, and w4 are the dynamic weight coefficients of the corresponding risk factors, and satisfy the normalization constraint w1+ w2+ w3+ w4= 1. S143. Classify the early warning level according to the risk level L_risk, configure the corresponding response measures, and respond to the early warning according to the early warning level: When 0 ≤ L_risk < 0.3, the risk level is defined as Level I, and the response measure is to send an SMS / APP alert. When 0.3 ≤ L_risk < 0.6, the risk level is Level II, and the response measure is to notify the on-duty personnel for manual verification. When L_risk is defined as 0.6 ≤ L_risk < 0.8, the risk level is Level III, and the response measure is to instruct the drone to automatically conduct a follow-up inspection. When L_risk ≥ 0.8, the risk level is Level IV, and the response measures are to interrupt the power supply to the transmission line, set up isolation zones, and implement dispatch and control.

9. A method for early warning of wildfires along power transmission lines based on multimodal image fusion according to claim 8, characterized in that: When the following conditions occur, the current weather forecast and power grid operation and maintenance status are loaded using a rule engine. Multi-source information is then parsed using predefined logical rules to optimize response measures. The weather forecast includes wind speed, precipitation probability, and drought index. The power grid operation and maintenance status includes real-time load rate, maintenance plan, and line voltage level, specifically including: When the grid load rate is >85%, the interruption of power supply to transmission lines in the Level IV response measures will be replaced by step-down operation + isolation zone designation; When the wind force is ≥ level 6, the Level III response measures shall include an instruction to shut down power transmission lines in fire-prone sections along the route; When the drought index SPI < -1.5, the Level II trigger threshold will be lowered from 0.3 to 0.2; During the maintenance period, the Level I trigger threshold was increased from 0.3 to 0.

4.

10. A method for early warning of wildfires on power transmission lines based on multimodal image fusion according to claim 1, characterized in that: The methods for acquiring the visible light image, infrared image, and radar image of the wildfire to be tested in S2 include: Based on the comprehensive risk indicators of mountainous areas, towers that are 500 meters apart or cross important sections on the power transmission lines in mountainous areas are designated as key towers, resulting in multiple key towers. A high-definition visible light camera and a thermal infrared imager are installed in the upper middle part of each key tower. The comprehensive risk indicators include terrain slope, vegetation density, historical fire frequency, drought index, and line voltage level. The important sections include steep slopes and forest belt intersection areas. Visible light cameras, thermal infrared imagers, and radar were used to acquire visible light images, infrared images, and radar images at every moment during the wildfire.

Images