Power transmission line multi-mode fusion intelligent forest fire monitoring method

Abstract

The invention relates to the field of power transmission line forest fire monitoring, and discloses a power transmission line multi-mode fusion intelligent forest fire monitoring method which is used for constructing an all-dimensional power transmission line forest fire prevention monitoring system and remarkably improving the intelligent level of power transmission line forest fire risk prevention and control. Comprising the steps of reasoning and identifying a smoke and fire target for a fused feature set by utilizing an improved YOLOv8 algorithm model, realizing accurate positioning of a fire point by adopting a haze penetration thermal imaging technology and a smoke cooperative monitoring technology in combination with spatial positioning, and establishing a temperature and humidity and forest fire occurrence probability correlation model based on historical data. According to the real-time temperature and humidity parameter dynamic adjustment algorithm model, the confidence threshold and the sensor acquisition frequency are detected, a dynamic threshold control method is adopted, an early warning threshold is adaptively adjusted according to parameters such as environment illumination, a final early warning result is output after cross validation of a multi-modal data fusion strategy, and the safety of the power transmission line is effectively guaranteed.

Description

Technical Field

[0001] This invention relates to the field of wildfire monitoring of power transmission lines, and in particular to a multimodal fusion intelligent wildfire monitoring method for power transmission lines. Background Technology

[0002] With the rapid development of my country's economy, the demand for electricity continues to grow, and power transmission lines, as the key carriers of power transmission, are expanding in scale. Many transmission lines traverse complex terrain areas such as mountainous and forested regions. These areas are densely vegetated and, due to factors such as dry climates and frequent lightning, have a high risk of wildfires. Once a wildfire breaks out, it spreads rapidly, posing a serious threat to the safe operation of transmission lines. Relevant statistics show that in recent years, transmission line failures caused by wildfires have been on the rise, posing a significant challenge to the stable power supply of the power system. In mountainous areas, transmission line tripping accidents caused by wildfires lead to large-scale power outages, affecting not only the normal lives of residents but also causing huge economic losses to industrial production.

[0003] Traditional wildfire monitoring methods primarily rely on fixed-point shooting by ordinary cameras supplemented by manual inspections. While ordinary cameras are deployed in fixed locations for continuous monitoring, their core problem lies in their heavy dependence on manual image review for fire assessment. In the unique and complex environment of mountainous areas, such as frequent fog, drastic changes in lighting, and the swaying and obstruction of dense vegetation, ordinary cameras are highly susceptible to misinterpreting environmental interference as flames or smoke, leading to a high false detection rate. Furthermore, limited by camera resolution and the challenges posed by the mountainous environment itself, the system lacks the ability to capture irregularly shaped and dynamically changing flames and smoke, especially small or blurred targets, resulting in a significant risk of missed detections and monitoring blind spots. On the other hand, traditional manual inspection measures, including 24-hour duty by line guards, manual clearing of weeds and shrubs, patrolling high-risk areas such as sacrificial sites to reduce fire hazards, and creating firebreaks, while playing a role in ensuring stable power supply and blocking fires, also have obvious limitations: high labor costs, effectiveness depends on personal experience and judgment, and it is difficult to achieve comprehensive and seamless coverage at night or in vast and complex mountainous areas, making it difficult to guarantee efficiency and reliability.

[0004] The application of unmanned intelligent inspection technology represents a significant advancement in the field of wildfire monitoring. Drones, equipped with various advanced sensors, conduct aerial inspections, making them particularly suitable for monitoring critical, high-risk areas such as power transmission line corridors. Their core advantage lies in overcoming terrain limitations, allowing them to flexibly penetrate rugged mountains, valleys, and other areas inaccessible to vehicles and personnel, achieving wide-area coverage. Simultaneously, drones possess multi-angle, three-dimensional observation capabilities, enabling aerial or lateral scanning, effectively avoiding vegetation obstruction problems often encountered by fixed ground cameras, and obtaining more comprehensive on-site information. However, drone inspection technology also faces inherent challenges and limitations. The most prominent constraint is limited flight time. Currently, most civilian multi-rotor drones typically have a single flight time of around 1 to 3 hours. For wildfire prevention tasks requiring long-term, large-area continuous monitoring, frequent battery replacements or deployment of multiple drones in relay significantly increase operational complexity and cost. Its effectiveness is highly dependent on weather conditions. In strong winds, flight safety risks are extremely high, and takeoff may even be impossible, leading to monitoring interruptions. Furthermore, the peak season for wildfires is often accompanied by dry, windy weather, posing a severe challenge to the availability of drones. In addition, achieving truly all-weather, uninterrupted continuous monitoring remains very difficult and costly under current technological conditions. The operation, real-time data transmission, and analysis of drones also require personnel with specialized skills; maintenance costs and the professional threshold are factors that cannot be ignored when promoting their application.

[0005] Improving monitoring accuracy is a core challenge that current wildfire monitoring technologies urgently need to overcome. Existing technologies urgently need to break through the following key bottlenecks: First, their ability to cope with complex background interference in mountainous areas is severely insufficient. Persistent clouds and fog, drastically changing lighting, and swaying vegetation, among other interference factors, make it difficult for traditional algorithms to effectively distinguish between real flames and interfering objects, leading to the particularly prominent problem of clouds and lights being misdetected as smoke or flames. Second, the detection accuracy of flames and smoke targets urgently needs improvement. These targets have irregular shapes and are dynamically changing; existing models perform poorly in identifying small, distant, or low-contrast blurred targets, making effective detection difficult in complex background scenes.

[0006] Therefore, we propose a multimodal fusion intelligent wildfire monitoring method for power transmission lines to solve the above problems. Summary of the Invention

[0007] This invention provides a multimodal fusion intelligent wildfire monitoring method for power transmission lines, which is used to build a comprehensive wildfire prevention monitoring system for power transmission lines and significantly improve the level of intelligence in wildfire risk prevention and control of power transmission lines.

[0008] The first aspect of this invention provides a multimodal fusion intelligent wildfire monitoring method for power transmission lines. The method includes: collecting multimodal data, preprocessing the multimodal data and fusing multidimensional features to generate a fused multimodal feature set; using an improved YOLOv8 algorithm model to infer the fused multimodal feature set and identify smoke and fire targets; performing fog-penetrating thermal imaging and extracting suspected fire points based on dynamic threshold segmentation; the smoke collaborative monitoring technology calculates the smoke diffusion direction using the concentration difference of multiple photoelectric smoke detectors; dynamically adjusting the detection confidence threshold of the algorithm model and the sensor acquisition frequency according to real-time temperature and humidity parameters; and using a dynamic threshold control method to adaptively adjust the warning threshold based on ambient light, smoke concentration, and temperature and humidity parameters, and outputting the final warning result.

[0009] Optionally, in a first implementation of the first aspect of the present invention, the method includes: acquiring raw multimodal data, wherein a starlight-level camera acquires visible light images, a thermal imaging sensor acquires infrared images, a smoke sensor acquires smoke concentration data, and a temperature and humidity sensor acquires temperature and humidity data; preprocessing the acquired raw multimodal data to generate preprocessed multimodal data, performing noise reduction and contrast enhancement processing on the visible light images, performing temperature normalization and pseudo-color mapping on the infrared images, and converting the smoke concentration data and temperature and humidity data into numerical features; and fusing multi-dimensional features into the preprocessed multimodal data to output a fused multimodal feature set.

[0010] Optionally, in the second implementation of the first aspect of the present invention, the multi-dimensional feature fusion includes: front-end fusion: concatenating 6-channel image data with 3-channel smoke-temperature-humidity feature maps to form 9-channel input data; middle fusion: extracting features through a CNN-Transformer dual-branch system, wherein the CNN branch processes 9-channel spatial features and the Transformer branch processes smoke and temperature-humidity temporal features, and using an iterative cross-attention mechanism to guide feature alignment; and back-end fusion: independently processing visual-thermal imaging features and smoke-temperature-humidity features, generating their respective detection probabilities, and then fusing the results through DS evidence theory.

[0011] Optionally, in a third implementation of the first aspect of the present invention, the method includes: inputting the fused multimodal feature set into an improved YOLOv8 algorithm model; performing weighted processing on the input features, dynamically adjusting the feature weights according to the real-time collected smoke concentration change rate and temperature change rate, and outputting a weighted feature map; inputting the weighted feature map into a parallel processing channel to simultaneously process image features and numerical features, and outputting fused features; optimizing the fused features through cross-modal and cross-layer bidirectional interactive connections, and outputting optimized detection features; performing smoke and fire target recognition based on the optimized detection features, and outputting the recognition result.

[0012] Optionally, in the fourth implementation of the first aspect of the present invention, the weighted feature map is input into a parallel processing channel to process image features and numerical features simultaneously. The numerical feature branch uses a fully connected layer and batch normalization to extract features from smoke concentration and temperature and humidity data, and performs deep fusion with image features to output fused features. The fused features are optimized through cross-modal and cross-layer bidirectional interactive connections. An improved CBAM hybrid attention mechanism is embedded in the feature pyramid network layer. Channel attention distinguishes the importance of different modal features, spatial attention locates the target region, and modal attention dynamically adjusts the fusion ratio of each modal feature to output optimized detection features.

[0013] Optionally, in the fifth implementation of the first aspect of the present invention, the method includes: acquiring infrared radiation data and generating a clear thermal imaging image using its fog-penetrating characteristics; performing dynamic threshold segmentation processing on the thermal imaging image, automatically adjusting the high-temperature area judgment threshold according to the real-time ambient temperature and background temperature distribution, and extracting temperature abnormal areas as suspected fire points; acquiring smoke concentration data in real time and calculating the concentration difference between adjacent detectors; determining the smoke concentration gradient change based on the concentration difference, and calculating the smoke diffusion direction vector in combination with the detector deployment location information; performing spatial correlation analysis between the suspected fire point area and the smoke diffusion direction vector, and determining the precise location coordinates of the fire point through the triangulation principle; in a high-humidity environment, correcting the smoke concentration threshold using a humidity compensation algorithm based on real-time humidity data, eliminating the influence of water vapor interference on the detection results, and outputting the final fire point location coordinates and confidence assessment.

[0014] Optionally, in the sixth implementation of the first aspect of the present invention, the method includes: collecting ambient temperature and humidity data in real time using a temperature and humidity sensor; inputting the ambient temperature and humidity data into a pre-established correlation model between temperature and humidity and the probability of wildfire occurrence; dynamically adjusting the detection confidence threshold of the algorithm model based on the output of the correlation model; synchronously adjusting the collection frequency of the smoke sensor according to the real-time temperature and humidity parameters; correcting the feature weights for fire point identification based on the environmental impact of temperature and humidity on fire point detection; and outputting the adjusted detection confidence threshold, sensor collection frequency, and feature weight parameters.

[0015] Optionally, in the seventh implementation of the first aspect of the present invention, the method includes: collecting ambient light data, smoke concentration data, and temperature and humidity data; inputting the ambient light data, smoke concentration data, and temperature and humidity data into a pre-trained LSTM model; adaptively adjusting the detection confidence threshold according to the output of the LSTM model; dynamically adjusting the warning threshold based on environmental parameters; and comprehensively considering the smoke and fire recognition results, smoke concentration changes, temperature and humidity parameters, and fire location information, performing cross-validation through a multimodal data fusion strategy, and outputting the final warning result and warning level.

[0016] The beneficial effects of this invention are as follows: Based on the YOLOv8 algorithm, an enhanced CA attention mechanism, a multimodal C2f-FasterNet module, and cross-modal and cross-layer bidirectional interactive connections are introduced to enhance the ability to identify smoke and fire targets. This enables more accurate identification of smoke and fire targets, especially in complex environments, and improves the detection effect of small smoke and fire or hidden fire sources, providing strong support for timely early warning and disposal. By combining fog penetration thermal imaging technology with smoke collaborative monitoring technology, the direction of smoke diffusion is calculated by using the fog penetration characteristics of long-wave infrared sensors and the concentration difference of multiple smoke detectors, so as to achieve precise fire point location. Even under severe weather conditions such as fog and haze, the fire point location can still be accurately determined, providing key information for rapid fire fighting and rescue, and reducing fire losses. A correlation model between temperature and humidity and the probability of wildfire occurrence is established based on historical data. The algorithm model detection confidence threshold and sensor acquisition frequency are dynamically adjusted according to real-time temperature and humidity parameters to make the monitoring system more adaptable to different environmental conditions. In high-risk wildfire environments, the monitoring sensitivity is improved, while in low-risk environments, energy consumption and false alarm rates are reduced, thus achieving optimal resource allocation. A dynamic threshold control method is adopted, which uses a pre-trained LSTM model to adaptively adjust the warning threshold based on ambient light, smoke concentration and temperature and humidity parameters. Combined with a multimodal data fusion strategy for cross-validation, the accuracy and timeliness of the warning are improved. The method can be flexibly adjusted according to environmental changes, avoiding the limitations of fixed threshold warnings and providing a reliable guarantee for the safe operation of transmission lines. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of an embodiment of the multimodal fusion intelligent wildfire monitoring method for transmission lines according to the present invention; Figure 2 This is a schematic diagram of the installation of a multimodal fusion intelligent wildfire monitoring system for power transmission lines in an embodiment of the present invention. Detailed Implementation

[0018] This invention provides a multimodal fusion intelligent wildfire monitoring method for power transmission lines, used to construct a comprehensive wildfire prevention monitoring system for power transmission lines, significantly improving the intelligent level of wildfire risk prevention and control for power transmission lines. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0019] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 One embodiment of the multimodal fusion intelligent wildfire monitoring method for transmission lines in this invention includes: 101. Multimodal data is collected by monitoring devices deployed on transmission lines, and the multimodal data is preprocessed and fused with multidimensional features to generate a fused multimodal feature set; the monitoring devices include starlight-level cameras, thermal imaging sensors, smoke sensors, and temperature and humidity sensors; multidimensional feature fusion adopts front-end fusion, intermediate fusion, or back-end fusion methods; It is understood that the executing entity of this invention can be a multimodal fusion intelligent wildfire monitoring device for power transmission lines, or it can be a terminal or a server; the specific implementation is not limited here. This embodiment of the invention will be described using a server as an example.

[0020] Specifically, raw multimodal data is collected through monitoring devices, including visible light images collected by a starlight-level camera, infrared images collected by a thermal imaging sensor, smoke concentration data collected by a smoke sensor, and temperature and humidity data collected by a temperature and humidity sensor. The collected raw multimodal data is preprocessed to generate preprocessed multimodal data. This includes denoising and contrast enhancement of visible light images, temperature normalization and pseudo-color mapping of infrared images, and conversion of smoke concentration data and temperature and humidity data into numerical features. The preprocessed multimodal data is fused using at least one of the following methods to achieve multidimensional feature fusion: Front-end fusion: The 6-channel image data is stitched together with the 3-channel smoke-temperature and humidity feature map to form 9-channel input data; Intermediate fusion: Features are extracted through a CNN-Transformer dual-branch approach, where the CNN branch processes 9-channel spatial features and the Transformer branch processes temporal features such as smoke and temperature / humidity. An iterative cross-attention mechanism is used to guide feature alignment. Backend fusion: Visual-thermal imaging features and smoke-temperature and humidity features are processed independently, and their respective detection probabilities are generated before the results are fused using DS evidence theory; The fused multimodal feature set is output for subsequent improved YOLOv8 algorithm model inference. 102. The improved YOLOv8 algorithm model is used to infer the fused multimodal feature set and identify fireworks targets. The improved YOLOv8 algorithm model includes an enhanced CA attention mechanism, a multimodal C2f-FasterNet module, and cross-modal and cross-layer bidirectional interactive connections. Specifically, the fused multimodal feature set is input into the improved YOLOv8 algorithm model, where the multimodal feature set includes visible light image features, infrared image features, smoke concentration data, and temperature and humidity data; The input features are weighted by an enhanced CA attention mechanism. This mechanism dynamically adjusts the feature weights based on the real-time collected smoke concentration change rate and temperature change rate. For regions where the smoke concentration suddenly increases beyond a set threshold or the temperature suddenly rises beyond a set threshold, the feature weights are increased by 20-30%, and the weighted feature map is output. The weighted feature map is input into the multimodal C2f-FasterNet module, which processes image features and numerical features simultaneously through parallel processing channels. The numerical feature branch uses fully connected layers and batch normalization to extract features from smoke concentration and temperature and humidity data, and then deeply fuses them with image features to output fused features. The fused features are further optimized through cross-modal and cross-layer bidirectional interactive connections. This connection embeds an improved CBAM hybrid attention mechanism in the feature pyramid network layer. Channel attention distinguishes the importance of different modal features, spatial attention locates the target region, and modal attention dynamically adjusts the fusion ratio of each modal feature to output optimized detection features. Based on the optimized detection features, smoke and fire targets are identified, and the identification results, including smoke and fire type, location information and detection confidence, are output. The identification results are used for subsequent accurate fire point location and early warning decision-making. 103. By combining fog and haze penetration thermal imaging technology with smoke collaborative monitoring technology and spatial positioning methods, the fire point can be accurately located. The fog and haze penetration thermal imaging technology uses an 8-14μm long-wave infrared sensor for fog penetration thermal imaging and extracts suspected fire points based on dynamic threshold segmentation. The smoke collaborative monitoring technology calculates the smoke diffusion direction by using the concentration difference of multiple photoelectric smoke detectors. Specifically, infrared radiation data is collected by an 8-14μm long-wave infrared sensor, and its fog-penetrating properties are used to generate clear thermal imaging images. Even in dense fog environments with visibility of less than 50 meters, it can still maintain 70-80% infrared transmittance. Dynamic threshold segmentation is performed on thermal imaging images. The threshold for judging high-temperature areas is automatically adjusted based on the real-time ambient temperature and background temperature distribution, and abnormal temperature areas are extracted as suspected fire points. By deploying multiple photoelectric smoke detectors in different locations, smoke concentration data is collected in real time, and the concentration difference between adjacent detectors is calculated. The smoke concentration gradient change is determined based on the concentration difference, and the smoke diffusion direction vector is calculated by combining the detector deployment location information. Spatial correlation analysis was performed between the suspected fire area and the smoke diffusion direction vector, and the precise location coordinates of the fire point were determined by the principle of triangulation. In high humidity environments, a humidity compensation algorithm is used to correct the smoke concentration threshold based on real-time humidity data, eliminating the influence of water vapor interference on the detection results, and outputting the final fire point location coordinates and confidence level assessment. 104. Adjustment technology based on temperature and humidity data: dynamically adjust the detection confidence threshold of the algorithm model and the acquisition frequency of the sensor according to real-time temperature and humidity parameters; the adjustment technology is based on a correlation model between temperature and humidity and the probability of wildfire occurrence established by historical data. Specifically, ambient temperature and humidity data are collected in real time using temperature and humidity sensors; Ambient temperature and humidity data are input into a pre-established correlation model between temperature and humidity and the probability of wildfire occurrence. This model is based on historical data. When the temperature is >30℃ and the humidity is <30%, the probability of wildfire occurrence is 5-8 times that of normal temperature and humidity environment. When the temperature is >40℃ and the humidity is <20%, the probability of wildfire occurrence reaches its peak. Based on the output of the association model, the detection confidence threshold of the algorithm model is dynamically adjusted: the detection confidence threshold is reduced to 0.3 in high temperature and low humidity environment (temperature > 35℃, humidity < 25%), and increased to 0.7 in low temperature and high humidity environment (temperature < 10℃, humidity > 70%). The sampling frequency of the smoke sensor is adjusted synchronously according to real-time temperature and humidity parameters: the sampling frequency is adjusted to 1 time / second in high temperature and low humidity environment (temperature > 35℃, humidity < 25%), and the sampling frequency is adjusted to 1 time / 5 seconds in low temperature and high humidity environment (temperature < 10℃, humidity > 70%). Based on the environmental impact of temperature and humidity on fire detection, the feature weights for fire identification are adjusted, with the temperature feature weights being increased by 10-15% in high-temperature and dry environments. The adjusted detection confidence threshold, sensor acquisition frequency, and feature weight parameters are output for subsequent dynamic threshold control. 105. Employing a dynamic threshold control method, the system adaptively adjusts the warning threshold based on ambient light, smoke concentration, and temperature and humidity parameters, and outputs the final warning result. Specifically, ambient light data is collected using a visible light camera, smoke concentration data is collected using a smoke sensor, and temperature and humidity data are collected using a temperature and humidity sensor. Ambient light data, smoke concentration data, and temperature and humidity data are input into a pre-trained LSTM model, which is trained based on a historical case database containing more than 100,000 meteorological events. This model is used to predict the optimal combination of confidence thresholds, smoke concentration thresholds, and temperature and humidity thresholds under different environments. Based on the output of the LSTM model, the detection confidence threshold is adaptively adjusted within the range of 0.25-0.75; The warning threshold is dynamically adjusted based on environmental parameters: in high temperature and low humidity environments (temperature > 35℃, humidity < 25%), the detection threshold is reduced and the smoke sampling frequency is increased to 1 time / second; in low temperature and high humidity environments (temperature < 10℃, humidity > 70%), the detection threshold is increased and the sampling frequency is reduced to 1 time / 5 seconds. When the wind speed is greater than 10 m / s, the detection thresholds for all items will be relaxed by 20%; when a fire point is detected, the detection thresholds for surrounding monitoring points will be automatically reduced to expand the monitoring range. By combining smoke and fire identification results, smoke concentration changes, temperature and humidity parameters, and fire location information, a multimodal data fusion strategy is used for cross-validation to dynamically output the final wildfire warning result and warning level.

[0021] Reference Figure 2 The hardware deployment of this invention will be described below: Front-end monitoring equipment: Wildfire monitoring equipment is installed on the power transmission lines. This equipment includes a starlight-level camera, a thermal imaging sensor, a smoke sensor (used to detect parameters such as smoke concentration and particle size), and a temperature and humidity sensor (monitoring wind speed, wind direction, temperature, and humidity). The smoke sensor uses a photoelectric smoke detector with a response time ≤3 seconds and a smoke concentration detection range of 0%~5%obs / m. The temperature and humidity sensor measures temperature from -40℃ to +85℃ and humidity from 0% to 100%RH, with accuracies of ±0.5℃ and ±2%RH, respectively.

[0022] Mid-range transmission equipment: 4G / 5G communication modules are used to achieve real-time data backhaul, while edge computing nodes (such as Android development boards) are deployed to perform local data preprocessing to reduce cloud computing pressure.

[0023] Backend server: Build a high-performance server cluster, configure storage hardware and central control system for data storage, model inference and early warning decision-making. It has powerful parallel computing capabilities and can process multimodal data from thousands of monitoring points at the same time.

[0024] Software environment setup and deep learning framework: A YOLOv8 model training and inference environment was built based on PyTorch, and the Ultralytics official library was integrated to achieve rapid development. The model structure was also extended to support the input and fusion of numerical features such as smoke, temperature and humidity.

[0025] Geographic Information System (GIS): Construct a three-dimensional digital twin model of the power transmission line, integrate BeiDou / GNSS positioning data and lidar point cloud data, and achieve accurate mapping of fire point locations.

[0026] Database: PostgreSQL is used to store historical monitoring data and model training samples, supporting efficient querying and analysis of spatiotemporal data. It features specially designed smoke concentration time series tables and temperature and humidity change record tables, which are associated with image data index tables to facilitate the backtracking and analysis of multimodal data.

[0027] Dataset preparation: Smoke recognition dataset: Collect 9,000 images covering various scenarios including day / night, strong light interference, dense smoke obstruction, and different temperature and humidity environments. Simultaneously record the smoke concentration value and temperature and humidity data under the corresponding scenarios. Label them in YOLO format (category 0: fire, 1: smoke) and divide them into training set, validation set and test set in an 8:1:1 ratio.

[0028] Principle and steps: Video smoke recognition: Multimodal data fusion: Image preprocessing: Noise reduction and contrast enhancement are performed on visible light images, and temperature normalization and pseudo-color mapping are performed on infrared images to highlight temperature differences. Simultaneously, smoke concentration, temperature and humidity data of the corresponding scene are correlated to build a multi-dimensional feature base.

[0029] Feature fusion strategy: A front-end fusion approach is adopted, integrating processed image features with numerical features such as smoke, temperature, and humidity before inputting them into the model. By enhancing the feature extraction capability of smoke and fire areas, the fusion efficiency of multi-scale features (such as small target flames and blurred smoke) is improved, achieving complementary enhancement of cross-modal information.

[0030] Model inference and optimization: Lightweight design: By optimizing the network module structure, the number of parameters is reduced while maintaining detection accuracy, meeting the real-time processing requirements of the edge.

[0031] Dynamic threshold adjustment: The detection confidence threshold (0.25-0.75) is dynamically adjusted according to ambient light intensity, smoke concentration and temperature and humidity conditions to balance detection sensitivity and false alarm rate in complex environments.

[0032] Early warning decision-making: When smoke or fire is detected, the system automatically captures images and makes a preliminary judgment through edge computing nodes. If the confidence level is ≥0.8, a level 1 warning is triggered, and the data is uploaded to the backend server for secondary verification. If the verification is successful, the warning information (including the location of the fire and the fire intensity) is immediately pushed to the operation and maintenance personnel.

[0033] Thermal imaging for fire location and temperature anomaly detection: Long-wave infrared imaging: Utilizing thermal imaging technology to penetrate low-visibility environments such as clouds and fog, capture tiny areas of temperature anomalies, and identify potential fire points.

[0034] Dynamic threshold segmentation: Based on real-time background temperature distribution and combined with environmental temperature and humidity conditions, the threshold for judging high-temperature areas is automatically adjusted to accurately extract suspected fire points.

[0035] 3D spatial positioning: Geometric positioning algorithm: Based on parameters such as the angle and distance between the monitoring equipment and the fire point, the relative position of the fire point and the transmission line is calculated through spatial geometric relationships, and then converted into specific coordinates by combining the tower positioning information.

[0036] GIS spatial analysis: Mapping the coordinates of the fire point to a 3D geographic information model, overlaying vegetation, topography and environmental parameters, to predict the fire spread path and the impact range on the line.

[0037] Coordinated monitoring of smoke and environmental parameters, smoke concentration detection and location assistance: Real-time concentration sensing: The smoke sensor captures smoke particle signals, converts them into concentration values, and records the trend of change, enabling sensitive detection of early smoke.

[0038] Multi-node collaboration: Calculate the smoke diffusion direction by using the concentration difference of multiple smoke sensors, which helps to narrow down the fire point location range and reduce the positioning error of a single device.

[0039] Temperature and humidity driven adaptive adjustment: Environmental correlation assessment: Based on the correlation between temperature, humidity, and the probability of wildfires, the sensitivity of the monitoring system is adjusted. For example, in hot and dry environments, the response speed to small fires is improved, while in high humidity environments, the risk of false alarms is reduced.

[0040] Dynamic parameter optimization: The smoke sampling frequency is automatically adjusted according to real-time temperature and humidity to balance energy consumption while ensuring monitoring timeliness and adapt to monitoring needs in different environments.

[0041] Multi-source data fusion decision making: By combining video smoke and fire recognition results, smoke concentration changes, temperature and humidity parameters, and fire location information, a multimodal data fusion strategy is used for cross-validation to dynamically output fire situation judgment results and warning levels, ensuring the accuracy and reliability of decision-making.

[0042] The technology used, and the improved YOLOv8 algorithm model: The improved YOLOv8 model has been further optimized for multimodal data (including smoke, temperature, and humidity) fusion, with key innovations including: Enhanced CA Attention Mechanism: Building upon the original CA attention mechanism, this mechanism embeds weighting factors for smoke concentration and temperature / humidity. When extracting features in the width and height directions using global average pooling, parameters such as smoke concentration gradient, temperature change rate, and humidity change rate are simultaneously introduced to adaptively adjust the feature weights for different regions. For example, the feature weights for regions with sudden increases in smoke concentration or rapid temperature rises are increased by 20-30%, enhancing the model's focus on key areas.

[0043] The multimodal C2f-FasterNet module adds a numerical feature processing branch to the original C2f-FasterNet module. It employs an ELAN structure to control the gradient flow path, combining group convolutions and residual connections to enhance image feature fusion. Simultaneously, it processes numerical features such as smoke, temperature, and humidity through fully connected layers and batch normalization, achieving deep fusion of image and numerical features. By adjusting the stacking number N (usually set to 4), network complexity and inference speed are balanced.

[0044] Cross-modal, cross-layer bidirectional interactive connectivity (CM-CBCC): An improved CBAM hybrid attention is embedded after the upsampling module and the C2f module in the Neck layer, adding a new modal attention dimension. Channel attention distinguishes the importance of visual, thermal imaging, smoke, and temperature and humidity features, spatial attention locates the target position, and modal attention adjusts the fusion weights of different modal features, achieving bidirectional enhancement of cross-layer and cross-modal features.

[0045] In the smoke detection task, the optimized YOLOv8 implementation is as follows: mAP@0.5 reaches 99.1%: an improvement of 7.4% over the baseline model, an improvement of 8.9% over the model that only integrates vision and thermal imaging, and a 60% reduction in false positive rate.

[0046] Inference speed: On the NVIDIA Jetson AGX Orin edge computing node, the average inference time for processing 9-channel input data is 18ms / frame, which meets the requirements for real-time monitoring.

[0047] Robustness: The model maintains high detection accuracy (≥98%) even in complex environments such as dense smoke, heavy fog, and strong light, which is more than 15% higher than that of traditional models.

[0048] Multimodal fusion networks, fusion strategies and advantages: Based on the characteristics of visible light (RGB), infrared (IR), smoke (S), and temperature / humidity (T / H) data, three fusion methods are supported, and the multimodal collaborative mechanism has been significantly optimized: Front-end fusion: The 6-channel image data (RGB+IR) and the 3-channel smoke-temperature and humidity feature map are stitched together to form a 9-channel input, realizing early data integration and preserving the integrity of the original information.

[0049] Intermediate fusion: Features are extracted using a CNN-Transformer dual-branch approach. The CNN branch processes the spatial features of 9 channels, while the Transformer branch processes the temporal features of smoke concentration, temperature, and humidity. An iterative cross-attention mechanism (ICAFusion) is used to guide the dynamic alignment of spatial and temporal features, focusing on spatial feature changes at key time points such as sudden increases in smoke concentration and rapid temperature rises.

[0050] Backend fusion: Visual-thermal imaging features and smoke-temperature and humidity features are processed independently to generate their respective detection probabilities. Detection results are fused using DS evidence theory to address the uncertainty of different modalities and improve the reliability of decision-making.

[0051] The multimodal fusion network introduces a modality-adaptive weight learning mechanism to dynamically adjust the weights of each modality based on the real-time environment. For example, in clear weather, the visual-thermal imaging modality has a higher weight (0.7); in dense smoke environments, the smoke-temperature and humidity modality weight increases to 0.6; and in hot and dry environments, the temperature feature weight is increased by 10-15% independently. This mechanism reduces the false negative rate (MR) to 3.2% on the multimodal wildfire monitoring dataset, a 12.5% ​​reduction compared to traditional fusion methods.

[0052] Haze penetration thermal imaging and smoke co-monitoring technology: The principle of fog penetration thermal imaging technology: The core advantages of military-grade 8-14μm long-wave infrared (LWIR) sensors: Fog penetration capability: In dense fog (visibility ≤ 50m), the infrared transmittance is 70-80%, and it can identify temperature anomalies of 0.05℃ at a distance of 400 meters (such as early fire points).

[0053] Imaging mechanism: An uncooled vanadium oxide focal plane detector (256×192 pixels) captures the maximum infrared radiation energy of the Earth's surface, and an electronic fog-penetrating algorithm dynamically enhances the image in a graded manner.

[0054] Fire detection: Maximum detection distance of 45 meters (35 meters for human targets), supports smoke detection; Smoke sensor collaborative enhancement: The smoke sensor uses the principle of photoelectric scattering. When smoke particles enter the detection chamber, the scattered infrared light causes the receiving tube to generate an electrical signal, and the smoke concentration is calculated by converting the signal strength. It works in conjunction with fog-penetrating thermal imaging technology. Complementary detection: Thermal imaging excels at detecting high-temperature fire spots, while smoke excels at detecting early-stage smoke. Combining the two enables full-process monitoring of wildfires from their inception to combustion.

[0055] Concentration gradient localization: The direction of smoke diffusion is calculated by the concentration difference of multiple smoke sensors, which helps thermal imaging determine the approximate location of the fire point and reduces the localization error of a single sensor.

[0056] Environmental adaptability compensation: In high humidity environments, thermal imaging may be affected by water vapor. Smoke sensors improve detection accuracy through humidity compensation algorithms (such as correcting the concentration threshold based on humidity values).

[0057] Temperature and humidity data-driven adjustment techniques; correlation model between temperature / humidity and wildfires: A correlation model between temperature, humidity, and the probability of wildfires was established based on historical data. The model shows that when the temperature is greater than 30℃ and the humidity is less than 30%, the probability of wildfires increases sharply (5-8 times that of normal temperature and humidity environments); when the temperature is greater than 40℃ and the humidity is less than 20%, the probability of wildfires reaches its peak. This model provides a basis for adjusting the sensitivity of the monitoring system.

[0058] Dynamic thresholds and parameter optimization: Detection threshold adjustment: In high temperature and low humidity environments (temperature > 35℃, humidity < 25%), the model confidence threshold is reduced to 0.3 and the smoke concentration alarm threshold is reduced to 2%obs / m to improve early warning capability; in low temperature and high humidity environments (temperature < 10℃, humidity > 70%), the thresholds are increased to 0.7 and 3.5%obs / m to reduce false alarms.

[0059] Sensor parameter optimization: The sampling frequency of the smoke sensor is automatically adjusted according to temperature and humidity (the sampling frequency is increased to 1 time / second in high temperature and low humidity environment, and 1 time / 5 seconds in normal temperature and humidity environment), balancing detection real-time performance and energy consumption.

[0060] Dynamic threshold control, adaptive mechanism: Parameter-driven: The LSTM model is trained using a historical case library (100,000+ meteorological events) to predict the optimal combination of confidence thresholds, smoke concentration thresholds, and temperature and humidity thresholds under different environments.

[0061] Real-time adjustment: The infrared sensitivity is dynamically adjusted according to humidity / visibility; the threshold is relaxed by 20% when the wind speed is >10m / s, and the threshold is relaxed by 15% to consider the rapid spread of fire. At the same time, the smoke concentration monitoring frequency is doubled; when a fire point is detected, the threshold of the surrounding monitoring points is automatically reduced to expand the monitoring range.

[0062] This invention integrates visual, thermal imaging, smoke, and temperature and humidity multimodal data fusion to construct a comprehensive and high-precision power transmission line wildfire monitoring system. By achieving deep fusion of multimodal features through an improved YOLOv8 model, and combining technologies such as fog-penetrating thermal imaging and smoke collaborative detection, temperature and humidity environmental adaptive adjustment, and dynamic threshold collaborative decision-making, significant technological breakthroughs have been achieved in smoke and fire identification accuracy, fire point location accuracy, system reliability, and early warning response speed. Its effectiveness and reliability have been verified in practical applications.

[0063] System Deployment: 1. Equipment installation steps: Installation and site selection: Install monitoring equipment on the power transmission line every 1-2 kilometers.

[0064] Hardware debugging: The temperature value of the thermal imaging sensor is calibrated by multi-point calibration using a blackbody radiation source (accuracy ±0.1℃); the smoke sensor is calibrated by concentration using a standard smoke generator (concentration 0-5%obs / m) to ensure that the measurement error is ≤0.1%obs / m; the temperature and humidity sensor is compared with data from a standard weather station, and the calibration error is reduced to ±0.3℃ (temperature) and ±1%RH (humidity).

[0065] Communication configuration: Configure the 4G module APN parameters on the edge computing node, test data upload latency ≤500ms, packet loss rate ≤1%.

[0066] 2. Model Inference Test: Smoke and fire recognition test: Validation was conducted in a laboratory environment using a test set containing 1000 images. Results showed: Daytime scene: mAP@0.5=98.5%, average inference time 12ms / frame. Nighttime scene: mAP@0.5=97.8%, average inference time 15ms / frame; Dense smoke scene (visibility <30m): mAP@0.5=97.5%, a 12% improvement over the model without smoke data.

[0067] Fire location test: Fire sources at different distances (100 meters to 3 kilometers) were set up in an outdoor area. The test results are as follows: Distance from fire source Visual + thermal imaging positioning error only Positioning error due to smoke and temperature / humidity 100 meters 3.2 meters 1.1 meters 500 meters 8.7 meters 3.5 meters 1000 meters 15.6 meters 4.8 meters 3000 meters 42.3 meters 12.5 meters System Operation Guide: 1. User Interface: Front-end display: The client interface is developed using PyQt5, displaying real-time video streams, temperature and humidity curves, smoke concentration curves, and temperature and humidity information for each monitoring point, with different colors indicating the warning level (red: Level 1 warning, yellow: Level 2 warning). The interface supports multi-modal data linkage viewing; clicking on the warning information of a specific monitoring point will simultaneously display the visible light image, infrared image, smoke concentration changes, and temperature and humidity data for that point.

[0068] GIS Interaction: Clicking the device icon on the 3D map allows viewing detailed monitoring data, supporting heat map analysis of fire locations (combined with smoke concentration distribution) and fire spread prediction simulation (combined with temperature, humidity, wind speed, and wind direction). Monitoring equipment parameters (such as camera focal length and smoke sensitivity) can be manually adjusted through the GIS interface.

[0069] 2. System Maintenance: Model updates: New monitoring data is collected monthly to incrementally train the YOLOv8 and LSTM models, ensuring that the models adapt to changes in the environment.

[0070] Sensitivity of the smoke sensor (tested using standard smoke, response time ≤ 3 seconds); temperature measurement accuracy of the thermal imaging sensor (compared with a blackbody radiation source, error ≤ 0.5℃).

[0071] The accuracy of the temperature and humidity sensor (compared with data from a standard weather station, the error is within the allowable range).

[0072] Data backup and cleanup: Monitoring data is automatically backed up to a cloud database daily and retained for one year. Redundant data (such as continuous monitoring data without anomalies) is cleaned up regularly to optimize database performance.

[0073] Operating environment: 1. Hardware Requirements: Edge computing nodes: NVIDIA Jetson AGX Orin (recommended) or equivalent computing power devices (such as Intel NUC11 Pro).

[0074] Cloud server: CPU ≥ 8 cores (such as Intel Xeon Silver 4316), memory ≥ 64GB, GPU ≥ NVIDIA RTX 3080 (for accelerating model inference).

[0075] 2. Software Requirements: Operating system: Ubuntu 20.04LTS (edge ​​and cloud), Windows 10 / 11 (client).

[0076] Environment: Python 3.8+, PyTorch 2.0+, OpenCV 4.7+, GDAL 3.6+ (GIS support).

[0077] 3. Network requirements: Front-end to cloud communication: 4G / 5G network, bandwidth ≥1Mbps, latency ≤500ms.

[0078] Client-cloud communication: wired network or Wi-Fi, bandwidth ≥10Mbps, latency ≤100ms.

[0079] This invention proposes a multimodal fusion network based on cross-modal, cross-layer bidirectional interactive connections (CM-CBCC), which achieves adaptive fusion of visible light, infrared, smoke, and temperature and humidity features through dynamic weight allocation. On a multimodal wildfire monitoring dataset, it improves mAP by 9.2% compared to traditional fusion methods, with significant advantages, especially in complex environments such as dense smoke and fog.

[0080] A quantitative relationship model between smoke concentration, temperature, humidity, and fire probability was constructed, and a three-dimensional early warning threshold matrix of "concentration-temperature-humidity" was proposed to achieve accurate early warning under different environments. Compared with single visual early warning, the false alarm rate was reduced to 0.008%, and the missed alarm rate was reduced to 0.005%.

[0081] By comprehensively considering smoke detection confidence levels, smoke concentration, temperature and humidity, drone verification results, and historical case data, the system dynamically adjusts warning levels and response strategies. The system can autonomously learn and optimize decision parameters based on environmental changes, adapting to the wildfire monitoring needs of different regions and seasons, achieving a decision accuracy rate of 99.5%.

[0082] By integrating spatial positioning through fog-penetrating thermal imaging, directional positioning based on smoke concentration gradients, environmental correction for temperature and humidity, and terrain analysis using 3D GIS, high-precision location of fire points (error ≤ 5 meters) is achieved. Combined with machine learning models to predict fire spread paths, considering multiple factors such as temperature, humidity, wind speed, terrain, and vegetation, the prediction accuracy reaches over 90%, providing a scientific basis for fire rescue.

[0083] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-modal fusion intelligent wildfire monitoring method for power transmission lines, characterized in that, The multimodal fusion intelligent wildfire monitoring method for transmission lines includes: Collect multimodal data, and preprocess the multimodal data and fuse it with multidimensional features to generate a fused multimodal feature set; The improved YOLOv8 algorithm model is used to infer the fused multimodal feature set to identify fireworks targets; Fog-penetrating thermal imaging is performed, and suspected fire points are extracted based on dynamic threshold segmentation. The smoke collaborative monitoring technology calculates the smoke diffusion direction by using the concentration difference of multiple photoelectric smoke detectors. The detection confidence threshold of the algorithm model and the sensor acquisition frequency are dynamically adjusted based on real-time temperature and humidity parameters. A dynamic threshold control method is adopted to adaptively adjust the warning threshold based on ambient light, smoke concentration, and temperature and humidity parameters, and output the final warning result.

2. The multi-modal fusion intelligent wildfire monitoring method for transmission lines according to claim 1, characterized in that, include: Raw multimodal data is collected, including visible light images from a starlight-level camera, infrared images from a thermal imaging sensor, smoke concentration data from a smoke sensor, and temperature and humidity data from a temperature and humidity sensor. The raw multimodal data collected is preprocessed to generate preprocessed multimodal data. The visible light image is denoised and contrast enhanced. The infrared image is normalized for temperature and pseudo-color mapped. The smoke concentration data and temperature and humidity data are converted into numerical features. The preprocessed multimodal data is fused with multidimensional features to output a fused multimodal feature set.

3. The multi-modal fusion intelligent wildfire monitoring method for transmission lines according to claim 2, characterized in that, Multi-dimensional feature fusion includes: Front-end fusion: The 6-channel image data is stitched together with the 3-channel smoke-temperature and humidity feature map to form 9-channel input data; Intermediate fusion: Features are extracted through a CNN-Transformer dual-branch approach, where the CNN branch processes 9-channel spatial features and the Transformer branch processes temporal features such as smoke and temperature / humidity. An iterative cross-attention mechanism is used to guide feature alignment. Backend fusion: Visual-thermal imaging features and smoke-temperature and humidity features are processed independently, and their respective detection probabilities are generated and then fused using DS evidence theory.

4. The multi-modal fusion intelligent wildfire monitoring method for transmission lines according to claim 2, characterized in that, include: The fused multimodal feature set is then input into the improved YOLOv8 algorithm model; The input features are weighted and dynamically adjusted according to the real-time collected smoke concentration change rate and temperature change rate, and the weighted feature map is output. The weighted feature map is input into the parallel processing channel to process image features and numerical features simultaneously, and the fused feature is output. The fused features are optimized through cross-modal and cross-layer bidirectional interactive connections, and the optimized detection features are output. Based on the optimized detection features, the smoke and fire target is identified, and the identification result is output.

5. The multi-modal fusion intelligent wildfire monitoring method for transmission lines according to claim 4, characterized in that, The weighted feature map is input into the parallel processing channel to process image features and numerical features simultaneously. The numerical feature branch uses a fully connected layer and batch normalization to extract features from smoke concentration and temperature and humidity data, and then deeply fuses them with image features to output fused features. The fused features are optimized through cross-modal and cross-layer bidirectional interactive connections. An improved CBAM hybrid attention mechanism is embedded in the feature pyramid network layer. Channel attention distinguishes the importance of different modal features, spatial attention locates the target region, and modal attention dynamically adjusts the fusion ratio of each modal feature to output the optimized detection features.

6. The multi-modal fusion intelligent wildfire monitoring method for transmission lines according to claim 4, characterized in that, include: Infrared radiation data is collected and its fog-penetrating properties are used to generate clear thermal imaging images. The thermal imaging image is subjected to dynamic threshold segmentation processing. The threshold for judging high temperature areas is automatically adjusted according to the real-time ambient temperature and background temperature distribution, and the temperature abnormal areas are extracted as suspected fire points. Real-time acquisition of smoke concentration data; calculation of concentration difference between adjacent detectors. The smoke concentration gradient change is determined based on the concentration difference, and the smoke diffusion direction vector is calculated by combining the detector deployment location information. Spatial correlation analysis was performed between the suspected fire point area and the smoke diffusion direction vector, and the precise location coordinates of the fire point were determined by the triangulation principle. In high humidity environments, a humidity compensation algorithm is used to correct the smoke concentration threshold based on real-time humidity data, eliminating the influence of water vapor interference on the detection results, and outputting the final fire point location coordinates and confidence level assessment.

7. The multi-modal fusion intelligent wildfire monitoring method for transmission lines according to claim 6, characterized in that, include: Real-time collection of ambient temperature and humidity data via temperature and humidity sensors; The ambient temperature and humidity data are input into a pre-established correlation model between temperature and humidity and the probability of wildfire occurrence. Based on the output of the association model, the detection confidence threshold of the algorithm model is dynamically adjusted; The sampling frequency of the smoke sensor is adjusted synchronously according to real-time temperature and humidity parameters; Based on the environmental impact of temperature and humidity on fire detection, the feature weights for fire identification are corrected. Output the adjusted detection confidence threshold, sensor acquisition frequency, and feature weight parameters.

8. The multi-modal fusion intelligent wildfire monitoring method for transmission lines according to claim 7, characterized in that, include: Collect ambient light data, smoke concentration data, and temperature and humidity data; The ambient light data, smoke concentration data, and temperature and humidity data are input into the pre-trained LSTM model; The detection confidence threshold is adaptively adjusted based on the output of the LSTM model. Dynamically adjust early warning thresholds based on environmental parameters; By combining smoke and fire identification results, smoke concentration changes, temperature and humidity parameters, and fire location information, a multimodal data fusion strategy is used for cross-validation to output the final warning result and warning level.

Images