Dual-light fusion method and system for monitoring forest fire in dark night based on SAM semantic enhancement

Abstract

The application discloses a kind of based on SAM semantic enhancement's black night power transmission mountain fire monitoring double light fusion method and system, method includes: collecting the visible light and infrared image in the periphery of power transmission line under black night environment and respectively pre-processing;Based on SAM model, extract the black night mountain fire special semantic from the visible light and infrared image after pre-processing, and obtain the corresponding feature map by semantic feature enhancement;According to feature map, hierarchical feature extraction and dynamic fusion are carried out, and the final fusion feature is obtained;The main network is constructed, and the quality of final fusion feature and mountain fire recognition precision are improved as the goal to train the main network;Sub-network for mountain fire recognition is constructed, and the fusion feature and mountain fire probability map distance between sub-network and main network are minimum as the goal to train the sub-network;Mountain fire early warning is carried out using the trained sub-network according to the collected image output mountain fire probability map.The application can realize the accurate identification and positioning of mountain fire in the periphery of power transmission line.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent monitoring of power systems and computer vision, and particularly relates to a night power transmission forest fire monitoring dual-light fusion method and system based on SAM semantic enhancement. BACKGROUND

[0002] The overhead transmission line is one of the important equipment of the power system, and plays an important role in the flow and transmission of electric energy. The stable operation of the overhead transmission line is the key to ensuring the reliable operation of the power system. However, the overhead transmission line often crosses vast regions, including forests, grasslands and other flammable areas. These areas are prone to forest fires in dry seasons, which pose a serious threat to the overhead transmission line. Forest fires can not only directly burn the overhead transmission line, but also cause line tripping, power failure and other accidents, affecting the normal operation of the power system. Although some forest fire monitoring technologies for power transmission have been proposed in recent years, the monitoring of forest fire hazards in the dark environment is still not high, mainly facing two technical bottlenecks:

[0003] First, the performance of a single sensor is limited. The signal-to-noise ratio (SNR) of a visible light camera is less than 20 dB when the light intensity is less than or equal to 5 lux in the dark, and the texture details (such as smoke edges, low-lying vegetation fires) are severely lost, and the recognition rate of dark fires (temperature 200-300℃) is not high. Although the infrared thermal imager can capture the thermal radiation (8-14μm spectral band) of the forest fire, the spatial resolution is low (mostly 640×512), it is easily disturbed by the background (such as the heat dissipation of the overhead transmission tower joint and the heat source of wild animals), the false detection rate is more than 40%, and it cannot distinguish between "forest fire smoke" and "morning mist / low cloud".

[0004] Second, the existing dual-light fusion technology based on visible light and infrared imaging is not good. Traditional dual-light fusion methods (such as wavelet transform and non-subsample contourlet transform) only rely on pixel-level intensity superposition and do not combine the semantic association of forest fires and overhead transmission lines, resulting in problems such as "fuzzy thermal targets" and "texture artifacts" in the fusion image; deep learning-based methods (such as U2Fusion and TarDAL) improve visual quality, but are not optimized for night forest fires. The loss function focuses on "global visual fidelity", ignoring key features of forest fires (such as local highlights of point-like bright fires and diffusion textures of smoke), and the recognition effect of forest fires in the dark environment is not good, which cannot meet the operation and maintenance needs of the power grid "15-minute early warning".

[0005] In recent years, the semantic prior of the SAM (Segment Anything Model) visual large model has been proven to effectively improve the performance of multi-modal fusion, but direct reuse has two major problems: first, the original semantic patch of SAM contains a large number of non-fire targets (such as trees, towers, and rocks), which are easy to dominate the fusion process in the dark night scene, resulting in the fire features being covered up; second, the complete SAM model (ViT-H backbone) takes more than 200ms per frame for inference, while the power inspection unmanned aerial vehicle requires real-time performance of less than 50ms per frame, which cannot meet the engineering deployment requirements.

[0006] In summary, the prior art cannot simultaneously meet the "semantic accuracy, scene adaptability, and real-time performance" requirements of dark night power line fire monitoring, and a targeted dual-light fusion method is urgently needed. SUMMARY

[0007] The technical problem to be solved by the present application is to provide a dual-light fusion method and system for dark night power line fire monitoring based on SAM semantic enhancement to realize accurate identification and positioning of power line perimeter fires.

[0008] To solve the above technical problems, the technical scheme adopted by the present application is:

[0009] A dual-light fusion method for dark night power line fire monitoring based on SAM semantic enhancement, comprising the steps of:

[0010] S1, collecting visible light images and infrared images around the power line in the dark night environment and pre-processing them respectively;

[0011] S2, constructing a dark night fire exclusive semantic extraction module based on the SAM model and extracting dark night fire exclusive semantics from the pre-processed visible light images and infrared images, and obtaining the corresponding feature map through semantic feature enhancement;

[0012] S3, constructing a dynamic attention fusion network and performing hierarchical feature extraction and dynamic fusion according to the feature map to obtain the final fusion feature;

[0013] S4, constructing a main network for dark night fire identification based on the dark night fire exclusive semantic extraction module and the dynamic attention fusion network, training the main network to improve the quality of the final fusion feature and the accuracy of fire identification, and obtaining the trained main network for outputting the fire probability map;

[0014] S5, a lightweight forest fire recognition subnetwork based on MobileNetV3 is constructed, and the subnetwork is trained to minimize the L2 distance between the fusion features of the subnetwork and the main network and the KL divergence of the forest fire probability map, to obtain a trained subnetwork, wherein the L2 distance between the fusion features output by the subnetwork and the final fusion features of the main network is constrained by a feature distillation function, and the KL divergence between the forest fire probability map output by the subnetwork and the forest fire probability map of the main network is constrained by a forest fire probability distillation function, and the output of the subnetwork is the fusion features and the forest fire probability map;

[0015] S6, using the trained subnetwork to output a forest fire probability map according to the collected image for forest fire early warning.

[0016] Further, in step S1, the specific steps of preprocessing are as follows:

[0017] The visible light image is enhanced by adaptive Gaussian filtering and multi-scale Retinex technology, the infrared image is denoised by median filtering and the potential forest fire hot area is screened by a temperature threshold, and preprocessed visible light images and infrared images are generated respectively.

[0018] The expression of the visible light image enhancement is as follows:

[0019] ;

[0020] In the above formula, k is the total number of scales, is a Gaussian blurred image including multiple scales, is the weight of each scale, is a very small positive number, is the preprocessed visible light image, is the visible light image before processing.

[0021] Further, step S2 includes:

[0022] SAM semantic patch generation: mark the power line corridor area in the visible light image and the infrared image as a point prompt word, mark the open fire, smoke and high temperature vegetation in the power line corridor as a text prompt word, and generate an initial infrared and visible light semantic patch with the same resolution as the input visible light image and infrared image based on the point prompt word and the text prompt word;

[0023] Semantic screening: constructing a forest fire semantic mask to filter non-forest fire semantics; wherein for the initial infrared semantic patch , the area with a hot target greater than a first preset value in the SAM category probability is retained to generate an infrared semantic patch ; for the initial visible light semantic patch , and the region where the smoke or vegetation anomaly in the SAM category probability is greater than a second preset value is reserved to generate a visible light semantic patch ;

[0024] Semantic feature enhancement: the infrared semantic patch is multiplied pixel by pixel with the preprocessed infrared image to highlight high-temperature regions, and then a convolution is performed to reduce the dimensionality to generate a thermal target feature map ; the visible light semantic patch is subjected to edge detection to strengthen the smoke edge, and then a convolution is performed to reduce the dimensionality to generate a texture feature map .

[0025] Further, step S3 comprises:

[0026] Multi-scale visual feature extraction: a multi-level wildfire-adaptive convolution block is constructed to extract multi-level features of the infrared and visible light images, respectively, to obtain infrared visual features and visible light visual features ;

[0027] Semantic-guided dynamic attention mechanism: for each level of visual features, the fusion semantic feature map obtained by cross-modal semantic fusion of the feature maps of the visible light image and the infrared image in step S2 is projected into a query matrix Q through a fully connected layer, and the infrared visual features and the visible light visual features are mapped through a convolution, and the scores of the infrared and visible light visual features after the convolution are weighted and adjusted based on the wildfire confidence parameter to obtain corresponding attention scores, the value matrix of the infrared and visible light visual features is weighted and fused with the attention scores as weights, and high and low level features are spliced through a skip connection to obtain the final fusion features .

[0028] Further, the scores of the infrared and visible light visual features after the convolution are weighted and adjusted based on the wildfire confidence parameter to obtain corresponding attention scores, and the expression is as follows:

[0029]

[0030]

[0031]

[0032] ;

[0033] In the above formula, d is a scaling factor, is the score of the infrared visual features, is the score of the visible light visual features, is the attention score of the infrared visual features, attention score of visible light visual feature, fire confidence, Q is a query matrix, key matrix of infrared visual feature, key matrix of visible light visual feature;

[0034] The value matrix of infrared and visible light visual features is weighted and fused with the attention score as the weight, and the expression is as follows:

[0035]

[0036] In the above formula, is the lth layer fused feature after fusion, is the value matrix of infrared visual feature, is the value matrix of visible light visual feature.

[0037] Further, the loss function expression used when training the main network in step S4 is as follows:

[0038]

[0039] In the above formula, is the total loss function of the main network, is the feature alignment loss, is the context loss, is the fire recognition loss, is an adjustable coefficient;

[0040] The expression of the feature alignment loss is as follows:

[0041]

[0042] In the above formula, is the final fused feature, is the infrared visual feature, is the visible light visual feature;

[0043] The expression of the context loss is as follows:

[0044]

[0045]

[0046]

[0047] In the above formula, is the image target edge information loss, is the image pixel loss, is the visible light image,​​​​ is an infrared image;

[0048] wildfire identification loss The expression is as follows:

[0049] ;

[0050] In the above formula, is the label of the pixel point (i, j) position, is the wildfire probability map of the pixel point (i, j) position.

[0051] Further, the loss function expression used when training the sub-network in step S5 is as follows:

[0052] ;

[0053] In the above formula, is the total loss function of the sub-network, is the feature distillation function, is the wildfire probability distillation function, is the adjustable coefficient;

[0054] The feature distillation function is used to constrain the fusion feature output by the sub-network and the final fusion feature of the main network L2 distance, the expression is as follows:

[0055] ;

[0056] The wildfire probability distillation function is used to constrain the KL divergence between the wildfire probability map of the sub-network and the wildfire probability map of the main network , the expression is as follows:

[0057] .

[0058] Further, step S6 includes:

[0059] The fusion feature output by the sub-network is restored to the original resolution through upsampling, and a normalized fusion image is obtained through Sigmoid activation , and the wildfire probability map is output at the same time.

[0060] When it is judged that the wildfire probability map of a certain area is higher than the early warning threshold, it is determined that there is a wildfire hazard, and the early warning is triggered.

[0061] A dual-light fusion system for monitoring forest fires at night based on SAM semantic enhancement, comprising a microprocessor and a memory connected to each other, the microprocessor being programmed or configured to perform the steps of the method for monitoring forest fires at night based on SAM semantic enhancement dual-light fusion.

[0062] A computer-readable storage medium having stored therein computer programs / instructions programmed or configured to perform the steps of the method for monitoring forest fires at night based on SAM semantic enhancement dual-light fusion by a processor.

[0063] Compared with the prior art, the advantages of the present application are:

[0064] The present application can fully exploit the key fire semantic features hidden in visible light and infrared images under extremely low light conditions by introducing a SAM-based night fire-specific semantic enhancement mechanism, significantly improving the distinguishability and saliency of the fire target; on this basis, the dynamic attention fusion network adaptively allocates weights according to the importance of different modal features, enabling multi-level, cross-modal deep feature collaborative expression, effectively overcoming the poor adaptability of traditional static fusion to complex night scenes; by constructing a main network and a subnetwork model, the fusion features and the final fire probability map can be kept consistent, thereby improving the feature expression quality and the robustness of fire detection. The present application can realize higher precision and higher stability of fire recognition and early warning in the dark, significantly improving the safety and reliability of night patrol of power transmission lines. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 The flowchart of the method for monitoring forest fires at night based on SAM semantic enhancement of the embodiments of the present application. DETAILED DESCRIPTION

[0066] In order to better understand the above technical solutions, the above technical solutions will be described in detail below in conjunction with the drawings and specific embodiments of the specification.

[0067] As shown in Figure 1 the method for monitoring forest fires at night based on SAM semantic enhancement of the embodiments of the present application comprises the steps of:

[0068] S1, collecting visible light images and infrared images around the power transmission line in the dark environment and pre-processing them respectively;

[0069] S2, constructing a night fire-specific semantic extraction module based on a SAM model and extracting night fire-specific semantics from the pre-processed visible light images and infrared images, and obtaining corresponding feature maps through semantic feature enhancement;

[0070] S3, a dynamic attention fusion network is constructed, and hierarchical feature extraction and dynamic fusion are performed according to the feature map to obtain final fusion features;

[0071] S4, a main network for night mountain fire recognition is constructed based on the night mountain fire special semantic extraction module and the dynamic attention fusion network, the main network is trained to improve the quality of the final fusion features and the mountain fire recognition accuracy, and a trained main network for outputting a mountain fire probability map is obtained;

[0072] S5, a lightweight mountain fire recognition subnetwork with MobileNetV3 as the backbone network is constructed, the subnetwork is trained to minimize the L2 distance of the fusion features between the subnetwork and the main network and the KL divergence of the mountain fire probability map, and a trained subnetwork is obtained, wherein the L2 distance of the fusion features output by the subnetwork and the final fusion features of the main network is constrained by a feature distillation function, and the KL divergence of the mountain fire probability map output by the subnetwork and the mountain fire probability map of the main network is constrained by a mountain fire probability distillation function, the output of the subnetwork is the fusion features and the mountain fire probability map;

[0073] S6, the trained subnetwork is used to output a mountain fire probability map according to the collected image to perform mountain fire early warning.

[0074] In this embodiment, in step S1, the specific steps of preprocessing are as follows:

[0075] The visible light image is enhanced by adaptive Gaussian filtering and multi-scale Retinex technology, the infrared image is denoised by median filtering and the potential mountain fire hot area is screened by a temperature threshold, and preprocessed visible light images and infrared images are generated, respectively.

[0076] The expression of the visible light image enhancement is as follows:

[0077] ; (1)

[0078] In the above formula, k is the total number of scales, is a Gaussian blurred image including multiple scales (for example, 5x5, 15x15, 25x25 three scales), is the weight of each scale (for example, 0.5, 0.3, 0.2, respectively), is a very small positive number (for example, to avoid the nonsense of logarithm), is the preprocessed visible light image, is the visible light image before processing.

[0079] In a specific application embodiment, step S1 is specifically nighttime dual-light data acquisition and preprocessing. A patrol device or transmission line visualization monitoring device equipped with a visible light camera and an infrared thermal imager can be used to acquire synchronous visible light images of the channel area around the transmission line and infrared images . The visible light images are enhanced by adaptive Gaussian filtering and multi-scale Retinex technology shown in formula (1) to suppress noise and improve smoke contrast. The infrared images are denoised by 5x5 median filtering and screened for potential forest fire hot areas by a temperature threshold of ≥ 150°C to exclude environmental low temperature interference, respectively generating preprocessed visible light images and infrared images .

[0080] In this embodiment, step S2 includes:

[0081] SAM semantic patch generation: mark the transmission line corridor area as a point prompt word in the visible light image and the infrared image, and mark the open fire, smoke, and high temperature vegetation in the transmission line corridor as a text prompt word. Based on the point prompt word and the text prompt word, an initial infrared and visible light semantic patch with the same resolution as the input visible light image and infrared image is generated;

[0082] Semantic filtering: construct a forest fire semantic mask to filter non-forest fire semantics. For the initial infrared semantic patch , the area where the hot target in the SAM category probability is greater than a first preset value is retained to generate an infrared semantic patch . For the initial visible light semantic patch , the area where the smoke or vegetation anomaly in the SAM category probability is greater than a second preset value is retained to generate a visible light semantic patch .

[0083] Semantic feature enhancement: multiply the infrared semantic patch with the preprocessed infrared image pixel by pixel to highlight the high temperature area, and then generate a hot target feature map through convolution dimension reduction . The visible light semantic patch is subjected to edge detection to strengthen the smoke edge, and then a texture feature map is generated through convolution dimension reduction .

[0084] In a specific application embodiment, step S2 is specifically constructing a transmission forest fire exclusive semantic extraction module and generating a feature map. To solve the problem of "strong generality and weak transmission forest fire pertinence" of the SAM semantic patch, a three-level processing flow of prompt (prompt word) guidance, semantic filtering, and feature enhancement is designed to extract exclusive semantics of nighttime forest fires. The specific steps are as follows:

[0085] S21, SAM semantic patch generation and filtering:

[0086] Use "point + text" joint prompt for SAM night forest fire semantic extraction:

[0087] S211, point prompt: mark the power line corridor area in the dual-light image (mark the line coordinates through the visual camera, and the range is the vertical area of the overhead line Z coordinate);

[0088] S212, text prompt: "open fire (point, high temperature), smoke (patch, uneven gray scale), high temperature vegetation (brown, temperature ≥180℃) in the power line corridor";

[0089] S213, input SAM model (ViT-H backbone), generate initial infrared and visible light semantic patches 、 (visible light), patch resolution consistent with input image.

[0090] S22, semantic screening:

[0091] S221, build a forest fire semantic mask , filter non-forest fire semantics;

[0092] S222, for infrared patch , keep the area where "hot target" ≥0.6 in SAM category probability, generate ;

[0093] S223, for visible light patch , keep the area where "smoke / vegetation anomaly" ≥0.5 in SAM category probability, generate ; The non-preserved area is set to 0 to ensure that the semantic patch only contains forest fire related targets.

[0094] S23, forest fire semantic feature enhancement:

[0095] Design "temperature-texture dual-driven" semantic enhancer (Fire-Sem Enhancer) to strengthen forest fire features:

[0096] S231, infrared semantic feature enhancement: multiply with infrared preprocessed image pixel by pixel to highlight high temperature areas, and then reduce dimension to 128 channels through 1x1 convolution to generate hot target feature map ;

[0097] S232, visible light semantic feature enhancement: for Edge detection (Canny operator, threshold 50-150) + morphological dilation (kernel 3x3) is used to strengthen the smoke edge, and then 1x1 convolution is used to reduce the dimension to 128 channels to generate a texture feature map ;

[0098] S233, cross-modal semantic fusion: introduce the forest fire confidence (normalized by the forest fire class probability output by the SAM, ), dynamically adjust the dual-light semantic weight, and the formula is as follows:

[0099] (2)

[0100] In the above formula, is the sine position encoding, which is used to retain the spatial position information of the semantic patch and avoid position deviation during fusion; represents the enhanced infrared thermal target weight, The larger the value is, the more significant the forest fire is; is used to balance the visible light texture weight.

[0101] In this embodiment, step S3 includes:

[0102] Multi-scale visual feature extraction: construct a multi-level forest fire adaptive convolution block to extract multi-level features of infrared and visible light respectively, to obtain infrared visual features and visible light visual features ;

[0103] Semantic-guided dynamic attention mechanism: for each level of visual feature, the fusion semantic feature map obtained by cross-modal semantic fusion of the feature maps of the visible light image and the infrared image in step S2 is projected into a query matrix Q through a fully connected layer, and the infrared visual features and the visible light visual features are mapped through convolution, and the scores of the infrared and visible light visual features after convolution are weighted and adjusted based on the forest fire confidence parameter to obtain corresponding attention scores. The value matrix of the infrared and visible light visual features is weighted and fused with the attention scores as weights, and the high and low level features are spliced through the jump connection to obtain the final fusion feature .

[0104] In this embodiment, the scores of the infrared and visible light visual features after convolution are weighted and adjusted based on the forest fire confidence parameter to obtain corresponding attention scores, and the expression is as follows:

[0105]

[0106]

[0107]

[0108] ; (3)

[0109] In the above formula, d is a scaling factor (which can be 256, used to avoid gradient disappearance), is the score of the infrared visual feature, is the score of the visible light visual feature, is the attention score of the infrared visual feature, is the attention score of the visible light visual feature, is the forest fire confidence, and Q is a query matrix, is the key matrix of the infrared visual feature, is the key matrix of the visible light visual feature;

[0110] The value matrix of the infrared and visible light visual features is weighted and fused with the attention score as the weight, and the expression is as follows:

[0111] ; (4)

[0112] In the above formula, is the fused l-th layer fused feature, is the value matrix of the infrared visual feature, is the value matrix of the visible light visual feature.

[0113] In a specific application embodiment, step S3 is specifically constructing a dynamic attention fusion network and outputting a fused feature. In combination with the characteristics of forest fires at night, a “multi-scale visual feature extraction-semantic guided dynamic attention” fusion structure is designed to realize precise complementation of dual light features, and the specific steps are as follows:

[0114] S31, multi-scale visual feature extraction:

[0115] A 4-level “fire-adapted convolution block” (Fire-Conv Block) is constructed to extract multi-level features of infrared and visible light respectively, and each level block contains “convolution-normalization-activation-residual connection”:

[0116] The first level (low level, resolution 1920x1080, channel 64): 3x3 convolution (step 1) + BN + LeakyReLU (negative slope 0.2) is adopted to retain the smoke edge, pole tower outline and other detail features of the visible light image;

[0117] The second level (low level, resolution 960x540, channel 128): 3x3 convolution (step 2) + BN + LeakyReLU is adopted to strengthen the point-like fire heat details of the infrared image;

[0118] Level 3 (high-level, resolution 480×270, 256 channels): 3×3 dilated convolution (dilation=2) + BN + LeakyReLU, to capture the mesoscale diffusion features of wildfire smoke;

[0119] Level 4 (high-level, resolution 240×135, channels 512): 3×3 dilated convolution (dilation=4) + BN + LeakyReLU, to extract the global distribution of wildfires (such as the trend of open flame spread).

[0120] Output infrared visual features Visual characteristics of visible light The superscript indicates the level.

[0121] S32, Semantic-guided dynamic attention mechanism (innovative weight calculation):

[0122] For each level of visual features, a Fire-Attention module is designed, consisting of "semantic query - dual light key value - dynamic weights". To guide and avoid single-modal dominance:

[0123] S321, Generate the Query-Key-Value matrix:

[0124] Query(Q): will Through fully connected layers ( The projection is the query matrix. Focusing on the semantic and visual features of wildfires;

[0125] Key (K) and Value (V): The two-light visual features are convolved through a 1×1 convolution ( , Mapping to the current level's channel count: ; .

[0126] S322, Calculate the dynamic attention weights:

[0127] Introducing wildfire confidence The weighted adjustment of the dual-light score solves the problem of insufficient single-modal information in the dark scene. The calculation expression is shown in formula (3).

[0128] S323, Generate hierarchical fusion features: According to formula (4), weighted fusion of dual light value matrix, high and low level features are spliced ​​by skip connection; upsample high level features ( (down to lower resolution) ,and By splicing, the final fused features are obtained. .

[0129] In this embodiment, the loss function expression used when training the main network in step S4 is as follows:

[0130] ; (5)

[0131] In the above formula, is the total loss function of the main network, is the feature alignment loss (restricting the cosine similarity of the fused feature and the binocular vision feature), is the context loss (including the map loss), is the forest fire recognition loss (based on cross-entropy), is the adjustable coefficient (for example, );

[0132] The expression of the feature alignment loss is as follows:

[0133] ; (6)

[0134] In the above formula, is the final fused feature, is the infrared vision feature, is the visible light vision feature;

[0135] The expression of the context loss is as follows:

[0136]

[0137]

[0138] ; (7)

[0139] In the above formula, is the image target edge information loss (used to retain the image target edge information), is the image pixel loss (used for image pixel fidelity), is the visible light image, is the infrared image;

[0140] The expression of the forest fire recognition loss is as follows:

[0141] ; (8)

[0142] In the above formula, is the label of the pixel point (i, j) position (Y(i, j) = 1, for forest fire, Y(i, j) = 0, for background), ​The fire probability map of the pixel point (i, j) position.

[0143] In this embodiment, the loss function expression used when training the sub-network in step S5 is as follows:

[0144] (9)

[0145] In the above formula, is the total loss function of the sub-network, is the feature distillation function, is the fire probability distillation function, is the adjustable coefficient (for example, );

[0146] The feature distillation function is used to constrain the L2 distance between the fusion features output by the sub-network and the final fusion features of the main network , and the expression is as follows:

[0147] (10)

[0148] The fire probability distillation function is used to constrain the KL divergence between the fire probability map of the sub-network and the fire probability map of the main network , and the expression is as follows:

[0149] (11)

[0150] It can be understood that after distillation, the inference time of the sub-network is greatly reduced, which can meet the real-time requirement.

[0151] In a specific application embodiment, for the night power transmission fire monitoring scene, a "main network-sub network" distillation architecture is designed through steps S4 and S5. The main network is responsible for learning the "fire semantic guided dual light fusion + fire identification" composite capability, and the sub-network migrates knowledge through distillation to realize lightweight real-time deployment.

[0152] The main network is built with "fire exclusive semantic extraction module + dynamic attention fusion network" as the main body, and a fire identification loss (based on cross-entropy) is introduced, and the training target is "fusion quality + fire identification accuracy". The loss function of the main network is shown in formulas (5)-(8).

[0153] The sub-network can use MobileNetV3 as the backbone, with only 0.15M parameters. Through "double distillation", the knowledge of the main network is migrated to realize lightweight real-time deployment, and its loss function is shown in formulas (9)-(11).

[0154] In this embodiment, step S6 (fusion image generation and wildfire early warning) includes:

[0155] The fusion features output by the subnetwork The image is restored to its original resolution through upsampling (bilinear interpolation) and then normalized and fused using Sigmoid activation. Simultaneously output a wildfire probability chart. ;

[0156] When determining the wildfire probability map for a certain area When the value exceeds the warning threshold (e.g., 0.7), it is determined that there is a potential wildfire hazard, triggering an early warning (by locating the GPS coordinates of the area and the distance to the power transmission line).

[0157] Compared with the prior art, the present invention has the following beneficial effects:

[0158] Addressing complex scenarios such as low light conditions at night, high humidity and fog, and obstruction by mountain vegetation, as well as the three key characteristics of nighttime wildfires—low light, prominent thermal targets, and unique semantic features—this invention employs a three-tiered technical architecture: "wildfire-specific semantic extraction, dynamic attention fusion, and lightweight distillation." This architecture retains the complementary advantages of infrared thermal radiation and visible light texture while achieving accurate wildfire identification and real-time deployment. The semantic extraction module optimizes the SAM application logic of the SAGE framework, integrating a cross-modal attention approach from a network-reused power vision fusion method to achieve precise complementarity of dual-light features. The distillation module balances accuracy and real-time performance. This invention is primarily applied to nighttime inspections of 110kV and above high-voltage transmission lines and can be extended to forest fire prevention, oil pipeline fire monitoring, and other fields, particularly suitable for areas difficult to cover using traditional monitoring methods, such as high-altitude mountains and virgin forests.

[0159] The advantages of this embodiment will be illustrated below with an exemplary experimental verification.

[0160] 1. Experimental Data and Environment

[0161] 1) Data source: Nighttime inspection data of 110kV-500kV high-voltage transmission lines were collected, covering scenarios such as low light (≤5 lux), fog, and vegetation obstruction. A total of 8,000 sets of samples were collected, including open flames, smoldering fires, and smoke. Among them, there were 6,400 sets of training data and 1,600 sets of test data. Visible light images (resolution 1920×1080) and infrared images (resolution 640×512) were collected simultaneously for each set, and the wildfire type, location, and temperature range were labeled.

[0162] 2) Experimental environment: CPU is Intel Core i9-13900K, GPU is NVIDIA RTX 4090 (24GB), deep learning framework is PyTorch 1.18, and inference time statistics are based on single frame image processing time.

[0163] 2. Comparison experiment design and results

[0164] Select the existing mainstream dual optical fusion and forest fire monitoring methods as the comparison objects, including traditional methods (wavelet transform, non-subsample contourlet transform), deep learning methods (U2Fusion, TarDAL), and SAM direct reuse methods (SAM+U2Fusion), and evaluate them from three dimensions of fusion image quality, forest fire recognition performance, and real-time performance.

[0165] 1) Fusion image quality evaluation

[0166] The peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and edge preservation (EP) are used as evaluation indexes, and the results are shown in Table 1:

[0167] Table 1

[0168]

[0169] The results show that the PSNR, SSIM, and EP of the fusion image of the present application are significantly higher than those of the comparison methods, indicating that the fusion image retains more rich thermal target detail and texture information, providing high-quality input for forest fire recognition.

[0170] 2) Forest fire recognition performance evaluation

[0171] The recognition rate (Recall), precision (Precision), F1 score, and false alarm rate (False Alarm Rate, FAR) are used as evaluation indexes, and the results are shown in Table 2:

[0172] Table 2

[0173]

[0174] The results show that the recognition rate and precision of the present application for nighttime forest fires are both more than 90%, and the false alarm rate is less than 6%, which is significantly better than existing methods, especially for the recognition ability of dark fire (200-300℃) and thin smoke, which is significantly improved, meeting the power grid early warning demand.

[0175] 3) Real-time performance evaluation

[0176] The single-frame image inference time of each method is counted, and the results are shown in Table 3:

[0177] Table 3

[0178]

[0179] The results show that the sub-network reasoning time of the application is only 38 ms / frame, meeting the real-time requirement (≤50 ms / frame) of the power transmission inspection unmanned aerial vehicle, while the time consumption of the SAM direct reuse method is more than 200 ms / frame, which cannot meet the engineering deployment requirements.

[0180] 3. Ablation experiment design and results

[0181] To verify the effectiveness of the core innovations (mountain fire exclusive semantic extraction module, dynamic attention fusion network, and light distillation architecture) of the application, an ablation experiment is designed, and the performance of the model is compared by removing each innovation module in turn. The results are shown in Table 4.

[0182] Table 4

[0183]

[0184] The experimental results show that: 1) after removing the mountain fire exclusive semantic extraction module, the recognition rate and precision rate decrease by about 11%, indicating that the targeted semantic extraction can effectively filter non-mountain fire interference and strengthen the mountain fire features; 2) after removing the dynamic attention fusion network, the F1 score decreases by 0.094, indicating that the dynamic weight adjustment guided by semantics can achieve accurate complementation of dual-light features and improve the recognition performance; 3) after removing the light distillation architecture, the reasoning time increases to 205 ms / frame, which cannot meet the real-time requirement, indicating that the distillation module can ensure the accuracy while realizing the lightness of the model.

[0185] The application further provides a dual-light fusion system for monitoring mountain fires at night based on SAM semantic enhancement, which comprises a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to perform the steps of the dual-light fusion method for monitoring mountain fires at night based on SAM semantic enhancement.

[0186] The application further provides a computer-readable storage medium having a computer program / instruction stored therein, and the computer program / instruction is programmed or configured to perform the steps of the dual-light fusion method for monitoring mountain fires at night based on SAM semantic enhancement by a processor.

[0187] The system and medium of the application correspond to the above-mentioned method and also have the advantages of the above-mentioned method.

[0188] The present application realizes all or part of the processes in the above-mentioned embodiment methods, and can also be completed by a computer program instructing related hardware. The computer program can be stored in a computer readable storage medium, and the computer program can realize the steps of the above-mentioned method embodiments when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The computer readable medium includes any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. The memory is used to store the computer program and / or modules, and the processor realizes various functions by running or executing the computer program and / or modules stored in the memory, and calling the data stored in the memory. The memory can include high-speed random access memory, and can also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device, etc.

[0189] The above-mentioned is only the preferred embodiment of the present application, and the protection scope of the present application is not limited to the above-mentioned embodiment. Any technical solution falling within the idea of the present application belongs to the protection scope of the present application. It should be noted that some improvements and refinements made by ordinary skilled in the art without departing from the principle of the present application should also be considered as the protection scope of the present application.

Claims

1. A method for monitoring forest fires based on SAM semantic enhancement and dual light fusion of night power transmission, characterized by, The method comprises the steps of: S1, collecting visible light images and infrared images around the power transmission line in the dark environment and respectively performing preprocessing; S2, constructing a dark night forest fire exclusive semantic extraction module based on a SAM model and extracting dark night forest fire exclusive semantics from the preprocessed visible light images and infrared images, and obtaining corresponding feature maps through semantic feature enhancement; S3, constructing a dynamic attention fusion network and performing hierarchical feature extraction and dynamic fusion according to the feature maps to obtain final fusion features; S4, constructing a main network for dark night forest fire recognition based on the dark night forest fire exclusive semantic extraction module and the dynamic attention fusion network, training the main network to improve the quality of the final fusion features and the accuracy of forest fire recognition, and obtaining a trained main network for outputting a forest fire probability map; S5, constructing a lightweight forest fire recognition subnetwork with MobileNetV3 as the backbone network, training the subnetwork to minimize the L2 distance of the fusion features between the subnetwork and the main network and the KL divergence of the forest fire probability map, and obtaining a trained subnetwork, wherein the L2 distance of the fusion features output by the subnetwork and the final fusion features of the main network is constrained by a feature distillation function, and the KL divergence of the forest fire probability map output by the subnetwork and the forest fire probability map of the main network is constrained by a forest fire probability distillation function, and the output of the subnetwork is the fusion features and the forest fire probability map; S6, outputting a forest fire probability map for forest fire warning by using the trained subnetwork according to the collected images.

2. The SAM semantic enhancement based dual optical fusion method for monitoring forest fire during night power transmission according to claim 1, characterized in that, In step S1, the specific steps of preprocessing are as follows: The visible light image is enhanced by using adaptive Gaussian filtering and multi-scale Retinex technology, the infrared image is denoised by using median filtering and screened by a temperature threshold for a potential forest fire heat area, and preprocessed visible light images and infrared images are generated respectively The expression of visible light image enhancement is as follows: ； In the above equation, k is the total number of scales, is a Gaussian blurred image including a plurality of scales, is a weight for each scale, is a very small positive number, is a visible light image after pre-processing, is a visible light image before processing.

3. The SAM semantic enhancement based dual optical fusion method for monitoring forest fire during night power transmission according to claim 1, characterized in that, Step S2 includes: SAM semantic patch generation: marking the power transmission line corridor area as a point prompt word in the visible light image and the infrared image, marking the open fire, smoke, and high-temperature vegetation in the power transmission line corridor as text prompt words, and generating initial infrared and visible light semantic patches with the same resolution as the input visible light image and infrared image based on the point prompt word and the text prompt word; Semantic screening: build a semantic mask of forest fire, filter non-forest fire semantics; wherein for the initial infrared semantic patch , the region where the hot target in the SAM category probability is greater than the first preset value is retained, and an infrared semantic patch is generated ; for the initial visible light semantic patch , the region where the smoke or vegetation anomaly in the SAM category probability is greater than the second preset value is retained, and a visible light semantic patch is generated ; Semantic feature enhancement: multiply the infrared semantic patch with the preprocessed infrared image pixel by pixel to highlight the high temperature area, and then generate a thermal target feature map through convolution dimension reduction ; visible light semantic patch Edge detection is used to strengthen the smoke edge, and then a texture feature map is generated through convolution dimension reduction .

4. The SAM semantic enhancement based dual optical fusion method for monitoring forest fire during night power transmission according to claim 1, characterized in that, Step S3 includes: Multi-scale visual feature extraction: a multi-level fire-adaptive convolution block is constructed to extract multi-level features of infrared and visible light respectively, obtaining infrared visual features and visible light visual features ; The semantic guidance dynamic attention mechanism: for each level of visual feature, the fusion semantic feature map obtained by performing cross-modal semantic fusion on the feature maps of the visible light image and the infrared image in step S2 The infrared visual feature and the visible light visual feature are projected into a query matrix Q through a full connection layer, and the scores of the infrared visual feature and the visible light visual feature after the convolution mapping are weighted and adjusted based on a forest fire confidence parameter to obtain corresponding attention scores. The value matrices of the infrared visual feature and the visible light visual feature are weighted and fused with the attention scores as weights, high-level and low-level features are spliced through a jump connection, and finally fused features are obtained.

5. The SAM semantic enhancement based dual optical fusion method for monitoring forest fire during night power transmission according to claim 4, characterized in that, Based on the forest fire confidence parameter, the scores of the infrared and visible light visual features after convolution mapping are adjusted to obtain corresponding attention scores, and the expression is as follows: ； In the above formula, d is a scaling factor, is a score of an infrared visual feature, is a score of a visible light visual feature, is an attention score of an infrared visual feature, is an attention score of a visible light visual feature, is a forest fire confidence, and Q is a query matrix, is a key matrix of an infrared visual feature, is a key matrix of a visible light visual feature; The value matrix of the infrared and visible light visual features is weighted and fused with the attention scores as weights, and the expression is as follows: ； In the above formula, is the value matrix of the infrared visual feature, is the value matrix of the infrared visual feature, is the value matrix of the visible light visual feature.

6. The SAM semantic enhancement based dual optical fusion method for monitoring forest fire during night power transmission according to claim 1, characterized in that, In step S4, the loss function expression used when training the main network is as follows: ； In the above formula, is a main network total loss function, is a feature alignment loss, is a context loss, is a forest fire recognition loss, is an adjustable coefficient; characteristic alignment loss The expression of the characteristic alignment loss is as follows: ； In the above formula, is a final fusion feature, is an infrared vision feature, is a visible light vision feature; Contextual loss The expression of the contextual loss is as follows: ； In the above formula, is an image target edge information loss, is an image pixel loss, is a visible light image, is an infrared image; Wildfire recognition loss The expression of the wildfire recognition loss is as follows: ； In the above formula, is a label of a pixel point (i, j) position, is a forest fire probability map of a pixel point (i, j) position.

7. The SAM semantic enhancement based dual optical fusion method for monitoring forest fire during night power transmission according to claim 1, characterized in that, In step S5, the loss function expression used when training the subnetwork is as follows: ； In the above formula, is a total loss function of the sub-network, is a feature distillation function, is a forest fire probability distillation function, is an adjustable coefficient; The feature distillation function Fused features for constraining subnetwork outputs Final fused features with the main network L2 distance, expressed as follows: ； The bushfire probability distillation function is used to constrain the subnetwork bushfire probability map with the main network bushfire probability map with the KL divergence, expressed as follows: 。 8. The SAM semantic enhancement based dual optical fusion method for monitoring forest fire during night power transmission according to claim 1, characterized in that, Step S6 includes: The fusion features output by the sub-networks The normalized fusion image is obtained through Sigmoid activation by restoring to the original resolution through upsampling The forest fire probability map is output simultaneously ; When it is judged that the fire probability map of a certain area is higher than the early warning threshold When it is judged that the fire probability map of a certain area is higher than the early warning threshold 9. A dual light fusion system based on SAM semantic enhancement for monitoring black night power transmission forest fire, comprising a microprocessor and a memory connected to each other, characterized in that, The microprocessor is programmed or configured to perform the steps of the SAM semantic enhancement based dark night power transmission forest fire monitoring dual light fusion method of any one of claims 1-8.

10. A computer-readable storage medium having stored therein computer programs / instructions, characterized in that, The computer program / instructions are programmed or configured to perform the steps of the SAM semantic enhancement based dark night power transmission forest fire monitoring dual light fusion method of any one of claims 1-8 by the processor.

Images