A fire point detection method and system based on cross-modal feature grouping

Abstract

The present application relates to the technical field of remote sensing image target detection, and provides a fire point detection method and system based on cross-modal feature grouping, which comprises collecting and registering simultaneous-phase visible light images and thermal infrared images to generate a registered image pair; based on the registered image pair, cross-modal feature fusion is performed through an adaptive feature grouping network to generate a feature grouping image containing information of two modalities; the feature grouping image is input into a Fire-Yolo detection network to obtain fusion features for fire point detection; and a fire point detection result is output through a target detection head of the Fire-Yolo detection network. Compared with existing fire point detection methods based on single modality or fixed weighted fusion, the present application can adaptively adjust the weights of two modalities according to the scene, enhance weak fire point response, and suppress cross-modal noise, and through the collaborative learning of a texture feature branch, a brightness feature branch and a cross-modal feature fusion branch, the detection accuracy in complex backgrounds such as smoke interference, strong reflection and ground thermal spot can be improved, and the false alarm rate and the missing detection rate can be reduced.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image target detection technology, and in particular to a fire detection method and system based on cross-modal feature grouping. Background Technology

[0002] Fire monitoring is one of the core tasks in environmental safety and disaster emergency management. Traditional fire detection mainly relies on single-modal data, such as visible light or infrared images. Visible light images have high resolution and rich texture details, and can capture information such as smoke and flame patterns, but they are prone to omissions or false detections at night, in shadows, or under strong light reflection conditions. Infrared images can reflect thermal radiation characteristics and are suitable for low-light or nighttime scenes, but they are prone to introducing false alarms in high-temperature backgrounds or under reflective conditions.

[0003] Single-modal fire detection algorithms typically extract features from the target region based on convolutional neural networks. However, due to the significant spectral differences between flames, smoke, and their backgrounds in different scenes, the discriminative features of a single modality are often insufficient. For example, fragmented clouds in visible light can be confused with the smoke presented by fire, while noise and other thermal reflections in infrared images are highly similar to the bright spots presented by fire, leading to a decrease in detection accuracy. Therefore, the existing technologies mainly have the following problems: (1) Visible light imaging is significantly affected by changes in illumination, shadows and reflections, while infrared imaging is constrained by low resolution and background thermal interference. It is difficult for a single mode to stably identify fire points in complex environments; (2) Clouds, fog, strongly reflective objects or hot spots on the ground are similar to fire points in different modes, which makes it easy for traditional detection algorithms to misjudge or miss detections; (3) Some studies have attempted to fuse visible light and infrared images, but most of them use simple pixel superposition or fixed weighting methods, lacking a mechanism to dynamically adjust the modal weights and feature layer semantic alignment according to the scene. They cannot make full use of the complementary information of the two modes, resulting in limited fusion effect and low detection accuracy. Therefore, how to make full use of the complementary characteristics of visible light and thermal infrared to construct a multimodal feature fusion framework to distinguish between fire points and false alarms with similar features has become the focus and difficulty of current research. Summary of the Invention

[0004] The purpose of this invention is to address the aforementioned technical problems. Therefore, this invention proposes a fire detection method and system based on cross-modal feature grouping. By grouping the smoke features of fire in visible light and the bright spot features in thermal infrared light into a joint feature, and designing a corresponding target detection network, it achieves the suppression of false alarm targets and the accurate identification of fire targets. The objective of this invention can be achieved through the following technical solutions:

[0005] This invention provides a fire detection method based on cross-modal feature grouping, comprising:

[0006] Step S1: Acquire and register the visible light image and thermal infrared image of the same phase to generate a registered image pair;

[0007] Step S2: Based on the registered image pairs, cross-modal feature fusion is performed through an adaptive feature grouping network to generate a feature grouping image containing information from two modalities, so as to enhance the joint representation of fire point features. The two modalities include texture information in the visible light image and thermal radiation information in the thermal infrared image.

[0008] Step S3: Input the feature grouped image into the Fire-Yolo detection network. The visible light texture features and infrared brightness features are extracted by the texture feature branch and the brightness feature branch, respectively. The feature fusion branch is used for alignment and collaborative enhancement to obtain fused features for fire detection.

[0009] Step S4: Perform fire point category prediction and bounding box regression on the fused features using the target detection head of the Fire-Yolo detection network to generate and output the fire point detection results.

[0010] Further, step S2 includes:

[0011] The registered visible light image and the registered thermal infrared image are extracted from the acquired registered image pairs respectively.

[0012] Adaptive feature parameters for grouping are calculated based on thermal radiation information from thermal infrared images. These adaptive feature parameters characterize the relative weights of two modal information in cross-modal feature fusion.

[0013] The visible light image and the thermal infrared image are weighted and fused according to the grouping adaptive feature parameters to generate a feature-grouped image. The formula is as follows.

[0014] ;

[0015] in, and These are the registered visible light image and the geometrically registered thermal infrared image, respectively. The image is grouped into groups based on the fused features; For adaptive feature parameters;

[0016] Multi-scale smoothing and gradient consistency constraints are applied to the feature grouped image to preserve the fire point edge structure while suppressing cross-modal fusion noise, thereby enhancing the joint representation of fire point features.

[0017] Furthermore, the adaptive feature parameters of the grouping are calculated based on the thermal radiation information of the thermal infrared image, using the following formula: ;in, For adaptive feature parameters, For the sigmoid function, and To adjust the parameters, It is an infrared thermal significance index.

[0018] Furthermore, the infrared thermal significance index is calculated using the following formula:

[0019] ;

[0020] in, Average brightness The standard deviation of brightness, To prevent tiny constants with a denominator of zero.

[0021] Further, in step S3, visible light texture features and infrared brightness features are extracted from the texture feature branch and the brightness feature branch, respectively, including:

[0022] The first 3×3 convolution is performed on the feature grouping image through the texture feature branch to obtain the first texture feature map. Then, based on the first texture feature map, the second 3×3 convolution, channel attention operation and spatial attention operation are performed in sequence to generate texture attention weights. At the fusion node, the texture attention weights are used to weight the first texture feature map to obtain visible light texture features including smoke, flame shape and edge information.

[0023] The first 3×3 convolution is performed on the feature grouping image by the brightness feature branch to obtain the first brightness feature map. Then, a second 3×3 convolution and average pooling-Sigmoid operation are performed on the first brightness feature map to generate brightness weights. At the fusion node, the first brightness feature map is weighted using the brightness weights to obtain the infrared brightness features that characterize the high temperature region and bright spot structure.

[0024] Further, in step S3, alignment and collaborative enhancement are performed through the feature fusion branch to obtain fused features for fire detection, including:

[0025] Visible light texture features and infrared brightness features are projected onto the feature space used to construct query vectors, key vectors, and value vectors through linear mapping;

[0026] The visible light texture features are used as the query vector, and the infrared brightness features are used as the key vector and value vector to calculate the cross-modal attention weights from texture to brightness. The cross-modal attention weights are then used to weight the infrared brightness features to generate brightness-guided texture enhancement features.

[0027] Infrared brightness features are used as query vectors, and visible light texture features are used as key and value vectors to calculate cross-modal attention weights from brightness to texture direction. The cross-modal attention weights are then used to weight the visible light texture features to generate texture-guided brightness enhancement features.

[0028] By fusing brightness-guided texture enhancement features with texture-guided brightness enhancement features, a fused feature for fire detection is obtained.

[0029] Furthermore, the Fire-Yolo detection network is trained using joint supervised loss during the training phase. The training steps include:

[0030] Fire point category prediction and bounding box regression are performed on the fused features by classifying subheads and regressing subheads respectively to obtain preliminary prediction results of fire point detection; visible light branch loss, infrared branch loss and fusion branch loss are calculated. The visible light branch loss and infrared branch loss are used to constrain the detection performance of the texture feature branch and the brightness feature branch respectively, and the fusion branch loss is used to evaluate the detection effect after cross-modal feature fusion.

[0031] The visible light branching loss, infrared branching loss, and fusion branching loss are weighted based on preset weighting coefficients to form a joint supervision loss;

[0032] The parameters of the Fire-Yolo detection network are updated based on the joint supervised loss; the final fire detection result is output based on the updated Fire-Yolo detection network.

[0033] Based on the same inventive concept, this invention provides a fire detection system based on cross-modal feature grouping, employing the fire detection method described above, including:

[0034] The acquisition module is used to acquire and register visible light images and thermal infrared images of the same phase to generate registered image pairs;

[0035] The feature grouping module is used to perform cross-modal feature fusion based on registered image pairs through an adaptive feature grouping network to generate a feature grouping image containing information from two modalities, thereby enhancing the joint representation of fire point features. The two modalities include texture information in the visible light image and thermal radiation information in the thermal infrared image.

[0036] The multi-branch feature extraction module is used to input the feature grouped image into the Fire-Yolo detection network. The texture feature branch and the brightness feature branch extract visible light texture features and infrared brightness features respectively. The feature fusion branch performs alignment and collaborative enhancement to obtain fused features for fire detection.

[0037] The detection output module is used to perform fire point category prediction and bounding box regression on the fused features through the target detection head of the Fire-Yolo detection network, and generate and output the fire point detection results.

[0038] Furthermore, the feature grouping module includes:

[0039] The modality extraction unit is used to extract the registered visible light image and the registered thermal infrared image from the acquired registered image pairs, respectively.

[0040] The adaptive weight calculation unit is used to calculate the grouped adaptive feature parameters based on the thermal radiation information of the thermal infrared image. The adaptive feature parameters characterize the relative weights of the two modal information in cross-modal feature fusion.

[0041] The cross-modal weighted fusion unit is used to weight and fuse the visible light image and the thermal infrared image according to the grouping adaptive feature parameters to generate a feature-grouped image, as shown in the formula.

[0042] ;

[0043] in, and These are the registered visible light image and the geometrically registered thermal infrared image, respectively. The image is grouped into groups based on the fused features; These are adaptive feature parameters.

[0044] Furthermore, it also includes a training module, used to perform fire point category prediction and bounding box regression on the fused features using a classification subhead and a regression subhead respectively, to obtain preliminary prediction results for fire point detection; calculate visible light branch loss, infrared branch loss, and fusion branch loss, whereby the visible light branch loss and infrared branch loss are used to constrain the detection performance of the texture feature branch and the brightness feature branch respectively, and the fusion branch loss is used to evaluate the detection effect after cross-modal feature fusion; weight the visible light branch loss, infrared branch loss, and fusion branch loss based on preset weight coefficients to form a joint supervision loss; update the parameters of the Fire-Yolo detection network according to the joint supervision loss, and output the final fire point detection result based on the updated Fire-Yolo detection network.

[0045] Compared with the prior art, the present invention has at least one of the following technical advantages:

[0046] This invention utilizes an adaptive feature grouping and cross-modal collaborative detection method to dynamically fuse texture and morphological features from visible light images with thermal radiation features from thermal infrared images, achieving joint representation of fire point features. Compared to existing fire point detection methods based on single-modal or fixed-weighted fusion, this invention can adaptively adjust the weights of the two modalities according to the scene, significantly enhancing the response to weak fire points and suppressing cross-modal noise. Simultaneously, by combining the collaborative learning of the texture feature branch, brightness feature branch, and cross-modal feature fusion branch of the Fire-Yolo detection network, it improves the detection accuracy under complex backgrounds such as smoke interference, strong reflection, and surface hot spots, while reducing false alarm and false negative rates. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below:

[0048] Figure 1 This is a flowchart of the fire detection method based on cross-modal feature grouping of the present invention;

[0049] Figure 2 This is a schematic diagram illustrating the working principle of the fire detection method in this embodiment of the invention.

[0050] Figure 3 This is a structural diagram of the texture feature branch and the brightness feature branch in an embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram of the feature fusion branch in an embodiment of the present invention;

[0052] Figure 5 This is a diagram showing the overall structure of the Fire-Yolo network in an embodiment of the present invention;

[0053] Figure 6 This is a schematic diagram of feature grouping and fire detection results in an embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0055] First Embodiment

[0056] In existing fire monitoring technologies, visible light and thermal infrared images are often used separately for fire detection: visible light images can provide rich information on texture, smoke morphology, and color variations, but their imaging quality is highly dependent on lighting conditions and is prone to misjudgment at night, in shadowed, or strongly reflective environments; thermal infrared images can reflect the thermal radiation characteristics of fire points, but are limited by spatial resolution and background thermal interference, often producing bright spots and false fires at high-temperature objects, building exterior reflections, and cloud heat spots. In real-world environments, the scale of fire points, the degree of smoke obstruction, and the complex variations in background spectral characteristics make it difficult for detection methods relying on a single mode to achieve a balance between stability and accuracy.

[0057] Existing technologies attempt to fuse visible light and infrared images, but they mainly rely on fixed weighting or simple pixel overlay methods, failing to dynamically adjust the importance of the two modalities according to scene changes. This results in false alarms and missed detections even when the fire point is weak, there is a lot of smoke, or the background is complex. In addition, existing deep learning detection networks mostly focus on single-modal feature extraction and do not fully utilize cross-modal structural complementarity, making it impossible for the smoke structure in visible light and the thermal brightness features in infrared to form a co-representation within the network.

[0058] After fully analyzing the above-mentioned technical defects, the inventors realized that the key to fire detection is not simply to fuse two images, but to how to make the information of visible light and infrared modes adaptively cooperate in the same feature space, so that texture information and thermal radiation information can complement and constrain each other, thereby forming a joint feature expression that is most discriminative for fire points.

[0059] Based on this idea, the inventors proposed a fire detection method based on cross-modal feature grouping. The core of the invention lies in: firstly, using an adaptive feature grouping network to achieve dynamic weighted fusion of visible light and infrared images at the pixel level, generating a joint representation that balances texture and thermal radiation features; then, utilizing the texture feature branch, brightness feature branch, and cross-modal feature fusion module in the Fire-Yolo detection network to achieve deep coupling and synergistic enhancement of the two types of features; finally, under a joint supervision mechanism, accurate and stable fire detection results are obtained. This not only solves the problems of insufficient modal fusion and susceptibility to interference in single-modal detection in traditional methods, but also maintains high detection accuracy under conditions of weak fire points, smoke obscuration, and strong background noise, effectively reducing false alarms caused by cloud reflection, bright spots, and thermal interference in single-modal detection. The specific implementation method is as follows:

[0060] like Figure 1 , 2 As shown, this invention provides a fire detection method based on cross-modal feature grouping, comprising:

[0061] Step S1: Acquire visible light and thermal infrared images simultaneously, register the visible light and thermal infrared images, establish pixel-level spatial correspondence, and generate registered image pairs;

[0062] Step S2: Based on the registered image pairs, cross-modal feature fusion is performed through an adaptive feature grouping network to generate a feature grouping image containing information from two modalities, so as to enhance the joint representation of fire point features. The two modalities include texture information in the visible light image and thermal radiation information in the thermal infrared image.

[0063] Specifically, in this embodiment, a pixel-level correspondence between the two modal images is established using a feature-point registration algorithm to achieve accurate feature matching. A feature-point matching algorithm is employed, and the affine transformation matrix is ​​calculated. Perform geometric transformations on the infrared image to make and Strict spatial alignment. Output is the registered image pair. This lays the foundation for subsequent feature grouping.

[0064] Step S3: Input the feature grouped image into the Fire-Yolo detection network. The visible light texture features and infrared brightness features are extracted by the texture feature branch and the brightness feature branch, respectively. The feature fusion branch is used for alignment and collaborative enhancement to obtain fused features for fire detection.

[0065] Step S4: Perform fire point category prediction and bounding box regression on the fused features using the target detection head of the Fire-Yolo detection network to generate and output the fire point detection results.

[0066] Specifically, this method combines visible light smoke features, visible light texture features, and infrared bright spot features to effectively distinguish false alarm sources in their respective modes, such as cloud and fog interference, sunlight reflection, high-temperature buildings or cloud reflection, thereby significantly improving detection accuracy and reducing false alarm rate.

[0067] The Fire-Yolo network includes: a texture feature branch for extracting smoke morphology, edge and color variations from the joint representation of fire; a brightness feature branch for enhancing the high-temperature region and its spatiotemporal distribution features in the joint representation of fire; a feature fusion branch for cross-modal feature alignment and collaborative learning; and a detection head module for outputting the fire location and confidence level. During the model training phase, the Fire-Yolo network simultaneously calculates the visible light branch loss function, the infrared branch loss function, and the fusion detection loss function, and employs a joint supervision mechanism to optimize the overall network performance.

[0068] Further, step S2 includes:

[0069] The registered visible light image and the registered thermal infrared image are extracted from the acquired registered image pairs respectively.

[0070] Adaptive feature parameters for grouping are calculated based on thermal radiation information from thermal infrared images. These adaptive feature parameters characterize the relative weights of two modal information in cross-modal feature fusion.

[0071] The visible light image and the thermal infrared image are weighted and fused according to the grouping adaptive feature parameters to generate a feature-grouped image. The formula is as follows.

[0072] ;

[0073] in, and These are the registered visible light image and the geometrically registered thermal infrared image, respectively. The image is grouped into groups based on the fused features; For grouping adaptive feature parameters;

[0074] Specifically, grouping adaptive parameters Based on the adaptive determination of the thermal feature intensity in the infrared image, when the fire point intensity in the infrared image is low or the thermal contrast is insufficient, it indicates that the infrared signal contributes weakly to the fire point. The system automatically increases the fusion weight of the infrared channel, thereby increasing the proportion of thermal information in the grouped image and strengthening the response characteristics of weak fire points. When the infrared signal is significant and the thermal target is prominent, the infrared weight is appropriately reduced to maintain the structural and texture details of the visible light image and achieve dynamic balance fusion of the two modal features.

[0075] Multi-scale smoothing and gradient consistency constraints are applied to the feature grouped image to preserve the fire point edge structure while suppressing cross-modal fusion noise, thereby enhancing the joint representation of fire point features.

[0076] Furthermore, the adaptive feature parameters of the grouping are calculated based on the thermal radiation information of the thermal infrared image, using the following formula: ;in, For grouping adaptive feature parameters, For the sigmoid function, and To adjust the parameters, It is an infrared thermal significance index.

[0077] Specifically, when the fire signal is weak or the contrast of the thermal target is insufficient in the infrared image, When the value is small, the system automatically increases the fusion weight of the infrared channel, thereby enhancing the response in the fire point area; when the infrared signal is significant and the fire point target is prominent, If the value is large, the system reduces the infrared weight to preserve visible light texture and structural details. This process can be achieved through standardized calculations, making... It exhibits good stability and transferability under different imaging conditions. Through this adaptive weighting mechanism, dynamic balance fusion of infrared and visible light features can be achieved in scenarios such as low fire intensity and complex lighting, effectively improving the detectability and robustness of fire point features.

[0078] Furthermore, the infrared thermal significance index is calculated using the following formula:

[0079] ;

[0080] in, Average brightness The standard deviation of brightness, To prevent the use of tiny constants with a denominator of zero, an adaptive function is used to determine the fusion weights based on the thermal significance index.

[0081] Further, in step S3, visible light texture features and infrared brightness features are extracted from the texture feature branch and the brightness feature branch, respectively, including:

[0082] The first 3×3 convolution is performed on the feature grouping image through the texture feature branch to obtain the first texture feature map. Then, based on the first texture feature map, the second 3×3 convolution, channel attention operation and spatial attention operation are performed in sequence to generate texture attention weights. At the fusion node, the texture attention weights are used to weight the first texture feature map to obtain visible light texture features including smoke, flame shape and edge information.

[0083] The first 3×3 convolution is performed on the feature grouping image by the brightness feature branch to obtain the first brightness feature map. Then, a second 3×3 convolution and average pooling-Sigmoid operation are performed on the first brightness feature map to generate brightness weights. At the fusion node, the first brightness feature map is weighted using the brightness weights to obtain the infrared brightness features that characterize the high temperature region and bright spot structure.

[0084] Specifically, such as Figure 3 As shown, the texture feature branch extracts smoke morphology features through convolutional layers. The main structure of this branch consists of convolutional layers, a channel attention module, and a spatial attention module, used to enhance the response to fine-grained fire point regions at the feature level and suppress background interference. The channel attention module highlights high-frequency flame textures and smoke edge information through global weighting of feature channels; the spatial attention module utilizes local saliency mapping to improve the localization accuracy of fire point regions.

[0085] The brightness feature branch enhances the feature response of high-temperature regions by extracting temperature gradient and bright spot structure information. This branch introduces a brightness attention mechanism in the feature mapping stage, applying weighted boosts to bright spot regions in the infrared channel to enhance the perception of the fire center and its diffusion area. During computation, a brightness response map is first obtained through average pooling; then, it is converted into normalized weight coefficients using the sigmoid mapping function; finally, this weight map is multiplied point-by-point with the feature map to enhance the features of the bright areas. The brightness branch and the texture branch maintain a hierarchical correspondence in structure to achieve precise alignment in the feature dimensions during the subsequent fusion stage.

[0086] Further, in step S3, alignment and collaborative enhancement are performed through the feature fusion branch to obtain fused features for fire detection, including:

[0087] Visible light texture features and infrared brightness features are projected onto the feature space used to construct query vectors, key vectors, and value vectors through linear mapping;

[0088] The visible light texture features are used as the query vector, and the infrared brightness features are used as the key vector and value vector to calculate the cross-modal attention weights from texture to brightness. The cross-modal attention weights are then used to weight the infrared brightness features to generate brightness-guided texture enhancement features.

[0089] Infrared brightness features are used as query vectors, and visible light texture features are used as key and value vectors to calculate cross-modal attention weights from brightness to texture direction. The cross-modal attention weights are then used to weight the visible light texture features to generate texture-guided brightness enhancement features.

[0090] By fusing brightness-guided texture enhancement features with texture-guided brightness enhancement features, a fused feature for fire detection is obtained.

[0091] Specifically, such as Figure 4 As shown, the feature fusion branch includes a texture-luminosity cross-attention mechanism for features extracted from visible light and thermal infrared light. This mechanism adjusts the weights of different modal features based on complementary properties, achieving collaborative representation of smoke features and thermal luminosity features. Specifically, let the texture feature map output by the visible light branch be... The brightness characteristics of the thermal infrared branch output are mapped as follows During the fusion phase, the modules first undergo linear transformation or... Convolution maps both to query (Q), key (K), and value (V) spaces respectively, constructing cross-modal attention pairs, including texture-to-luminance orientation. With brightness to texture direction With visible light texture features As a query Based on infrared brightness characteristics As a key AND value Calculate attention weights:

[0092] ;

[0093] in, This represents the weighted matrix of texture feature response to brightness feature, where each element reflects the degree of attention given to infrared brightness feature at each location within the visible light texture feature. This is achieved through weighted summation. This yields a texture enhancement representation guided by heat-induced brightness features, allowing smoke edges and flame textures to be emphasized in areas of abrupt brightness changes.

[0094] Approximately, the brightness to texture direction can be obtained. and its feature map Through this bidirectional cross-attention mechanism, the network establishes a dynamic feature coupling relationship between the texture domain and the brightness domain, realizing intermodal guidance and complementarity, thereby significantly improving the stability and robustness of detection in complex scenarios such as smoke obscuring, uneven brightness, or weak fire points.

[0095] Furthermore, the Fire-Yolo detection network is trained using joint supervised loss during the training phase. The training steps include:

[0096] Fire point category prediction and bounding box regression are performed on the fused features by classifying subheads and regressing subheads respectively to obtain preliminary prediction results of fire point detection; visible light branch loss, infrared branch loss and fusion branch loss are calculated. The visible light branch loss and infrared branch loss are used to constrain the detection performance of the texture feature branch and the brightness feature branch respectively, and the fusion branch loss is used to evaluate the detection effect after cross-modal feature fusion.

[0097] The visible light branching loss, infrared branching loss, and fusion branching loss are weighted based on preset weighting coefficients to form a joint supervision loss;

[0098] The parameters of the Fire-Yolo detection network are updated based on the joint supervised loss; the final fire detection result is output based on the updated Fire-Yolo detection network.

[0099] Specifically, a joint supervision mechanism is designed, utilizing a Fire-Yolo network to detect fire points in feature-grouped images and outputting the fire point detection results. The detection head module adopts a lightweight decoupled structure, consisting of a classification sub-head and a regression sub-head, used to predict the fire point category and location coordinates, respectively. The classification sub-head improves the recognition accuracy of small-area fire points through multi-scale feature stacking and focus loss function optimization; the regression sub-head combines CIoU loss and confidence regression mechanism to achieve high-precision localization of fire point bounding boxes.

[0100] Furthermore, such as Figure 5 As shown, Fire-Yolo simultaneously inputs registered visible light and infrared images, and the network outputs three sets of prediction results: visible light branch prediction, infrared branch prediction, and fused branch prediction. Correspondingly, the following loss terms are calculated:

[0101] Visible light branching loss It consists of classification loss and bounding box regression loss, and is used to constrain the detection accuracy of texture feature branches under visible light;

[0102] Infrared branching loss It also includes classification loss and regression loss, used to enhance the robustness of the infrared branch under conditions of weak fire point thermal features;

[0103] Fusion branch loss The detection results output after cross-modal feature fusion are matched with the real annotations to comprehensively reflect the fusion effect of multimodal information.

[0104] The above three losses are weighted and integrated through a joint monitoring mechanism:

[0105] ;

[0106] in, , , These are adjustable weighting coefficients used to balance the contribution ratio of each branch during the training process.

[0107] In the early stages of training, the weights of the single-modal loss can be increased to stabilize feature extraction; during the network convergence phase, the weights of the fusion branch are gradually increased to enhance the multimodal collaborative learning capability. Through this joint supervision mechanism, the Fire-YOLO network can achieve collaborative optimization of texture and brightness information in the multimodal feature space, enabling the visible light and infrared branches to compensate for each other and converge together, thereby significantly improving the detection accuracy and robustness of weak fire points and smoke targets.

[0108] like Figure 6 As shown, Figure 6 (a) is the input thermal infrared image, which shows that the fire point area exhibits a distinct bright structure in thermal radiation; Figure 6 (b) is a visible light image of the same scene. Due to the obscuring effect of smoke and the reflection from the ground, the location of the fire point is not prominent enough in terms of visual texture. Figure 6 (c) in the figure is a feature grouping image generated based on this method, which can simultaneously preserve visible light texture details and thermal infrared brightness response.

[0109] To further demonstrate the differences, Figure 6 (d) in the image shows the fire point details in the thermal infrared image, where the brightness is concentrated but the edges are blurred; Figure 6 (e) in the image shows the fire details in the visible light image, where the smoke texture is rich but the fire itself is not obvious; Figure 6 (f) shows the fire point details and detection results after feature grouping. It can be seen that the fire point area is enhanced in terms of texture and brightness, the fire point boundary is clearer, and the detection box accurately covers the fire point position, proving that the cross-modal feature grouping of this scheme can effectively improve the visibility and detectability of fire points.

[0110] Second Embodiment

[0111] Based on the same inventive concept, this invention provides a fire detection system based on cross-modal feature grouping, employing the fire detection method described above, including:

[0112] The acquisition module is used to acquire and register visible light images and thermal infrared images of the same phase to generate registered image pairs;

[0113] The feature grouping module is used to perform cross-modal feature fusion based on registered image pairs through an adaptive feature grouping network to generate a feature grouping image containing information from two modalities, thereby enhancing the joint representation of fire point features. The two modalities include texture information in the visible light image and thermal radiation information in the thermal infrared image.

[0114] The multi-branch feature extraction module is used to input the feature grouped image into the Fire-Yolo detection network. The texture feature branch and the brightness feature branch extract visible light texture features and infrared brightness features respectively. The feature fusion branch performs alignment and collaborative enhancement to obtain fused features for fire detection.

[0115] The detection output module is used to perform fire point category prediction and bounding box regression on the fused features through the target detection head of the Fire-Yolo detection network, and generate and output the fire point detection results.

[0116] Furthermore, the feature grouping module includes:

[0117] The modality extraction unit is used to extract the registered visible light image and the registered thermal infrared image from the acquired registered image pairs, respectively.

[0118] The adaptive weight calculation unit is used to calculate the grouped adaptive feature parameters based on the thermal radiation information of the thermal infrared image. The adaptive feature parameters characterize the relative weights of the two modal information in cross-modal feature fusion.

[0119] The cross-modal weighted fusion unit is used to weight and fuse the visible light image and the thermal infrared image according to the grouping adaptive feature parameters to generate a feature-grouped image, as shown in the formula.

[0120] ;

[0121] in, and These are the registered visible light image and the geometrically registered thermal infrared image, respectively. The image is grouped into groups based on the fused features; These are the adaptive feature parameters for grouping.

[0122] Furthermore, it also includes a training module, used to perform fire point category prediction and bounding box regression on the fused features using a classification subhead and a regression subhead respectively, to obtain preliminary prediction results for fire point detection; calculate visible light branch loss, infrared branch loss, and fusion branch loss, whereby the visible light branch loss and infrared branch loss are used to constrain the detection performance of the texture feature branch and the brightness feature branch respectively, and the fusion branch loss is used to evaluate the detection effect after cross-modal feature fusion; weight the visible light branch loss, infrared branch loss, and fusion branch loss based on preset weight coefficients to form a joint supervision loss; update the parameters of the Fire-Yolo detection network according to the joint supervision loss, and output the final fire point detection result based on the updated Fire-Yolo detection network.

[0123] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

Claims

1. A fire detection method based on cross-modal feature grouping, characterized in that, include: Step S1: Acquire and register the visible light image and thermal infrared image of the same phase to generate a registered image pair; Step S2: Based on the registered image pair, cross-modal feature fusion is performed through an adaptive feature grouping network to generate a feature grouping image containing two modal information to enhance the joint representation of fire point features. The two modal information include texture information in the visible light image and thermal radiation information in the thermal infrared image. Step S3: The feature grouped image is input into the Fire-Yolo detection network. Visible light texture features, including smoke, flame morphology, and edge information, and infrared brightness features representing high-temperature regions and bright spot structures are extracted by the texture feature branch and brightness feature branch, respectively. Alignment and collaborative enhancement are then performed through the feature fusion branch to obtain fused features for fire detection. Alignment and collaborative enhancement are performed through feature fusion branches to obtain fused features for fire detection, including: The visible light texture features and the infrared brightness features are projected onto the feature space used to construct the query vector, key vector, and value vector by linear mapping, respectively. The visible light texture features are used as the query vector, the infrared brightness features are used as the key vector, and the value vector are used to calculate the cross-modal attention weights from texture to brightness direction. The cross-modal attention weights are then used to weight the infrared brightness features to generate brightness-guided texture enhancement features. The infrared brightness feature is used as the query vector, the visible light texture feature is used as the key vector, and the value vector is used to calculate the cross-modal attention weight from brightness to texture direction. The visible light texture feature is then weighted using the cross-modal attention weight to generate texture-guided brightness enhancement features. The brightness-guided texture enhancement feature is fused with the texture-guided brightness enhancement feature to obtain the fused feature used for fire detection; Step S4: Perform fire point category prediction and bounding box regression on the fused features using the target detection head of the Fire-Yolo detection network to generate and output the fire point detection results.

2. The fire detection method according to claim 1, characterized in that, Step S2 includes: The registered visible light image and the registered thermal infrared image are extracted from the acquired registered image pair, respectively. Based on the thermal radiation information of the thermal infrared image, grouping adaptive feature parameters are calculated, and the adaptive feature parameters characterize the relative weights of the two modal information in the cross-modal feature fusion; The visible light image and the thermal infrared image are weighted and fused according to the adaptive feature parameters to generate the feature-grouped image, as shown in the formula. ； in, and These are the registered visible light image and the geometrically registered thermal infrared image, respectively. The fused feature grouped image; These are the adaptive feature parameters; Multi-scale smoothing and gradient consistency constraints are applied to the feature grouped image to preserve the fire point edge structure while suppressing cross-modal fusion noise, thereby enhancing the joint representation of the fire point features.

3. The fire detection method according to claim 2, characterized in that, The grouping adaptive feature parameters are calculated based on the thermal radiation information from the thermal infrared image, using the following formula: ； Among them, the The adaptive feature parameters are... For the sigmoid function, and To adjust the parameters, It is an infrared thermal significance index.

4. The fire detection method according to claim 3, characterized in that, The infrared thermal saliency index is calculated using the following formula: ； in, Average brightness The standard deviation of brightness, To prevent tiny constants with a denominator of zero.

5. The fire detection method according to claim 1, characterized in that, In step S3, visible light texture features and infrared luminance features are extracted from the texture feature branch and the luminance feature branch, respectively, including: The texture feature branch performs a first 3×3 convolution on the feature grouping image to obtain a first texture feature map. Based on the first texture feature map, a second 3×3 convolution, channel attention operation, and spatial attention operation are performed sequentially to generate texture attention weights. At the fusion node, the texture attention weights are used to weight the first texture feature map to obtain the visible light texture features including the smoke, flame shape, and edge information. The first 3×3 convolution is performed on the feature grouping image through the brightness feature branch to obtain a first brightness feature map. Then, a second 3×3 convolution and average pooling-Sigmoid operation are performed on the first brightness feature map to generate brightness weights. At the fusion node, the first brightness feature map is weighted using the brightness weights to obtain the infrared brightness features characterizing the high-temperature region and bright spot structure.

6. The fire detection method according to claim 1, characterized in that, The Fire-Yolo detection network is trained using joint supervised loss during the training phase. The training steps include: The fire point category prediction and bounding box regression are performed on the fused features by the classification subhead and regression subhead respectively to obtain the preliminary prediction results of the fire point detection; the visible light branch loss, the infrared branch loss and the fusion branch loss are calculated. The visible light branch loss and the infrared branch loss are used to constrain the detection performance of the texture feature branch and the brightness feature branch respectively. The fusion branch loss is used to evaluate the detection effect after the cross-modal feature fusion. The visible light branch loss, the infrared branch loss, and the fusion branch loss are weighted based on preset weighting coefficients to form the joint supervision loss; The parameters of the Fire-Yolo detection network are updated based on the joint supervised loss; the final fire detection result is output based on the updated Fire-Yolo detection network.

7. A fire detection system based on cross-modal feature grouping, employing the fire detection method as described in any one of claims 1 to 6, characterized in that, include: The acquisition module is used to acquire and register visible light images and thermal infrared images of the same phase to generate registered image pairs; The feature grouping module is used to perform cross-modal feature fusion through an adaptive feature grouping network based on the registered image pair to generate a feature grouping image containing two modal information to enhance the joint representation of fire point features. The two modal information include texture information in the visible light image and thermal radiation information in the thermal infrared image. The multi-branch feature extraction module is used to input the feature grouped image into the Fire-Yolo detection network, extract visible light texture features and infrared brightness features by the texture feature branch and the brightness feature branch respectively, and perform alignment and collaborative enhancement through the feature fusion branch to obtain fused features for fire detection; The detection output module is used to perform fire point category prediction and bounding box regression on the fused features through the target detection head of the Fire-Yolo detection network, and generate and output the fire point detection results.

8. The fire detection system according to claim 7, characterized in that, The feature grouping module includes: A modal extraction unit is used to extract the registered visible light image and the registered thermal infrared image from the acquired registered image pair, respectively. An adaptive weight calculation unit is used to calculate grouping adaptive feature parameters based on the thermal radiation information of the thermal infrared image, wherein the adaptive feature parameters characterize the relative weights of the two modal information in the cross-modal feature fusion; A cross-modal weighted fusion unit is used to weight and fuse the visible light image and the thermal infrared image according to the adaptive feature parameters to generate the feature-grouped image, using the following formula: ； in, and These are the registered visible light image and the geometrically registered thermal infrared image, respectively. The fused feature grouped image; The adaptive feature parameters are defined as follows.

9. The fire detection system according to claim 8, characterized in that, It also includes a training module, used to perform fire point category prediction and bounding box regression on the fused features using a classification subhead and a regression subhead respectively, to obtain preliminary prediction results for fire point detection; calculate visible light branch loss, infrared branch loss, and fusion branch loss, wherein the visible light branch loss and the infrared branch loss are used to constrain the detection performance of the texture feature branch and the brightness feature branch respectively, and the fusion branch loss is used to evaluate the detection effect after cross-modal feature fusion; weight the visible light branch loss, infrared branch loss, and fusion branch loss based on preset weight coefficients to form a joint supervision loss; update the parameters of the Fire-Yolo detection network according to the joint supervision loss, and output the final fire point detection result based on the updated Fire-Yolo detection network.

Images