Fire-fighting high hotspot detection method based on AR glasses bimodal cross attention fusion

Abstract

The invention discloses a fire-fighting high hotspot detection method based on AR glasses bimodal cross attention fusion. The method comprises the following steps: constructing a bimodal (RGB and IR) space attention matching module; constructing a bimodal high-hotspot feature fusion module based on cross attention; building a bimodal fusion network based on cross attention and a corresponding high-hotspot target detection model; and finally, using the trained high-hotspot detection model to detect high-hotspot articles on fire-fighting AR glasses. According to the method, the cross attention mechanism is effectively utilized to fuse the advantages of infrared and visible light mode complementary information, so that the capability of sensing high hotspots in a complex fire scene environment is enhanced, and the problem of multi-source image alignment in actual hardware deployment is solved through a network. According to the method, the firefighter can be prevented from switching between the fire-fighting lance and the thermal infrared imager, the re-ignition point can be quickly found and positioned only through observation of the AR glasses, efficient cooperation of identification and removal is achieved, and the operation safety and efficiency are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of multimodal image processing technology, and in particular to a method for detecting high-hot spots in fires based on dual-modal cross-attention fusion of AR glasses. Background Technology

[0002] In modern fire emergency rescue missions, the rapid identification and accurate location of reignition points are crucial for ensuring the safety of rescue personnel and preventing the resurgence of the disaster. Traditional reignition point detection relies on firefighters manually scanning with handheld infrared thermal imagers and then switching to fire hoses for extinguishing. This method requires repeated switching between detection equipment and firefighting tools, significantly increasing operational risks and response time in high-temperature, dense smoke fire environments. In recent years, although augmented reality (AR) technology has been gradually applied in the firefighting field, its supporting intelligent recognition software still has significant shortcomings. Especially at the hardware integration level, the spatial perspective difference between dual-modal (visible light / RGB and infrared / IR) cameras makes it difficult to accurately align cross-modal images. Traditional registration algorithms rely on manual calibration and have weak anti-interference capabilities. At the algorithm level, existing solutions mostly use single visible light modality recognition models, lacking effective fusion of infrared thermal image features, resulting in a significant drop in recognition accuracy in complex fire environments such as dense smoke obstruction and high-temperature heat wave interference. These shortcomings make current firefighting AR equipment unable to meet the stringent requirements for real-time detection and spatial location of reignition points in actual combat.

[0003] Therefore, there is an urgent need to construct a dual-modal intelligent recognition architecture deeply integrated into AR hardware platforms. This architecture should address the spatial calibration challenges of multi-source imaging devices through an adaptive cross-modal feature fusion mechanism, collaboratively leveraging the complementary advantages of visible light texture information and infrared radiation features to enhance the robustness of heat source target perception. A dual-modal registration and recognition network based on cross-attention can achieve end-to-end pixel-level spatial transformation and semantic-level feature interaction. While eliminating hardware perspective bias, it significantly improves the detection accuracy and real-time performance of reignition points in complex fire scenes, providing firefighters with an integrated "detection-location-response" capability without the need to switch equipment. Summary of the Invention

[0004] The purpose of this invention is to provide a fire-fighting high-hotspot target detection method based on dual-modal cross-attention fusion. This method is based on a dual-modal registration and recognition network with cross-attention, which can realize end-to-end pixel-level spatial transformation and semantic-level feature interaction. While eliminating hardware perspective bias, it significantly improves the detection accuracy and real-time performance of reignition points in complex fire scenes, providing firefighters with an integrated "detection-location-response" combat capability without switching equipment.

[0005] The specific technical solution for achieving the objective of this invention is as follows:

[0006] A fire hotspot detection method based on bimodal cross-attention fusion, comprising the following steps:

[0007] S1. Processing dual-modal video streams;

[0008] S2. Construct a bimodal spatial attention matching module;

[0009] S3. Construct a dual-modal high-hotspot feature fusion module based on cross-attention;

[0010] S4. Based on the bimodal spatial attention matching module and feature fusion module, a bimodal fusion network and a corresponding high-hotspot target detection model are built.

[0011] S5. Train the high hotspot target detection model and use the trained model to perform real-time detection and output of high hotspot targets in the dual-modal images collected by the fire-fighting AR glasses;

[0012] Step S1 specifically includes: aligning the video stream output to ensure that the number of frames for RGB modal data and IR modal data is the same; taking the dual-modal data from the same frame and performing uniform size processing; if the image size is smaller than the target size, upsampling is performed; if the image size is larger than the target size, downsampling is performed. Let the original image size be... The processed image is Upsampling is represented as Downsampling is represented as Upsampling Using linear interpolation, downsampling Max pooling is used;

[0013] Step S2 specifically involves the following: The constructed bimodal spatial attention matching module includes a block-coding convolution module and an asymmetric attention module. Using IR and RGB modal images as input, the block-coding convolution module performs block-coding convolution processing on the bimodal images: converting the RGB modal image into a query matrix (Query, Q) through convolution; converting the IR modal image into a key matrix (Key, K) through convolution; initializing the built-in trainable texture matrix (Pattern, P) to follow a normal distribution N(0, 0.02) with a mean of 0 and a variance of 0.02. Let the RGB modal image be... IR mode diagram is The specific operation is represented by the following formula: , , ,in , , It is a trainable matrix; where the convolution uses overlapping processing, for example, if the convolution kernel size is (2s) x (2s), then the stride is set to s, where s is a positive integer and the size of s is set according to the experiment;

[0014] The block-encoded convolution obtained and The spatial correlation model is performed using the input asymmetric attention module, and the attention weight matrix is ​​calculated. Where T represents the matrix transpose, d represents the number of hidden convolutional layers encoding the convolution, and then the final texture encoding information is obtained by weighting the texture matrix P using the weight matrix Score. ;

[0015] The bimodal spatial attention matching module ultimately outputs texture encoding information aligned between the IR and RGB spaces. It is then applied to the original modality map for positional encoding, and an RGB feature map is output. and IR feature map ;

[0016] In step S3, the specific process of constructing the bimodal high-hotspot feature fusion module based on cross-attention is as follows:

[0017] Feature fusion is performed using the feature map with positional encoding information output by the bimodal spatial attention matching module as input. The input RGB feature map is then... IR feature map is The specific cross-attention fusion operation can then be expressed as the following formula: , , ,in , , It is a trainable matrix. It is by Mapped query matrix It is by The mapping of the key matrix, (Value) is determined by The value matrix of the mapping. The feature fusion result is expressed by the formula: The feature map is further processed for feature extraction, resulting in feature fusion at a higher dimension. The extraction method is expressed by the formula: , , will get , Perform as The same cross-attention fusion operation yields Similarly, , Feature extraction was performed to obtain and Obtained through cross-attention fusion operation ; will obtain characteristics ,feature ,feature After being flattened along the channel dimension, the components are stitched together, as shown in the formula. Finally, a fully connected layer is used to obtain the feature fusion output of the module;

[0018] Step S4 specifically involves: reshaping the feature fusion output obtained in step S3. Let the number of detected categories be i, then the output vector after feature reshaping is [...]. , , , , , ],in As the starting point of the width, As the starting point of the long period, The proportion of width, The proportion of length, , They are the 1st and 2nd respectively. The score for detecting target class i; the prediction confidence is obtained by inputting the score into the Sigmoid function. Let the output score of the model for the selected object being class k be... Then the confidence level The output features, including wide starting points, long starting points, width proportion, and length proportion, are removed based on confidence levels. Low-confidence boxes are removed, and non-maximum suppression is applied to the remaining boxes. The result is the target point.

[0019] Furthermore, the block coding convolution module allows overlapping portions of the encoding between image blocks, thereby enabling the encoding information to flow between image blocks and enhancing the extraction of image block information.

[0020] Furthermore, the query matrix Q and key matrix K in the spatial attention matching module are obtained from RGB and IR images through... , It is mapped from, and the texture matrix P is an independent trainable texture matrix. .

[0021] Furthermore, in step S4, the feature fusion process is a dual-branch structure where a convolutional network is responsible for extracting high-dimensional features, and a cross-attention mechanism is responsible for feature fusion.

[0022] This invention uses a dual-branch network and employs cross-attention fusion at different feature stages to fully capture the correlation between low-dimensional and high-dimensional features of different modalities.

[0023] This invention uses a learnable texture matrix to replace the original hardware modal graph alignment, reducing hardware computational burden and debugging costs.

[0024] This invention employs a lightweight design for each module, ensuring that performance remains essentially unchanged while reducing the number of model parameters and accelerating the real-time inference speed of AR glasses. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the process of the present invention;

[0026] Figure 2 A schematic diagram of the spatial attention matching module provided in an embodiment of the present invention;

[0027] Figure 3 A schematic diagram of the cross-attention fusion module provided in an embodiment of the present invention;

[0028] Figure 4 A schematic diagram of the cross-attention structure provided in an embodiment of the present invention;

[0029] Figure 5 This is a schematic diagram of a bimodal high hotspot detection model based on cross-attention provided in an embodiment of the present invention;

[0030] Figure 6 The diagram shows the laboratory simulation test results provided in this embodiment of the invention. Detailed Implementation

[0031] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0032] Example

[0033] When a fire breaks out, after firefighters have conducted initial extinguishing, they can re-enter the scene wearing fire-fighting AR glasses. Utilizing a fire-fighting high-hotspot detection method based on dual-modal cross-attention fusion of AR glasses, as described in this invention, they can extinguish suspected reignition points with high heat levels. The overall process is as follows: Figure 1 As shown, the specific process is as follows:

[0034] The fire-fighting AR glasses capture video streams using both an RGB camera and an IR camera. The input bimodal image undergoes uniform size processing. Assuming the target size is the same as the original RGB modal image (256x256x3), the output RGB modal image size is also 256x256x3. The original IR modal image (96x96x3) is then upsampled. Linear interpolation was used to obtain a 256x256x3 IR modal diagram. The processed data was then fed into the model. The overall model schematic diagram is shown below. Figure 5 As shown.

[0035] The RGB and IR modal images, after being processed to a uniform size, are fed into the bimodal high-hotspot detection model. They first pass through the spatial attention matching module, such as... Figure 2 As shown, assuming the RGB and IR modal maps are both processed into 256x256x3, corresponding to the length, width, and number of channels respectively, the block coding convolutional module uses the formula... , , Generate the corresponding query vector key vector and texture vectors ,at this time , All are 256x256x3. Let the parameters of the block-coding convolution be: 16 hidden channels, 16x16 kernel size, and 8 stride. Then the generated... It is 32x32x16. The value is 32x32x16, and P is 32x32x16. Afterwards, it passes through an asymmetric attention module, using... Calculate the block matching degree, at this time The flattened size is 1024x16. The flattened size is 1024x16, resulting in a matching score of 1024x1024. Finally, texture information is calculated. ,get The image is 1024x16, which contains the texture information of 1024 blocks. This information is added back to the original image. Finally, the video stream input to the spatial attention matching module is two 256x256x3 images, and the output is also two 256x256x3 images. The difference is that the output images have position-encoded texture information added to them.

[0036] The output of the previous layer is then fed into a dual-modal high-hotspot feature fusion module based on cross-attention, such as... Figure 3 As shown, through the formula , , Perform feature mapping, such as Figure 4 As shown, at this time , Given a 256x256x3 matrix, and assuming the feature mapping layer has 16 hidden channels, a 16x16 kernel size, and a stride of 16, the generated... , , It is 16x16x16, through This yields a feature with a shape of 16x16x16. ;Will , Each convolutional layer is processed separately. Assuming the kernel size is 3x3, the stride is 2, and the padding is 1, then... It can be obtained It is 128x128x3, the same. It is 128x128x3, and then it is subjected to... This yields a feature with a shape of 8x8x16. ; Will , Each convolutional layer is processed separately. Assuming the kernel size is 3x3, the stride is 2, and the padding is 1, then... It can be obtained It is 64x64x3, the same. It is 64x64x3, and then it is subjected to... This yields a feature with a shape of 4x4x16. splice these features This yields features of (16x16+8x8+4x4)x16, or 336x16 dimensions. Finally, after passing through a fully connected layer, the output features correspond to the number of categories plus 4.

[0037] The output features are processed using Sigmoid to obtain a confidence score. Non-maximum suppression (NMS) is then used to process the obtained wide starting point, long starting point, width proportion, and length proportion information. The final information is converted into corresponding bounding boxes, and the confidence score is labeled. The output is then projected onto VR glasses to guide firefighters in cooling down high-heat locations.

[0038] The entire fire scenario was simulated in the laboratory, using a cup of hot water as a hot spot to simulate various obstruction scenarios, and the results were compared with classic algorithms, such as... Figure 6 As shown, the first column is the input IR modality map, the second column is the input RGB modality map, the third column is the result of the classic YOLOv8 detection algorithm, and the fourth column is the detection result of our proposed method. Figure 6 The test in (a) shows that when there is no occlusion, both YOLOv8 and our method can identify the results, but our method has a higher confidence level, indicating that the IR mode can help make better judgments. Figure 6 The test in (b) shows that when the target is occluded, YOLOv8 cannot accurately identify the target because it lacks RGB information, while this method can infer the location of the heat source and detect it based on IR information.

Claims

1. A method for detecting high-hotspot areas in fire safety based on dual-modal cross-attention fusion of AR glasses, characterized in that, The method includes the following steps: S1. Processing dual-modal video streams; S2. Construct a bimodal spatial attention matching module; S3. Construct a dual-modal high-hotspot feature fusion module based on cross-attention; S4. Based on the bimodal spatial attention matching module and feature fusion module, a bimodal fusion network and a corresponding high-hotspot target detection model are built. S5. Train the high hotspot target detection model and use the trained model to perform real-time detection and output of high hotspot targets in the dual-modal images collected by the fire-fighting AR glasses; Step S1 specifically includes: aligning the video stream output to ensure that the number of frames for RGB modal data and IR modal data is the same; taking the dual-modal data from the same frame and performing uniform size processing; if the image size is smaller than the target size, upsampling is performed; if the image size is larger than the target size, downsampling is performed. Let the original image size be... The processed image is Upsampling is represented as Downsampling is represented as Upsampling Using linear interpolation, downsampling Max pooling is used; Step S2 specifically involves: the constructed bimodal spatial attention matching module includes a block-coding convolution module and an asymmetric attention module. Taking the IR modality map and the RGB modality map as input, the block-coding convolution module performs block-coding convolution processing on the bimodal maps: converting the RGB modality map into a query matrix Q through convolution; converting the IR modality map into a key matrix K through convolution; initializing the built-in trainable texture matrix P, which follows a normal distribution N(0,0.02) with a mean of 0 and a variance of 0.02; and the RGB modality map being... IR mode diagram is The specific operation is represented by the following formula: , , ,in , , It is a trainable matrix; the convolution uses overlapping processing, the kernel size is (2s) x (2s), then the stride is set to s, where s is a positive integer; The block-encoded convolution obtained and The spatial correlation model is performed using the input asymmetric attention module, and the attention weight matrix is ​​calculated. Where T represents the matrix transpose, d represents the number of hidden convolutional layers encoding the convolution, and then the texture encoding information is obtained by weighting the texture matrix P using the weight matrix Score. Finally, output texture encoding information aligned between IR and RGB spaces. It is then applied to the original modality map for positional encoding, and an RGB feature map is output. and IR feature map ; In step S3, the specific process of constructing the bimodal high-hotspot feature fusion module based on cross-attention is as follows: Feature fusion is performed using the feature map with positional encoding information output by the bimodal spatial attention matching module as input. The input RGB feature map is then... IR feature map is The specific cross-attention fusion operation can then be expressed as the following formula: , , ,in , , It is a trainable matrix. It is by Mapped query matrix It is by The mapping of the key matrix, It is by The value matrix of the mapping, and the feature fusion result are expressed by the formula: The feature map is further processed for feature extraction, resulting in feature fusion at a higher dimension. The extraction method is expressed by the formula: , , will get , Perform as The same cross-attention fusion operation yields Then , Feature extraction was performed to obtain and Obtained through cross-attention fusion operation ; will obtain characteristics ,feature ,feature After being flattened along the channel dimension, the components are stitched together, as shown in the formula. Finally, a fully connected layer is used to obtain the feature fusion output of the module; Step S4 specifically involves: reshaping the feature fusion output obtained in step S3. Let the number of detected categories be i, then the output vector after feature reshaping is [...]. , , , , , ],in As the starting point of the width, As the starting point of the long period, The proportion of width, The proportion of length, , They are the 1st and 2nd respectively. The score for detecting target class i; the prediction confidence is obtained by inputting the score into the Sigmoid function. Let the output score of the model for the selected object being class k be... Then the confidence level The output features, including wide starting points, long starting points, width proportion, and length proportion, are removed based on confidence levels. Low-confidence boxes are removed, and non-maximum suppression is applied to the remaining boxes. The result is the target point.

2. The fire-fighting high-temperature spot detection method according to claim 1, characterized in that, The block coding convolution module allows overlapping portions of the encoding between image blocks, thereby enabling the encoding information to flow between image blocks and enhancing the extraction of image block information.

3. The fire-fighting high-temperature spot detection method according to claim 1, characterized in that, The query matrix Q and key matrix K in the spatial attention matching module are obtained from RGB and IR images through... , It is mapped from, and the texture matrix P is an independent trainable texture matrix. .

4. The fire-fighting high-temperature spot detection method according to claim 1, characterized in that, In step S4, the feature fusion process is a dual-branch structure where a convolutional network is responsible for extracting high-dimensional features, and a cross-attention mechanism is responsible for feature fusion.

Images