Human body target discrimination algorithm system for dense smoke environment
By combining the dominant unidirectional cross-attention and self-attention mechanisms of thermal imaging and near-infrared images, the modal alignment error and dominant mechanism problems of human target detection in complex environments are solved, and high-robustness detection and low-latency inference are achieved in dense smoke environments.
Patent Information
- Application Number
- CN202510750577.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-06
- Publication Date
- 2025-09-19
AI Technical Summary
Existing multimodal fusion methods fail to fully utilize the advantages of the main modality in complex environments, suffer from modal alignment errors and lack of a dominant mechanism, resulting in poor performance of human target detection in dense smoke environments.
By combining thermal imaging and near-infrared dual-modal images with dominant unidirectional cross-attention mechanism and self-attention mechanism, high robustness detection of human targets is achieved through video acquisition, data preprocessing, dominant modality feature enhancement and detection modules.
In complex smoke environments, the stable perception advantages of thermal imaging are fully utilized to improve fusion robustness and adaptability, reduce dependence on alignment accuracy, and support low-latency reasoning and actual scenario deployment.
Smart Images
Figure CN120673441A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of multimodal perception and target detection, and more particularly to a human target discrimination algorithm system for dense smoke environments. Background Art
[0002] In complex environments like fire, smoke, and low illumination, traditional single-modal image detection methods (such as visible light-based object detection) often suffer from performance degradation due to insufficient lighting or smoke obstruction. To improve detection accuracy, researchers are exploring the use of multimodal data fusion techniques, specifically combining thermal imaging with near-infrared imagery to leverage the complementary information from different modalities.
[0003] Existing multimodal fusion methods mainly include three strategies: early fusion, late fusion, and mid-term fusion. Early fusion methods directly fuse multimodal data at the input stage, which may cause interference between modal information; late fusion methods fuse at the decision layer, which makes it difficult to fully utilize the fine-grained information between modalities; mid-term fusion methods fuse at the feature extraction stage, which can partially solve the above problems, but still has the following shortcomings:
[0004] 1. Modal equivalence assumption: Most methods assume that each modality contributes information equally, but fail to consider the information advantage of a certain modality in a specific environment, resulting in poor fusion effect.
[0005] 2. Alignment sensitivity: Traditional fusion methods have high requirements for spatial alignment between modalities. In practical applications, due to factors such as sensor differences and changes in viewing angles, alignment errors often exist between modalities, affecting the fusion effect.
[0006] 3. Lack of dominant mechanism: Existing methods fail to dynamically adjust the dominant relationship between modalities according to environmental conditions and are unable to fully utilize the advantages of the dominant modality in specific scenarios.
[0007] Therefore, there is an urgent need for a target detection system that can fully utilize the advantages of the main modality in complex environments, has alignment robustness, and can effectively integrate multimodal information to improve the detection performance of human targets in dense smoke environments. Summary of the Invention
[0008] In view of this, the present invention provides a human target discrimination algorithm system for dense smoke environments, which can utilize thermal imaging and near-infrared dual-modal image input, combined with a dominant unidirectional cross-attention mechanism and a self-attention mechanism to achieve highly robust detection of human targets.
[0009] To achieve the above objectives, the present invention adopts the following technical solution: a human target discrimination algorithm system for a dense smoke environment, comprising:
[0010] A video acquisition module, a data preprocessing module, a dominant modality feature enhancement module, and a detection module connected in sequence;
[0011] The video acquisition module is used to collect thermal imaging modality videos and near-infrared modality videos and transmit them to the data preprocessing module;
[0012] The data preprocessing module preprocesses the collected video and outputs spatially aligned image frame pairs, and transmits the thermal imaging and near-infrared image frame pairs to the dominant modality feature enhancement module;
[0013] The dominant modal feature enhancement module is used to process the thermal imaging image frame pair and the near-infrared image frame pair to obtain an enhanced feature map and input it into the detection module;
[0014] The detection module is used to receive the enhanced feature map and output the target box position and category confidence to realize the detection of human targets.
[0015] Preferably, the video acquisition module includes a video deframing unit and a spatiotemporal synchronization unit. The video deframing unit is used to cut the thermal imaging modality video and the near-infrared modality video into image sequences according to a set frame rate; the spatiotemporal synchronization unit is used to perform frame-level time alignment and pixel-level spatial alignment on the thermal imaging and near-infrared image sequences.
[0016] Preferably, the spatiotemporal synchronization unit includes a time synchronization subunit and a spatial registration subunit. The time synchronization subunit is used to determine the starting frame pair of the thermal imaging image sequence and the near-infrared image sequence, and to construct a virtual timestamp sequence based on the frame rate difference to achieve dual-modal frame-level time alignment. The spatial registration subunit uses the matching point pairs in the starting frame pair to estimate the optimal homography matrix. The near-infrared image sequence is registered to the thermal imaging image space through perspective transformation to achieve pixel-level alignment.
[0017] Preferably, the homography matrix The expression is as follows:
[0018]
[0019] in, is the homogeneous coordinates of the i-th pair of matching points in the near-infrared and thermal imaging images; N is the number of matching points, is the estimated optimal homography matrix, and H is the variable matrix involved in the optimization process.
[0020] Preferably, the dominant modality feature enhancement module includes a unidirectional cross attention submodule, a self-attention enhancement submodule and a feature fusion submodule;
[0021] The unidirectional cross-attention submodule is used to extract the original feature map of the thermal imaging image frame and the feature map of the near-infrared image frame, and uses the thermal imaging feature map as the query vector Q and the near-infrared feature map as the key vector K and value vector V to build a multi-head attention mechanism, learn the cross-modal detail compensation relationship, and generate an enhanced thermal imaging feature map;
[0022] The self-attention enhancement submodule performs multi-head self-attention enhancement on the original feature map of thermal imaging, constructs its own semantic consistency modeling expression, and obtains the self-attention enhancement result;
[0023] The feature fusion submodule is used to fuse the enhanced thermal imaging features with the self-attention enhancement results, and output the final thermal imaging enhancement feature map using channel splicing, weighted summation or gating mechanism.
[0024] Preferably, the step of calculating the enhanced thermal imaging feature comprises:
[0025] Input near infrared feature map T NIR and thermal imaging original feature map T T ;
[0026] Construct query vector Q = T T W Q , key vector K = T NIR W K , value vector V = T NIR W V , where W Q , W K , W V They are the vector spaces that map the original feature map to the query vector, key vector, and value vector respectively;
[0027] The attention weight is calculated using the inner product of the query vector Q and the key vector K and normalized by softmax:
[0028]
[0029] Where A is the calculated attention score matrix, which represents the correlation between the query vector and the key vector; d k is the dimension of the key vector;
[0030] Use the attention matrix A to perform weighted summation on the value vector V to obtain the weighted feature map:
[0031] Z T =AV
[0032] Z T Through linear mapping and thermal imaging original feature map T T Perform residual fusion to obtain the thermal imaging feature map F enhanced by the unidirectional cross attention mechanism sub-moduleT , the formula is as follows:
[0033] F T =αZ T W O +βT T
[0034] Among them, α and β are learnable coefficients that control the proportion of weighted fusion; W O It is the output mapping matrix, which is used to map the fused features to a space suitable for subsequent processing.
[0035] Preferably, the self-attention enhancement submodule uses h attention heads to parallelly calculate the subspace similarity, wherein each attention head is respectively for the thermal imaging feature map T T and near-infrared characteristic map T NIR The mapping and fusion of query vector, key vector and value vector are performed on different sub-channels, and the fusion output is expressed as:
[0036] T' T =Concat(head1,head2,head3,...,head h )·W O
[0037] Among them, W O is the output transformation matrix, h is the number of attention heads;
[0038] For each attention head i , from the thermal imaging feature map F T Generate query vector, key vector and value vector respectively:
[0039]
[0040] in, is the transformation matrix of the query vector, key vector, and value vector of the i-th attention head.
[0041] Preferably, the enhanced thermal imaging feature is fused with the self-attention enhancement result, and the expression is as follows:
[0042] F=Concat(F T ,T' T )
[0043] Among them, F T is the thermal imaging feature map enhanced by the unidirectional cross attention submodule, T' T is the fused output representation from the self-attention enhancement submodule, and F is the fused feature map used for subsequent detection.
[0044] It can be seen from the above technical solutions that, compared with the prior art, the present invention provides a human target discrimination algorithm system for a dense smoke environment, which has the following beneficial effects:
[0045] (1) Adapting to complex smoke scenes: Build an asymmetric fusion mechanism based on thermal imaging to give full play to its stable perception advantage in smoke-obstructed environments;
[0046] (2) Improve fusion robustness: By combining unidirectional cross attention with main modality self-attention, the main modality feature expression is enhanced and the interference to the weak modality is reduced;
[0047] (3) Reduce dependence on alignment accuracy: adopt cross-modal feature matching and spatial perspective registration strategies to effectively improve the system's adaptability to non-strict alignment scenarios;
[0048] (4) Support for deployment and implementation: The model is lightweight and has a clear structure, making it easy to integrate into existing embedded platforms and supporting low-latency reasoning and alarming in actual scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0050] Figure 1 The overall structural diagram of the system provided by the present invention;
[0051] Figure 2 A flow chart of the space-time synchronization unit provided by the present invention;
[0052] Figure 3 A structural diagram of the unidirectional cross attention submodule provided by the present invention;
[0053] Figure 4 This is a structural diagram of the self-attention enhancement submodule provided by the present invention. DETAILED DESCRIPTION
[0054] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0055] like Figure 1As shown, the embodiment of the present invention discloses a human target discrimination algorithm system for a dense smoke environment, comprising:
[0056] A video acquisition module, a data preprocessing module, a dominant modality feature enhancement module, and a detection module connected in sequence;
[0057] The video acquisition module is used to collect thermal imaging modality videos and near-infrared modality videos and transmit them to the data preprocessing module;
[0058] The data preprocessing module preprocesses the collected video and outputs spatially aligned image frame pairs, and transmits the thermal imaging and near-infrared image frame pairs to the dominant modality feature enhancement module;
[0059] The dominant modal feature enhancement module is used to process the thermal imaging image frame pair and the near-infrared image frame pair to obtain an enhanced feature map and input it into the detection module;
[0060] The detection module is used to receive the enhanced feature map and output the target box position and category confidence to realize the detection of human targets.
[0061] Specifically, the video acquisition module includes a video deframing unit and a spatiotemporal synchronization unit. The video deframing unit is used to cut the thermal imaging modality video and the near-infrared modality video into image sequences according to a set frame rate; the spatiotemporal synchronization unit is used to perform frame-level time alignment and pixel-level spatial alignment on the thermal imaging and near-infrared image sequences.
[0062] Specifically, the spatiotemporal synchronization unit includes a time synchronization subunit and a spatial registration subunit. The time synchronization subunit is used to determine the starting frame pair of the thermal imaging image sequence and the near-infrared image sequence, and to construct a virtual timestamp sequence based on the frame rate difference to achieve dual-modal frame-level time alignment; the spatial registration subunit uses the matching point pairs in the starting frame pair to estimate the optimal homography matrix. The near-infrared image sequence is registered to the thermal imaging image space through perspective transformation to achieve pixel-level alignment.
[0063] In another specific embodiment, the data preprocessing module takes officially collected paired dual-modal videos as input, and the video types include thermal imaging video (Thermal) and filtered infrared video (NIR), and the shooting environment covers bright smoke-free, bright light smoke, white smoke under dark light conditions, yellow smoke under dark light conditions, and black smoke under dark light conditions.
[0064] The specific processing flow is as follows Figure 2 As shown:
[0065] (1) Video deframing: The input video is cut into image frame sequences at a fixed frame rate (e.g., 30fps for Thermal and 25fps for NIR) and stored in numbered form.
[0066] (2) Frame pair screening and starting frame pair selection: Based on the frame segment where the target typically appears (about 10 meters), feature points (ORB) are extracted from the two-modal image frames, the matching degree is calculated, and the frame pair with the best bidirectional matching score is selected as the starting point of spatiotemporal synchronization.
[0067] (3) Time synchronization: Based on the start frame, a virtual timestamp sequence of the two modalities is constructed, and linear time alignment is performed based on the frame rate difference (e.g., 25fps and 30fps) to match the nearest time frame pair.
[0068] (4) Spatial registration: Using the point pairs extracted from the starting frame, calculate the homography matrix from NIR to Thermal Perspective transformation (Warp) is performed on all NIR image sequences to achieve pixel-level spatial alignment.
[0069] Specifically, here It represents a 3-row, 3-column matrix space in the real field. That is, the matrix H is a 3×3 real matrix (more specifically, this homography matrix is a 3×3 real matrix used to represent the perspective transformation between two images).
[0070] (5) Output: The output is the aligned NIR–Thermal image pair for subsequent model training.
[0071] Specifically, the optimal homography matrix The expression is as follows:
[0072]
[0073] in, is the homogeneous coordinates of the i-th pair of matching points in the near-infrared and thermal imaging images; N is the number of matching points, is the estimated optimal homography matrix, and H is the variable matrix involved in the optimization process. The optimal homography matrix can realize the spatial mapping from the near-infrared image coordinate system to the thermal imaging image coordinate system, which is used to complete pixel-level registration. represents a point in the near-infrared image, The point coordinates after the transformation by applying the homography matrix are transformed by this matrix, and the point of the near-infrared image is is mapped into the coordinate system of the thermal image.
[0074] In another specific embodiment, in the spatial registration subunit, the homography matrix H between the thermal imaging image and the near-infrared image is obtained by estimating the matching point set through RANSAC, and is further optimized by the Levenberg–Marquardt nonlinear least squares algorithm to obtain the final registration homography matrix
[0075] Specifically, the dominant modality feature enhancement module includes a unidirectional cross attention submodule, a self-attention enhancement submodule and a feature fusion submodule;
[0076] The unidirectional cross-attention submodule is used to extract the original feature map of the thermal imaging image frame and the feature map of the near-infrared image frame, and uses the thermal imaging feature map as the query vector Q and the near-infrared feature map as the key vector K and value vector V to build a multi-head attention mechanism, learn the cross-modal detail compensation relationship, and generate an enhanced thermal imaging feature map;
[0077] The self-attention enhancement submodule performs multi-head self-attention enhancement on the original feature map of thermal imaging, constructs its own semantic consistency modeling expression, and obtains the self-attention enhancement result;
[0078] The feature fusion submodule is used to fuse the enhanced thermal imaging features with the self-attention enhancement results, and output the final thermal imaging enhancement feature map using channel splicing, weighted summation or gating mechanism.
[0079] Specifically, such as Figure 3 As shown, the step of calculating the enhanced thermal imaging feature includes:
[0080] Input near infrared feature map T NIR and thermal imaging original feature map T T ;
[0081] represents the original feature map extracted from the thermal imaging image frame;
[0082] represents the original feature map extracted from the near-infrared image frame;
[0083] The feature map at this stage is still in a fused state and serves as the input basis for the subsequent attention mechanism.
[0084] Construct query vector Q = T T W Q , key vector K = T NIR W K , value vector V = T NIR W V , where W Q , W K , W V They are the vector spaces that map the original feature map to the query vector, key vector, and value vector respectively;
[0085] The attention weight is calculated using the inner product of the query vector Q and the key vector K and normalized by softmax:
[0086]
[0087] Where A is the calculated attention score matrix, which represents the correlation between the query vector and the key vector; d k is the dimension of the key vector;
[0088] Use the attention matrix A to perform weighted summation on the value vector V to obtain the weighted feature map:
[0089] Z T =AV
[0090] Z T Through linear mapping and thermal imaging original feature map T T Perform residual fusion to obtain the thermal imaging feature map F enhanced by the unidirectional cross attention mechanism sub-module T , the formula is as follows:
[0091] F T =αZ T W O +βT T
[0092] Among them, α and β are learnable coefficients that control the proportion of weighted fusion; W O It is the output mapping matrix, which is used to map the fused features to a space suitable for subsequent processing.
[0093] In another specific embodiment, the input is the thermal imaging feature map and the registered near-infrared feature map. T As the query vector Q, the registered near-infrared feature map T NIR As key and value vectors K, V, calculate the cross-modal attention output:
[0094]
[0095] in, d k is the latitude of the key vector, d v is the latitude of the value vector, n q and n k are the number of queries and keys, respectively.
[0096] After attention calculation, the enhanced features are output, that is, the output is:
[0097] Z T =Attention(Q T ,K T ,V T )
[0098] Among them, Z TIndicates the output enhancement feature, which is the feature calculated by the attention mechanism. It contains the fusion result of information from the query, key, and value; it is the final output used to enhance the thermal imaging feature. T K represents the query vector (Query) after the projection of the thermal imaging modal feature map, which is used to ask questions or obtain information. T The key vector (Key) represents the projected thermal imaging modal feature map, which is the query vector Q T What to match. V T Represents the value vector (Value) of the thermal imaging modal feature map after projection, and the key vector K T Correspondingly, information associated with the query vector is provided.
[0099] The final output is:
[0100] F T =αZ T W O +βT T
[0101] Among them, Table F T It is the thermal imaging feature map enhanced by the unidirectional cross attention submodule. W O Represents the output mapping matrix, which is used to map the weighted attention result to the original feature map T of thermal imaging T In the same space. α and β represent the learnable weight coefficients, which control the cross attention enhancement result and the original thermal imaging feature map T respectively. T The proportion in the final output.
[0102] Specifically, such as Figure 4 As shown in the figure, the self-attention enhancement submodule uses h attention heads to parallelly calculate the subspace similarity, where each attention head is used to calculate the original feature map T of the thermal imaging. T and near-infrared characteristic map T NIR The mapping and fusion of query vector, key vector and value vector are performed on different sub-channels, and the fusion output is expressed as:
[0103] T' T =Concat(head1,head2,head3,...,head h )·W O
[0104] Among them, W O is the output transformation matrix, h is the number of attention heads;
[0105] For each attention head i , from the enhanced thermal imaging feature map F TGenerate query vector, key vector and value vector respectively:
[0106] in, is the transformation matrix of the query vector, key vector, and value vector of the i-th attention head.
[0107] Specifically, the enhanced thermal imaging feature is fused with the self-attention enhancement result, and the expression is as follows:
[0108] F=Concat(F T ,T' T )
[0109] Among them, F T is the thermal imaging feature map enhanced by the unidirectional cross attention submodule, T' T is the fused output representation from the self-attention enhancement submodule, and F is the fused feature map used for subsequent detection.
[0110] In another specific embodiment, the detection module uses a multi-scale detection head structure compatible with YOLOv8. This input feeds enhanced thermal imaging features into the detection head and performs object detection. This multi-scale object detection capability enables the system to effectively detect objects of varying sizes. Ultimately, the detection module outputs the object's location and category confidence.
[0111] Furthermore, in another specific embodiment, the present invention also includes a model deployment module responsible for exporting the trained detection model into ONNX or RKNN formats, facilitating deployment and inference on edge device platforms. By deploying the model to devices such as intelligent firefighting robots, portable security terminals, embedded AI boxes, or high-temperature wearable computing platforms, the system can handle target detection tasks in dense smoke environments in real time.
[0112] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.
[0113] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A human target discrimination algorithm system for a dense smoke environment, characterized by: include: A video acquisition module, a data preprocessing module, a dominant modality feature enhancement module, and a detection module connected in sequence; The video acquisition module is used to collect thermal imaging modality videos and near-infrared modality videos and transmit them to the data preprocessing module; The data preprocessing module preprocesses the collected video and outputs spatially aligned image frame pairs, and transmits the thermal imaging and near-infrared image frame pairs to the dominant modality feature enhancement module; The dominant modal feature enhancement module is used to process the thermal imaging image frame pair and the near-infrared image frame pair to obtain an enhanced feature map and input it into the detection module; The detection module is used to receive the enhanced feature map and output the target box position and category confidence to realize the detection of human targets.
2. The human target identification algorithm system for dense smoke environment according to claim 1 is characterized in that: The video acquisition module includes a video deframing unit and a spatiotemporal synchronization unit. The video deframing unit is used to cut the thermal imaging modality video and the near-infrared modality video into image sequences according to a set frame rate; the spatiotemporal synchronization unit is used to perform frame-level time alignment and pixel-level alignment on the thermal imaging and near-infrared image sequences.
3. The human target identification algorithm system for dense smoke environment according to claim 2 is characterized in that: The spatiotemporal synchronization unit includes a time synchronization subunit and a spatial registration subunit. The time synchronization subunit is used to determine the starting frame pair of the thermal imaging image sequence and the near-infrared image sequence, and to construct a virtual timestamp sequence based on the frame rate difference to achieve dual-modal frame-level time alignment. The spatial registration subunit uses the matching point pairs in the starting frame pair to estimate the optimal homography matrix The near-infrared image sequence is registered to the thermal imaging image space through perspective transformation to achieve pixel-level alignment.
4. The human target identification algorithm system for dense smoke environment according to claim 3 is characterized in that: The optimal homography matrix The expression is as follows: in, is the homogeneous coordinates of the i-th pair of matching points in the near-infrared and thermal imaging images; N is the number of matching points, is the estimated optimal homography matrix, and H is the variable matrix involved in the optimization process.
5. The human target identification algorithm system for dense smoke environment according to claim 1 is characterized in that: The dominant modality feature enhancement module includes a unidirectional cross attention submodule, a self-attention enhancement submodule and a feature fusion submodule; The unidirectional cross-attention submodule is used to extract the original feature map of the thermal imaging image frame and the feature map of the near-infrared image frame, and uses the thermal imaging feature map as the query vector Q and the near-infrared feature map as the key vector K and value vector V to build a multi-head attention mechanism, learn the cross-modal detail compensation relationship, and generate an enhanced thermal imaging feature map; the self-attention enhancement submodule performs multi-head self-attention enhancement on the original thermal imaging feature map, constructs its own semantic consistency modeling expression, and obtains the self-attention enhancement result; The feature fusion submodule is used to fuse the enhanced thermal imaging features with the self-attention enhancement results, and output the final thermal imaging enhancement feature map using channel splicing, weighted summation or gating mechanism.
6. The human target identification algorithm system for dense smoke environment according to claim 5 is characterized in that: The step of calculating the enhanced thermal imaging feature comprises: Input near infrared feature map T NIR and thermal imaging original feature map T T ; Construct query vector Q = T T W Q , key vector K = T NIR W K , value vector V = T NIR W V , where W Q , W K , W V They are the vector spaces that map the original feature map to the query vector, key vector, and value vector respectively; The attention weight is calculated using the inner product of the query vector Q and the key vector K and normalized by softmax: Where A is the calculated attention score matrix, which represents the correlation between the query vector and the key vector; d k is the dimension of the key vector; Use the attention matrix A to perform weighted summation on the value vector V to obtain the weighted feature map: Z T =OFF Z T Through linear mapping and thermal imaging original feature map T T Perform residual fusion to obtain the thermal imaging feature map F enhanced by the unidirectional cross attention mechanism sub-module T , the formula is as follows: F T =αZ T IN O +βT T Among them, α and β are learnable coefficients that control the proportion of weighted fusion; W O It is the output mapping matrix, which is used to map the fused features to a space suitable for subsequent processing.
7. The human target identification algorithm system for dense smoke environment according to claim 6 is characterized in that: The self-attention enhancement submodule uses h attention heads to parallelly calculate the subspace similarity, where each attention head is used to calculate the thermal imaging feature map T T and near-infrared characteristic map T NIR The mapping and fusion of query vector, key vector and value vector are performed on different sub-channels, and the fusion output is expressed as: T ' T =Concat(head1,head2,head3,...,head h )·W O Among them, W O is the output transformation matrix, h is the number of attention heads; For each attention head i , from the thermal imaging feature map F T Generate query vector, key vector and value vector respectively: in, is the transformation matrix of the query vector, key vector, and value vector of the i-th attention head.
8. The human target identification algorithm system for dense smoke environment according to claim 7 is characterized in that: The enhanced thermal imaging feature is fused with the self-attention enhancement result, and the expression is as follows: F=Concat(F T ,T′ T ) Among them, F T is the thermal imaging feature map enhanced by the unidirectional cross attention submodule, T ' T is the fused output representation from the self-attention enhancement submodule, and F is the fused feature map used for subsequent detection.