Cross-modal smoke-occluded human recognition method and device based on dual-engine cooperation

Abstract

This invention discloses a cross-modal smoke-covered human body recognition method and device based on dual-engine collaboration. The method employs the MambaPPA-MuYolo model to experimentally identify humans in dense smoke environments. A multimodal data preprocessing framework based on SECycleGAN is used to remove smoke from the data, optimizing human detection performance in dense smoke conditions. In the model architecture, the backbone network uses dual-branch data loading to fully integrate feature information from both modalities. The neck network is a BIFPN multi-scale fusion feature pyramid structure. Furthermore, a GT-Mamba feature information acquisition module, including a local spatial multi-core attention branch and a state space model based on selected scanning, along with multiple multi-branch feature extraction modules (PPA) to improve small target detection, are embedded into the neck module. This enhances spatial information capture and improves information interaction between different layers. This invention optimizes model performance, significantly improving accuracy in harsh smoke environments, providing reliable technical support for fire and rescue operations.

Description

Technical Field

[0001] This invention relates to the field of human body recognition technology, specifically to a cross-modal smoke-masked human body recognition method and device based on dual-engine collaboration. Background Technology

[0002] With the rapid development of intelligent technology, the fire and rescue field is also constantly exploring the application of advanced robotics technology in actual rescue missions. Especially in high-risk fire scenes, traditional manual rescue methods face enormous challenges and risks. Factors such as dense smoke, low visibility, and high temperatures often lead to low rescue efficiency, seriously hindering firefighters from quickly locating and rescuing trapped personnel, increasing the difficulty and danger of rescue, and even threatening the lives of rescuers.

[0003] Traditional methods for removing dense smoke can be divided into two categories. The first category is based on image enhancement algorithms, such as histogram equalization, Retinex theory, and wavelet transform algorithms. These methods simply adjust the contrast and saturation of the image without fully considering the physical characteristics of smoke, resulting in poor smoke removal performance in practical applications. The second category is based on atmospheric scattering models, such as algorithms combining dark channel priors with atmospheric scattering models. While these algorithms can remove smoke to some extent, they require extremely high accuracy in parameter estimation. When certain areas of the image do not meet the dark channel prior assumptions, color distortion and halo phenomena can easily occur, affecting subsequent target detection.

[0004] For the problem of target recognition, traditional single-modal target recognition techniques, such as those using RGB visible light or infrared light, have achieved good results in some situations. However, they lack the common problem of information complementarity between different modalities, which makes the target detection performance generally poor in real-world environments, and target features may be lost.

[0005] Current traditional algorithms are insufficient for fire rescue in terms of target recognition and smoke removal capabilities. Their limitations and shortcomings can be summarized as follows: Figure 1 As shown; The above issues can be summarized into three points: Problem 1: The processing of dense smoke is difficult, and traditional algorithms are prone to distortion in the harsh environment of a fire scene, which cannot meet the needs of reality.

[0006] Question 2: Existing target detection algorithms are not robust enough in harsh environments with dense smoke, resulting in high rates of missed detections and false detections, and low error tolerance.

[0007] Question 3: Existing methods cannot fully consider the fire scene conditions. Environmental factors such as high temperature can affect infrared imaging and image feature detection, resulting in poor model adaptability.

[0008] To address the challenge of accurately identifying human bodies in harsh environments characterized by dense smoke, low light, and high temperatures, this invention proposes a multimodal human body recognition scheme based on edge intelligence for dense smoke perception, utilizing the Internet of Things, artificial intelligence, and platform development. This scheme enables human body detection under adverse conditions on an edge computing development board (NLE-AI800 AI development board). Summary of the Invention

[0009] To address the challenge of accurately identifying human bodies in harsh environments characterized by dense smoke, low light, and high temperatures, this invention proposes a cross-modal smoke-covered human body recognition method based on dual-engine collaboration. Utilizing IoT, AI, and platform development, the method enables human body detection under adverse conditions on an edge computing development board (NLE-AI800 AI development board).

[0010] To achieve the above objectives, the present invention adopts the following technical solution: A cross-modal smoke-covered human body recognition method based on dual-engine collaboration, which executes the following steps via computer equipment: The experiment consists of two parts: model building and platform loading. The model building uses SEcycleGan (Self-Attention-Recurrent Generative Adversarial Network) and MambaPPA-MuYolo (Mamba Parallel Patch Perceptual Attention-Multimodal YOLO) models to conduct experiments on human recognition in dense smoke environments. The model input uses a data preprocessing framework based on SECYcleGAN (Self-Attention-Recurrent Generative Adversarial Network) to remove smoke from visible light data, optimizing human detection performance in dense smoke environments. In the model architecture, the backbone network uses dual-branch data loading and feature-level fusion to fully integrate feature information from both modalities. The neck network is a bidirectional feature pyramid structure with multi-scale fusion of BIFPN (Dynamically Weighted Bidirectional Feature Pyramid Network). A GT-Mamaba (Gated Lightweight Mamba) feature information acquisition module containing local spatial branches and Mamba (Selective Scan State Space Model) branches, along with multiple multi-branch feature extraction modules (PPA) to improve small target detection, are embedded into the Neck module. These modules are used to enhance spatial information capture and improve information interaction capabilities between different layers. The platform loading process utilizes edge intelligent computing development equipment to configure the environment, loads the optimized MambaPPA-MuYolo (Mamba Parallel Patch Perceptual Attention-Multimodal YOLO) model into the JupyterLab environment of the development board for model inference, and finally uses the USB camera of the edge computing platform to capture smoke image frames and call the model for detection.

[0011] Furthermore, model building includes, First, the backbone network is reconstructed by dual-stream heterogeneous feature fusion, and the original single-modal feature extraction architecture is extended into a dual-channel processing structure that supports RGB and enhanced features. The intermodal feature complementarity is achieved by constructing a three-level cross-modal feature fusion module. Secondly, the GT-Mamba attention mechanism is introduced into the connection layer between the backbone network and the feature pyramid, and its global context modeling capability is used to recalibrate the multi-scale features output by the backbone network. Finally, bidirectional feature pyramid optimization is implemented on the neck network to upgrade the traditional PAN-FPN structure to a BiFPN architecture that includes a PPA module, thereby enhancing robustness in complex backgrounds through multi-branch parallel feature extraction.

[0012] Furthermore, the input end receives the raw image data and uses a multimodal data preprocessing framework based on SECYcleGAN to preprocess the data, i.e., to perform smoke removal. Next, the preprocessed image data is fed into a Backbone network based on the MambaPPA-MuYolo model for feature extraction; The Backbone network consists of a Conv module, a C2f module, and a Mu-Fusion multimodal fusion module. The feature information extracted by the Backbone network is fed into the Neck network from four different network layers. This module is responsible for fusing the multi-scale features extracted by the Backbone. First, the feature information is processed by GT-Mamba and then fed into the multi-scale feature fusion network of the BIFPN structure. The multi-scale fusion module uses learnable weights Bi-Fusion to assign corresponding weights to data at different levels and fuse them. After fusion, the data enters the PPA multi-branch feature extraction network to capture feature information. Finally, the features processed by the Neck module are passed to the Head module for final regression prediction. The Head (detection head network) module uses feature maps extracted by the Backbone (main backbone network) to detect the location and category of the target. By training and optimizing the network parameters of the Head part, the model can be better adapted to the target detection task. Furthermore, the relevant formulas for the GT-Mamba attention mechanism are expressed as follows: (1) (2) (3) (4) Formula above LayNoem This indicates normalization processing; Wu This represents a learnable linear projection matrix; W o SSMConvolution represents the linear output projection matrix; it represents the application of the SSM state-space model to the entire sequence input, where parameter A represents the state transition matrix, B represents the input control matrix, and C represents the output projection matrix.

[0013] Furthermore, the GT-Mamba module consists of two parallel branches: One is the local spatial multi-core attention branch, and the other is the main Mamba branch of this module; In the upper branch network of the module, the focus is on acquiring local spatial information at multiple scales, specifically for a dimension of... The feature matrix X1 is first normalized using BatchNorm, and then enters a local spatial attention layer. This layer first uses 1×1 convolutions to reduce channels, resulting in a more efficient computational tensor. Simultaneously, a local attention mechanism is applied, using three different kernel sizes for expanded convolutions: 3×3, 5×5, and 7×7, with expansion rates of 1, 2, and 3. Subsequently, each attention map is activated by SiLU and then concatenated along the channel dimension. This process is represented by the following formula: (5) (6) (7) In the lower branch network of the module, global attention is extracted by analyzing the long-distance dependencies of the entire feature map. The feature matrix X2 is first normalized and then fed into the 2D selection scanning layer model to scan the global information of each pixel in each spectral image in four directions, namely from top to bottom, from bottom to top, from left to right, and from right to left, in order to capture global correlation information. After selecting the scan layer in 2D, the output features are normalized again by layer normalization, and then multiplied element-wise with the initial normalized input features of the branch. This process can be represented by the following formula: (8) (9) The GT-Mamba module ultimately concatenates the outputs of the local and global branches along the channel dimension. This concatenation integrates local and global spatial information, providing the model with a comprehensive understanding of the input feature map. After concatenation, the channels are shuffled to promote interaction between different feature dimensions, further enhancing the model's expressive power. This process is represented by the following formula: (10).

[0014] Furthermore, the PPA-BiFPN composite structure includes a PPA network and a BiFPN network; The PPA network contains three parallel branches, involving local, global, and convolutional branch structures. The processing flow includes features based on a feature matrix. First, it is processed through a convolution... Then, the three are calculated through the three branches respectively. , , The feature matrices are then added together to obtain ; After feature extraction, the design employs feature fusion and attention mechanisms for adaptive feature enhancement. The attention mechanism module primarily includes a series of efficient channel attention and spatial attention mechanisms. This process is mathematically represented by processing through a one-dimensional channel attention and a two-dimensional spatial attention, respectively: (11) (12) here and These represent ReLU and BatchNorm, respectively. M represents element-wise multiplication. c and M s F represents channel attention and spatial attention, respectively. c With F s These represent the features after channel selection; BiFPN, or Dynamically Weighted Bidirectional Feature Pyramid Network, incorporates skip connections. If the original input and output nodes are at the same level, an extra edge is added between them to fuse more features without increasing cost. If a node has only one input and no feature fusion occurs, its contribution to the overall feature network is minimal, and it is removed. Furthermore, instead of the traditional multi-layer fusion strategy of directly stacking the same weight dimensions, this part uses a Bi-Fusion module based on learnable weights to dynamically adjust features at different scales, assigning appropriate weights to different inputs so that the network can better highlight and capture data information.

[0015] Furthermore, the Bi-Fusion module dynamically adjusts the importance of features at different scales through learnable weights, for feature images F1, F2, F3, ..., F... nThe input introduces a set of trainable weight parameters ω1, ω2, ω3, ..., ω n Then, normalization is performed using the Swish function, a non-linear activation function, as shown in the formula below. (13) The corresponding learning weight formula is shown below: (14) Here, Wi represents the learnable feature fusion weights, and c is a very small number to prevent the denominator from being 0; The final multi-scale fusion matrix calculation formula is shown below: (15) The proposed Bi-Fusion module achieves multi-scale feature map fusion by first inputting feature maps. F i Multiply by the normalized weights, and then sum them up.

[0016] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0017] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0018] As can be seen from the above technical solution, this invention, based on target recognition technology and image generation and conversion technology, achieves accurate human body recognition under harsh conditions of dense smoke and low light on an edge computing development board (NLE-800 AI development board). Firstly, this invention proposes a multimodal feature fusion method for human body detection based on dense smoke perception. Specifically, for the input visible light and infrared images, a CycleGAN enhanced with a self-attention mechanism is first used to refine the multimodal data of the input dense smoke environment, providing high-quality input feature images for the human body detection network. To maximize the accuracy of human body detection at fire scenes, the model adopts a dual-branch image loading structure, performing three-stage multimodal feature fusion to achieve complementary fusion of feature maps from the two modalities, enhancing the feature extraction capability of the detection network. To further enhance the model's target detection capability, a bidirectional multi-scale channel-spatial hybrid attention feature pyramid structure is proposed, combined with the Mamba model and PPA network. This design significantly enhances the model's ability to fuse multi-scale features and improves the model's accuracy in detecting humans.

[0019] Experiments show that on the dense smoke dataset, the structural similarity between the desmoked image and the original image using a self-attention enhanced recurrent generative adversarial network (CGAN) is 89.02%, which is 1.63 dB and 3.17% higher than the original CGAN in peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), respectively. The multimodal feature fusion detection method for dense smoke perception also shows better precision, recall, and mAP compared to the YOLOv8 model. 50 and mAP 50-95 These figures represent increases of 8.7, 11.6, 10.9, and 14.8 percentage points, respectively. Secondly, the project team developed a graphical user interface (GUI) based on PyQt5 modules, with features including selecting image folders or videos, accessing the camera to take photos, and real-time monitoring options.

[0020] Compared to traditional image processing and recognition algorithms, the method of this invention combines the commonalities of two modal feature maps, resulting in higher accuracy. This greatly improves the efficiency of human detection and recognition in harsh environments such as fires, providing an advanced intelligent solution for urban safety and fire protection.

[0021] Specifically, this invention, combining large-scale artificial intelligence models, computer vision, and edge computing hardware platforms, offers the following advantages: Preprocessing of dense smoke data based on the SECycleGan model: The SECycleGan model was independently designed based on CycleGan and Self-Attention. This model combines the advantages of recurrent generative adversarial networks in image data transformation and the ability of self-attention mechanism to capture feature data. It improves peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) by 1.63dB and 3.17% respectively, providing a good solution for dense smoke removal in data preprocessing.

[0022] This invention relates to a human recognition network based on the MambaPPA-MuYolo model. It utilizes the Mamba model, a PPA multi-branch feature extraction network, a BIFPN bidirectional multi-scale channel-spatial hybrid attention feature pyramid structure, and a Mu-Fusion multimodal feature fusion network to independently design the MambaPPA-MuYolo model. This model breaks through the limitations of traditional single-modal image recognition by innovatively employing a dual-stream loaded triple multimodal feature fusion backbone network. Furthermore, it incorporates several innovative modules to optimize model performance, resulting in a significant improvement in accuracy under harsh, smoky conditions, providing reliable technical support for fire and rescue operations.

[0023] Leveraging the PyQt5 user interface and edge computing platform support, this invention employs a graphical user interface (GUI) developed using PyQt5 modules. Users can easily select image folders or video files via the GUI, and can also directly access the camera for live shooting, enabling flexible switching between multiple data sources. The edge computing platform provides an excellent hardware foundation for this design project, ensuring rapid model processing response and high system stability. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of a traditional algorithm; Figure 2 This is a partial example of the dataset in an embodiment of the present invention; Figure 3 This is a flowchart illustrating the technical architecture of an embodiment of the present invention; Figure 4 This is a schematic diagram of the MambaPPA-MuYolo architecture according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the GT-Mamba architecture according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the PPA architecture according to an embodiment of the present invention; Figure 7a This is a display of smoke data processing when removing smoke and simultaneously removing the target person; Figure 7b It is a process of processing and displaying dense smoke data while preserving as much of the target person's characteristic information as possible and removing interfering backgrounds such as smoke. Figure 8 This is a schematic diagram of the SECycleGan generator architecture according to an embodiment of the present invention; Figure 9 This is a schematic diagram of the Backbone structure according to an embodiment of the present invention; Figure 10 This is a schematic diagram of the Attention architecture according to an embodiment of the present invention; Figure 11 This is the MambaPPA-MuYolo neck network architecture of this invention; Figure 12 This is a system flowchart of an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0026] like Figure 1As shown in this embodiment, the cross-modal smoke-covered human body recognition method based on dual-engine collaboration innovatively proposes a dual-engine collaborative architecture of "data constraint enhancement + multimodal spatiotemporal perception". Based on cutting-edge computer vision and artificial intelligence large-scale model technology, an end-to-end intelligent fire rescue human body recognition system with environmental adaptability is constructed. Its core technological breakthroughs are reflected in the following two innovative models: 1. Smoke Image Data Enhancing Process: SECycelGan Data Augmentation Model By embedding a Self-Attention (SE) mechanism module in the Cyclegan generator, the model's ability to capture feature information is enhanced, thereby strengthening the detailed reconstruction of the smoke-covered region.

[0027] 2. Multimodal target recognition: MambaPPA-MuYolo adaptive detection model An innovative dual-branch multimodal feature extraction backbone network is constructed, employing a visible-infrared dual-modal parallel input architecture. A three-layer cascaded feature fusion layer achieves cross-modal feature complementarity, compensating for the shortcomings of single-modality features. Furthermore, in the Neck network improvement, the Mamba model and PPA multi-branch feature extraction network are embedded, and the Bi-Fusion multi-scale fusion module is replaced with the Concat module to further enhance model performance.

[0028] Combining the solutions proposed above can effectively address the three core pain points in fire rescue: high identification errors, significant detection difficulties, and poor environmental adaptability. By integrating multi-source sensing, intelligent identification technology, and real-time smoke removal processing of fire scene images, real-time detection and location of trapped personnel can be achieved in harsh environments such as dense smoke, low light, and obstructions, thus providing more accurate technical support for fire rescue. This is of great significance for improving the intelligence level of emergency rescue systems and building safe cities.

[0029] The following is a detailed explanation: Dataset Introduction The quality of the dataset has a significant impact on the accuracy of the model's recognition. Since this project requires a multimodal dataset with smoke, there are few high-quality datasets available. Therefore, the initial dataset for this project was created using real-world photography. Furthermore, to improve the model's generalization ability, the multimodal LLVIP dataset containing RGB and infrared images was incorporated. The total dataset contains 24,213 images.

[0030] To meet project requirements, the dataset is categorized into one class (person), but with two modal data settings: RGB and infrared. Furthermore, the dataset used in the artwork encompasses diverse and complex environmental environments, covering various locations including low-light, bright-light, and smoky conditions. Specific data is shown in Table 1, with examples of dataset images. Figure 2 As shown.

[0031] Table 1 Dataset Information Table

[0032] Technical Architecture: The technical approach of this project is divided into two parts: model building and platform loading. Model building uses the MambaPPA-MuYolo model to experiment with human recognition in dense smoke environments. The model input uses SECycleGan to remove smoke from the data, optimizing human detection performance in dense smoke environments. In the model architecture, the backbone uses dual-branch data loading to fully integrate feature information from both modalities. The neck network uses a BIFPN multi-scale fusion feature pyramid structure, and embeds a GT-Mamaba feature information acquisition module containing local spatial branches and Mamba-annotated branches, along with multiple multi-branch feature extraction modules (PPA) to improve small target detection, into the Neck module. These modules are used to enhance spatial information capture and improve information interaction capabilities between different layers.

[0033] The platform loading process utilizes edge computing development equipment for environment configuration (including Python environment and related library files). The optimized MambaPPA-MuYolo is loaded into the JupyterLab environment on the development board for model inference. Finally, the USB camera on the edge computing platform is used to capture smoke image frames for model detection. The overall project flowchart is as follows: Figure 3 As shown: This invention proposes the MambaPPA-MuYOLO and SECycleGan models. Addressing the limitations of YOLOv8 in target detection under complex environments, this invention proposes an improved MambaPPA-MuYolo model based on multimodal feature fusion. Through systematic analysis of the shortcomings of YOLOv8 in feature extraction and scale transformation mechanisms, particularly the limited feature capture capability and low efficiency of cross-scale information fusion in scenarios with dense smoke obscuration, low illumination, and complex background interference, this paper innovatively improves upon these shortcomings from two dimensions: network architecture optimization and multimodal data processing.

[0034] In terms of model architecture design, this embodiment of the invention constructs a three-level progressive improvement scheme: First, the backbone network is reconstructed by dual-stream heterogeneous feature fusion, expanding the original single-modal feature extraction architecture into a dual-channel processing structure that supports RGB and enhanced features, and achieving intermodal feature complementarity by constructing a three-level cross-modal feature fusion module; Second, the GT-Mamba attention mechanism is introduced into the connection layer between the backbone network and the feature pyramid, and its global context modeling capability is used to recalibrate the multi-scale features output by the backbone network; Finally, the neck network is optimized by bidirectional feature pyramid, upgrading the traditional PANet-FPN structure to a BiFPN architecture that includes a PPA (Paralleized Patch-Aware Attention) module, and enhancing robustness in complex backgrounds through multi-branch parallel feature extraction.

[0035] The specific technical implementation includes three innovations: 1) Proposing a multimodal data preprocessing framework based on SECycleGAN, which enhances visual features and performs modal transformation under harsh environments through adversarial generative networks; 2) Designing a cross-modal feature interaction unit, establishing a learnable feature mapping matrix at key levels of the backbone network to achieve effective fusion of multi-source heterogeneous features; 3) Constructing a PPA-enhanced BiFPN structure, embedding multi-branch feature refining modules with parallel channel attention and spatial attention at each level of the feature pyramid, and integrating a Bi-Fusion learnable dynamic weight adjustment module to replace the original Concat.

[0036] like Figure 4 As shown, the model architecture adopts a hierarchical design: the data preprocessing network uses SECycelGan to refine the smoke before feeding it into the backbone network; the backbone network uses a dual-stream data loading structure to extract multimodal features and achieves intermodal information interaction through a cross-modal feature fusion layer; the GT-Mamba module serves as a feature recalibration unit, which enhances the response of key features through a gated temporal attention mechanism; the neck network adopts a PPA-BiFPN composite structure, which improves the detection accuracy of multi-scale targets through bidirectional cross-scale connections between feature pyramid layers and an attention-weighted fusion mechanism.

[0037] The following provides a detailed explanation of each module of the model: (1) GT-Mamba model Some designs in this invention are based on the Mamba model, which makes it more powerful in image processing and better at capturing image feature information.

[0038] Mamba's core relies on the discretization and optimization of the State-Space Model (SSM). In particular, the Structured State-Space Sequence Model (S4) has become an important method for building efficient deep networks, demonstrating superior performance in long sequence data analysis and mitigating problems such as gradient vanishing and exploding, as well as efficiency degradation with long inputs. It also alleviates the need for large computational resources in Transformer models when processing large datasets or high-resolution images. The selectively scanned State-Space Model (S6), such as Mamba... Figure 5 As shown, S4 is enhanced through a selection mechanism that emphasizes relevant data in an input-dependent manner. The relevant formula is implemented as follows: (1) (2) (3) (4) Formula above LayNoem This indicates normalization processing; Wu This represents a learnable linear projection matrix; Wo SSMConvolution represents the linear output projection matrix; it represents the application of the SSM state-space model to the entire sequence input, where parameter A represents the state transition matrix, B represents the input control matrix, and C represents the output projection matrix. This invention combines the advantages of Mamba in image processing with a local spatial attention network to form a dual-attention module network, GT-Mamba. GT-Mamba is developed based on the SS-Conv-SSM framework and is a dual-attention module consisting of two parallel branches: a local spatial multi-kernel attention branch and a GT-Mamba feature information acquisition module based on a selected scan state space model. This design can further extend the capture of spatial information and combine global and local attention perspectives. The module structure is as follows... Figure 5 As shown.

[0039] In the upper branch network of the module, the focus is on acquiring local spatial information at multiple scales. Specifically, for a dimension of... The feature matrix X1 is first normalized using BatchNorm, and then enters a local spatial attention layer. Here, a 1×1 convolution is first used to reduce the number of channels, resulting in a more efficient computational tensor. Simultaneously, a local attention mechanism is applied, using three different kernel sizes for expanded convolutions: 3×3, 5×5, and 7×7, with expansion rates of 1, 2, and 3, respectively. Subsequently, each attention map is activated by SiLU and then concatenated along the channel dimension. This process can be represented by the following formula: (5) (6) (7) In the lower branch of the module, a dedicated network is used to extract global attention by analyzing the long-range dependencies of the entire feature map. The feature matrix X2 is first normalized, then processed by a 5×5 convolution before being fed into a 2D Selective Scanning Layer (SS2D) model to scan the global information of each pixel in each spectral image in four directions (top to bottom, bottom to top, left to right, and right to left) to capture global correlation information. After passing through the SS2D layer, the output features are normalized again by layer normalization, and then multiplied element-wise with the initial normalized input features of this branch. This process can be represented by the following formula: (8) (9) The GT-Mamba module ultimately concatenates the outputs of the local and global branches along the channel dimension. This concatenation integrates local and global spatial information, providing the model with a comprehensive understanding of the input feature map. After concatenation, the channels are shuffled to facilitate interaction between different feature dimensions, further enhancing the model's expressive power. This process can be represented by the following formula: (10) (2) PPA-BiFPN multi-branch feature extraction and multi-scale fusion feature pyramid network In computer vision tasks such as object detection and semantic segmentation, feature fusion at different scales is a key step in improving model detection accuracy. Traditional Feature Pyramid Networks (FPNs) enhance the semantic representation of low-level features by fusing feature maps from top to bottom. However, FPNs struggle to transmit low-level information to higher levels. Therefore, the improved PAN-FPN path aggregation network structure was proposed, which adds a bottom-up path compared to FPN. However, the information acquired by a single node in PAN-FPN is limited, and the increased model parameters do more harm than good. Therefore, the weighted bidirectional feature pyramid network (BiFPN) structure was further proposed. PPA employs a parallel multi-branch feature capture method, with different scales and levels of feature extraction in each branch. This extraction strategy helps capture multi-scale features of objects, thereby improving the accuracy of detecting smaller and blurred targets. Therefore, combining the advantages of both approaches to propose "PPA-BiFPN" further enhances the model's ability to capture and process data feature information. The PPA-BiFPN structure can be referenced. Figure 4 As shown in (b), the PPA structure is as follows: Figure 6 As shown.

[0040] That is, the PPA-BiFPN composite structure includes PPA (Parallel Patch Aware Attention Module) and BiFPN (Dynamically Weighted Bidirectional Feature Pyramid Network). The advantages of BiFPN are as follows: If a node has only one input and no feature fusion, its contribution to the overall feature network is relatively small, and it can be removed; a skip connection can be added. If the original input and output nodes are at the same level, an extra edge can be added between the original input and output nodes to fuse more features without increasing the cost; The PPA network contains three parallel branches: local, global, and convolutional branches. Its structure is as follows: Figure 6 As shown. The processing flow is as follows. Based on a feature matrix First, it is processed through a convolution... Then, the three are calculated through the three branches respectively. , , The feature matrices are then added together to obtain .

[0041] After feature extraction, the design employs feature fusion and attention mechanisms for adaptive feature enhancement. The attention mechanism module mainly comprises a series of efficient channel attention and spatial attention mechanisms. This is achieved by applying one-dimensional channel attention and two-dimensional spatial attention separately. This process can be represented as: (11) (12) here and These represent ReLU and BatchNorm, respectively. This represents element-wise multiplication, with Mc and Ms representing channel attention and spatial attention, respectively.

[0042] (3) Bi-Fusion learnable weights dynamically adjust the multi-scale fusion module of features at different scales The Bi-Fusion module dynamically adjusts the importance of features at different scales through learnable weights. For feature images F1, F2, F3, ..., F n The input is a set of trainable weight parameters ω1, ω2, ω3, ..., ω. n Then, normalization is performed using the Swish function. Swish is a non-linear activation function, and the formula is shown below. (13) The corresponding learning weight formula is shown below: (14) Here, Wi represents the learnable feature fusion weights, and c is a very small number used to prevent the denominator from being 0.

[0043] The final multi-scale fusion matrix calculation formula is shown below: (15) The proposed Bi-Fusion module achieves multi-scale feature map fusion by first inputting feature maps. F i Multiply by the normalized weights, and then sum them up.

[0044] Evaluation indicators In the data processing model test, structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) were used to evaluate the performance of the SECycleGAN model.

[0045] Structural Similarity Index (SSIM) is a metric used to measure the similarity between two images in terms of brightness, contrast, and structural information. Its value ranges from 0 to 1; the closer to 1, the more similar the structures of the two images, and the better the detail is preserved after desmoking. The SSIM calculation consists of three parts: Luminance, Contrast, and Structure; the specific formulas are as follows: (16) In the formula above, l(x,y) represents brightness contrast; c(x,y) represents contrast ratio; and s(x,y) represents structure ratio. Generally, a value of 1 is used for weighting; Peak Signal-to-Noise Ratio (PSNR) assesses image quality loss by calculating the pixel value difference between a desmoked image and the original image. The unit is decibels (dB), with higher values ​​indicating better image quality. The calculation formula is as follows: PSNR is calculated based on Mean Squared Error (MSE): (17) (18) In the formula above x ij This represents the value at pixel position (i,j) in the original image; y ij This represents the value at the same location in the reconstructed image; M and N Indicates the width and height of the image; MAX I This represents the maximum possible pixel value in the image; Accuracy (P), Recall (R), Mean Precision (mAP50), and mAP 50-95 These metrics are used to comprehensively and accurately evaluate the performance of other YOLO models. The formulas for calculating accuracy, recall, and mean precision are as follows: (19) (20) (twenty one) Here's an explanation of TP and FN. Since sample prediction can include both positive and negative samples, TP means the sample was predicted to be positive and actually was, TN means it was predicted to be negative and actually was, i.e., the prediction was correct. FN means it was predicted to be negative but actually was, and FP means it was predicted to be positive but actually was negative.

[0046] Table 2 Prediction of Positive and Negative Samples

[0047] The following is a specific implementation technical route of the embodiments of the present invention: Implementation of the SECycleGan model for data preprocessing: The application scenario for this project is in a smoky, low-light environment, therefore the requirements for the model are relatively high. The original CycleGan model, when preprocessing feature images, can cause some damage to the target person; for example, it may remove the target person along with the smoke during smoke removal. Figure 7a As shown, the original CycleGan model cannot meet the design requirements of this project.

[0048] Therefore, to address the problems described above, our team designed the SECYcleGAN model, which aims to preserve as much of the target person's feature information as possible while removing interfering background elements such as smoke. Figure 7b As shown, the SECycleGan model fully leverages the performance of the self-attention mechanism by embedding it before and after the ResNet residual block in the CycleGan model's generator—that is, after the downsampling layer and before upsampling—to enhance the global representation of features and the ability to capture image features. The generator part of the improved SECycleGan model is shown below. Figure 8 As shown.

[0049] Implementation of the MambaPPA-MuYolo model for object detection The model is an improvement on YOLOv8. Considering the need to load multimodal data and integrate GT-Mamba and PPA models, the backbone and neck parts of the original network were redesigned to enhance the model's feature extraction capabilities and practical performance. The following is a detailed explanation of the MambPPA-MuYolo implementation.

[0050] Backbone network section In the backbone, the network adopts a dual-stream loading structure and designs a three-level multimodal feature fusion layer named "Mu-Fusion". The operating mechanism of each Mu-Fusion layer is as follows. The backbone structure is as follows: Figure 9As shown.

[0051] The feature matrices loaded from the dual-branch backbone network pass through layers P0-P2 and then enter the feature fusion layer "Mu-Fusion". Here, the RGB-loaded matrix is ​​denoted as X1, and the infrared-loaded matrix as X2. Upon entering "Mu-Fusion", X1 and X2 are first fused and concatenated along the first dimension to obtain feature matrix X3. Then, X3 is fed into a feature extractor "Attention" structure, as shown below. Figure 10 As shown, this module enhances the network's ability to capture feature matrix information while promoting the interaction and fusion of data from different modalities. The mathematical expression of the feature extractor "Attention" is as follows: (twenty two) Where S c This indicates a channel. c The average eigenvalue represents the importance of the channel in the global context. H and W Here, C represents the height and width of the feature image, and C represents the number of channels. Then, a two-layer fully connected (FC) structure is used to model the channel relationships and calculate the channel weights W on the matrix after global average pooling. The first FC layer performs dimensionality reduction, decreasing the number of channels to C / r, followed by ReLU activation. Then, the second FC layer performs dimensionality increase, restoring the number of channels to the original number C. The formula for the channel weights W is as follows: (twenty three) W1 and W2 This is the weight matrix of the fully connected layer, and r is the channel compression size. δ The expression represents the ReLU activation function, and σ represents the Sigmoid activation function that normalizes the weights to the range [0,1]. The final channel weights are then obtained. W When applied to the input feature matrix X, the formula is as follows: (twenty four) To further enhance the information exchange between channels and better obtain image features, the channels of the feature matrix Y obtained above need to be segmented and rearranged: (25) (26) (27) Finally, residuals are used to connect the input feature matrix X with the fused feature matrix. Yfused Add them together: (28) The processed feature matrix is ​​as follows:X 4. Finally, X 4 is obtained by splitting it into two according to the first dimension. X 5 and X 6. Then... X 5 and X 6 is fed as a new feature matrix into the next feature fusion layer. The Backbone network with the above-designed dual-stream structure possesses three levels of multimodal feature fusion layers at different scales, which greatly promotes the capture of information from feature images.

[0052] (29) (30) here Attention This represents the feature extractor; Split indicates that the matrix is ​​divided, dim = 1 indicates that it is divided according to the first dimension.

[0053] Neck network section In the Neck section, this paper has three design elements, the structure of which is as follows: Figure 11 As shown.

[0054] First, the design of the network structure. Addressing the problems of low feature reuse efficiency and significant cross-layer information flow attenuation in the traditional PAN-FPN structure, this study proposes an improved bidirectional feature pyramid network, PPA-BiFPN, based on multi-branch feature extraction. In this paper, the Neck of the original PAN-FPN structure is replaced with the PPA-BiFPN structure.

[0055] The second GT-Mamba module is incorporated. Based on the improved SS-Conv-SSM (Spatial-Semantic Convolutional State Space Model) framework, it constructs a dual-stream heterogeneous feature processing path with local spatial attention branches. This design innovatively deploys this module at the Backbone and Neck feature transfer hubs of the object recognition network. By constructing a space-state joint modeling mechanism, it achieves synergistic optimization of global semantic association and local detail enhancement across different feature levels.

[0056] The third is the embedding of the learnable weights dynamically adjusted multi-scale fusion module Bi-Fusion. Addressing the issues of feature space alignment bias and insufficient information exchange in traditional feature concatenation (Concat) operations for multimodal data fusion, this study innovatively designs a dynamic feature fusion module (Bi-Fusion) and deeply integrates it into the PPA-BiFPN network architecture, placing it at the original Concat location, thus breaking through the static fusion paradigm of traditional Concat operations.

[0057] Overall system processing flow The operation flow of the multimodal human recognition system for dense smoke perception designed in this embodiment of the invention is as follows: the input end receives raw image data and performs preprocessing operations on the data, such as using the SECYcleGAN model proposed by our team to remove smoke from the image data. Next, the preprocessed image data is fed into the Backbone network of the MambaPPA-MuYolo model designed by our team for feature information extraction. The Backbone network mainly consists of a Conv (CBS) module, a C2f module, and a Mu-Fusion multimodal fusion module. The feature information extracted by the Backbone is fed into the Neck network from four different network layers. This module is responsible for fusing the multi-scale features extracted by the Backbone. First, the feature information is processed by GT-Mamba and then fed into the multi-scale feature fusion network with a BIFPN structure, where the fusion module is Bi-Fusion, and after fusion, it enters the PPA multi-branch feature extraction network for feature information capture.

[0058] Finally, the features processed by the Neck module are passed to the Head module for final regression prediction. The Head module uses the feature maps extracted from the Backbone to detect the location and category of the target. By training and optimizing the network parameters of the Head part, the model is better adapted to the target detection task. The flowchart is as follows: Figure 12 As shown; Experimental data analysis The data preprocessing model proposed in this invention integrates self-attention and CycleGAN, incorporating the self-attention mechanism into the CycleGAN generator. Simultaneously, the proposed target recognition model, MambaPPA-MuYolo, integrates the Mamba mechanism, introduces a dual-branch multimodal loaded backbone network, and improves the Neck network structure. To reasonably evaluate the two proposed models, corresponding ablation experiments and comparative experiments are designed to better demonstrate the feasibility of the models designed in this paper.

[0059] ablation experiment The ablation experimental results of the MumbaPPA-MuYolo model on LLVIP multimodal data are shown in Table 3 below.

[0060] Table 3. Ablation Experiment Results of MambaPPA-MuYolo LLVIP Data

[0061] Note: × indicates that this improvement method is not used, √ indicates that this improvement method is used, P represents accuracy, mAP 50This represents the mean precision (mAP) when the IoU threshold is 0.5. 50-95 This represents the average precision when the IoU threshold ranges from 0.5 to 0.95, as shown in the following table.

[0062] The ablation experiment results of the MumbaPPA-MuYolo model on multimodal smoke data are shown in Table 4 below.

[0063] Table 4. Ablation Experiment Results of MambaPPA-MuYolo Smoke Data

[0064] Combining the data in Tables 3 and 4, it can be seen that the proposed method improves the human recognition performance in harsh environments with dense smoke from 46.9% to 61.7% on the LLVIP dataset (mAP@0.5:0.95) and from 57.2% to 87.4% on the dense smoke dataset, proving the effectiveness of the model improvement measures. Compared with the original model, the addition of the Mu-Fusion multimodal feature fusion network improves the performance (mAP@0.5:0.95) by 13.3 percentage points on the LLVIP dataset, the embedding of GT-Mamba improves it by 0.6 percentage points, and the integration of the PPA multi-branch feature extraction network further improves the model performance by 0.9 percentage points. This demonstrates that through multimodal data fusion, Mamba data capture, and PPA multi-branch data extraction, the semantic information contained in the feature maps is enriched, enhancing the expressive power of the features.

[0065] Comparative experiment The performance comparison between the SECycleGan data preprocessing model and the original CycleGan model is shown in Table 5 below.

[0066] Table 5 Comparison and Evaluation Indicators of Smoke Removal Algorithms

[0067] The results in Table 5, obtained through repeated verification with a large amount of data, show that the proposed SECYcleGan model improves the SSIM structural similarity by 3.17 percentage points and the PSNR by 1.63 dB compared to the original CycleGan model for smoke clarity processing. This demonstrates the feasibility of the SECYcleGan model.

[0068] The comparative experimental results of the MumbaPPA-MuYolo model on LLVIP multimodal data are shown in Table 6 below.

[0069] Table 6. Comparison Results of LLVIP Data Experiments

[0070] Note: P represents precision, R represents recall, and mAP represents accuracy. 50 This represents the mean precision (mAP) when the IoU threshold is 0.5. 50-95 This represents the average precision when the IoU threshold ranges from 0.5 to 0.95, as shown in the following table.

[0071] The comparative experimental results of the MumbaPPA-MuYolo model on multimodal data of dense smoke are shown in Table 7 below.

[0072] Table 7. Experimental Comparison Results of Smoke Data

[0073] The data in Tables 6 and 7 above show that the proposed method improves the human recognition performance of the model in dense smoke environments from 57.2% to 87.4% for mAP@0.5:0.95, and from 84.5% to 99.2% for mAP@0.5:0.95. For LLVIP data, the mAP@0.5:0.95 value increases from 46.9% to 61.7%, and the mAP@0.5 value rises from 85.0% to 95.9%. This significant improvement over other models is attributed to the MambaPPA-MuYolo model's dual-branch multimodal data extraction backbone network and its inheritance of advanced algorithms from the Mamba and PPA models. This enhances the model's ability to extract features and capture global information, enabling it to maintain high-precision human target recognition even in harsh environments with limited visibility.

[0074] In summary, the embodiments of the present invention, based on target recognition and image generation and conversion technologies, have achieved significant results in the field of multimodal human target discrimination in dense smoke environments, and have the following outstanding features and performance advantages: Excellent data preprocessing performance: In the data preprocessing section, our system independently developed the SECYcleGAN model. This model combines the unsupervised image transformation capability of the CycleGAN model with the excellent feature information capture capability of the self-attention mechanism, thereby improving the ability to sharpen dense smoke images.

[0075] 2. Highly efficient target detection capability: The object detection network in this system adopts a multi-network fusion approach, using a self-designed MambaPPA-MuYolo model. This model possesses multimodal loading and fusion capabilities while inheriting the powerful processing power of the Mamba model. A BiFPN network structure is introduced in the neck part, and the multi-layer fusion module is replaced with a Bi-Fusion module, followed by a PPA multi-branch feature extraction network. Experiments demonstrate that the object detection model used in this system outperforms the original object recognition model.

[0076] 3. Visualized deployment of edge intelligence platform: This system deploys the model on an edge intelligence platform. This deployment brings the theoretical model to life, and its excellent hardware support accelerates model inference, providing faster response times, higher stability, and support for offline operation, making it applicable to other object detection scenarios.

[0077] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0078] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0079] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the dual-engine collaborative cross-modal smoke occlusion human body recognition methods described above.

[0080] It is understood that the systems, devices, and storage media provided in the embodiments of the present invention correspond to the methods provided in the embodiments of the present invention, and the explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

[0081] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0082] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0083] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0084] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-modal smoke-occluded human identification method based on dual-engine collaboration, characterized in that, The model consists of two parts: model building and platform loading. The model building uses the self-attention enhanced recurrent generative adversarial network SEcycleGan and the MambaPPA-MuYolo model to conduct experiments on human recognition in a dense smoke environment. The model input uses a data preprocessing framework based on the self-attention enhanced recurrent generative adversarial network SECYcleGAN to remove smoke from visible light data, thereby optimizing the human detection effect in dense smoke environments. In the model architecture, the backbone network uses dual-branch data loading and feature-level fusion to fully integrate feature information from both modalities. The neck network is a bidirectional feature pyramid structure with BIFPN multi-scale fusion. A gated lightweight Mamba GT-Mamba feature information acquisition module containing local spatial multi-core attention branches and a state space model based on selective scanning, along with multiple multi-branch feature extraction modules (PPA) to improve small target detection, are embedded into the Neck module. These modules are used to enhance spatial information capture and improve information interaction capabilities between different layers, respectively. The platform loading process utilizes edge intelligent computing development equipment to configure the environment, loads the optimized MambaPPA-MuYolo model into the JupyterLab environment of the development board for model inference, and finally uses the USB camera of the edge computing platform to capture smoke image frames and call the model for detection.

2. The dual-engine synergy based cross-modal smog-occluded human recognition method according to claim 1, characterized in that: Model building includes, First, the backbone network is reconstructed by dual-stream heterogeneous feature fusion, and the original single-modal feature extraction architecture is extended into a dual-channel processing structure that supports RGB and enhanced features. The intermodal feature complementarity is achieved by constructing a three-level cross-modal feature fusion module. Secondly, a gated lightweight Mamba attention mechanism is introduced into the connection layer between the backbone network and the feature pyramid, and its global context modeling capability is used to recalibrate the multi-scale features output by the backbone network. Finally, bidirectional feature pyramid optimization is performed on the neck network to upgrade the traditional PAN-FPN structure to a BiFPN architecture that includes a PPA module, thereby enhancing robustness in complex backgrounds through multi-branch parallel feature extraction.

3. The cross-modal smoke occlusion human body recognition method based on dual-engine collaboration according to claim 2, characterized in that: The input end receives raw image data and uses a multimodal data preprocessing framework based on the self-attention enhanced recurrent generative adversarial network SECycleGAN to preprocess the data, i.e., perform smoke removal. Next, the preprocessed image data is fed into a Backbone network based on the MambaPPA-MuYolo model for feature extraction; The Backbone network consists of a Conv module, a C2f module, and a Mu-Fusion multimodal fusion module. The feature information extracted by the Backbone network is fed into the Neck network from four different network layers. This module is responsible for fusing the multi-scale features extracted by the Backbone. First, the feature information is processed by the gated lightweight Mamba (GT-Mamba) and then fed into the multi-scale feature fusion network with the BIFPN structure. The multi-scale fusion module uses learnable weights (Bi-Fusion) to assign corresponding weights to data at different levels and fuses them. After fusion, the data enters the PPA multi-branch feature extraction network to capture feature information. Finally, the features processed by the Neck module are passed to the Head module for final regression prediction. The Head module uses the feature maps extracted from the Backbone to detect the location and category of the target. By training and optimizing the network parameters of the Head part, the model can be better adapted to the target detection task.

4. The dual-engine synergy based cross-modal smog-occluded human identification method according to claim 2, characterized in that: The relevant formula for the gated lightweight Mamba GT-Mamba attention mechanism is as follows: （1） （2） （3） （4） In formula (1) LayNoem This indicates normalization processing; Wu This represents a learnable linear projection matrix; Wo Represents the linear output projection matrix; SSMConvolution represents applying the SSM state-space model to the entire sequence input, where parameter A represents the state transition matrix, B represents the input control matrix, and C represents the output projection matrix.

5. The cross-modal smoke-covered human body recognition method based on dual-engine collaboration according to claim 4, characterized in that: The gated lightweight Mamba GT-Mamba module consists of two parallel branches: One is the local spatial multi-core attention branch, and the other is the main Mamba branch of this module; In the upper branch network of the module, the focus is on acquiring local spatial information at multiple scales, specifically for a dimension of... The feature matrix X1 is first normalized using BatchNorm, and then enters a local spatial attention layer. This layer first uses 1×1 convolutions to reduce channels, resulting in a more efficient computational tensor. Simultaneously, a multi-kernel local attention mechanism is applied, using three different kernel sizes (3×3, 5×5, and 7×7) for expanded convolutions, with expansion rates of 1, 2, and 3. Subsequently, each attention map is activated by SiLU and then concatenated along the channel dimension. This process is represented by the following formula: （5） （6） （7） In the lower branch network of the module, global attention is extracted by analyzing the long-distance dependencies of the entire feature map. The feature matrix X2 is first normalized and then fed into the 2D selection scanning layer model to scan the global information of each pixel in each spectral image in four directions, namely from top to bottom, from bottom to top, from left to right, and from right to left, in order to capture global correlation information. After selecting the scan layer in 2D, the output features are normalized again by layer normalization, and then multiplied element-wise with the initial normalized input features of the branch. This process can be represented by the following formula: （8） （9） The gated lightweight Mamba GT-Mamba module ultimately concatenates the outputs of the local and global branches along the channel dimension. This concatenation integrates local and global spatial information, providing the model with a comprehensive understanding of the input feature map. After concatenation, the channels are shuffled to promote interaction between different feature dimensions, further enhancing the model's expressive power. This process is represented by the following formula: （10）。 6. The cross-modal smoke-covered human body recognition method based on dual-engine collaboration according to claim 5, characterized in that: The PPA-BiFPN composite structure includes a PPA network and a BiFPN network; The PPA network contains three parallel branches, involving local, global, and convolutional branch structures. The processing flow includes features based on a feature matrix. First, it is processed through a convolution. Then, the three are calculated through the three branches respectively. , , The feature matrices are then added together to obtain ; After feature extraction, the design employs feature fusion and attention mechanisms for adaptive feature enhancement. This process is mathematically represented by processing through a one-dimensional channel attention and a two-dimensional spatial attention, respectively: （11） （12） here and These represent ReLU and BatchNorm, respectively. M represents element-wise multiplication. c and M s F represents channel attention and spatial attention, respectively. c With F s These represent the features after channel selection; BiFPN, or Dynamically Weighted Bidirectional Feature Pyramid Network, incorporates skip connections. If the original input and output nodes are at the same level, an extra edge is added between them to fuse more features without increasing cost. If a node has only one input and no feature fusion occurs, its contribution to the overall feature network is minimal, and it is removed. Furthermore, instead of the traditional multi-layer fusion strategy of directly stacking the same weight dimensions, this part uses a Bi-Fusion module based on learnable weights to dynamically adjust features at different scales, assigning appropriate weights to different inputs so that the network can better highlight the captured data information.

7. The cross-modal smoke-covered human body recognition method based on dual-engine collaboration according to claim 6, characterized in that: The Bi-Fusion module dynamically adjusts the importance of different scale features through learnable weights. For feature images F1, F2, F3, …, F n The Bi-Fusion module dynamically adjusts the importance of different scale features through learnable weights. For feature images F1, F2, F3, …, F n , and is normalized by a Swish function, which is a nonlinear activation function and is shown in the following formula. （13） The corresponding learning weight formula is shown below: （14） Here, Wi represents the learnable feature fusion weights, and c is a preset constant used to prevent the denominator from being 0; The final multi-scale fusion matrix calculation formula is shown below: （15） The proposed Bi-Fusion module achieves multi-scale feature map fusion by first inputting feature maps. F i Multiply by the normalized weights, and then sum them up.

8. A computer-readable storage device storing a computer program, characterized in that, When the computer program is executed by a processor, the processor performs the steps of the method as described in any one of claims 1 to 6.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, the processor performs the steps of the method as described in any one of claims 1 to 6.

Images