Human body recognition method and device in smoke scene, terminal equipment and medium

Abstract

The invention provides a human body recognition method and device in a smoke scene, terminal equipment and a medium. The method comprises the following steps: synchronously acquiring a multi-modal smoke image sample in the smoke scene; training a pre-constructed smog human body recognition model by using the multi-modal smog image sample until the loss value of the smog human body recognition model is smaller than a preset loss threshold, and obtaining a trained smog human body recognition model; and inputting collected smoke image data to be recognized into the trained smoke human body recognition model to obtain a human body recognition result corresponding to the smoke image data to be recognized. The accuracy of human body recognition in the smoke scene can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and recognition technology, and specifically relates to a method, device, terminal equipment and medium for human body recognition in a smoke scene. Background Art

[0002] In a fire, a smoke-filled environment will greatly affect rescue efforts. Human target recognition in a smoke-filled environment is of great significance to improving fire rescue efficiency and people's life safety.

[0003] Traditional human recognition algorithms in smoky scenes, such as dark channel prior algorithms and convolutional neural networks, can only perform to a limited or specific degree in dense smoke environments. The main reason why dark channel prior algorithms are so limited in fog processing is that they inaccurately estimate transmittance in these conditions, making it difficult to completely remove the smoke, which in turn affects human recognition accuracy. Convolutional neural networks, as black-box models, make it difficult to intuitively understand how the model removes smoke and restores images. This poor interpretability makes it difficult to flexibly adjust the model, which in turn affects human recognition accuracy.

[0004] Therefore, there is an urgent need for a method that can perform accurate human body recognition in smoke scenes. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method, apparatus, terminal device and medium for human body recognition in a smoke scene, so as to improve the accuracy of human body recognition in a smoke scene.

[0006] In a first aspect, the present invention provides a method for human body recognition in a smoke scene, the method comprising the following steps:

[0007] Synchronously collecting multimodal smoke image samples in a smoke scene; the multimodal smoke image samples include at least visual image samples and brightness temperature image samples;

[0008] A pre-built smoke human recognition model is trained using multimodal smoke image samples until the loss value of the smoke human recognition model is less than a preset loss threshold, thereby obtaining a trained smoke human recognition model. The smoke human recognition model includes a feature extraction module for extracting multi-scale features from smoke image data, a feature fusion module for fusing the extracted multi-scale features, a generative adversarial module for generating a smoke-free image based on the multi-scale feature fusion results, and a human recognition module for identifying the smoke-free image. The loss value is used to measure the quality of the smoke human recognition model's smoke removal effect and the accuracy of human recognition.

[0009] The collected smoke image data to be identified is input into the trained smoke human body recognition model to obtain the human body recognition result corresponding to the smoke image data to be identified.

[0010] Optionally, a pre-built smoke human recognition model is trained using multimodal smoke image samples until a loss value of the smoke human recognition model is less than a preset loss threshold, thereby obtaining a trained smoke human recognition model, including:

[0011] Convolutional neural networks are used to extract shallow features of multimodal smoke image samples. Shallow features are low-level visual information, mainly corresponding to local edges, textures, brightness changes, and simple shape structures in the image.

[0012] The residual network is used to extract deep features of multimodal smoke image samples from shallow features. The deep feature representation models the global attributes of the human body structure, spatial relationships between parts, and posture and movement in the entire image.

[0013] Reduce the dimensionality of deep features and generate multi-scale feature maps;

[0014] Perform channel splicing on the multi-scale feature map to obtain the multi-scale feature fusion result;

[0015] Generate adversarial networks to reconstruct the desmoked image data corresponding to the multi-scale feature fusion results, and calculate the reconstruction loss between the desmoked image data and the pre-acquired real smoke-free image data;

[0016] Extract the image features of the smoke-free image data and globally pool the image features to obtain the recognition vector for human recognition;

[0017] Obtaining human body recognition results corresponding to multimodal smoke image samples based on recognition vectors;

[0018] Calculate the distribution loss corresponding to the human recognition result;

[0019] The smoke human body recognition model is trained according to the reconstruction loss, distribution loss and Adam optimizer to obtain the trained smoke human body recognition model.

[0020] Optionally, channel concatenation is performed on the multi-scale feature maps to obtain a multi-scale feature fusion result, including:

[0021] By calculating the formula

[0022] F z =F l *M l +F m *M m +F h *M h

[0023] Among them, F z Represents the multi-scale feature fusion result, F l represents the low-scale feature map extracted from the multimodal smoke image sample, represents the medium-scale feature map extracted from the multimodal smoke image sample, and F h represents the high-scale feature map extracted from the multimodal smoke image sample, M l Represents the feature response weight map corresponding to the low-scale feature map, which is used to measure the importance of low-scale features in the current input image. m Represents the feature response weight map corresponding to the mesoscale feature map, which is used to measure the importance of the mesoscale feature in the current input image. h Represents the feature response weight map corresponding to the high-scale feature map, which is used to measure the importance of high-scale features in the current input image.

[0024] Optionally, the loss value is calculated as

[0025] L=λ m L mse +λ g L gan

[0026] L mse =y I -G I 9x IQUP )2

[0027] L gan =-logD9G I (x IQUP ))

[0028] Among them, L represents the loss value, L mse Denotes the distribution loss, L gan represents the reconstruction loss, λ m ,λ g Represents the weighted coefficient of distribution loss and reconstruction loss in the total loss, y I Denotes the real smoke-free image sample corresponding to the current input image, which is used as the supervision signal, G I (x IQUP ) represents the generator G I The fused multi-scale image features x IQUP The generated desmog image, D represents the discriminator module in the generative adversarial network, which is used to evaluate whether the desmog image is close to the real image.

[0029] Optionally, a generative adversarial network is used to reconstruct the desmog image data corresponding to the multi-scale feature fusion result, including:

[0030] A generative adversarial network (GAN) architecture is constructed. The GAN consists of two submodules: a generator and a discriminator. The generator takes the result of multi-scale feature fusion as input and uses an encoder-decoder structure to reconstruct the image. The encoder extracts deep semantic information from the fused features through multi-layer convolution and gradually compresses the spatial resolution. The decoder restores the image's spatial structure through deconvolution or upsampling operations, while using skip connections to transfer low-level edge information to preserve image details. The discriminator takes the real smoke-free image and the generated image as input, uses a convolutional discriminant structure to extract discriminative features, and outputs true or false probabilities to evaluate the authenticity of the generated image.

[0031] By jointly training the generator and discriminator, the reconstruction error between the generated image and the real image is minimized while maximizing its true probability in the discriminator, thereby improving the visual quality and semantic consistency of the desmog image and enhancing the recognizability of the human target area.

[0032] Optionally, the convolutional neural network includes multiple sequentially connected feature extraction submodules; the feature extraction submodule includes a Conv layer, a BN layer, and a ReLU layer.

[0033] Optionally, the residual network is ResNet-18.

[0034] In a second aspect, the present invention provides a human body recognition device in a smoke scene, comprising:

[0035] A sample acquisition module is used to synchronously acquire multimodal smoke image samples in a smoke scene; the multimodal smoke image samples include at least visual image samples and brightness temperature image samples;

[0036] A model training module is used to train a pre-built smoke human recognition model using multimodal smoke image samples until the loss value of the smoke human recognition model is less than a preset loss threshold, thereby obtaining a trained smoke human recognition model; the smoke human recognition model includes a feature extraction module for extracting multi-scale features from smoke image data, a feature fusion module for fusing the extracted multi-scale features, a generative adversarial module for generating a smoke-free image based on the multi-scale feature fusion results, and a human recognition module for identifying the smoke-free image. The loss value is used to measure the smoke removal effect of the smoke human recognition model and the accuracy of human recognition;

[0037] The human body recognition module is used to input the collected smoke image data to be identified into the trained smoke human body recognition model to obtain the human body recognition result corresponding to the smoke image data to be identified.

[0038] In a third aspect, the present invention provides a terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method when executing the computer program.

[0039] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, which implements the above method when executed by a processor.

[0040] The beneficial effects of the present invention are:

[0041] The present invention provides a method for human body recognition in a smog scene. A smog human body recognition model constructed by the method utilizes a feature extraction module to perform multi-scale feature extraction on smog image data, utilizes a feature fusion module to fuse the extracted multi-scale features, and then utilizes a generative adversarial module to generate a desmog image based on the multi-scale feature fusion result. The multi-scale features are fused by utilizing the difference in sensitivity of smoke image data of different modalities to smoke, which is beneficial to improving the accuracy of the multi-scale feature fusion result, thereby improving the excellence of the desmog effect and obtaining a clear desmog image. Furthermore, human body recognition is performed based on the clear desmog image, which can avoid the interference of thick smoke and fog on human body recognition, and significantly improve the accuracy of human body recognition in smog scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flow chart of a method for human body recognition in a smoke scene in one embodiment of the present application;

[0043] Figure 2 This is a structural diagram of a smoke human body recognition model in one embodiment of the present application;

[0044] Figure 3 This is a structural diagram of a human body recognition device in a smoke scene in one embodiment of the present application;

[0045] Figure 4 This is a schematic diagram of the structure of a terminal device in one embodiment of the present application. DETAILED DESCRIPTION

[0046] In response to the problem that traditional human recognition methods have low accuracy in smoke scenes, the present invention provides a human recognition method, device, terminal equipment and medium in smoke scenes. The smoke human recognition model constructed by this method uses a feature extraction module to extract multi-scale features from smoke image data, uses a feature fusion module to fuse the extracted multi-scale features, and then uses a generative adversarial module to generate a smoke-free image based on the multi-scale feature fusion results. The multi-scale features are fused by utilizing the difference in sensitivity of smoke image data of different modalities to smoke, which is conducive to improving the accuracy of the multi-scale feature fusion results, thereby improving the excellence of the smoke removal effect and obtaining a clear smoke-free image. Furthermore, human recognition based on the clear smoke-free image can avoid the interference of thick smoke and fog on human recognition, and significantly improve the accuracy of human recognition in smoke scenes.

[0047] The following combination Figure 1 The human body recognition method in a smoke scene in this embodiment is described.

[0048] like Figure 1 As shown, the human body recognition method in a smoke scene provided by the present invention includes steps 11 to 13.

[0049] Step 11: synchronously collect multimodal smoke image samples in the smoke scene.

[0050] In an embodiment of the present invention, the multimodal smoke image samples include at least visual image samples and brightness temperature image samples. Exemplarily, the visual image samples are RGB image samples, and the brightness temperature image samples are infrared images, which can be collected using a visual camera and an infrared camera, respectively, in different smoke scenarios. Smoke scenarios refer to scenarios where people are evacuated from crowded places such as hospitals, shopping malls, and office buildings when a fire or heavy smoke occurs. Human body recognition in these scenarios is used to accurately locate trapped individuals, enabling timely and accurate rescue and ensuring their personal safety.

[0051] To ensure data consistency in subsequent processing steps, high-precision clock synchronization equipment can be used to provide unified timing for multimodal data acquisition devices, such as visual and infrared data, ensuring precise alignment of data in the temporal dimension. Simultaneously, a timestamp calibration algorithm is designed within the data acquisition system to monitor and adjust the time offset of each modality in real time.

[0052] In one feasible implementation, the multimodal smoke image samples also include video data. The present invention implements keyframe sampling on the video data to capture video keyframes to reduce interference from redundant data. Exemplarily, video keyframes can be acquired at fixed intervals. It should be understood that video keyframes are essentially a form of image data.

[0053] In another feasible implementation, multimodal smoke image samples in smoke scenes may be obtained from a public dataset.

[0054] It's worth noting that the present invention uses multimodal smoke image samples as training data, which can overcome the limitations of traditional human recognition methods that rely on single-mode visible light or infrared light. This is because when using single-mode visible light for human recognition, smoke obscures the human outline, significantly interfering with recognition. Conversely, when using single-mode infrared light for human recognition, the combined effects of thermal radiation signal attenuation and smoke particle scattering can cause deviations in the captured infrared image, reducing recognition accuracy.

[0055] Step 12: Use multimodal smoke image samples to train the pre-built smoke human recognition model until the loss value of the smoke human recognition model is less than a preset loss threshold, thereby obtaining a trained smoke human recognition model.

[0056] like Figure 2 As shown, in an embodiment of the present invention, the smoke human body recognition model includes a feature extraction module 21, a feature fusion module 22, a generative adversarial module 23, and a human body recognition module 24 which are connected in sequence.

[0057] The functions of each module are described below.

[0058] The feature extraction module 21 is used to perform multi-scale feature extraction on smoke image data.

[0059] Specifically, a convolutional neural network is used to extract shallow features of multimodal smoke image samples, and a residual network is used to extract deep features of multimodal smoke image samples from the shallow features.

[0060] Shallow features are low-level visual information extracted from images by the first few layers of a convolutional neural network. They primarily correspond to local edges, textures, brightness variations, and simple shape structures within the image. Physically, shallow features reflect the most intuitive and primitive visual elements in an image, such as the boundary between the human body and the background, texture details of clothing, grayscale variations in different regions, and differences in local illumination or thermal intensity. In smoky scenes, despite occlusion affecting overall image quality, these low-level features can still retain some key geometric and contrast information, helping the model initially identify possible human body regions and providing foundational support for subsequent feature extraction and discrimination.

[0061] Deep features are high-level semantic information extracted and abstracted from shallow features by deeper neural network structures (such as residual networks). Their physical meaning is primarily reflected in the modeling of global attributes such as human structure, spatial relationships between body parts, and posture and motion across the entire image. Deep features no longer focus on pixel-level details, but instead represent the semantic attributes of human targets in a more abstract and generalized manner. For example, they can represent the compositional relationships between the head, torso, and limbs, the overall contour trends of the human body, or common regions extracted between different modal images (such as visible light and brightness temperature images). Furthermore, in the process of deconstructing a human body after deconstruction using a generative adversarial network, deep features can also characterize salient regions of the human body after deconstruction—key parts that are more easily distinguished from the background visually or thermally. Deep features play a crucial role in improving recognition accuracy under complex occlusion conditions because they carry a higher-level semantic representation of the target concept "human."

[0062] In an embodiment of the present invention, the convolutional neural network includes a plurality of sequentially connected feature extraction submodules; the feature extraction submodule includes a Conv layer, a BN layer, and a ReLU layer; and the residual network is ResNet-18.

[0063] In a feasible implementation, features of the visual image can also be extracted through a pre-configured visual encoder (ViT, Vision Transformer). The extracted features include but are not limited to human body geometric features (such as human body contour, limb length ratio, etc.), texture features (skin texture, facial expression, etc.), and context features (the relative position relationship between the human body and the surrounding environment); features of the brightness temperature image can be extracted through a U-Net network. The extracted features include but are not limited to thermal radiation value and penetrability features.

[0064] To reduce computational complexity and improve performance, after obtaining deep features, the deep features are further reduced in dimension to generate multi-scale feature maps. For example, in this embodiment of the present invention, by reducing the dimension of the deep features, feature maps at three scales (low (1 / 4), medium (1 / 2), and high (1*1)) are obtained.

[0065] The feature fusion module 22 is used to fuse the extracted multi-scale features.

[0066] It should be understood that different modal images exhibit different physical properties in smoke scenarios. Infrared images can penetrate smoke, thermal images can capture heat sources within the smoke, and visual images can capture contour information. Therefore, fusing visual and infrared images can help recover objects obscured by smoke, thereby improving the accuracy of human recognition. By fusing thermal and infrared images, objects obscured by smoke can be recovered from different angles, especially in high-temperature areas, where thermal imaging can accurately capture the outline of a person.

[0067] Specifically, in the embodiment of the present invention, the process of fusing the extracted multi-scale features is as follows: channel splicing is performed on the multi-scale feature map to obtain a multi-scale feature fusion result. This process can be expressed as:

[0068] By calculating the formula

[0069] F z =F l *M l +G m *M m +G h *M h

[0070] Among them, G z Represents the multi-scale feature fusion result, F l represents the low-scale feature map extracted from the multimodal smoke image sample, represents the medium-scale feature map extracted from the multimodal smoke image sample, and F h represents the high-scale feature map extracted from the multimodal smoke image sample, M l Represents the feature response weight map corresponding to the low-scale feature map, which is used to measure the importance of low-scale features in the current input image. m Represents the feature response weight map corresponding to the mesoscale feature map, which is used to measure the importance of the mesoscale feature in the current input image. h Represents the feature response weight map corresponding to the high-scale feature map, which is used to measure the importance of high-scale features in the current input image.

[0071] In another feasible implementation, multi-scale features can also be fused based on a dynamic gated attention mechanism.

[0072] Specifically, first, the similarity matrix between feature maps of each scale is calculated.

[0073] Then, dynamic weights are generated by gated LSTM. The formula is: W gate =σ(LSTM([F l ,F m ,F h ])).

[0074] Finally, by calculating the formula Get the multi-scale feature fusion result F z Among them, F S Represents feature maps of different scales, Represents the dynamically generated gating weight matrix, which is used to measure the contribution weight of each scale feature.

[0075] The generative adversarial module 23 is used to generate a smoke-free image based on the multi-scale feature fusion result.

[0076] In order to reduce the impact of smoke on human body recognition, in an embodiment of the present invention, a generative adversarial network is used to reconstruct the smoke-free image data corresponding to the multi-scale feature fusion result.

[0077] The specific process is as follows: constructing a generative adversarial network structure, which includes two sub-modules: a generator and a discriminator. The generator takes the multi-scale feature fusion result as input and uses an encoder-decoder structure to reconstruct the image. The encoder extracts deep semantic information from the fused features through multi-layer convolution and gradually compresses the spatial resolution. The decoder restores the image spatial structure through deconvolution or upsampling operations, while using jump connections to transfer low-level edge information to preserve image details, and finally outputs a smoke-free image. The discriminator takes the real smoke-free image and the generated image as input, uses a convolutional discriminant structure to extract discriminative features and outputs true or false probabilities to evaluate the authenticity of the generated image.

[0078] By jointly training the generator and discriminator, the reconstruction error between the generated image and the real image is minimized while maximizing its true probability in the discriminator, thereby improving the visual quality and semantic consistency of the desmog image and enhancing the recognizability of the human target area.

[0079] The human body recognition module 24 is used to recognize the smoke-free image and obtain a human body recognition result.

[0080] Specifically, in an embodiment of the present invention, the desmog image is recognized through a fully connected layer network to obtain a human body recognition result. The process includes: extracting image features of the desmog image data, and globally pooling the image features to obtain a recognition vector for human body recognition, and then obtaining a human body recognition result corresponding to the multimodal smoke image sample based on the recognition vector. The human body recognition result includes the target human body bounding box and the category score. Among them, the category score is used to represent the model's confidence in the recognition result, that is, the probability that the recognized target belongs to the "human body" category. For example, when the category score is 0.92, it means that the model's confidence in judging that the area is a human body is 92%; when the category score is 0.45, it means that the model is not sure enough about the judgment of the recognition result, and the confidence is only 45%. The category score can be used for subsequent target screening or alarm trigger threshold judgment.

[0081] The following describes the training process of the smoke human body recognition model in an embodiment of the present invention. In this embodiment, the smoke human body recognition model is trained based on reconstruction loss, distribution loss, and Adam optimizer to obtain a trained smoke human body recognition model.

[0082] The above loss value is used to measure the effectiveness of the smoke removal model and the accuracy of human recognition. The calculation formula is:

[0083] L=λ m L mse +λ g L gan

[0084] L mse =y I -G I (x IQUP )2

[0085] L gan = -logD(G I (x IQUP ))

[0086] Among them, L represents the loss value, L mse Denotes the distribution loss, L gan represents the reconstruction loss, λ m ,λ g Represents the weighted coefficient of distribution loss and reconstruction loss in the total loss, y I Denotes the real smoke-free image sample corresponding to the current input image, which is used as the supervision signal, G I (x IQUP ) represents the generator G I The fused multi-scale image features x IQUPThe generated desmog image, D, represents the discriminator module in the generative adversarial network, which is used to evaluate whether the desmog image is close to the real image. In one embodiment, the Adam optimizer optimization strategy is configured as follows: learning rate: 0.001, random cropping (64×64 to 416×416) to enhance generalization, and 200 training rounds.

[0087] The optimization strategies of Adam optimization are different in different embodiments and are not limited here.

[0088] Step 13: input the collected smoke image data to be identified into the trained smoke human body recognition model to obtain a human body recognition result corresponding to the smoke image data to be identified.

[0089] The above human body recognition results can provide data support for personnel evacuation and rescue in fire rescue and disaster relief scenarios.

[0090] In order to verify the effectiveness of the human body recognition method in a smoky scene provided by the present invention, in another embodiment of the present invention, the desmoking effect of the method was compared with that of traditional human body recognition algorithms (polarization defogging algorithm and deep learning algorithm). The results are as follows:

[0091] Traditional polarization dehazing algorithms [9, 12, 58, 59] rely on atmospheric scattering models, with PSNR (peak signal-to-noise ratio) < 20 dB and SSIM (structural similarity index) < 0.5, resulting in ineffective smoke removal and image distortion.

[0092] Among the deep learning algorithms, GCANet and FFANet performed best (PSNR 24.56 / 25.90, SSIM 0.7013 / 0.7162), but the details in the smoke area were severely lost.

[0093] The method provided by the present invention: PSNR reaches 28.01 (compared with the global strategy +2.11dB), SSIM is improved to 0.9258, processing speed is optimized by 53%, smoke removal effect is obvious, and performance is significantly improved, which fully demonstrates that the human body recognition method in smoke scenes provided by the present invention can help reduce the interference of smoke and improve the accuracy of human body recognition in smoke scenes.

[0094] In one embodiment of the present invention, the effectiveness of the smoke human body recognition model was further verified, and the results are as follows:

[0095] The PSNR of multi-channel input (multimodal smoke image) is improved by 5.04dB compared with that of single channel, verifying the effectiveness of polarization information fusion.

[0096] The results of the ablation experiment are shown in Table 1.

[0097] Table 1

[0098]

[0099] As shown in Table 1, the feature fusion module and the generative adversarial module in the smoke human body recognition model constructed by the present invention contribute to PSNR improvements of 4.48dB / 2.99dB respectively.

[0100] Qualitative comparison shows that: in thick smoke scenes, target details are restored more completely, color balance is better than GCANet / FFANet, and structural clarity in thin smoke scenes is improved by 30%.

[0101] In summary, the present invention provides a method for human body recognition in a smoke scene. The smoke human body recognition model constructed by the method utilizes a feature extraction module to perform multi-scale feature extraction on smoke image data, utilizes a feature fusion module to fuse the extracted multi-scale features, and then utilizes a generative adversarial module to generate a smoke-free image based on the multi-scale feature fusion result. The multi-scale features are fused by utilizing the difference in sensitivity of smoke image data of different modalities to smoke, which is beneficial to improving the accuracy of the multi-scale feature fusion result, thereby improving the excellence of the smoke removal effect and obtaining a clear smoke-free image. Furthermore, human body recognition is performed based on the clear smoke-free image, which can avoid the interference of thick smoke and fog on human body recognition, and significantly improve the accuracy of human body recognition in smoke scenes.

[0102] In another embodiment of the present invention, in order to further improve the human body recognition and emergency response capabilities of fire scenes, the DeepSeek model is combined with the original technical route for full fine-tuning to adapt to applications in complex environments such as fire scenes.

[0103] Specifically, it includes steps I to III.

[0104] In step I, a large number of fire scene datasets were collected, including different fire types, different fire stages (such as flame initiation, flame spread, fire extinguishing, etc.), different environments (indoor and outdoor fires), and multimodal data (visible light, infrared, thermal imaging).

[0105] The dataset covers a large number of fire images and related text descriptions, ensuring that the model can understand various situations at the fire scene.

[0106] Step II: Use the collected fire scene data to fully fine-tune DeepSeek.

[0107] During the fine-tuning process, the model was refined and adjusted according to the characteristics of the fire scene, especially in target detection (such as human recognition and fire source detection) and scene reasoning (such as fire spread and fire extinguishing progress assessment).

[0108] In step III, in order to further improve the performance of the model, especially in complex situations such as smoke and flame occlusion, multimodal data (visible light images, thermal imaging, and infrared images) are integrated for joint training.

[0109] Multimodal fusion can provide more fire scene information and help DeepSeek make accurate judgments in different environments.

[0110] Based on the fine-tuned DeepSeek model and combined with the ability of visual question answering, it can provide more layers of support for disaster management:

[0111] Real-time feedback: In the early stages of a fire, rescuers can ask questions through the system to obtain real-time information about the fire source, affected people, fire spread, etc., providing immediate feedback for rescue operations.

[0112] Automatic Recommendations: DeepSeek can automatically generate emergency response recommendations based on the progress of a fire. For example, based on the location of the fire source, the model can suggest "immediate fire extinguishing" or "recommend evacuation of personnel in the area," providing reference for decision makers.

[0113] Disaster prediction: By predicting the path of fire spread, DeepSeek can provide areas where fires may spread in the future, predict fire development trends in advance, and help deploy firefighting resources and rescue teams.

[0114] This intelligent emergency response not only improves the efficiency of post-disaster assessment, but also greatly helps rescue workers to formulate more effective emergency measures in advance and ensure the smooth progress of rescue work.

[0115] Verification results using the DeepSeek model show that in a simulated fire environment, the system completed single-frame image question-answering in an average time of 320ms, achieved a fire source location accuracy (IoU) of 92.7%, and increased the adoption rate of emergency recommendations to 89%. Compared to traditional firefighting systems, DeepSeek-VQA achieved a 34% improvement in question-answering accuracy (F1-score) under complex smoke interference, significantly reducing the risk of misjudgment. The system has been integrated into smart fire helmets and drone platforms, providing real-time decision support for frontline rescuers and helping transform fire prevention and control from passive response to proactive prediction.

[0116] The human body recognition device in a smoke scene provided by the present invention is described below.

[0117] like Figure 3 As shown, the human body recognition device 300 includes:

[0118] The sample collection module 301 is used to synchronously collect multimodal smoke image samples in a smoke scene; the multimodal smoke image samples include at least visual image samples and brightness temperature image samples;

[0119] A model training module 302 is configured to train a pre-built smoke human recognition model using multimodal smoke image samples until the loss value of the smoke human recognition model is less than a preset loss threshold, thereby obtaining a trained smoke human recognition model. The smoke human recognition model includes a feature extraction module for extracting multi-scale features from smoke image data, a feature fusion module for fusing the extracted multi-scale features, a generative adversarial module for generating a smoke-free image based on the multi-scale feature fusion results, and a human recognition module for identifying the smoke-free image. The loss value is used to measure the smoke human recognition model's smoke removal performance and the accuracy of human recognition.

[0120] The human body recognition module 303 is used to input the collected smoke image data to be recognized into the trained smoke human body recognition model to obtain a human body recognition result corresponding to the smoke image data to be recognized.

[0121] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiments of the present application. Their specific functions and technical effects can be found in the method embodiment section and will not be described in detail here. Those skilled in the art will clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the above-mentioned method embodiment and will not be described in detail here.

[0122] like Figure 4 As shown, an embodiment of the present invention provides a terminal device, such as Figure 4 As shown, the terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 4Only one processor is shown in the figure), a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, wherein the processor D100 implements the steps of any of the above method embodiments when executing the computer program D102.

[0123] Specifically, when the processor D100 executes the computer program D102, it simultaneously collects multimodal smoke image samples in a smoke scene; uses the multimodal smoke image samples to train a pre-constructed smoke human recognition model until the loss value of the smoke human recognition model is less than a preset loss threshold, thereby obtaining a trained smoke human recognition model; and inputs the collected smoke image data to be identified into the trained smoke human recognition model to obtain a human recognition result corresponding to the smoke image data to be identified. The constructed smoke human recognition model uses a feature extraction module to extract multi-scale features from the smoke image data, uses a feature fusion module to fuse the extracted multi-scale features, and then uses a generative adversarial module to generate a de-smoked image based on the multi-scale feature fusion results. The multi-scale features are fused based on the differences in sensitivity of smoke image data of different modalities to smoke, which is conducive to improving the accuracy of the multi-scale feature fusion results, thereby improving the excellence of the de-smoked effect and obtaining a clear de-smoked image. Furthermore, performing human recognition based on the clear de-smoked image can avoid interference from thick smoke and fog on human recognition, significantly improving the accuracy of human recognition in smoke scenes.

[0124] The processor D100 may be a central processing unit (CPU), or may be another general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0125] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may also be an external storage device of the terminal device D10, such as a plug-in hard disk, a smart memory card (SMC, SmartMedia Card), a secure digital (SD, Secure Digital) card, a flash card, etc. equipped on the terminal device D10. Furthermore, the memory D101 may also include both an internal storage unit of the terminal device D10 and an external storage device. The memory D101 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory D101 may also be used to temporarily store data that has been output or is to be output.

[0126] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned various method embodiments can be implemented.

[0127] An embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0128] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of protection of the present application is limited to these examples. In line with the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0129] The one or more embodiments of this application are intended to encompass all such substitutions, modifications, and variations that fall within the broad scope of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this application should be included in the scope of protection of this application.

Claims

1. A human body recognition method in a smoke scene, characterized in that: include: Synchronously collecting multimodal smoke image samples in a smoke scene; the multimodal smoke image samples include at least visual image samples and brightness temperature image samples; A pre-constructed smoke human body recognition model is trained using the multimodal smoke image samples until a loss value of the smoke human body recognition model is less than a preset loss threshold, thereby obtaining a trained smoke human body recognition model; the smoke human body recognition model includes a feature extraction module for performing multi-scale feature extraction on smoke image data, a feature fusion module for fusing the extracted multi-scale features, a generative adversarial module for generating a smoke-free image based on the multi-scale feature fusion result, and a human body recognition module for identifying the smoke-free image; the loss value is used to measure the quality of the smoke human body recognition model's smoke removal effect and the accuracy of human body recognition; The collected smoke image data to be identified is input into the trained smoke human body recognition model to obtain a human body recognition result corresponding to the smoke image data to be identified.

2. The human body recognition method according to claim 1, characterized in that: The method of training a pre-built smoke human body recognition model using the multimodal smoke image samples until a loss value of the smoke human body recognition model is less than a preset loss threshold, thereby obtaining a trained smoke human body recognition model, includes: Extracting shallow features of the multimodal smoke image sample using a convolutional neural network; the shallow features represent low-level visual information, mainly corresponding to local edges, textures, brightness changes, and simple shape structures in the image; Extracting deep features of the multimodal smoke image sample from the shallow features using a residual network; the deep features represent the modeling of global attributes such as human body structure, spatial relationships between parts, and posture and movement in the entire image; Performing dimensionality reduction on the deep features to generate a multi-scale feature map; Performing channel splicing on the multi-scale feature map to obtain the multi-scale feature fusion result; Reconstructing the smoke-free image data corresponding to the multi-scale feature fusion result using a generative adversarial network, and calculating the reconstruction loss between the smoke-free image data and the real smoke-free image data acquired in advance; Extracting image features of the smoke-free image data and globally pooling the image features to obtain a recognition vector for human body recognition; Obtaining a human body recognition result corresponding to the multimodal smoke image sample based on the recognition vector; Calculating the distribution loss corresponding to the human body recognition result; The smoke human body recognition model is trained according to the reconstruction loss, the distribution loss, and the Adam optimizer to obtain the trained smoke human body recognition model.

3. The human body recognition method according to claim 2, characterized in that: The performing channel splicing on the multi-scale feature map to obtain the multi-scale feature fusion result includes: By calculating the formula F z =F l *M l +F m *M m +F h *M h Among them, F z Represents the multi-scale feature fusion result, F l represents the low-scale feature map extracted from the multimodal smoke image sample, F m represents the mid-scale feature map extracted from the multimodal smoke image sample, F h represents the high-scale feature map extracted from the multimodal smoke image sample, M l Represents the feature response weight map corresponding to the low-scale feature map, which is used to measure the importance of low-scale features in the current input image. m Represents the feature response weight map corresponding to the mesoscale feature map, which is used to measure the importance of the mesoscale feature in the current input image. h represents the feature response weight map corresponding to the high-scale feature map, which is used to measure the importance of high-scale features in the current input image.

4. The human body recognition method according to claim 3, characterized in that: The calculation formula of the loss value is: L=λ m L mse +λ g L gan L mse =y I -G I (x IQUP )2 L gan =-logD(G I (x IQUP )) Wherein, L represents the loss value, L mse represents the distribution loss, L gan represents the reconstruction loss, λ m ,λ g Represents the weighted coefficient of distribution loss and reconstruction loss in the total loss, y I Denotes the real smoke-free image sample corresponding to the current input image, which is used as the supervision signal, G I (x IQUP ) represents the generator G I The fused multi-scale image features x IQUP The generated desmog image, D represents the discriminator module in the generative adversarial network, which is used to evaluate whether the desmog image is close to the real image.

5. The human body recognition method according to claim 4, characterized in that: The reconstructing the smoke-free image data corresponding to the multi-scale feature fusion result by using a generative adversarial network includes: Constructing a generative adversarial network structure, the generative adversarial network includes two sub-modules: a generator and a discriminator. The generator uses the multi-scale feature fusion result as input and adopts an encoder-decoder structure to reconstruct the image. The encoder extracts deep semantic information from the fused features through multi-layer convolution and gradually compresses the spatial resolution. The decoder restores the image spatial structure through deconvolution or upsampling operations, while using jump connections to transfer low-level edge information to preserve image details, and finally outputs a smoke-free image. The discriminator uses a real smoke-free image and a generated image as input, uses a convolutional discriminant structure to extract discriminative features, and outputs true or false probabilities for evaluating the authenticity of the generated image. By jointly training the generator and discriminator, the reconstruction error between the generated image and the real image is minimized while maximizing its true probability in the discriminator, thereby improving the visual quality and semantic consistency of the desmog image and enhancing the recognizability of the human target area.

6. The human body recognition method according to claim 2, characterized in that: The convolutional neural network includes a plurality of sequentially connected feature extraction submodules; The feature extraction submodule includes a Conv layer, a BN layer, and a ReLU layer.

7. The human body recognition method according to claim 2, characterized in that: The residual network is ResNet-18.

8. A human body recognition device in a smoke scene, characterized in that: include: The sample acquisition module is used to synchronously acquire multimodal smoke image samples in smoke scenes; The multimodal smoke image samples include at least visual image samples and brightness temperature image samples; A model training module is configured to train a pre-constructed smoke human body recognition model using the multimodal smoke image samples until the loss value of the smoke human body recognition model is less than a preset loss threshold, thereby obtaining a trained smoke human body recognition model; the smoke human body recognition model includes a feature extraction module for performing multi-scale feature extraction on smoke image data, a feature fusion module for fusing the extracted multi-scale features, a generative adversarial module for generating a desmoked image based on the multi-scale feature fusion result, and a human body recognition module for identifying the desmoked image; the loss value is used to measure the quality of the smoke human body recognition model's desmoking effect and the accuracy of human body recognition; The human body recognition module is used to input the collected smoke image data to be recognized into the trained smoke human body recognition model to obtain the human body recognition result corresponding to the smoke image data to be recognized.

9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images