Fire helmet environment perception auxiliary device based on multi-modal fusion

Abstract

This application discloses a fire helmet environmental perception auxiliary device based on multimodal fusion, including a visible light acquisition module for acquiring real-time visible light images; an infrared sensing module for acquiring real-time infrared images; an environmental information acquisition module for acquiring real-time smoke particulate matter concentration; an inertial measurement unit for outputting six-axis inertial data; a processor for extracting the local infrared entropy of each pixel in the real-time infrared image; the processor calculates the infrared fusion weight of each pixel based on the real-time smoke particulate matter concentration, a preset smoke concentration sleep threshold, a preset gain coefficient, and the infrared local entropy, obtaining an infrared pixel-level fusion weight matrix; and an optical display module for receiving and executing masking instructions. Thus, pixel-level adaptive fusion is achieved through real-time smoke concentration, infrared local entropy, and heat flow evolution gradient, avoiding image fusion errors caused by traditional fusion methods that mask physical textures or miss hidden heat sources in localized dense smoke areas.

Description

Technical Field

[0001] This application relates to the technical field of fire-fighting auxiliary devices, and in particular to a fire helmet environmental perception auxiliary device based on multimodal fusion. Background Technology

[0002] Firefighters face extremely complex visual environments during fire rescue operations: high temperatures, dense smoke, low light, water mist, and other factors combine to severely limit the direct observation capabilities of the human eye. To improve firefighters' environmental awareness, modern fire helmets typically integrate visible light camera modules and infrared thermal imaging modules. These modules project enhanced images of the scene into the firefighter's field of vision through an optical display system, assisting them in penetrating smoke, identifying trapped individuals, and identifying hazards.

[0003] However, single visible light devices are almost completely ineffective in dense smoke or low-light environments, while pure infrared devices, although able to penetrate smoke to detect heat sources, are prone to losing scene texture details and spatial depth. Currently, the few auxiliary devices that use dual-light fusion mostly rely on fixed global fusion weights or simple image stitching, which cannot make real-time adaptive adjustments based on the dynamic and spatially uneven smoke concentration and local heat flow evolution in the fire scene, resulting in the fused image still being blurry or losing key information in some areas.

[0004] Furthermore, during strenuous activities such as running and climbing in a fire, the movement of firefighters' helmets can easily cause changes in parallax between visible light and infrared cameras, resulting in motion blur in the images. Existing technologies lack dynamic compensation mechanisms based on real-time motion posture, often leading to ghosting and misalignment in the fused images. Meanwhile, traditional equipment mostly projects the raw video stream directly onto the screen, requiring firefighters to identify hazards under high stress. It lacks intelligent semantic recognition and visual enhancement warnings for abnormal physical structures or high-temperature hazard areas, making human-computer interaction extremely unintuitive and significantly increasing the cognitive load on firefighters. Summary of the Invention

[0005] This application aims to at least partially address one of the technical problems in the related art.

[0006] To achieve the above objectives, the first aspect of this application proposes a fire helmet environmental perception auxiliary device based on multimodal fusion, comprising: a visible light acquisition module, installed on the fire helmet, for acquiring real-time visible light images; an infrared sensing module, installed on the fire helmet, for acquiring real-time infrared images; an environmental information acquisition module, installed on the fire helmet, for acquiring real-time smoke particulate matter concentration; an inertial measurement unit, installed inside the fire helmet, for outputting six-axis inertial data; and a processor, connected to the visible light acquisition module, the infrared sensing module, the environmental information acquisition module, and the inertial measurement unit respectively; the processor is used to extract the local infrared information entropy of each pixel in the real-time infrared image; the processor, based on the real-time... The infrared fusion weight of each pixel is calculated based on the smoke particulate concentration, a preset smoke concentration dormancy threshold, a preset gain coefficient, and the infrared local information entropy, resulting in an infrared pixel-level fusion weight matrix. The processor determines a visible light pixel-level fusion weight matrix based on the infrared pixel-level fusion weight matrix. The real-time infrared image is weighted using the infrared pixel-level fusion weight matrix, and the real-time visible light image is weighted using the visible light pixel-level fusion weight matrix, and then superimposed to obtain a multimodal fusion enhanced image. The multimodal fusion enhanced image is input into a scene semantic recognition network to extract abnormal state contour coordinates and generate masking instructions. An optical display module, mounted on a fire helmet and connected to the processor, is used to receive and execute the masking instructions.

[0007] In addition, the fire helmet environmental perception auxiliary device based on multimodal fusion proposed in this application may also have the following additional technical features:

[0008] In one embodiment of this application, the coordinates in the infrared pixel-level fusion weight matrix are ( The value of the infrared fusion weight of the pixel. Determined by the following formula: ;in, The real-time smoke particulate matter concentration, The preset smoke concentration dormancy threshold, The preset gain coefficient; The coordinates are ( The infrared local information entropy at the pixel point; The coordinates are ( The pixels at the current timestamp The temporal heat flux evolution gradient is calculated from the gray-level change rate of the same pixel in multiple consecutive infrared images; and These are the preset entropy adjustment coefficient and heat flow adjustment coefficient, respectively.

[0009] In one embodiment of this application, the processor is further configured to calculate the pose transformation matrix of the visible light acquisition module and the infrared sensing module based on the six-axis inertial data, generate a spatial mapping relationship by combining the intrinsic parameter matrix of the visible light acquisition module and the intrinsic parameter matrix of the infrared sensing module; perform pixel coordinate back projection compensation on the real-time infrared image and the real-time visible light image according to the spatial mapping relationship to obtain a spatially aligned infrared image and a visible light image, and extract the infrared local information entropy based on the spatially aligned infrared image.

[0010] In one embodiment of this application, when the processor determines the visible light pixel-level fusion weight matrix based on the infrared pixel-level fusion weight matrix, the specific steps include: normalizing the value of each infrared fusion weight in the infrared pixel-level fusion weight matrix to obtain a normalized infrared weight matrix; calculating the pose jitter variance within the current time window based on the six-axis inertial data, and generating a high-frequency pose jitter penalty term; and determining the coordinates in the visible light pixel-level fusion weight matrix as... The value of visible light fusion weight of the pixel , ;in, For the corresponding element in the normalized infrared weight matrix, This is the high-frequency pose jitter penalty term. This is the preset jitter penalty sensitivity constant.

[0011] In one embodiment of this application, the processor extracts the infrared local information entropy by: dividing the real-time infrared image into multiple image blocks of preset size; calculating a gray-level co-occurrence matrix for each image block; calculating the local information entropy of the image block based on the joint probability density of each gray-level combination in the gray-level co-occurrence matrix; and determining the local information entropy as the infrared local information entropy of all pixels in the image block.

[0012] In one embodiment of this application, the processor is further configured to: calculate the global infrared weight mean of the infrared pixel-level fusion weight matrix; when the global infrared weight mean is greater than a preset degradation protection threshold and the duration exceeds a preset time threshold, cut off the data acquisition link of the visible light acquisition module, and directly use the real-time infrared image as the multimodal fusion enhancement image.

[0013] In one embodiment of this application, the masking instructions include at least one of the following: a contour highlight overlay instruction, used to draw the abnormal state contour with a preset color on the multimodal fusion enhancement image; a region fill overlay instruction, used to fill the area surrounded by the abnormal state contour with a warning color with a preset transparency; and a flashing warning instruction, used to instruct the abnormal state contour or the filled area to flash at a preset frequency.

[0014] To achieve the above objectives, a second aspect of this application proposes a multimodal fusion-based environmental perception assistance method for fire helmets, comprising: acquiring real-time visible light images output by a visible light acquisition module, real-time infrared images output by an infrared sensing module, real-time smoke particulate matter concentration output by an environmental information acquisition module, and six-axis inertial data output by an inertial measurement unit, all mounted on the fire helmet; extracting the local infrared entropy of each pixel in the real-time infrared image; calculating the infrared fusion weight of each pixel based on the real-time smoke particulate matter concentration, a preset smoke concentration dormancy threshold, a preset gain coefficient, and the infrared local entropy, to obtain an infrared pixel-level fusion weight matrix; determining a visible light pixel-level fusion weight matrix based on the infrared pixel-level fusion weight matrix; weighting the real-time infrared image using the infrared pixel-level fusion weight matrix, weighting the real-time visible light image using the visible light pixel-level fusion weight matrix, and superimposing them to obtain a multimodal fusion enhanced image; inputting the multimodal fusion enhanced image into a scene semantic recognition network, extracting abnormal state contour coordinates, generating mask instructions, and sending them to an optical display module for display.

[0015] This invention provides a multimodal fusion-based environmental perception auxiliary device and method for fire helmets. This invention overcomes the limitations of traditional globally fixed weights by achieving pixel-level adaptive fusion through the integration of real-time smoke concentration, infrared local information entropy, and heat flow evolution gradient. This avoids image fusion errors caused by traditional methods, such as obscuring physical textures or missing hidden heat sources in dense smoke areas, thus reducing the risk of firefighters misjudging terrain and fire conditions due to blind spots. To address firefighter motion interference, this invention utilizes inertial data to generate a high-frequency pose jitter penalty term and performs pixel back-projection compensation, isolating the visual pollution of the fused image caused by visible light blurring frames from vigorous running and eliminating spatial coordinate misalignment caused by the physical distance between dual-lens cameras. Simultaneously, the system, through scene semantic recognition and mask annotation, directly eliminates the manual judgment step of firefighters visually searching for abnormal contours in complex fire scene images, shortening the reaction time from environmental detection to hazard avoidance. In addition, the system triggers a degradation protection mechanism through global infrared weight monitoring, which automatically disconnects the link in extremely dense smoke where visible light fails. This prevents invalid noise data from crowding out the processor's operations, avoids the risk of system crashes caused by high equipment load, and improves the system's operational stability and the reliability of continuous data output under extreme conditions.

[0016] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0017] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0018] Figure 1 This is a system architecture diagram of the fire helmet environmental perception auxiliary device based on multimodal fusion according to this application;

[0019] Figure 2 This is a flowchart illustrating the steps of the fire helmet environmental perception assistance method based on multimodal fusion in this application;

[0020] Figure 3 This is a schematic diagram of the structure of the fire helmet environmental perception auxiliary device based on multimodal fusion according to this application. Detailed Implementation

[0021] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0022] The following description, in conjunction with the accompanying drawings, describes an embodiment of the fire helmet environmental perception auxiliary device based on multimodal fusion.

[0023] like Figure 1 As shown in the embodiment of this application, the fire helmet environmental perception auxiliary device based on multimodal fusion aims to solve the problems of limited perception in complex fire scene environments (such as dense smoke, high temperature, low light, etc.) caused by single perception modality in the prior art, as well as image ghosting, motion blur, and loss of key information caused by simple superposition of traditional multimodal methods. It may include:

[0024] The visible light acquisition module, installed on the fire helmet, is used to acquire real-time visible light images;

[0025] The visible light acquisition module can be a miniature high-definition CMOS or CCD camera, which is fixed at the center of the forehead or the side of the fire helmet. It can acquire the physical texture and structural details of the fire scene environment at a high frame rate, thereby continuously outputting the real-time visible light image.

[0026] An infrared sensing module, installed on a fire helmet, is used to collect real-time infrared images;

[0027] Specifically, the infrared sensing module is preferably a thermal imager such as an uncooled microbolometer, and its installation position is adjacent to the visible light acquisition module. This module can penetrate dense smoke and sense heat sources in the environment (such as open flames, high-temperature danger zones, and human bodies), and output the real-time infrared image containing the ambient temperature field distribution.

[0028] An environmental information acquisition module, installed on a firefighter's helmet, is used to collect real-time smoke particulate matter concentration. The environmental information acquisition module can be a photoelectric smoke sensor or a smoke particulate matter concentration sensor based on the principle of laser scattering. It can monitor the smoke density of the environment where the firefighter is located in real time and convert the physical quantity into an electrical signal, and output the quantified real-time smoke particulate matter concentration in real time.

[0029] An inertial measurement unit (IMU) is installed inside a fire helmet. The IMU is used to output six-axis inertial data. The IMU has a built-in three-axis accelerometer and a three-axis gyroscope, which can sense the motion state of the fire helmet in three-dimensional space in real time (including translational acceleration and rotational angular velocity in three directions) and output continuous six-axis inertial data to provide a real-time kinematic reference for subsequent image stabilization and spatial alignment.

[0030] Preferably, the output frequency of the inertial measurement unit is not less than 200Hz, and each frame of inertial data is timestamped to ensure synchronization with the timestamps of the real-time visible light image and the real-time infrared image.

[0031] The processor is connected to the visible light acquisition module, the infrared sensing module, the environmental information acquisition module, and the inertial measurement unit, respectively.

[0032] The processor is used to extract the local infrared information entropy of each pixel in the real-time infrared image;

[0033] It is understandable that the information entropy of an image reflects the local texture complexity and information richness of the image. The processor obtains the infrared local information entropy by performing statistical analysis on local regions of the real-time infrared image to quantitatively evaluate the complexity of the heat source distribution in the neighborhood of each pixel.

[0034] The processor calculates the infrared fusion weight of each pixel based on the real-time smoke particulate concentration, the preset smoke concentration sleep threshold, the preset gain coefficient, and the infrared local information entropy, to obtain an infrared pixel-level fusion weight matrix.

[0035] The processor determines the visible light pixel-level fusion weight matrix based on the infrared pixel-level fusion weight matrix; it weights the real-time infrared image using the infrared pixel-level fusion weight matrix and the real-time visible light image using the visible light pixel-level fusion weight matrix, and then superimposes them to obtain a multimodal fusion enhanced image; the multimodal fusion enhanced image is input into a scene semantic recognition network to extract abnormal state contour coordinates and generate mask instructions.

[0036] The scene semantic recognition network employs a semantic segmentation architecture based on convolutional neural networks to perform pixel-level classification of key targets in fused images, such as high-temperature hazard zones, flame boundaries, trapped personnel, obstacles, and safety passages. The network includes an encoder and a decoder: the encoder can use a lightweight backbone network such as MobileNetV3 or EfficientNet to extract multi-scale feature maps; the decoder uses a feature pyramid network structure, fusing shallow details and deep semantic information through lateral connections, and upsampling layer by layer to restore the input resolution. The network output is a multi-channel probability map, with each channel corresponding to a preset abnormal state category, and the semantic label map is obtained through argmax.

[0037] Specifically, for the training process of this model, a multimodal fusion image dataset of fire scene is pre-constructed, including image pairs acquired and registered simultaneously in visible light and infrared, as well as fused image samples generated by the system's fusion algorithm. All samples are manually labeled at the pixel level, with labels including fire source, high-temperature area, trapped personnel, safety passage, obstacles, and background. During training, the input images are uniformly scaled to 640×480 pixels and subjected to data augmentation such as random horizontal flipping and brightness / contrast perturbation. The loss function uses a combination of weighted cross-entropy loss and Dice loss, where the weighted cross-entropy loss assigns higher weights to rare categories (such as trapped personnel), and the Dice loss is used to optimize region overlap. The optimizer is Adam, with an initial learning rate of 1×10⁻⁶. -4 A cosine annealing learning rate decay strategy was employed. The convergence condition was set as follows: training stopped when the loss did not reach a new low for 10 consecutive training epochs on the validation set. After training, the model was quantized using INT8 to adapt it for real-time inference on embedded processors.

[0038] During the inference phase, the processor inputs the multimodal fusion enhanced image into the trained scene semantic recognition network to obtain the category prediction results of each pixel, and then extracts the contour coordinates of each category region, for example, by obtaining the abnormal state contour through connected component analysis, and then generates the corresponding mask instructions according to the category mapping.

[0039] It should be noted that when extracting the infrared local entropy, the processor divides the real-time infrared image into several image blocks of preset sizes, and independently calculates the entropy of each block to represent the texture richness of the neighborhood of each pixel. Subsequently, a bilinear interpolation algorithm is used to smoothly upsample the local entropy of each block to the pixel level, consistent with the resolution of the original image. This reduces computational load while eliminating the mosaic effect and target edge truncation caused by shared entropy values ​​in the fused image. The processor is an embedded high-performance image processing unit, which can be an FPGA or a SoC chip with a GPU, such as the NVIDIA Jetson series.

[0040] The processor uses the Sigmoid function. The real-time smoke particulate concentration This is mapped to a global modulation factor with a value between 0 and 1. When much smaller When the factor approaches 0, the system primarily relies on visible light images; when Exceed When the factor smoothly transitions to near 1, the system will adaptively tilt the fusion strategy toward the infrared image.

[0041] For example, the preset smoke concentration dormancy threshold Based on engineering experience with visibility changes in dense smoke environments at fire scenes, the target value was set between 300 μg / m³ and 500 μg / m³. The gain coefficient k was then selected within the range of 0.005 to 0.02 using a grid search method. A smoke concentration dormancy threshold was then established. Taking a concentration of 400 μg / m³ as an example, when k = 0.01, the smoke concentration C increases from 0.5... (200 μg / m³) increased to 1.5 During the process of achieving a concentration of 600 μg / m³, the global modulation factor successfully transitioned from approximately 0.12 to approximately 0.88, covering the decision-making switching range from visible light dominance to infrared dominance. Based on actual fire scene simulation testing, k=0.01 was ultimately selected as the default gain coefficient. It should be noted that the specific value of k can be fine-tuned according to the response characteristics of different smoke sensor models to adapt to varying sensitivity requirements.

[0042] An optical display module, mounted on the firefighter's helmet and connected to the processor, is used to receive and execute the masking commands. It should be noted that the optical display module can be mounted inside the firefighter's helmet visor, enabling the commands defined by the masking commands to be displayed in the firefighter's actual field of vision.

[0043] In one embodiment of this application, the coordinates in the infrared pixel-level fusion weight matrix are ( The value of the infrared fusion weight of the pixel. Determined by the following formula:

[0044] ;

[0045] in, The real-time smoke particulate matter concentration, The preset smoke concentration dormancy threshold, The preset gain coefficient;

[0046] The preset smoke concentration dormancy threshold and the preset gain coefficient are both pre-stored in the processor memory based on experience with fire scene environments. The preset gain coefficient... Used to control the smoothness of transitions.

[0047] The coordinates are ( The infrared local information entropy at the pixel point;

[0048] The coordinates are ( The pixels at the current timestamp The temporal heat flux evolution gradient is calculated from the gray-level change rate of the same pixel in multiple consecutive infrared images; and These are the preset entropy adjustment coefficient and heat flow adjustment coefficient, respectively.

[0049] Through the reconstructed calculation formula described above, the global smoke modulation factor is constrained only by the static spatial texture term (i.e., local information entropy), while the dynamic heat flux term exists independently as an additive component. This design ensures that even in a smoke-free environment (where the modulation factor tends to 0), when the temperature in a certain area rises sharply, the significantly increased [modulation factor] [is effectively mitigated]. It can still forcefully increase the value of the infrared fusion weight, ensuring that firefighters can clearly observe the rapidly heating danger zone and gain valuable time to detect flashover in advance.

[0050] Furthermore, this embodiment introduces the dynamic feature of time-series heat flow evolution gradient to detect rapid temperature changes in the fire field that are indicative of flashover precursors.

[0051] To eliminate spurious gradients caused by field-of-view shifts due to firefighter head movements, before calculating the time-series heat flux evolution gradient, the processor can first use the six-axis inertial data output by the inertial measurement unit to perform field-of-view motion compensation based on affine transformation on the previous frame of infrared image, ensuring that the images of the previous and next frames are perfectly aligned in the same real three-dimensional physical coordinates. The calculation method is as follows: In the formula, Current timestamp The real-time infrared image described below is in coordinates grayscale value at that location This is the grayscale value of the corresponding position in the previous frame.

[0052] In a preferred embodiment, in order to suppress the amplification effect of transient noise of the microbolometer on the frame difference, before calculating the time-series heat flow evolution gradient, the median time-domain filter with a window size of 3 is first applied to the continuous multiple frames of infrared images to smooth the gray value sequence, and then the gradient is calculated according to the above formula.

[0053] This gradient value directly reflects the rate of change of thermal radiation intensity over time within the spatial region corresponding to that pixel. When the value is positive, it indicates that the area is in a state of rising temperature, and the infrared fusion weight needs to be increased to enhance the warning; when... When the value is negative, it indicates that the region is in a cooling state, and the infrared fusion weight should be reduced accordingly to allocate visual attention reasonably.

[0054] When the temperature in a certain area rises sharply Significantly increased, even if the texture of the region is relatively smooth. Lower, through the addition term in the formula + It can still forcibly increase the value of the infrared fusion weight, ensuring that firefighters can clearly observe the rapidly heating dangerous area in the fused image, thus gaining valuable time for early detection of flashover and timely evacuation.

[0055] in, The entropy adjustment coefficient is used to adjust the weight of the static texture item; The heat flux adjustment coefficient is used to adjust the weight of the dynamic heat flux term. Together, they determine the relative importance of the infrared local information entropy and the temporal heat flux evolution gradient.

[0056] In a preferred embodiment, the evaluation index is the sharpness of the trapped person's outline and the contrast of thermal anomaly areas in the fused image. A grid search method can be used to set this index. Take 0.7, The value is set to 0.3. Specifically, contour sharpness is quantized by the average gradient magnitude of the edge region of the trapped person, and contrast of the thermal anomaly region is quantized by the average grayscale difference between the thermal anomaly region and the adjacent background region. The search process iterates through... ∈[0.5,0.9]、 =1− For discrete combinations, calculate the weighted score of the evaluation index under each combination, and the highest score is the selected value.

[0057] This formula forms a pixel-level fusion strategy that combines spatial texture and temporal heat flow modulation, achieving a leap from "seeing clearly" to "seeing danger".

[0058] It should be noted that, in this embodiment, to ensure the value of the infrared fusion weight...

[0059] Always within the [0,1] range, the processor first processes the infrared local information entropy separately before performing fusion calculations. and the time-series heat flux evolution gradient Min-Max normalization is performed to map the numerical range of both values ​​to the [0,1] interval; simultaneously, for The negative part is truncated to zero using the ReLU function. After the above preprocessing, the weighted combination term... The output range is constrained to [0,1], and then multiplied by the Sigmoid global modulation factor with a value in (0,1) to ultimately ensure... It remains compatible with the normalization constraints of the basic fusion strategy.

[0060] In one embodiment of this application, the processor is further configured to calculate the pose transformation matrix of the visible light acquisition module and the infrared sensing module based on the six-axis inertial data, and generate a spatial mapping relationship by combining the intrinsic parameter matrix of the visible light acquisition module and the intrinsic parameter matrix of the infrared sensing module.

[0061] The real-time infrared image and the real-time visible light image are back-projected with pixel coordinates according to the spatial mapping relationship to obtain spatially aligned infrared and visible light images, and the infrared local information entropy is extracted based on the spatially aligned infrared image.

[0062] To achieve high-precision fusion of heterogeneous images, this embodiment performs real-time spatial alignment of the two modules, fundamentally eliminating fusion ghosting and contour misalignment caused by installation position differences and operational vibrations. The intrinsic parameter matrix of the visible light acquisition module... The intrinsic parameter matrix of the infrared sensing module and the fixed extrinsic parameter matrix between the two. The external parameter matrix is ​​obtained in advance through joint calibration using Zhang's calibration method and stored in the processor. The static mounting posture relationship between the visible light acquisition module and the infrared sensing module on the helmet is described, including relative rotation and relative translation.

[0063] During operation, the processor reads the six-axis inertial data output by the inertial measurement unit, and uses accelerometer and gyroscope data to obtain the overall attitude change of the fire helmet relative to the geodetic coordinate system in real time through quaternion attitude calculation and integration. .

[0064] In one specific implementation, the quaternion attitude calculation employs the Mahony complementary filtering algorithm. This algorithm fuses the attitude quaternion obtained by gyroscope integration with the gravity vector observed by the accelerometer, utilizing the low-frequency stability of the accelerometer to correct the high-frequency drift of the gyroscope. For example, the complementary filter gain coefficient can be set to 0.5, thereby obtaining real-time, stable, and accurate overall helmet attitude change. .

[0065] The processor utilizes the overall posture change of the helmet. For the fixed extrinsic parameter matrix Perform dynamic corrections to obtain the pose transformation matrix at the current timestamp. This dynamic correction compensates for global attitude disturbances caused by vibration or impact on the helmet to the external parameters between cameras, effectively reflecting the instantaneous relative pose relationship between the two modules under actual operating conditions.

[0066] In a preferred embodiment, based on the characteristics of the main targets (such as the ground, walls, and trapped personnel) in a fire scene being relatively far from the helmet and having relatively small depth variations, a homography matrix is ​​used to establish the spatial mapping relationship. Specifically, the processor uses the intrinsic parameter matrix of the visible light acquisition module... The intrinsic parameter matrix of the infrared sensing module and the pose transformation matrix Construct homography mapping matrix The calculation formula is: Before calculating the homography mapping matrix, distortion correction processing was performed on the visible light image and the infrared image. Based on the homography mapping matrix, the spatial mapping relationship is expressed as follows: In the formula Let be the homogeneous coordinates of a pixel in the infrared image. These are the corresponding pixel coordinates of the visible light image. Performing this mapping compensation on each pixel in the infrared image yields the spatially aligned infrared image, precisely registered with the visible light image. Subsequent infrared local entropy extraction and fusion weighting are based on this spatially aligned image pair, ensuring the geometric accuracy of the calculation.

[0067] In one embodiment of this application, when the processor determines the visible light pixel-level fusion weight matrix based on the infrared pixel-level fusion weight matrix, it specifically includes:

[0068] The value of each infrared fusion weight in the infrared pixel-level fusion weight matrix is ​​normalized to obtain a normalized infrared weight matrix.

[0069] The pose jitter variance within the current time window is calculated based on the six-axis inertial data, and a high-frequency pose jitter penalty term is generated.

[0070] The coordinates in the visible light pixel-level fusion weight matrix are determined as follows: The value of visible light fusion weight of the pixel , ;

[0071] in, For the corresponding element in the normalized infrared weight matrix, This is the high-frequency pose jitter penalty term. This is the preset jitter penalty sensitivity constant.

[0072] To maintain the quality of the fused image under conditions of vigorous movement such as firefighters running and climbing, this embodiment incorporates a motion quality assessment mechanism based on inertial perception when determining the visible light fusion weights. This mechanism actively suppresses the contamination of the fusion result by visible light motion blur caused by helmet shaking. This embodiment is a preferred improvement of the basic fusion strategy described in claim 1. After introducing a motion penalty term, the sum of the infrared fusion weight and the visible light fusion weight at the same pixel location is not forced to be always equal to 1. Instead, when the risk of motion blur is high, the visible light weight is reduced overall to ensure the clarity of the fused image.

[0073] In one embodiment, to avoid overall darkening of the fused image due to unilaterally reducing the visible light weight, the processor introduces a dynamic global brightness adaptive compensation mechanism when generating the final multimodal fused enhanced image. Histogram equalization is used to stretch the brightness channels of the fused image to ensure that the output image still has suitable brightness for human visual perception even when the penalty term is activated.

[0074] The normalized infrared weight matrix The infrared pixel-level fusion weight matrix is ​​obtained by performing maximum-minimum value normalization, so that the value range of each element is mapped to [0,1], which facilitates subsequent normalization constraints with visible light weights.

[0075] The physical meaning of this formula is: when the helmet does not shake, The visible light weight remains at 0. This naturally complements the infrared weighting; when the helmet shakes violently, Increase, exponential decay term The value approaches 0, thus making the visible light fusion weight value approach 0. The blurring was significantly reduced. This is equivalent to using inertial data to proactively predict the motion blur that may occur in visible light images before fusion, and suppressing it at the pixel level to prevent blurred pixels from contaminating the fusion result.

[0076] For example, the jitter penalty sensitivity constant γ is used to adjust the intensity of suppression, through simulating the quality test of fused images under different motion states. The optimal value is 2.0. This mechanism effectively ensures the clarity and stability of the output image under dynamic working environments.

[0077] In one embodiment of this application, the processor extracts the infrared local information entropy by:

[0078] The real-time infrared image is divided into multiple image blocks of preset sizes;

[0079] Calculate the gray-level co-occurrence matrix for each of the image blocks;

[0080] The local information entropy of the image block is calculated based on the joint probability density of each gray level combination in the gray-level co-occurrence matrix, and this local information entropy is determined as the infrared local information entropy of all pixels in the image block.

[0081] It should be noted that the infrared local information entropy processing scheme in this embodiment avoids the computational burden caused by pixel sliding window calculation.

[0082] In a preferred embodiment, the processor first divides the spatially aligned infrared image into non-overlapping image blocks of 16×16 pixels. For each block, to fully capture texture directionality, four gray-level co-occurrence matrices (GLCMs) with a pixel distance of 1 are generated (0°, 45°, 90°, 135°). The average value of corresponding elements in the four directional GLCMs is taken as the comprehensive GLCM for that block to eliminate direction sensitivity. Each element in this comprehensive GLCM... This represents the joint probability density of a certain combination of gray levels.

[0083] Based on this, the local information entropy is calculated using the following formula: The calculated scalar entropy value The value is assigned to all pixels within the image block as the infrared local information entropy corresponding to each pixel, i.e., the entire... The 256 pixels within a block share the same entropy value. This method reduces the high complexity of pixel-by-pixel sliding windows to block-by-block computation, significantly reducing computational load while maintaining texture differentiation capabilities, thus meeting the real-time requirements of embedded platforms. This entropy value accurately reflects the degree of disorder in grayscale distribution within the block; regions with high entropy values ​​correspond to textured, information-rich target contours or edges, while regions with low entropy values ​​correspond to smooth backgrounds, providing a reliable spatial basis for subsequent weight modeling.

[0084] In one embodiment of this application, the processor is further configured to:

[0085] Calculate the global infrared weight mean of the infrared pixel-level fusion weight matrix; when the global infrared weight mean is greater than the preset degradation protection threshold and the duration exceeds the preset time threshold, cut off the data acquisition link of the visible light acquisition module, and directly use the real-time infrared image as the multimodal fusion enhancement image.

[0086] To cope with extremely harsh visual environments (such as dense smoke or sustained high temperatures), this embodiment incorporates a robust degradation protection mechanism to prevent interference from invalid data and waste of computing power. The global infrared weight mean is obtained by calculating the arithmetic mean of all elements in the infrared pixel-level fusion weight matrix, and is used to macroscopically measure the degree of suppression of visible light imaging by the current environment.

[0087] Specifically, before averaging the elements of the infrared pixel-level fusion weight matrix, the processor first performs a pruning process on the matrix, truncating all element values ​​greater than 1 to 1 and all element values ​​less than 0 to 0, to ensure that the global infrared weight mean is always within the [0,1] interval. The preset degradation protection threshold is exemplarily set to 0.85, and the preset time threshold is set to 3 seconds.

[0088] The degradation protection threshold of 0.85 is determined based on prior knowledge from fire protection experts. This means that when the arithmetic mean of the infrared fusion weights of all pixels exceeds 0.85, it indicates that after cropping to [0,1], the infrared contribution of most pixels is already at a high level, the overall signal-to-noise ratio of the visible light channel is severely reduced, and effective texture and color information cannot be provided. Triggering degradation protection at this point is reasonable. The duration is set to 3 seconds to filter out false triggers caused by short-term flame flashes, explosion impacts, or instantaneous sensor noise, ensuring the stability and reliability of the degradation switching action and avoiding frequent screen jumps that could cause discomfort to firefighters.

[0089] Furthermore, when degradation protection is triggered, the processor continuously monitors the real-time smoke particulate matter concentration output by the environmental information acquisition module. When the real-time smoke particulate matter concentration drops back to the preset smoke concentration dormancy threshold, the processor will continue to monitor the smoke particulate matter concentration. If the duration exceeds a preset recovery time threshold (e.g., 5 seconds), the system automatically restores the data link of the visible light acquisition module and re-enters the multimodal fusion mode. This self-recovery mechanism is based on the direct environmental perception parameter of smoke concentration, with rigorous logic, ensuring fully automated operation of the system without manual intervention.

[0090] When the smoke concentration at the fire scene is extremely high and the temperature distribution is abnormal, causing the infrared fusion weight to remain at a high value for a prolonged period, it indicates that the visible light image can hardly provide any effective information and may introduce a large amount of scattering noise. In this situation, the processor actively disconnects the data link of the visible light acquisition module, for example, by stopping the reading of its image data or cutting off its power supply, and directly outputs the spatially aligned infrared image of the current frame as the multimodal fusion enhanced image to the subsequent scene semantic recognition network and optical display module. This not only avoids invalid calculations, reduces system power consumption and heat generation, and extends the effective operating time of battery-powered equipment, but also ensures that firefighters always obtain stable, high signal-to-noise ratio thermal images.

[0091] In one embodiment of this application, the masking instructions include at least one of the following:

[0092] The outline highlight overlay command is used to draw the outline of the abnormal state on the multimodal fusion enhanced image with a preset color;

[0093] The area fill overlay command is used to fill the area enclosed by the abnormal state outline with a warning color with a preset transparency.

[0094] A flashing warning instruction is used to instruct the outline or filled area of ​​the abnormal state to flash at a preset frequency.

[0095] Specifically, when generating the outline highlight overlay instruction, the processor generates a set of Bézier curve vector paths based on the outline coordinates of the abnormal state, and specifies the line color and line width. In one specific implementation, the color mapping strategy is as follows: red (RGB:255,0,0) represents a fire source or high-temperature hazard area, green (RGB:0,255,0) represents a safety passage or exit, and yellow (RGB:255,255,0) represents trapped personnel or obstacles. The line width is set to 3 pixels to have sufficient visual salience on the helmet display module, driving the optical display module to overlay and draw on the fused image.

[0096] When the region fill overlay instruction is generated, the processor fills the region inside the outline with a semi-transparent warning color with a transparency of 40%, making the spatial range and size of the danger zone clear at a glance, while not completely obscuring the scene information in the background.

[0097] When the flashing warning command is generated, the processor alternately displays and hides the outline or filled area at a frequency of 2Hz. The flashing mechanism can quickly capture the peripheral visual attention of the human eye, and is especially suitable for alerting trapped personnel or sudden dangerous targets.

[0098] During each display cycle, the above-mentioned multiple instructions can be used in combination according to the priority and danger level of the anomaly category. For example, for the detected "trapped personnel", both outline highlighting (green) and area filling (semi-transparent green) are used simultaneously, supplemented by 2Hz flashing, to achieve the strongest warning effect; for "flame boundaries", only outline highlighting (red) is used to avoid excessive warning and information overload.

[0099] like Figure 2 As shown, a multimodal fusion-based environmental perception assistance method for fire helmets includes:

[0100] The system acquires real-time visible light images from the visible light acquisition module installed on the fire helmet, real-time infrared images from the infrared sensing module, real-time smoke particulate matter concentration from the environmental information acquisition module, and six-axis inertial data from the inertial measurement unit.

[0101] Extract the local infrared information entropy of each pixel in the real-time infrared image;

[0102] Based on the real-time smoke particulate matter concentration, the preset smoke concentration dormancy threshold, the preset gain coefficient, and the infrared local information entropy, the infrared fusion weight of each pixel is calculated to obtain the infrared pixel-level fusion weight matrix.

[0103] The visible light pixel-level fusion weight matrix is ​​determined based on the infrared pixel-level fusion weight matrix.

[0104] The real-time infrared image is weighted using the infrared pixel-level fusion weight matrix, and the real-time visible light image is weighted using the visible light pixel-level fusion weight matrix. The two images are then superimposed to obtain a multimodal fusion enhanced image.

[0105] The multimodal fusion enhanced image input scene semantic recognition network extracts the abnormal state contour coordinates, generates mask instructions, and sends them to the optical display module for display.

[0106] In summary, the fire helmet environmental perception auxiliary device based on multimodal fusion in this application improves the perception quality and robustness of complex fire scenes by introducing environmental sensors and inertial measurement units and combining them with innovative dynamic algorithms. This invention breaks away from the traditional globally fixed weight fusion method, innovatively combining real-time smoke particle concentration, infrared local information entropy extracted based on the gray-level co-occurrence matrix, and temporal heat flow evolution gradient to independently calculate the fusion weight for each pixel. This automatically increases the infrared weight in dense smoke or high-heat areas to highlight the heat source, while preserving visible light details in textured areas, achieving pixel-level adaptive fusion and maximizing the restoration of the overall fire scene.

[0107] To address motion blur and dual-spectral parallax issues, this invention utilizes six-axis inertial data to calculate pose jitter variance in real time and generate a high-frequency pose jitter penalty term. During periods of intense motion, it automatically reduces the visible light fusion weights and combines this with intrinsic parameter matrices for pixel coordinate back-projection compensation, effectively eliminating motion ghosting and spatial misalignment. In terms of human-computer interaction, this invention inputs high-quality multimodal fusion images into a scene semantic recognition network, accurately extracting abnormal state contours and generating mask instructions such as contour highlighting, transparency filling, or flashing. This provides an intuitive presentation of warning information through augmented reality, significantly reducing the cognitive load on firefighters in highly stressful environments. Furthermore, this invention incorporates a degradation protection mechanism for extreme environments. When the global infrared weights continuously exceed limits (i.e., in extreme dense smoke environments where visible light is completely ineffective), the system automatically cuts off the visible light data acquisition link, avoiding unnecessary computational resource consumption, effectively reducing system power consumption, and ensuring the sustained and stable operation of core equipment.

[0108] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0109] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fire helmet environmental perception auxiliary device based on multimodal fusion, characterized in that, include: The visible light acquisition module, installed on the fire helmet, is used to acquire real-time visible light images; An infrared sensing module, installed on a fire helmet, is used to collect real-time infrared images; An environmental information acquisition module, installed on a fire helmet, is used to collect real-time smoke particulate matter concentration. An inertial measurement unit, installed inside a fire helmet, is used to output six-axis inertial data; The processor is connected to the visible light acquisition module, the infrared sensing module, the environmental information acquisition module, and the inertial measurement unit, respectively. The processor is used to extract the local infrared information entropy of each pixel in the real-time infrared image; The processor calculates the infrared fusion weight of each pixel based on the real-time smoke particulate concentration, the preset smoke concentration sleep threshold, the preset gain coefficient, and the infrared local information entropy, to obtain an infrared pixel-level fusion weight matrix. The processor determines the visible light pixel-level fusion weight matrix based on the infrared pixel-level fusion weight matrix; it then uses the infrared pixel-level fusion weight matrix to weight the real-time infrared image, uses the visible light pixel-level fusion weight matrix to weight the real-time visible light image, and superimposes them to obtain a multimodal fusion enhanced image. The multimodal fusion enhanced image input scene semantic recognition network is used to extract the abnormal state contour coordinates and generate mask instructions; An optical display module, mounted on a fire helmet, is connected to the processor and is used to receive and execute the mask instructions.

2. The fire helmet environmental perception auxiliary device based on multimodal fusion according to claim 1, characterized in that, The coordinates in the infrared pixel-level fusion weight matrix are ( The value of the infrared fusion weight of the pixel. Determined by the following formula: ； in, The real-time smoke particulate matter concentration, The preset smoke concentration dormancy threshold, The preset gain coefficient; The coordinates are ( The infrared local information entropy at the pixel point; The coordinates are ( The pixels at the current timestamp The temporal heat flux evolution gradient is calculated from the gray-level change rate of the same pixel in multiple consecutive infrared images; and These are the preset entropy adjustment coefficient and heat flow adjustment coefficient, respectively.

3. The fire helmet environmental perception auxiliary device based on multimodal fusion according to claim 1, characterized in that, The processor is also used to calculate the pose transformation matrix of the visible light acquisition module and the infrared sensing module based on the six-axis inertial data, and generate a spatial mapping relationship by combining the intrinsic parameter matrix of the visible light acquisition module and the intrinsic parameter matrix of the infrared sensing module. The real-time infrared image and the real-time visible light image are back-projected with pixel coordinates according to the spatial mapping relationship to obtain spatially aligned infrared and visible light images, and the infrared local information entropy is extracted based on the spatially aligned infrared image.

4. The fire helmet environmental perception auxiliary device based on multimodal fusion according to claim 1, characterized in that, When the processor determines the visible light pixel-level fusion weight matrix based on the infrared pixel-level fusion weight matrix, the specific steps include: The value of each infrared fusion weight in the infrared pixel-level fusion weight matrix is ​​normalized to obtain a normalized infrared weight matrix. The pose jitter variance within the current time window is calculated based on the six-axis inertial data, and a high-frequency pose jitter penalty term is generated. The coordinates in the visible light pixel-level fusion weight matrix are determined as follows: The value of visible light fusion weight of the pixel , ; in, For the corresponding element in the normalized infrared weight matrix, This is the high-frequency pose jitter penalty term. This is the preset jitter penalty sensitivity constant.

5. The fire helmet environmental perception auxiliary device based on multimodal fusion according to claim 1 or 3, characterized in that, The processor extracts the infrared local information entropy including: The real-time infrared image is divided into multiple image blocks of preset sizes; Calculate the gray-level co-occurrence matrix for each of the image blocks; The local information entropy of the image block is calculated based on the joint probability density of each gray level combination in the gray-level co-occurrence matrix, and this local information entropy is determined as the infrared local information entropy of all pixels in the image block.

6. The fire helmet environmental perception auxiliary device based on multimodal fusion according to claim 1, characterized in that, The processor is also used for: Calculate the global infrared weight mean of the infrared pixel-level fusion weight matrix; when the global infrared weight mean is greater than the preset degradation protection threshold and the duration exceeds the preset time threshold, cut off the data acquisition link of the visible light acquisition module, and directly use the real-time infrared image as the multimodal fusion enhancement image.

7. The fire helmet environmental perception auxiliary device based on multimodal fusion according to claim 1, characterized in that, The masking instructions include at least one of the following: The outline highlight overlay command is used to draw the outline of the abnormal state on the multimodal fusion enhanced image with a preset color; The area fill overlay command is used to fill the area enclosed by the abnormal state outline with a warning color with a preset transparency. A flashing warning instruction is used to instruct the outline or filled area of ​​the abnormal state to flash at a preset frequency.

8. A method for assisting environmental perception in fire helmets based on multimodal fusion, characterized in that, include: The system acquires real-time visible light images from the visible light acquisition module installed on the fire helmet, real-time infrared images from the infrared sensing module, real-time smoke particulate matter concentration from the environmental information acquisition module, and six-axis inertial data from the inertial measurement unit. Extract the local infrared information entropy of each pixel in the real-time infrared image; Based on the real-time smoke particulate matter concentration, the preset smoke concentration dormancy threshold, the preset gain coefficient, and the infrared local information entropy, the infrared fusion weight of each pixel is calculated to obtain the infrared pixel-level fusion weight matrix. The visible light pixel-level fusion weight matrix is ​​determined based on the infrared pixel-level fusion weight matrix. The real-time infrared image is weighted using the infrared pixel-level fusion weight matrix, and the real-time visible light image is weighted using the visible light pixel-level fusion weight matrix. The two images are then superimposed to obtain a multimodal fusion enhanced image. The multimodal fusion enhanced image input scene semantic recognition network extracts the abnormal state contour coordinates, generates mask instructions, and sends them to the optical display module for display.

Images