A multimodal image fusion method based on fire mask and fire mask

Abstract

A multimodal image fusion method based on a fire mask and a fire mask, which relates to the general field of image data processing or generation, and comprises: using an image acquisition device and an infrared sensor to respectively acquire on-site environment images containing information on the edge of a flame and the outline of a building, as well as abnormal temperature distribution information containing temperature values ​​and position information, inputting a dynamic weight adjustment neural network model, accurately identifying smoke-dense areas and areas with clear environmental details, and then improving the image fusion weight of the corresponding area accordingly, superimposing the temperature and outline information to obtain a regional differentiated fusion image, highlighting the heat distribution and building structural features, and finally enhancing the boundaries of key areas and displaying them in real time. The implementation of this method can improve the intuitiveness of the presentation of fire scene information.

Description

Technical Field

[0001] The present application relates to the general field of image data processing or generation, and in particular to a multimodal image fusion method based on a fire mask and a fire mask. Background Art

[0002] As firefighting and rescue missions become more complex and diverse, firefighters need to quickly and accurately grasp environmental information when performing tasks at the fire scene to ensure the safety and effectiveness of rescue operations. In a smoke-filled fire environment, the acquisition and processing of environmental information is of great significance to the life safety of firefighters.

[0003] In the relevant technology, the technical solutions of headlight illumination and single sensor perception can be adopted. Among them, some intelligent fire fighting equipment obtains temperature information by configuring infrared thermal imagers, collects environmental images by visible light cameras, and presents this information to firefighters for reference.

[0004] However, in actual applications, due to the complex and changeable fire environment, the scattered presentation of single information makes it impossible to provide fire scene information to firefighters intuitively, resulting in firefighters needing to pay attention to and process multiple information sources at the same time in a short period of time, which increases the cognitive burden and prolongs the information processing time. Summary of the invention

[0005] The present application provides a multimodal image fusion method based on a fire mask and a fire mask, which are used to improve the intuitiveness of fire scene information presentation.

[0006] In a first aspect, the present application provides a multimodal image fusion method based on a fire mask, which is applied to a fire mask. The method comprises: in response to the wearing operation of the fire mask, collecting a scene environment image based on an image acquisition device, and collecting temperature distribution abnormality information based on an infrared sensor, wherein the scene environment image includes a flame edge image and building contour line information, and the temperature distribution abnormality information includes a temperature value and position information; inputting the scene environment image and the temperature distribution abnormality information into a dynamic weight adjustment neural network model to obtain a smoke-dense area and an area with clear environmental details, and the dynamic weight adjustment neural network model is used to identify the smoke-dense area and the area with clear environmental details in the scene environment image; The image fusion weight of the temperature distribution anomaly information in the smoke-dense area is increased to a first preset threshold, and the image fusion weight of the contour line information in the area with clear environmental details is increased to a second preset threshold, so as to obtain an image fusion weight allocation scheme for each area of ​​the on-site environmental image; according to the image fusion weight allocation scheme, the temperature distribution anomaly information and the contour line information are superimposed to obtain a regional differentiated fusion image, which includes heat distribution characteristics and building structure characteristics; the boundaries of the areas where the temperature exceeds the preset temperature threshold and the structural edges in the contour line information of the building are enhanced to obtain an edge-enhanced image, and the edge-enhanced image is displayed in real time on the display of the fire mask.

[0007] By adopting the above technical solution, the fire mask can respond in time after being worn, and use the image acquisition device and infrared sensor to respectively collect the on-site environmental images containing the flame edge and the building contour line information, as well as the temperature distribution abnormality information containing the temperature value and position information, and input the dynamic weight adjustment neural network model to accurately identify the smoke-dense areas and the areas with clear environmental details. Then, the image fusion weight of the corresponding area is improved, and the temperature and contour information are superimposed to obtain the regional differentiated fusion image, highlighting the heat distribution and building structure characteristics. Finally, the boundaries of key areas are enhanced and displayed in real time, so that firefighters can quickly and intuitively grasp the on-site environment and improve rescue safety and efficiency.

[0008] In combination with some embodiments of the first aspect, in some embodiments, before inputting the scene environment image and the temperature distribution abnormality information into the dynamic weight adjustment neural network model to obtain the smoke-dense area and the environment detail clear area, the method also includes: obtaining the light intensity information of the scene environment image; generating a first preset threshold and a second preset threshold based on the light intensity information.

[0009] By adopting the above technical solution, after the on-site environment image and temperature distribution abnormality information are input into the dynamic weight adjustment neural network model, the on-site environment light intensity information is obtained, and the first preset threshold and the second preset threshold are dynamically generated according to the light intensity. When the light intensity is low, the temperature information is highlighted, and when the intensity is high, the contour information is enhanced. It can adapt to different lighting environments and ensure that the image fusion weights are reasonably distributed under various complex conditions, such as strong light, weak light and even dark environments, thereby improving the stability of the display effect.

[0010] In combination with some embodiments of the first aspect, in some embodiments, after the step of displaying the edge enhanced image in real time on the display of the fire mask, the method also includes: extracting temperature feature points in the temperature distribution anomaly information and contour feature points in the contour line information; establishing a correspondence between the temperature feature points and the contour feature points to obtain a feature correspondence; spatially aligning the heat distribution features in the temperature distribution anomaly information with the building structure features in the contour line information according to the feature correspondence to obtain a heat contour overlay map; and displaying the heat contour overlay map in real time on the display of the fire mask.

[0011] By adopting the above technical solution, after the edge-enhanced image is displayed in real time, the temperature feature points in the temperature distribution anomaly information and the contour feature points in the contour line information are extracted, and the corresponding relationship between the two is established to achieve spatial registration to obtain the heat contour overlay map and display it in real time, so that the temperature distribution and the building contour are accurately aligned in space, and the relationship between the high-temperature area and the building structure is intuitively presented. Firefighters can use this to quickly understand the relationship between the high-temperature area and the building layout, and predict danger in advance.

[0012] In combination with some embodiments of the first aspect, in some embodiments, after the step of displaying the thermal contour overlay image in real time on the display of the fire mask, the method also includes: defogging the smoke-dense area and contrast enhancing the area with clear environmental details to obtain an enhanced image; switching the enhanced image and the edge-enhanced image to be displayed on the display of the fire mask.

[0013] By adopting the above technical solution, after the heat contour overlay map is displayed in real time, the smoke-dense area is dehazed to restore the image details obscured by the smoke, improve the image quality of the area, enhance the contrast of the area with clear environmental details, and make the image visual effect clearer. The enhanced image and the edge-enhanced image are then switched and displayed on the fire mask display. Firefighters can view different images as needed to obtain more environmental details.

[0014] In combination with some embodiments of the first aspect, in some embodiments, after the step of displaying the edge enhanced image in real time on the display of the fire mask, the method also includes: when it is detected that the temperature value in the temperature distribution abnormality information exceeds the preset temperature threshold or the environmental toxic gas concentration exceeds the preset toxic gas concentration threshold, obtaining the target high temperature position corresponding to the temperature value, and obtaining the target toxic gas position corresponding to the environmental toxic gas concentration, the target high temperature position being the position where the temperature exceeds the preset temperature threshold; displaying the target high temperature position and the target toxic gas position on the display; and sending a warning prompt information containing the target high temperature position and the target toxic gas position.

[0015] By adopting the above technical solution, when the temperature value in the temperature distribution abnormality information exceeds the preset temperature threshold or the ambient toxic gas concentration exceeds the preset toxic gas concentration threshold, the fire mask can accurately obtain the target high temperature position and toxic gas position corresponding to the temperature value, and display them clearly on the display. Firefighters can know the dangerous location and situation in time, and the command center can quickly obtain information.

[0016] In combination with some embodiments of the first aspect, in some embodiments, after the step of displaying the edge-enhanced image in real time on the display of the fire mask, the method also includes: obtaining the real-time position coordinate information of the fire mask; matching the real-time position coordinate information with the building structure features in a preset terrain database to obtain the specific spatial position of the fire mask; constructing an indoor path topology map based on the building structure features, mapping the temperature distribution anomaly information to each path node of the indoor path topology map, generating path traffic weights according to the temperature values ​​of each path node, and calculating the optimal evacuation route based on the path traffic weights; superimposing the optimal evacuation route on the edge-enhanced image, and updating the display in real time on the display of the fire mask.

[0017] By adopting the above technical solution, the fire mask obtains real-time position coordinate information, realizes high-precision positioning through the combined positioning module and fusion algorithm, and obtains the specific spatial position by matching with the preset terrain database, providing accurate position reference for firefighters, and constructs the indoor path topology map based on the structural characteristics of the building. The risk attributes are generated for the path nodes according to the abnormal temperature distribution information, and the path weight is calculated based on the distance, channel width and risk attributes. The optimal evacuation route is planned with the current position of the fire mask and the nearest safe exit as nodes and updated and displayed in real time. Firefighters can clearly understand their own position and the optimal escape route and effectively avoid dangerous areas.

[0018] In combination with some embodiments of the first aspect, in some embodiments, the indoor path topology map is constructed based on the structural characteristics of the building, the temperature distribution anomaly information is mapped to each path node of the indoor path topology map, the path traffic weight is generated according to the temperature value of each path node, and the step of calculating the optimal evacuation route based on the path traffic weight specifically includes: extracting passages, doors, stairs, corner points and safety exits in the structural characteristics of the building as path nodes, and generating an indoor path topology map according to the path nodes; generating a node risk attribute for each path node according to the temperature distribution anomaly information at each path node; setting a path basic weight according to the distance between each path node and the passage width, setting a risk weight based on the node risk attribute, and combining the path basic weight with the risk weight to obtain a path comprehensive weight; taking the current position of the fire mask as the starting node, the safety exit closest to the current position as the target node, and selecting the node sequence with the smallest path comprehensive weight as the optimal evacuation route.

[0019] By adopting the above technical solution, key nodes are extracted from the structural characteristics of the building to generate a topological map, the node risk attributes are determined according to the temperature distribution, the optimal route is planned according to the weight, and the firefighters' positions and exits are combined. The various features cooperate with each other to provide firefighters with accurate evacuation paths and improve evacuation efficiency.

[0020] In a second aspect, an embodiment of the present application provides a fire mask, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to make the fire mask perform the method described in the first aspect and any possible implementation method of the first aspect.

[0021] In a third aspect, an embodiment of the present application provides a computer program product comprising instructions. When the computer program product is run on a fire mask, the fire mask executes the method described in the first aspect and any possible implementation method of the first aspect.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions. When the instructions are executed on a fire mask, the fire mask executes the method described in the first aspect and any possible implementation method of the first aspect.

[0023] It can be understood that the firefighting mask provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the method provided in the embodiment of the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, and will not be repeated here.

[0024] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0025] 1. This application enables the fire mask to respond promptly after being worn, and uses an image acquisition device and an infrared sensor to respectively collect on-site environmental images containing information on the edge of the flame and the outline of the building, as well as temperature distribution abnormality information containing temperature values ​​and position information, and inputs the dynamic weight adjustment neural network model to accurately identify areas with dense smoke and areas with clear environmental details, and then improves the image fusion weight of the corresponding area based on this, superimposes the temperature and contour information to obtain a regional differentiated fusion image, highlights the heat distribution and building structure characteristics, and finally enhances the boundaries of key areas and displays them in real time, so that firefighters can quickly and intuitively grasp the on-site environment and improve rescue safety and efficiency.

[0026] 2. This application extracts the temperature feature points in the temperature distribution anomaly information and the contour feature points in the contour line information after real-time display of the edge-enhanced image, establishes a corresponding relationship between the two to achieve spatial registration, obtains a heat contour overlay map and displays it in real time, so that the temperature distribution and the building contour are accurately aligned in space, and the relationship between the high-temperature area and the building structure is intuitively presented. Firefighters can quickly understand the relationship between the high-temperature area and the building layout and predict danger in advance.

[0027] 3. The present application obtains real-time location coordinate information through a fire mask, realizes high-precision positioning through a combined positioning module and a fusion algorithm, obtains a specific spatial position by matching with a preset terrain database, provides accurate location reference for firefighters, constructs an indoor path topology map based on the structural characteristics of the building, generates risk attributes for path nodes based on temperature distribution anomaly information, calculates path weights based on distance, channel width and risk attributes, plans the optimal evacuation route with the current position of the fire mask and the nearest safe exit as nodes, and updates and displays it in real time. Firefighters can clearly understand their own position and the optimal escape route, and effectively avoid dangerous areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of a multimodal image fusion method based on a fire mask in an embodiment of the present application;

[0029] Figure 2 is another flowchart of a multimodal image fusion method based on a fire mask in an embodiment of the present application;

[0030] Figure 3 It is a schematic diagram of the physical device structure of a fire mask in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification of the present application, the singular expressions "one", "a kind of", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to any or all possible combinations comprising one or more of the listed items.

[0032] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0033] For ease of understanding, the following is a description of the process of the method provided by this implementation. Figure 1 , which is a flow chart of a multimodal image fusion method based on fire masks in an embodiment of the present application.

[0034] S101. In response to the wearing operation of the fire mask, an image acquisition device is used to acquire a scene environment image, and an infrared sensor is used to acquire temperature distribution abnormality information. The scene environment image includes a flame edge image and building contour line information. The temperature distribution abnormality information includes a temperature value and position information.

[0035] Among them, the fire mask refers to an intelligent protective device worn by firefighters; the wearing operation refers to the action of correctly putting the fire mask on the head; the image acquisition device refers to the visible light camera installed on the mask; the infrared sensor refers to the thermal imaging sensor used to measure temperature; the on-site environment image refers to the real-time image of the fire scene obtained by the visible light camera; the flame edge image represents the dividing line between the flame and the surrounding environment; the building contour line information represents the contour, edge and other characteristic lines of the building; the temperature distribution abnormality information represents the regional distribution of temperatures exceeding the normal range; the temperature value represents the specific temperature data; the position information represents the spatial coordinates of the temperature abnormality point.

[0036] After the fire mask is put on, it is necessary to start collecting environmental information immediately to ensure that firefighters can grasp the situation on the scene in time. Specifically, when it is detected that the mask is in contact with the head and the fixed buckle is locked, the system automatically starts the image acquisition device and infrared sensor. The image acquisition device collects color images with a resolution of 1920×1080 at a rate of 30 frames per second, and the infrared sensor collects temperature data at a resolution of 384×288 and a frequency of 25Hz. The system preprocesses the collected raw data, extracts the flame edge and building outline features, and records the values ​​and three-dimensional coordinates of abnormal temperature points.

[0037] In some embodiments, data collection and preprocessing can be achieved in the following ways: Optionally, the contact between the mask and the face is detected by a pressure sensor, and when the contact pressure exceeds a threshold and lasts for more than 3 seconds, it is determined to be effectively worn; Optionally, a binocular camera is used to collect stereo images, and depth information is obtained by parallax calculation, and spatial registration is performed in combination with infrared sensor data. It is understandable that other methods can also be used to achieve data collection and preprocessing functions, which are not limited here.

[0038] S102. Input the scene environment image and the abnormal temperature distribution information into a dynamic weight adjustment neural network model to obtain smoke-dense areas and areas with clear environmental details. The dynamic weight adjustment neural network model is used to identify smoke-dense areas and areas with clear environmental details in the scene environment image.

[0039] Among them, the dynamic weight adjustment neural network model represents a deep learning model that can adaptively adjust weights according to the characteristics of the input data; the dense smoke area refers to the area where the line of sight is severely blocked by smoke; the area with clear environmental details refers to the area with high visibility that is not affected by smoke; recognition refers to the process of analyzing the image and judging the area type.

[0040] The system needs to classify different areas to optimize the display effect. Specifically, the preprocessed visible light image and temperature distribution data are aligned according to the timestamp and input into the pre-trained neural network model. The model contains three main modules: feature extraction, region segmentation, and category judgment. ResNet50 is used as the backbone network. Through multi-scale feature fusion and attention mechanism, the model can accurately identify the boundaries of dense smoke areas and clear environment areas, and output regional segmentation masks.

[0041] In some embodiments, region recognition can be achieved in the following ways: Optionally, a multi-dimensional feature vector is constructed using image grayscale value, contrast and texture features, and a support vector machine is used for region classification; Optionally, a fully convolutional neural network is used to directly output pixel-level region segmentation results, and then morphological processing is performed to optimize the boundaries. It is understandable that other methods can also be used to achieve the region recognition function, which is not limited here.

[0042] It should be noted that the dynamic weight adjustment neural network model can process on-site environmental images (including flame edge and building outline information) and temperature distribution anomaly information (including temperature value and location information), and accurately identify areas with dense smoke and clear environmental details. The model uses ResNet50 as the backbone network, including feature extraction, region segmentation and category judgment modules. It identifies region boundaries through multi-scale feature fusion and attention mechanism, and outputs segmentation masks.

[0043] The fire mask pre-processes the visible light image and temperature distribution data by aligning the timestamps, and then inputs them into the pre-trained model. Pre-training enables the model to learn the relationship between fire image features and regional types, and can quickly process new input data;

[0044] The feature extraction formula is: , and Respectively represent the position coordinates of the pixels in the image in the horizontal and vertical directions. For the input image, the model uses the convolution kernel The convolution operation is performed by sliding on the image. The and are the indices of the convolution kernel in the horizontal and vertical directions. This is the basic method of feature extraction, which can extract features of different scales and directions of the image. For example, convolution kernels of different sizes can capture the macroscopic shape features of the flame (large convolution kernel) and the detailed features of the flame edge (small convolution kernel).

[0045] Introduced factor, is the standard deviation of the pixel values ​​in the neighborhood (e.g., neighborhood) around the pixel at position (i, j), which reflects the degree of local change in the image. In a fire scene, the pixel values ​​in key areas such as the edge of the flame and the area of ​​temperature mutation usually change greatly, and their standard deviation is also large. By dividing by the reference value And add 1, so that these areas with drastic changes can get higher weights when extracting features. For example, when When it is larger, the value of this factor will increase significantly, thereby enhancing the ability to extract features of these key areas, allowing the model to more keenly capture the characteristic performance of flames, high-temperature areas, and building contours in the image.

[0046] The region segmentation formula is:

[0047] Indicates that the pixel at position (i, j) belongs to a dense smoke area. Approximately equal to 1, or the area with clear environmental details is approximately equal to 0, Indicates the number of feature maps, which determines the total number of feature maps involved in the region segmentation calculation. Weight With bias After processing, logistic regression is performed.

[0048] ( Control cycle frequency, Controlled phase shift) simulates the periodicity of flame flickering and assists in accurately identifying area boundaries, especially in areas with dynamic changes in flame edges.

[0049] In the category judgment module, the attention weight in the category judgment formula is:

[0050] Represents the attention weight of the pixel at position (i, j), reflecting the relative importance of the pixel in the category judgment process, represents the number of feature maps, feature maps With weight After multiplication, based on the distance decay factor , ( is the distance from the pixel to the center of the high temperature area, which controls the degree of influence). At the fire scene, the high temperature area is closely related to the smoke and flame areas and is an important feature area. This attenuation factor makes the pixels close to the high temperature area obtain higher attention weights. The model focuses on the key area features and ignores irrelevant backgrounds to improve the accuracy of category judgment. For example, the pixel features close to the high temperature area have higher weights in category judgment. Based on this, the model can more accurately judge the area to which the pixel belongs. After softmax normalization, , and the region segmentation probability Multiply to get the final category judgment result , optimize judgment.

[0051] The model uses the common cross-entropy loss function for training optimization. During the training process, the model continuously adjusts its internal parameters based on a large amount of fire scene image data to minimize the cross-entropy loss function, so that the model can achieve optimal performance in the area recognition task, thereby accurately identifying areas with dense smoke and areas with clear environmental details.

[0052] In some embodiments, after step S102, the following steps may also be included:

[0053] Obtaining illumination intensity information of the scene environment image;

[0054] A first preset threshold and a second preset threshold are generated according to the light intensity information.

[0055] Among them, the light intensity information represents the brightness level of light in the environment, and the unit is lux; the first preset threshold refers to the upper limit value of the image fusion weight of the temperature distribution anomaly information, and the value range is 0.6-0.9; the second preset threshold refers to the upper limit value of the image fusion weight of the contour line information, and the value range is 0.5-0.8.

[0056] Before performing image fusion, the fire mask collects ambient light data in real time through the built-in light sensor. The light sensor uses a 16-bit resolution digital light meter with a measurement range of 0-100000 lux and a sampling frequency of 10Hz. When the light intensity is lower than 100 lux, the first preset threshold is set to 0.9 and the second preset threshold is set to 0.5 to highlight the temperature information; when the light intensity is in the range of 100-1000 lux, the first preset threshold is linearly reduced to 0.8 and the second preset threshold is linearly increased to 0.6; when the light intensity is in the range of 1000-10000 lux, the first preset threshold is further reduced to 0.7 and the second preset threshold is increased to 0.7; when the light intensity exceeds 10000 lux, the first preset threshold is set to 0.6 and the second preset threshold is set to 0.8 to enhance the display effect of contour information. The dynamic adjustment of the threshold adopts a piecewise linear interpolation method, which is updated every 100ms, and the adjustment step does not exceed 0.05 to ensure a smooth transition of the display effect. Under extreme lighting conditions, such as when the light intensity is less than 10 lux or more than 50,000 lux, the mask will start the fill light LED or automatically adjust the display brightness to ensure the stability of the image fusion effect. The lighting data and threshold parameters are simultaneously recorded in the local memory of the mask for subsequent data analysis and system optimization.

[0057] S103, increasing the image fusion weight of the abnormal temperature distribution information in the smoke-dense area to a first preset threshold, and increasing the image fusion weight of the contour line information in the area with clear environmental details to a second preset threshold, to obtain an image fusion weight allocation scheme for each area of ​​the scene environment image.

[0058] Among them, the image fusion weight indicates the importance of different types of image information in the fusion process; the first preset threshold refers to the maximum weight value of temperature information in the smoke-dense area; the second preset threshold refers to the maximum weight value of contour information in the clear environment area; the weight allocation scheme represents the set of weight parameters set for different areas; weight boosting refers to the process of gradually increasing the weight value of a certain type of information to the preset threshold; contour line information refers to structural features such as the edges and outlines of buildings.

[0059] Weight allocation needs to be performed immediately after the region identification is completed to ensure the accuracy of the subsequent fusion process. Specifically, the system first reads the region segmentation mask and divides the image into smoke-dense areas and clear environment areas. For smoke-dense areas, the temperature information weight starts from the baseline value of 0.3 and gradually increases to the first preset threshold of 0.8 in steps of 0.1; for clear environment areas, the contour information weight starts from the baseline value of 0.4 and increases to the second preset threshold of 0.7 in steps of 0.1. A 5-pixel transition zone is set at the region boundary, and the weight value is linearly interpolated to ensure a smooth transition. Finally, a weight matrix with the same size as the original image is generated, and each pixel position contains two parameters: temperature information weight and contour information weight.

[0060] In some embodiments, weight allocation can be achieved in a variety of ways: optionally, first calculate the average brightness and contrast in the area, dynamically adjust the weight reference value based on these characteristic values, then apply Gaussian blur to smooth the weight distribution, and finally perform normalization to ensure that the sum of the weights is 1; optionally, adopt a distance-based weight allocation strategy, calculate the distance from each pixel to the region boundary, map the distance to a weight value, introduce a temperature gradient as an adjustment factor, and finally obtain the weight value through exponential function mapping. It is understandable that other methods can also be used to achieve intelligent weight allocation, which is not limited here.

[0061] S104: superimposing the temperature distribution anomaly information and the contour line information according to the image fusion weight allocation scheme to obtain a regional differential fusion image, wherein the regional differential fusion image includes heat distribution characteristics and building structure characteristics.

[0062] Among them, superposition means weighted fusion of different types of image information according to weights; regional differentiated fusion image refers to the composite image obtained by using different fusion strategies in different regions; heat distribution characteristics represent the spatial distribution of temperature; building structural characteristics refer to the geometric shape and spatial structure information of the building; image fusion refers to the process of comprehensively processing multi-source image information to form a new image.

[0063] Image fusion is performed immediately after the weight distribution scheme is determined to generate a comprehensive information display that is easy to observe. Specifically, the system first converts the temperature distribution data into a pseudo-color image, using a gradient color band from blue to red to represent temperature changes. Then the contour line information is converted into a grayscale image and edge enhancement is performed. According to the weight distribution scheme, the weighted sum is calculated at each pixel position: output pixel value = temperature image pixel value × temperature weight + contour image pixel value × contour weight. To improve the visual effect, the fusion result is contrast stretched and detail enhanced, and color annotations are added to illustrate the corresponding relationship between temperature values.

[0064] In some embodiments, image fusion can be achieved in a variety of ways: optionally, first perform wavelet transform on the image, perform weighted fusion at each scale, then perform inverse transform to reconstruct the image, and finally apply adaptive histogram equalization to enhance the display effect; optionally, use a fusion algorithm based on pulse coupled neural network to achieve adaptive fusion of information through dynamic connections between neurons, and introduce local contrast factors to optimize detail performance. It is understandable that other methods can also be used to achieve multimodal image fusion processing, which is not limited here.

[0065] S105, enhancing the boundary of the area where the temperature exceeds the preset temperature threshold and the structure edge in the building outline information to obtain an edge-enhanced image, and displaying the edge-enhanced image in real time on the display of the fire mask.

[0066] Among them, the preset temperature threshold refers to the standard temperature value used to judge the dangerous area, which is usually set to 85°C; the area boundary refers to the outer contour line of the temperature abnormality area; the structural edge refers to the contour line of the walls, doors and windows and other components of the building; edge enhancement means using image processing technology to improve the contrast and clarity of the edge part; edge-enhanced image refers to the image after edge enhancement processing; real-time display means updating the display content at a speed that the human eye cannot detect the delay; the fire mask display refers to the transparent display screen installed on the mask.

[0067] Edge enhancement processing is performed immediately after image fusion is completed to highlight the boundary information of dangerous areas and building structures. Specifically, the system first performs threshold segmentation on the temperature data and marks the area with a temperature exceeding 85°C as a high-temperature area. The Canny operator is used to extract the boundary contour of the high-temperature area to obtain a binary boundary image. At the same time, the Sobel operator is applied to the building contour line information for gradient calculation to extract significant structural edges. The two types of edge information are superimposed, the high-temperature area boundary is represented by red, and the building structure edge is represented by white. Morphological processing is performed on the edge lines, including thinning, connecting breakpoints, and removing noise. Finally, an adaptive contrast enhancement algorithm is applied to improve the significance of the edge. The edge line width is set to 3 pixels and the brightness value is increased by 50%.

[0068] In some embodiments, edge enhancement and display can be achieved in a variety of ways: optionally, multi-scale edge detection is first performed, and edge information of multiple scales is extracted by combining the Laplacian operator and the Gaussian difference operator, and then the enhanced edge image is obtained by weighted fusion, and finally the edge is adaptively thresholded to remove false edges; optionally, edge features are extracted by using a deep learning-based edge detection network, and edge performance is enhanced by residual connections and attention mechanisms, and color space conversion is introduced to improve the visual prominence of the edge. It is understandable that other methods can also be used to achieve edge enhancement processing and display effect optimization, which are not limited here.

[0069] The following is a more detailed description of the process of the method provided by this implementation. Figure 2 , is another flow chart of the multimodal image fusion method based on fire masks in an embodiment of the present application.

[0070] S201 , enhancing the boundary of the region where the temperature exceeds a preset temperature threshold and the structure edge in the building outline information to obtain an edge-enhanced image.

[0071] It can be understood that step S201 is similar to step S105 and will not be described in detail here.

[0072] S202: extracting temperature feature points in the temperature distribution anomaly information and contour feature points in the contour line information.

[0073] The abnormal temperature distribution information is collected by the infrared sensor, which records the temperature value and location information of each point in the fire environment. The contour line information is obtained by the image acquisition device, which includes the structural edge and spatial contour information of the building. The feature point is the key reference position for subsequent registration.

[0074] The specific implementation of the extraction process is as follows: For temperature feature points, the temperature data is first divided into 16×16 grid units, and the average and standard deviation of the temperature values ​​are calculated in each grid. When the difference between the temperature value of a point and the grid average value exceeds 2 times the standard deviation, and the temperature value of the point is greater than that of the 8 adjacent points, it is marked as a temperature feature point. At the same time, the temperature value, two-dimensional coordinate position and grid number of the point are recorded. For contour feature points, the contour line image is divided into grids of the same size, and the intersections and mutation points of the lines are detected in each grid. Specifically, by calculating the variance of the grayscale value in the neighborhood of each pixel point, when the variance exceeds the set threshold and the grayscale value of the point is the local maximum / minimum, it is marked as a contour feature point. The position coordinates, grayscale value and grid affiliation information of the feature point are also recorded. Finally, two sets of feature point sets and their attribute information are obtained.

[0075] S203: Establish a correspondence between the temperature feature point and the contour feature point to obtain a feature correspondence.

[0076] The correspondence between feature points reflects the matching relationship between temperature features and structural features in spatial position, and is the basis for subsequent spatial registration.

[0077] The corresponding relationship establishment process adopts a partition pairing strategy: first, the two groups of feature points are grouped according to the grid number, and matching is performed only in the same or adjacent grids, which can reduce mismatching and improve efficiency. For each temperature feature point, the Euclidean distance between it and all contour feature points in the same area is calculated, and the contour feature point with the smallest distance and less than 5 pixel units is selected as its corresponding point. If a contour feature point has been matched by other temperature feature points and the new distance is smaller, the matching relationship is updated. For each pair of matching points, a data structure is established to record: the temperature feature point number, the corresponding contour feature point number, the spatial coordinates of the two points, the temperature value, the grayscale value, and the distance between them. Finally, all matching pairs are sorted from small to large according to the distance value, and the first 80% with the smallest distance are retained as valid corresponding relationships.

[0078] S204: spatially align the heat distribution features in the temperature distribution anomaly information with the building structure features in the contour line information according to the feature correspondence to obtain a heat contour overlay map.

[0079] The goal of spatial registration is to accurately align the temperature distribution information to the building outline so that the two types of information match perfectly in spatial position. The registered overlay image can intuitively show the spatial association between the high temperature area and the building structure.

[0080] The registration process is divided into the following specific steps: First, the coordinate transformation parameters are calculated based on the correspondence of the feature points. The first 20 pairs of matching points with the smallest distance value are selected to construct an overdetermined set of equations. The unknown quantities of the equations are the rotation angle θ, the horizontal offset dx, and the vertical offset dy. The optimal transformation parameters are obtained by solving the set of equations using the least squares method. Then, the temperature distribution data is transformed: for each temperature sampling point, the obtained transformation parameters are applied to calculate its new position in the contour coordinate system. The transformed temperature data is bilinearly interpolated to generate a temperature distribution map with a resolution consistent with the contour image. Finally, the temperature distribution map is converted to a pseudo-color display: temperature values ​​of 35-45℃ are mapped to blue, 45-60℃ are mapped to green, 60-80℃ are mapped to yellow, and above 80℃ are mapped to red. The pseudo-color temperature map is composited with the contour line image to obtain the final heat contour overlay map.

[0081] S205: Display the heat profile overlay image in real time on the display of the fire mask.

[0082] The display of the fire mask uses a transparent OLED display with a resolution of 1920×1080 pixels, a refresh rate of 30Hz, and supports 8-bit color depth display. The heat contour overlay is a composite image containing temperature distribution and building contours. Each pixel contains RGB three-channel color information and a transparency channel. Real-time display means that the display content is updated every 16.7ms to ensure the continuity and timeliness of image display.

[0083] The specific display process includes: first, the heat contour overlay is scaled according to the display resolution, keeping the image aspect ratio unchanged. The scaled image is processed by color mapping: the temperature information is represented by a gradient of red, yellow and green, and the building outline is represented by a white line. The transparency of the image is adjusted, the temperature distribution area is set to 70% opacity, the building outline is set to 90% opacity, and the background area is completely transparent. The processed image data is written to the display buffer, and the display controller scans and outputs it to the OLED panel line by line. After each frame of the display is drawn, the vertical synchronization signal is triggered to prompt the processor to prepare the next frame of image data. The entire display process transmits data through DMA to reduce the burden on the processor. The display position is fixed in the central area of ​​the display, covering a field of view of 120 degrees horizontally and 90 degrees vertically.

[0084] S206: Defogging the densely smoked area and contrast enhancing the area with clear environmental details to obtain an enhanced image.

[0085] Smoke-dense areas refer to areas in the image that are severely obscured by smoke, which are characterized by low image contrast and blurred details. Areas with clear environmental details refer to areas that are not obscured by smoke, but may have insufficient contrast due to insufficient lighting. Dehazing is intended to restore image details in areas obscured by smoke, and contrast enhancement is used to improve the visual clarity of the image.

[0086] The processing steps are as follows: For densely smoked areas, first estimate the atmospheric illumination value A. Divide the image into 15×15 local windows, and take the minimum channel dark primary color value in each window. Perform median filtering on the dark primary color image with a window size of 31×31 pixels to obtain a transmittance map. Calculate the defogging gain coefficient for each pixel based on the transmittance map. Apply the gain coefficient to the three RGB channels of the original image for brightness compensation. For areas with clear environmental details, calculate the cumulative histogram of the image, and determine the stretching range based on the histogram distribution. Perform histogram equalization on the three RGB channels, with a mapping range of [0.1, 0.9]. Finally, perform boundary transition smoothing on the two processed areas to avoid obvious boundaries between areas. The entire process outputs an enhanced image of the same size as the original image.

[0087] S207: Switch and display the enhanced image and the edge enhanced image on the display of the fire mask.

[0088] The enhanced image highlights the details in the scene, and the edge-enhanced image emphasizes the structural edge features. Switching display means that the two images are displayed alternately on the display according to specific rules. The display of the fire mask also uses the aforementioned transparent OLED screen.

[0089] The specific implementation method of switching display is: establish a double buffer structure, buffer A stores enhanced images, and buffer B stores edge enhanced images. The display controller switches the display content 10 times per second, that is, once every 100ms. The switching moment is triggered by the timer interrupt, and the interrupt service program modifies the data source address of the display controller to point to the new buffer. To avoid screen flickering, buffer switching is performed during vertical blanking. A cross-fade effect is used when switching: it takes 50ms to complete the gradual change of transparency from 0% to 100%. The fire mask is set with a manual switching button. Pressing the button pauses the automatic switching and fixes the current image. Press the button again to resume the automatic switching mode. The entire switching process is smooth and continuous, and does not affect the firefighter's observation environment.

[0090] S208. When it is detected that the temperature value in the temperature distribution abnormality information exceeds the preset temperature threshold or the ambient toxic gas concentration exceeds the preset toxic gas concentration threshold, the target high temperature position corresponding to the temperature value is obtained, and the target toxic gas position corresponding to the ambient toxic gas concentration is obtained. The target high temperature position is the position where the temperature exceeds the preset temperature threshold.

[0091] The target high temperature position is the position where the temperature exceeds the preset temperature threshold, which is 85°C. Temperatures exceeding this threshold are considered target high temperature areas. Environmental toxic gases include carbon monoxide, hydrogen sulfide, and hydrogen cyanide, with preset concentration thresholds of 100ppm, 10ppm, and 10ppm, respectively. The target high temperature position is located by the temperature sensor array, accurate to 0.5 meters. The target toxic gas position is located by the gas sensor array, accurate to 1 meter.

[0092] The detection process is performed as follows: The temperature sensor array samples once every 100ms to form a 32×32 temperature data matrix. The temperature value of each sampling point in the matrix is ​​analyzed, and all positions exceeding 85℃ are marked. Connected domain analysis is performed on the marked positions, and adjacent high-temperature points are classified as a target area. The centroid coordinates and area of ​​each target area are calculated, and the areas with an area greater than 4 sampling points are selected as valid high-temperature targets. The spatial position, temperature mean and area size of each valid target are recorded. The gas sensor array contains 3 groups of sensors, which detect carbon monoxide, hydrogen sulfide and hydrogen cyanide respectively. Each group of sensors takes turns to sample at an interval of 200ms, and the sampling data is temperature compensated and humidity corrected. When the concentration of any gas exceeds the corresponding threshold, the positioning algorithm is started. Based on the concentration difference of multiple sensors, the location of the toxic gas source is calculated using the trilateral positioning method. The positioning results are converted into azimuth and distance relative to the fire mask.

[0093] S209, displaying the target high temperature position and the target poisonous gas position on the display.

[0094] The display uses a transparent OLED screen with a resolution of 1920×1080, which supports the superposition of multiple layers of image information. The target location is marked with specific icons, high-temperature targets are represented by red triangles, and the three types of poisonous gases are represented by yellow, green and blue circles respectively.

[0095] The display process includes the following steps: Establish a display coordinate system with the coordinate origin directly in front of the firefighter's perspective. Calculate the projection position of the target on the display screen based on the target's azimuth and distance information. Determine the icon size, use 32×32 pixel icons within 2 meters, 24×24 pixel icons for 2-5 meters, and 16×16 pixel icons for 5 meters. Draw the corresponding icon at the target position, and the icon is semi-transparent with a transparency set to 60%. Add a text label next to the icon to display the specific temperature value or gas concentration, and the font size changes with distance. When multiple targets overlap, they are displayed in order of threat level, with temperature taking precedence over gas, and gases sorted by toxicity. The icon position is updated every 200ms to ensure real-time tracking of target movement.

[0096] S210: Send warning information including the target high temperature position and the target poisonous gas position.

[0097] Warning information adopts the standardized JSON data format, including fields such as target type, location coordinates, threat level, etc. Information is sent through the wireless communication module, supporting two transmission modes: 4G network and short-distance ad hoc network.

[0098] The specific implementation of the sending process is as follows: construct a warning message in JSON format, including fields such as sending timestamp, fire mask ID, target type code, spatial coordinates, detection value, threat level, etc. The data is encrypted with AES-256 to ensure transmission security. The data is sent to the command center server via the 4G network first. If the sending fails, the self-organizing network mode is enabled. In the self-organizing network mode, a mesh topology is formed between the fire masks, and a hopping routing protocol is used to transmit data. Each warning message is retransmitted three times, with intervals of 1 second, 2 seconds, and 4 seconds respectively. After receiving the data, the server returns a confirmation message, and the fire mask stops retransmitting after receiving the confirmation. The data is also saved in the local storage as a historical record. The entire sending process ensures that the warning information is conveyed to the command center in a timely and reliable manner.

[0099] S211. Obtain the real-time position coordinate information of the fire mask.

[0100] The fire mask is equipped with a combined positioning module, including an inertial measurement unit (IMU) and a UWB positioning unit. The inertial measurement unit integrates a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer, as well as a module with Beidou positioning function (compatible with a variety of satellite positioning systems to meet the needs of precise positioning in different environments), with a sampling frequency of 200Hz. The UWB positioning unit operates in the 6.5GHz frequency band and consists of 4 fixed base stations and a mobile tag on the mask, with a positioning accuracy of 0.3 meters. The real-time position coordinates use a right-handed coordinate system, including position data and heading angle data in the three directions of x, y, and z.

[0101] The positioning process is divided into the following steps: IMU collects acceleration and angular velocity data, fuses magnetometer data through Kalman filter algorithm, and calculates the attitude angle of the mask. Gravity compensation and zero bias correction are performed on the acceleration data, and the position increment is obtained by double integration. The UWB system calculates the distance from the tag to each base station by measuring the signal flight time, and uses trilateral measurement to calculate the absolute position of the tag. The position increment calculated by the IMU is fused with the absolute position measured by the UWB. The IMU data is used to supplement the position change between the UWB data, and the UWB data is used to periodically correct the accumulated error. The fused position data contains the three-dimensional coordinates (x, y, z) relative to the origin of the base station and the mask orientation angle φ. The coordinate resolution is 0.1 meters, the angle resolution is 0.1 degrees, and the data update rate is 10Hz.

[0102] S212: Match the real-time position coordinate information with the building structure features in a preset terrain database to obtain the specific spatial position of the fire mask.

[0103] The preset terrain database stores the 3D structural model of the building, including the geometric information and attribute information of components such as walls, doors, windows, and stairs. The structural characteristics of the building include data such as space dimensions, component types, and material attributes. The specific spatial location refers to the floor number, room number, and relative position description within the building.

[0104] The matching process is performed as follows: First, coordinate system conversion is performed to convert the location data in the UWB base station coordinate system to the building model coordinate system. The conversion uses a rigid body transformation matrix, which includes rotation and translation. The transformation parameters are calculated using the known coordinates of the four base stations. The converted location data is subjected to spatial collision detection with the three-dimensional model to determine the current spatial region. The collision detection uses an octree algorithm to recursively divide the space into sub-regions. Traverse from the top-level node until the smallest sub-region containing the current location is found. Query the spatial attribute information associated with the region to obtain the floor number and room number. Determine the vertical position based on the altitude, and determine the functional area based on the room type. Finally, a formatted location description is output, including the expression form of "x floor y area z room".

[0105] S213, extracting passages, doors, stairs, corner points and emergency exits from the structural features of the building as path nodes, and generating an indoor path topology map according to the path nodes.

[0106] Path nodes refer to key locations in the traversable area of ​​a building. Passages include corridors and the centerline of passages. Doors include the center points of room doors and fire doors. Stairs include the center points of stairways and stair landings. Corner points are the turning points of passages. Emergency exits are exit locations pointing to the outdoors. Indoor path topology is a network structure composed of nodes and connecting lines, which represents the connection relationship of indoor traversable paths.

[0107] The extraction process includes the following steps: Separate the ground layer model from the three-dimensional model of the building and project it into a two-dimensional plane map. Refine the plane map, extract the center line of the channel, and set nodes at a sampling interval of 2 meters for the center line. Identify the locations of all doors and set nodes at the center of the door. Extract the starting point, end point and middle platform of the stairs and set nodes. Detect the corners of the channel and set nodes at the corners. Mark all emergency exit locations and set nodes. Assign a unique number and type attribute to each node. Analyze the connectivity between nodes, and establish a connection relationship when there is a direct channel between two nodes. The connection relationship records the straight-line distance and channel width between two nodes. All nodes and connection relationships form an undirected graph structure and are stored in the form of an adjacency matrix. The element values ​​in the matrix represent the distance between nodes, and the distance between unconnected nodes is set to infinity.

[0108] S214: Generate a node risk attribute for each path node according to the temperature distribution anomaly information at each path node.

[0109] The temperature distribution anomaly information comes from the data collected by the infrared sensor, which records the temperature value of each spatial location. The node risk attribute is a value from 0 to 100, reflecting the degree of danger at the path node, 0 means safe, and 100 means extremely dangerous. Path nodes include channel nodes, door nodes, stair nodes, corner nodes, and exit nodes.

[0110] The risk attribute generation process is as follows: For each path node, a detection area with a radius of 3 meters is set around it. The data of all temperature sampling points in the area are extracted to calculate the maximum temperature, average temperature and temperature gradient in the area. The maximum temperature is mapped to the risk value, and the temperature below 85°C is mapped to the 0-30 interval, the temperature between 85-120°C is mapped to the 31-60 interval, and the temperature above 120°C is mapped to the 61-100 interval. The risk value increases by 10 points for every 10°C / meter increase in the temperature gradient. If the node is located at a door or staircase, the basic risk value increases by 20 points, considering the special importance of these locations. When the risk value of a node reaches 90 or above, it is marked as a prohibited node. For the risk values ​​of adjacent nodes, the distance weighted average is used for smoothing to avoid sudden changes in the risk value. Finally, each node obtains a risk attribute value for subsequent path planning.

[0111] S215. Set a path basic weight according to the distance between each node on the path and the channel width, set a risk weight based on the risk attribute of the node, and combine the path basic weight with the risk weight to obtain a path comprehensive weight.

[0112] The path base weight reflects the basic difficulty of the path and is related to the distance and width of the channel. The risk weight reflects the danger level of the path and is related to the risk attribute of the node. The path comprehensive weight is a weighted combination of these two weights and is used to evaluate the overall travel cost of the path.

[0113] The weight calculation process is performed as follows: The basic weight of the path is calculated using a combined formula of distance and width. The straight-line distance between the two nodes is divided by the channel width to get the basic value, with the distance unit in meters and the width unit in meters. When the channel width is less than 1.2 meters, the basic weight increases by 50%. For door nodes, the basic weight increases by 5 points. For stair nodes, the basic weight increases according to the vertical height, and the weight increases by 2 points for every increase of 1 meter. The risk weight is determined by the risk attribute values ​​of the nodes at both ends of the path, and the maximum value of the two risk attributes is taken as the risk weight of the path segment. The basic weight and risk weight are weighted and summed, with the basic weight accounting for 40% and the risk weight accounting for 60%, to obtain the comprehensive weight of the path. When the risk attribute of any node at either end of the path exceeds 90, the comprehensive weight of the path is set to infinity, indicating that the path is prohibited.

[0114] S216: Taking the current position of the fire mask as the starting node and the safe exit closest to the current position as the target node, a node sequence with the smallest comprehensive weight of the path is selected as the optimal evacuation route.

[0115] The origin node is the nearest path node corresponding to the current location of the fire mask. The safe exit node is the exit location leading to the outdoors. The optimal evacuation route is a series of continuous path nodes from the origin node to the target node, and the sum of the path comprehensive weights between these nodes is the smallest.

[0116] The route planning process is implemented as follows: First, determine the starting node, compare the current position of the fire mask with all path nodes, and select the node with the smallest Euclidean distance as the starting node. Scan all safe exit nodes, calculate their straight-line distance to the starting node, and select the nearest exit as the target node. Use the Dijkstra algorithm to calculate the shortest path, and use the comprehensive weight of the path as the weight of the edge. The algorithm starts from the starting node and maintains a set of visited nodes and a set of nodes to be visited. Each time, select the node with the smallest distance to the starting node from the set to be visited, add it to the visited set, and update the distance value of its adjacent nodes. Repeat this process until the target node is found. The shortest path obtained by backtracking is the optimal evacuation route. Record the node sequence in the route in order, including node number, location coordinates, and node type.

[0117] S217: superimpose and display the optimal evacuation route on the edge-enhanced image, and update and display it in real time on the display of the fire mask.

[0118] The optimal evacuation route is a guide path formed by a series of path nodes. The data structure contains node coordinate sequences and path direction information. Edge-enhanced images are processed images that highlight the contour features of buildings. The display of the fire mask uses a transparent OLED screen with a resolution of 1920×1080 and a refresh rate of 30Hz. Real-time update means refreshing the display content every 16.7 milliseconds.

[0119] The specific implementation of the superimposed display process is as follows: First, draw the evacuation route as an arrow guide line with a width of 12 pixels, using fluorescent green (RGB value: 0, 255, 0) and setting the opacity to 80%. The guide line is drawn using the B-spline curve algorithm to smoothly transition the straight line segments between adjacent path nodes. Draw a 30×30 pixel arrow mark at each turning point, and the arrow direction is consistent with the direction of the path. Draw a circular mark with a radius of 8 pixels at the node position, and different types of nodes use different colors: channel nodes are white, door nodes are blue, stair nodes are yellow, and safety exits are red. Draw a dynamic guide arrow between the current position and the next node, and the arrow flashes twice per second. Draw a moving light spot every 50 pixels on the path, and the light spot flows along the path direction at a speed of 60 pixels per second. The straight-line distance to the nearest safety exit and the required time are displayed in the upper right corner of the display, with the distance accurate to 0.1 meters and the time in seconds. Alpha blend the drawn route layer with the edge-enhanced image, with the route layer on top and the edge-enhanced image on the bottom. When the wearer's position changes, the relative position from the current position to each node is recalculated and the route display is updated. The entire display process uses a double buffering mechanism, and the background buffer is drawn before switching to the foreground display to avoid screen tearing.

[0120] In some embodiments, steps S202 to S217 may be performed after step S201 or after step S105.

[0121] The display of the fire mask adopts a multi-layer display structure, which includes the real-time camera original image, edge enhancement or defogging enhanced image, thermal contour overlay layer, target identification layer and evacuation route navigation layer from the bottom to the top. Each layer has an independent display buffer and transparency control, and the clear presentation of information is ensured through precise transparency settings. The bottom image is completely opaque, the edge enhancement or defogging layer is set to 40% transparency, the thermal contour layer is 30% transparency, and the target identification layer and navigation layer are each 20% transparent. This transparency configuration ensures the recognizability of each layer of information while avoiding visual interference.

[0122] Layer switching is achieved through intelligent control. The edge-enhanced image and the defogging-enhanced image are switched through the button on the right side of the mask. The switching process is designed with a 50-millisecond gradient transition to ensure a smooth visual experience. When a dangerous target is detected, the target identification layer automatically becomes visible. Similarly, when the evacuation route is calculated, the navigation layer is automatically displayed. Each layer supports display or hiding control through voice commands, providing a convenient human-computer interaction method.

[0123] The display uses triple buffering technology, including three buffers: the current display frame, the next display frame, and the frame being synthesized. The buffer switching is controlled by the vertical synchronization signal to effectively prevent screen tearing. The frame synthesis process is strictly carried out in layer order: first write the real-time camera image, then superimpose the edge enhancement or defogging image, then add the thermal contour information, then superimpose the target identification symbol, and finally draw the navigation route. The entire process is accelerated by the GPU to ensure display performance.

[0124] The heat contour uses a red and yellow gradient color code, the dangerous target uses a 1Hz frequency flashing effect to increase the warning, the navigation route is designed with a flowing light effect to enhance the guidance effect, and the switching between layers uses a smooth gradient transition. The mask supports multiple interaction methods, including button control, voice command and automatic mode, which can automatically optimize the display content according to environmental conditions.

[0125] In terms of data processing, the fire mask receives RGB images from the camera at 30fps, temperature matrices from thermal imaging at 25fps, sensor status updates at 30Hz, and location information updates at 10Hz. The display output maintains an overall refresh rate of 60Hz, with basic images updated at 30Hz, thermal data updated at 25Hz, and navigation information updated at 10Hz. This differentiated update frequency design not only ensures the real-time nature of key information, but also optimizes the efficiency of hardware resource usage.

[0126] The mask is designed with a complete fault handling solution. When the layer synthesis is abnormal, it automatically falls back to the basic display mode; when the display temperature is too high, the protection mechanism is activated to automatically reduce the brightness; if the layer refresh times out, the display module is automatically reset. These mechanisms ensure the reliable operation of the display function in various situations. Through this multi-level display management solution, the fire mask can provide users with clear, stable, and real-time visual information support.

[0127] The firefighting mask in the embodiment of the present invention is described below from the perspective of hardware processing. Figure 3 , is a schematic diagram of the physical device structure of a fire mask in an embodiment of the present application.

[0128] It should be noted that Figure 3 The structure of the fire mask shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0129] like Figure 3As shown, the fire mask includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage part 308 to the random access memory (RAM) 303, such as executing the method described in the above embodiment. In the RAM 303, various programs and data required for system operation are also stored. The CPU 301, the ROM 302 and the RAM 303 are connected to each other through the bus 304. The input / output (I / O) interface 305 is also connected to the bus 304.

[0130] The following components are connected to the I / O interface 305: an input section 306 including an audio input device, a button switch, etc.; an output section 307 including a liquid crystal display (LCD) and an audio output device, an indicator light, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed so that a computer program read therefrom is installed into the storage section 308 as needed.

[0131] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 309, and / or installed from a removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, various functions defined in the present invention are performed.

[0132] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0133] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. Each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings.

[0134] Specifically, the fire mask of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, the multimodal image fusion method based on the fire mask provided in the above embodiment is implemented.

[0135] As another aspect, the present invention further provides a computer-readable storage medium, which may be included in the fire mask described in the above embodiment; or may exist independently without being assembled into the fire mask. The above storage medium carries one or more computer programs, and when the above one or more computer programs are executed by a processor of the fire mask, the fire mask implements the multimodal image fusion method based on the fire mask provided in the above embodiment.

[0136] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0137] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0138] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. A multimodal image fusion method based on fire masks, characterized in that: Applied to firefighting masks, the method comprises: In response to the wearing operation of the fire mask, an image acquisition device is used to acquire a scene environment image, and an infrared sensor is used to acquire temperature distribution abnormality information, wherein the scene environment image includes a flame edge image and building outline information, and the temperature distribution abnormality information includes a temperature value and position information; Inputting the scene environment image and the temperature distribution abnormality information into a dynamic weight adjustment neural network model to obtain a smoke-dense area and an area with clear environmental details, wherein the dynamic weight adjustment neural network model is used to identify the smoke-dense area and the area with clear environmental details in the scene environment image; In the dense smoke area, the image fusion weight of the abnormal temperature distribution information is increased to a first preset threshold, and in the area with clear environmental details, the image fusion weight of the contour line information is increased to a second preset threshold, so as to obtain an image fusion weight allocation scheme for each area of ​​the scene environment image; The temperature distribution anomaly information is superimposed on the contour line information according to the image fusion weight allocation scheme to obtain a regional differential fusion image, wherein the regional differential fusion image includes heat distribution features and building structure features; The boundaries of the regions where the temperature exceeds the preset temperature threshold and the structural edges in the building outline information are enhanced to obtain edge-enhanced images, and the edge-enhanced images are displayed in real time on the display of the fire mask.

2. The method according to claim 1, characterized in that After the step of inputting the scene environment image and the abnormal temperature distribution information into the dynamic weight adjustment neural network model to obtain the smoke-dense area and the environment detail-clear area, the method further includes: Acquire illumination intensity information of the scene environment image; A first preset threshold and a second preset threshold are generated according to the light intensity information.

3. The method according to claim 1, characterized in that After the step of displaying the edge-enhanced image on the display of the firefighting mask in real time, the method further comprises: Extracting temperature feature points in the temperature distribution anomaly information and contour feature points in the contour line information; Establishing a correspondence between the temperature feature points and the contour feature points to obtain a feature correspondence; According to the feature correspondence, the heat distribution features in the temperature distribution anomaly information and the building structure features in the contour line information are spatially registered to obtain a heat contour overlay map; The heat profile overlay is displayed in real time on the display of the firefighting mask.

4. The method according to claim 3, characterized in that After the step of displaying the heat profile overlay in real time on the display of the fire mask, the method further comprises: Defogging the dense smoke area and enhancing the contrast of the clear environment detail area to obtain an enhanced image; The enhanced image and the edge enhanced image are switched and displayed on the display of the fire fighting mask.

5. The method according to claim 1, characterized in that After the step of displaying the edge-enhanced image on the display of the firefighting mask in real time, the method further comprises: When it is detected that the temperature value in the temperature distribution abnormality information exceeds a preset temperature threshold or the ambient toxic gas concentration exceeds a preset toxic gas concentration threshold, a target high temperature position corresponding to the temperature value is obtained, and a target toxic gas position corresponding to the ambient toxic gas concentration is obtained, wherein the target high temperature position is a position where the temperature exceeds the preset temperature threshold; Displaying the target high temperature position and the target poisonous gas position on the display; Send a warning message including the target high temperature position and the target poisonous gas position.

6. The method according to claim 1, characterized in that After the step of displaying the edge-enhanced image on the display of the firefighting mask in real time, the method further comprises: Obtaining real-time position coordinate information of the fire mask; Matching the real-time position coordinate information with the building structure features in a preset terrain database to obtain the specific spatial position of the fire mask; Building an indoor path topology map based on the structural characteristics of the building, mapping the temperature distribution abnormality information to each path node of the indoor path topology map, generating a path pass weight according to the temperature value of each path node, and calculating the optimal evacuation route based on the path pass weight; The optimal evacuation route is superimposed and displayed on the edge-enhanced image, and is updated and displayed in real time on the display of the fire mask.

7. The method according to claim 6, characterized in that The steps of constructing an indoor path topology map based on the structural characteristics of the building, mapping the temperature distribution abnormality information to each path node of the indoor path topology map, generating a path traffic weight according to the temperature value of each path node, and calculating the optimal evacuation route based on the path traffic weight specifically include: Extracting passages, doors, stairs, corner points and emergency exits from the structural features of the building as path nodes, and generating an indoor path topology map according to the path nodes; Generating a node risk attribute for each of the path nodes according to the temperature distribution anomaly information at each of the path nodes; A path basic weight is set according to the distance between each of the path nodes and the channel width, a risk weight is set based on the risk attribute of the node, and the path basic weight is combined with the risk weight to obtain a path comprehensive weight; The current position of the fire mask is taken as the starting node, the safety exit closest to the current position is taken as the target node, and the node sequence with the smallest comprehensive weight of the path is selected as the optimal evacuation route.

8. A firefighting mask, characterized in that: The fire mask comprises: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions to make the fire mask execute the method as described in any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that: When the instruction is executed on the fire-fighting mask, the fire-fighting mask is caused to execute the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that When the computer program product is run on a fire-fighting mask, the fire-fighting mask is enabled to perform the method according to any one of claims 1 to 7.

Images