Fire-fighting hidden danger real-time monitoring and early warning method and system combined with image recognition

Abstract

This invention relates to the field of fire monitoring technology, specifically disclosing a method and system for real-time monitoring and early warning of fire hazards combined with image recognition. The method includes: acquiring visible light and infrared thermal images of a monitored area based on image recognition; performing spatial registration and temporal alignment on the visible light and infrared thermal images to obtain pixel-by-pixel corresponding bimodal image pairs and the registration relationship between the visible light and infrared thermal images; acquiring the heat diffusion area of ​​the infrared thermal image based on the bimodal image pairs; filtering areas that meet the point heat source morphology conditions based on the heat diffusion area and identifying them as suspected heat hazard areas; and mapping the suspected heat hazard areas to the corresponding positions in the visible light image according to the registration relationship. This invention uses an open flame verification process to perform bimodal consistency verification on the initial open flame category, effectively reducing the false alarm rate for open flames.

Description

Technical Field

[0001] This invention relates to the field of fire monitoring technology, and in particular to a method and system for real-time monitoring and early warning of fire hazards that combines image recognition. Background Technology

[0002] Fire safety is a crucial component of public safety, and early detection and timely warning of fire hazards are key to preventing major fire accidents. With the development of image recognition technology, video image-based fire hazard monitoring methods, due to their advantages of being non-contact, having wide coverage, and responding quickly, are gradually becoming an important supplement to traditional smoke and temperature sensors. In particular, dual-light monitoring schemes combining visible light and infrared thermal imaging can simultaneously acquire visual texture information and temperature distribution information of a scene, theoretically possessing the potential to detect abnormally heated targets at the nascent stage of a potential hazard.

[0003] Existing fire hazard monitoring methods based on dual-light images still face several key challenges in practical applications. First, in the suspected hazard area extraction stage, existing methods often employ simple temperature threshold segmentation or connected component analysis, lacking precise constraints on the unique thermal field morphology characteristics of early point heat source hazards. This easily leads to the misidentification of normal heat sources, such as HVAC pipes and large-area heat-generating equipment, as hazard candidates, resulting in a large amount of interference in the input data for subsequent identification stages. Second, in the hazard identification stage, existing methods often perform simple channel splicing or separate processing and fusion of visible light and infrared features, failing to deeply explore the consistency of feature responses of the two modalities at corresponding spatial locations. This makes it difficult to effectively distinguish between real open flames and optical interference objects with fire-like textures but no corresponding high-temperature distribution, resulting in a persistently high false alarm rate for open flames. Furthermore, in the time-series analysis and early warning stage, existing methods mostly rely on static threshold judgment based on single frames or simple multi-frame averaging. They lack multi-level dynamic analysis of the temperature rise rate, area expansion rate, and acceleration trend of the hidden danger target in the continuous time dimension. This makes it impossible to accurately determine the development stage of the hidden danger, resulting in delayed early warning or inability to distinguish between hidden dangers of different urgency levels. Consequently, it is difficult to meet the needs of hierarchical early warning management from early detection to emergency alarm. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for real-time monitoring and early warning of fire hazards that combines image recognition, so as to solve the technical problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A method for real-time monitoring and early warning of fire hazards combining image recognition, comprising: Visible light images and infrared thermal images of the monitored area are obtained based on image recognition, and spatial registration and temporal alignment are performed on the visible light images and the infrared thermal images to obtain pixel-by-pixel corresponding dual-modal image pairs and the registration relationship between the visible light images and the infrared thermal images; The thermal diffusion area of ​​the infrared thermal image is obtained based on the dual-modal image pair, and the area that meets the point heat source morphology condition is selected based on the thermal diffusion area and regarded as the suspected thermal hazard area. The suspected thermal hazard area is mapped to the corresponding position in the visible light image according to the registration relationship, and the corresponding visible light image block and infrared image block are cropped from the dual-modal image pair; The hazard identification result of the current frame is obtained based on the visible light image block and the infrared image block, and the hazard category is obtained based on the hazard identification result; The hidden danger targets in the hidden danger identification results of multiple consecutive frames are tracked to obtain the temporal feature sequence of each hidden danger instance, and a graded early warning signal is generated based on the hidden danger development trend reflected by the temporal feature sequence.

[0006] Preferably, the step of obtaining the heat diffusion region of the infrared thermal image based on the dual-modal image pair, and filtering out regions that meet the point heat source morphology conditions based on the heat diffusion region, and identifying these regions as suspected heat hazard areas, includes: The horizontal and vertical temperature gradients of each pixel block in the infrared thermal image are obtained based on the dual-modal image, and a gradient direction pattern is generated based on the horizontal and vertical temperature gradients. The temperature value of each pixel block is obtained based on the infrared thermal image, and all local temperature maxima are obtained based on the temperature value of each pixel block. The obtained local temperature maxima are determined as candidate pixel blocks. Obtain the background temperature threshold. Starting from the candidate pixel block, check whether the gradient direction of the adjacent pixel blocks on the current heat diffusion region boundary points to the candidate pixel block according to the gradient direction map. Also, check whether the temperature value of the adjacent pixel block is greater than the background temperature threshold according to the infrared thermal image. If both conditions are met, the adjacent pixel block candidate point is included in the thermal diffusion region and continues to grow outward until no new pixel meets the conditions, thus obtaining the thermal diffusion region corresponding to the pixel block candidate point. The region compactness is obtained based on each heat diffusion region, and the thermal center offset of the heat diffusion region is calculated based on the geometric center position of each heat diffusion region and the position of the highest temperature pixel within the region. The consistency of the gradient direction of the heat diffusion region is obtained by comparing the gradient direction of each pixel within each heat diffusion region with the reference direction of each pixel pointing to the candidate point of its pixel block. Based on the compactness, thermal center offset, and gradient direction consistency of each heat diffusion region, a comparison is made with a preset point heat source morphology threshold. Regions that meet the point heat source morphology threshold are selected from all heat diffusion regions as suspected heat hazard regions.

[0007] Preferably, the step of checking whether the gradient directions of adjacent pixel blocks on the current heat diffusion region boundary point to the candidate pixel block, starting from the candidate pixel block, according to the gradient direction map, includes: Based on the coordinates of the adjacent pixels in the infrared thermal image and the coordinates of the candidate point of the pixel block in the infrared thermal image, a reference direction vector is calculated with the adjacent pixels as the starting point and the candidate point of the pixel block as the ending point. The gradient direction vectors of adjacent pixel blocks are obtained based on the gradient direction map. Calculate the angle between the reference direction vector and the gradient direction vector; When the included angle is less than a preset angle threshold, it is determined that the gradient direction of the adjacent pixel points to the candidate point of the pixel block.

[0008] Preferably, the step of obtaining the hazard identification result of the current frame based on the visible light image block and the infrared image block, and obtaining the hazard category based on the hazard identification result, includes: Multiple pixel values ​​are obtained from the infrared image block, and the temperature distribution feature map and the highest temperature value of the infrared image block are obtained from the pixel values. The color channel value corresponding to each pixel value is obtained from the visible light image block, and the texture feature map of the visible light image block is obtained from the color channel value. Based on the temperature distribution feature map of the infrared image block and the texture feature map of the visible light image block, the feature response intensity of each local region corresponding to the spatial position in the two feature maps is compared to obtain a cross-modal consistency feature map that characterizes the consistency of the feature response of the two modes at corresponding spatial positions. The temperature margin feature value, which characterizes the degree to which the current temperature is close to the danger temperature, is obtained based on the highest temperature value. The temperature margin feature value is then aggregated with the cross-modal consistency feature map to obtain a fused feature set. Based on the fused feature set, the confidence scores of the regions corresponding to the visible light image patch and the infrared image patch belonging to each preset hazard category are obtained respectively; Hazard categories are generated based on the confidence scores.

[0009] Preferably, the step of generating hazard categories based on the confidence score includes: Based on the confidence scores of each preset hazard category obtained by the classification processing of the fused feature set, the category with the highest confidence score is determined as the initial hazard category; Determine whether the preliminary hazard category belongs to a preset set of open flame-related categories; If it does not belong to the category, the preliminary hazard category will be directly output as the hazard category in the hazard identification result; If it is found, the open flame verification process is triggered. In the open flame verification process, the pixel positions in the visible light image block that are determined to be flame texture regions by the texture feature map are obtained according to the cross-modal consistency feature map. Based on the pixel positions of the flame texture regions, the feature value statistics of the corresponding positions in the cross-modal consistency feature map are obtained. The verification result is obtained by comparing the statistics with the preset open flame verification threshold. When the review result is passed, the initial hazard category is output as the hazard category in the hazard identification result; When the review result is not passed, the initial hazard category is excluded from the current candidate categories, and the hazard category is re-determined and output based on the confidence score of the remaining categories.

[0010] Preferably, the step of tracking the hazard targets in the hazard identification results of multiple consecutive frames and obtaining the temporal feature sequence of each hazard instance includes: Obtain the preset hazard trajectory, and obtain the position overlap value based on the position area of ​​each hazard target in the hazard identification result of the current frame and the historical position area of ​​the most recent frame in the preset hazard trajectory; The feature similarity value is obtained based on the texture feature distance between the visible light image patch of each of the potential hazards and the visible light image patch corresponding to the most recent frame of each potential hazard trajectory; The hazard targets and hazard trajectories are associated and matched based on the location overlap value and feature similarity value. Successfully matched hazard targets are incorporated into the corresponding hazard trajectories, and new hazard trajectories are created for unmatched hazard targets to obtain an updated hazard trajectory set. Based on the set of hidden danger trajectories, multiple updated hidden danger trajectories are obtained. According to the infrared image blocks associated with the hidden danger targets in each historical frame, the highest temperature value in each frame's infrared image block is extracted one by one and arranged in frame order to form a temperature evolution sequence. Based on the updated hazard trajectory, the suspected thermal hazard area corresponding to the hazard target in each historical frame is extracted one by one, and the total number of pixels contained in the suspected thermal hazard area of ​​each frame is arranged in frame order to form an area evolution sequence. Based on the confidence scores of the hazard categories of the corresponding hazard targets in each historical frame, the updated hazard trajectories are arranged in frame order to form a confidence evolution sequence. The temperature evolution sequence, area evolution sequence, and confidence evolution sequence are then combined to form a temporal feature sequence of the corresponding hazard instance.

[0011] Preferably, the step of generating a graded early warning signal based on the development trend of potential hazards reflected by the time-series feature sequence includes: The temperature rise rate between each adjacent frame is obtained based on the temperature evolution sequence, and a temperature rise rate sequence is generated based on the temperature rise rate between each adjacent frame. The area change rate between each adjacent frame is obtained based on the area evolution sequence, and an area expansion rate sequence is generated based on the area expansion rate between each adjacent frame. The temperature rise acceleration between each adjacent frame is obtained based on the temperature rise rate sequence. The area expansion acceleration between each adjacent frame is obtained based on the area expansion rate sequence; The temperature rise acceleration and the area expansion acceleration are normalized to obtain temperature acceleration index and area acceleration index respectively. A grading judgment value is obtained based on the temperature acceleration index and the area acceleration index, and it is determined whether the grading judgment value is greater than a preset threshold. If it is less than or equal to, the hidden danger instance remains unchanged at its current development stage; If the value is greater than the specified value, the potential hazard instance is determined to have entered the next development stage, wherein the development stage includes the incubation period, the development period, and the danger period. Based on the development stage and the hazard category corresponding to the hazard instance, a corresponding level of graded early warning signal is generated according to the preset early warning mapping rules.

[0012] This invention also discloses a real-time monitoring and early warning system for fire hazards that combines image recognition, comprising: The image pair acquisition module is used to acquire visible light images and infrared thermal images of the monitored area based on image recognition, and to perform spatial registration and temporal alignment on the visible light images and the infrared thermal images to obtain pixel-by-pixel corresponding dual-modal image pairs and the registration relationship between the visible light images and the infrared thermal images; The hidden danger area acquisition module is used to acquire the heat diffusion area of ​​the infrared thermal image based on the dual-modal image pair, and to filter out the areas that meet the point heat source morphology conditions based on the heat diffusion area, and to regard them as suspected heat hidden danger areas. The image patch acquisition module is used to map the suspected thermal hazard area to the corresponding position in the visible light image according to the registration relationship, and to crop the corresponding visible light image patch and infrared image patch from the dual-modal image pair; The hazard category acquisition module is used to obtain the hazard identification result of the current frame based on the visible light image block and the infrared image block, and to obtain the hazard category based on the hazard identification result; The graded early warning signal acquisition module is used to track the hidden danger targets in the hidden danger identification results of multiple consecutive frames, acquire the temporal feature sequence of each hidden danger instance, and generate a graded early warning signal based on the hidden danger development trend reflected by the temporal feature sequence.

[0013] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of a method for real-time monitoring and early warning of fire hazards combined with image recognition.

[0014] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of a method for real-time monitoring and early warning of fire hazards combined with image recognition.

[0015] The beneficial effects of this application are as follows: This invention constructs pixel-by-pixel corresponding bimodal image pairs by strictly registering visible light images and infrared thermal images, providing a complementary and geometrically consistent fusion data foundation for hazard analysis. Based on this, local temperature maxima are extracted from the infrared thermal images. Through region growing with directional and temperature constraints and point heat source morphology screening, suspected thermal hazard areas conforming to the thermal morphological characteristics of early fire hazards are accurately obtained, effectively filtering out large-area non-point-like normal heat source interference. After mapping the suspected thermal hazard areas to the visible light images and cropping the corresponding bimodal image blocks, temperature distribution feature maps and texture feature maps are extracted respectively. Cross-modal consistency analysis is used to generate cross-modal consistency feature maps characterizing the degree of synergy between the two modal feature responses, and these are aggregated with temperature margin features, enabling the identification model to not only perceive the absolute temperature hazard level but also to use visible light texture information to distinguish flame textures and fire-like interference. The bimodal consistency verification of the initial open flame category is performed through an open flame review process, effectively reducing the false alarm rate for open flames. The system tracks continuous frames of potential hazards and constructs a three-dimensional temporal feature sequence of temperature, area, and confidence level. Based on the acceleration of temperature and area rise, it extracts the accelerated deterioration signal of hazard development, realizing dynamic stage determination of incubation period, development period, and danger period. Finally, it generates graded early warning signals according to preset mapping rules based on hazard category, upgrading traditional passive monitoring to active situational awareness and graded early warning, significantly improving the accuracy of early detection of fire hazards and the timeliness of early warning. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of a method flow according to an embodiment of this application.

[0017] Figure 2This is a schematic diagram of the system structure according to an embodiment of this application.

[0018] Figure 3 This is a schematic diagram of the internal structure of a computer device according to an embodiment of this application.

[0019] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0021] like Figure 1 As shown, this application provides a method for real-time monitoring and early warning of fire hazards combined with image recognition, including: S1. Based on image recognition, obtain visible light images and infrared thermal images of the monitored area, and perform spatial registration and temporal alignment on the visible light images and the infrared thermal images to obtain pixel-by-pixel corresponding dual-modal image pairs and the registration relationship between the visible light images and the infrared thermal images; S2. Obtain the thermal diffusion area of ​​the infrared thermal image based on the dual-modal image pair, and select the area that meets the point heat source morphology condition based on the thermal diffusion area, and regard it as the suspected thermal hazard area. For example, in the early stages of fire hazard development, it often manifests as one or more abnormally high temperature points in a localized area. This invention analyzes infrared thermal images from a dual-modal image pair, first obtaining the temperature value represented by each pixel in the image. Then, a temperature anomaly detection algorithm is applied to locate candidate locations of thermal hazards.

[0022] S3. Map the suspected thermal hazard area to the corresponding position in the visible light image according to the registration relationship, and crop the corresponding visible light image block and infrared image block from the dual-modal image pair; For example, the suspected thermal hazard area obtained in step S2 is a set of connected pixels defined in the coordinate system of the infrared thermal image. This invention first utilizes the spatial registration relationship between the visible light image and the infrared thermal image calculated and saved in step S1 to transform the coordinates of the region's outline or bounding box from the infrared image coordinate system to the visible light image coordinate system. In this way, the spatial location corresponding to the suspected thermal hazard area is accurately located on the visible light image.

[0023] S4. Obtain the hazard identification result of the current frame based on the visible light image block and the infrared image block, and obtain the hazard category based on the hazard identification result; S5. Track the hidden danger targets in the hidden danger identification results of multiple consecutive frames, obtain the temporal feature sequence of each hidden danger instance, and generate a graded early warning signal based on the hidden danger development trend reflected by the temporal feature sequence. For example, a fire hazard typically develops over time, from its emergence to the occurrence of a fire. This step first tracks the hazard targets in multiple consecutive frames of hazard identification results. Using a multi-target tracking algorithm, the hazard targets identified in the current frame are correlated and matched with previously identified and recorded hazard targets in previous frames. The correlation can be based on the overlap of the targets' positions across consecutive frames and the visual feature similarity of their visible light image patches. Through correlation, it can be determined whether a hazard target in the current frame is a continuation of a previous hazard instance or a new hazard instance. For the same hazard instance that appears consistently across multiple consecutive frames, its identification data in each historical frame is concatenated to form a continuous observation record of that hazard instance.

[0024] As described in steps S1-S5 above, this invention utilizes a dual-target calibration technique, employing a specially designed calibration board (such as a checkerboard pattern with both visible light texture and infrared heating grid) to simultaneously acquire multiple sets of corresponding calibration points under both types of images. This allows for the determination of the spatial transformation relationship between the two sensors, i.e., the homography matrix or rigid body transformation matrix. Then, based on this transformation relationship, the infrared thermal image is mapped to the coordinate system of the visible light image, or both are resampled to a unified virtual imaging plane. After this processing, any pixel position in the resulting image pair refers to the same physical point in the real scene on both the visible light and infrared thermal images, achieving pixel-by-pixel correspondence. Subsequently, visual features of the visible light image are used for auxiliary screening. By identifying candidate temperature anomalies obtained from the infrared thermal image, and based on the registration relationship established in step S1, their corresponding positions are found in the visible light image. Image recognition technology is then used to determine whether a visual morphology matching a known normal heating device exists at that position. If the morphology matches a normal device, it is excluded; if the morphology is unfamiliar or cannot be matched, it is retained. Ultimately, areas that exhibit abnormally high temperatures in infrared thermal images but lack the visual characteristics of normal equipment will be defined as suspected thermal hazard areas.

[0025] Subsequently, a bounding box (e.g., a minimum bounding rectangle) is defined to completely enclose the region, and can be appropriately extended outward by a certain number of pixels to ensure that the contextual information of the region's edges is not lost. Based on the coordinates of this unified extended bounding box, two corresponding small-sized images are cropped from the bimodal image pair obtained in step S1: one is a visible light image patch from the visible light image, which preserves the color, texture, and shape details of the objects within the region; the other is an infrared image patch from the infrared thermal image, which preserves the temperature distribution details of the objects within the region. These two image patches are spatially pixel-by-pixel corresponding, forming a pair of bimodal image patches focusing on the same local hazard candidate object.

[0026] This invention uses the paired visible light image patch and infrared image patch obtained in step S3 as input to a pre-trained multimodal recognition model. The multimodal recognition model is trained in the following way: Obtain a training sample set, each sample including a spatially registered visible light image patch and an infrared image patch, as well as a corresponding hazard category label; the hazard categories include smoldering ignition point, overheating of electrical circuits, overheating of flammable material accumulation, open flame, and normal without hazard; A two-stream convolutional neural network was constructed as the initial model, including visible light and infrared branches. Both branches used ResNet-50 as the backbone network. The output feature maps were fused through a cross-attention fusion layer and then output as classification results through a fully connected layer. The cross-entropy loss function is used as the objective function, and the Adam optimizer is employed with an initial learning rate of 1×10⁻⁶. -3 With a batch size of 32, the training samples are input into the initial model for iterative training until the loss function converges, resulting in a trained multimodal recognition model.

[0027] Specifically, one branch of the multimodal recognition model processes visible light image patches to extract visual features that reflect attributes such as object edges, textures, and colors; the other branch processes infrared image patches to extract thermodynamic features that reflect attributes such as the object's highest temperature, temperature distribution gradient, and high-temperature area. The features extracted by the two branches are concatenated or cross-attention calculated in the model's fusion layer to obtain a fused feature vector that characterizes the comprehensive attributes of the local region under dual-modal information.

[0028] Based on the fused feature vector, the model classifies whether the region corresponding to the image patch constitutes a fire hazard and outputs a confidence score indicating whether it belongs to each preset hazard category. Examples of preset hazard categories include: smoldering ignition points, overheating of electrical wiring, overheating of flammable material accumulation, open flames, high-temperature pipes, and normally heating equipment.

[0029] Finally, based on the classification confidence scores, the hazard identification result for the current frame is determined. This result is a comprehensive data object that, for each input image patch, includes at least the bounding box location corresponding to the image patch and the hazard category determined based on the highest confidence score. For example, if the model determines that an image patch has a 95% probability of belonging to "overheating electrical wiring," then the hazard category is identified as "overheating electrical wiring." If the confidence scores of all preset hazard categories are lower than the preset judgment threshold, the area is identified as "normal" and not included in the subsequent tracking range.

[0030] This invention obtains a temporal feature sequence for each hazard instance. This temporal feature sequence is a set of sequence data characterizing the changes in features over time that reflect the development state of the hazard. For example, from the infrared image patch associated with each frame of the hazard instance, the highest temperature value is extracted and arranged chronologically to form a temperature evolution sequence; the area or number of pixels is extracted to form an area evolution sequence; and the confidence score of the hazard category in step S4 is extracted to form a confidence evolution sequence. These sequences together constitute a multidimensional temporal feature sequence describing the development state of the hazard instance.

[0031] Finally, based on the development trend of the hidden danger reflected by the time-series feature sequence, a graded early warning signal is generated. This invention pre-determines the development trend, assessing the risk level by performing trend analysis on the time-series feature sequence. For example, when the temperature evolution sequence shows a continuous rise in temperature at an accelerating rate, and the area evolution sequence shows a continuous expansion of the high-temperature area, it indicates that the hidden danger is in a rapidly developing dangerous period, and a corresponding high-level early warning signal is generated accordingly. If the temperature remains stable at an abnormal level slightly above the background level but without a significant upward trend, a low-level, suggestive early warning signal is generated. The specific grade of the early warning signal can be flexibly set according to the safety management needs of the actual application scenario, ultimately achieving multi-level, proactive fire hazard control from early warning to emergency alarm.

[0032] In one embodiment, the step of acquiring the thermal diffusion region of the infrared thermal image based on the dual-modal image pair, filtering out regions that meet the point heat source morphology conditions based on the thermal diffusion region, and identifying these regions as suspected thermal hazard areas includes: S201. Obtain the horizontal and vertical temperature gradients of each pixel block in the infrared thermal image based on the dual-modal image, and generate a gradient direction pattern based on the horizontal and vertical temperature gradients. For example, the present invention first divides the infrared thermal image in the current frame's dual-modal image pair into regularly arranged pixel blocks. Each pixel block can be a single pixel or a small pixel neighborhood (e.g., a 2x2 or 3x3 pixel grid) containing multiple adjacent pixels. The specific granularity of the division can be adjusted according to the image resolution and computational performance requirements. For each pixel block, its temperature change rate in the horizontal and vertical directions is calculated. The horizontal temperature gradient can be approximated by the temperature difference between the pixel block and its left and right adjacent pixel blocks; similarly, the vertical temperature gradient can be approximated by the temperature difference between the pixel block and its upper and lower adjacent pixel blocks.

[0033] S202. Obtain the temperature value of each pixel block based on the infrared thermal image, and obtain all local temperature maxima points based on the temperature value of each pixel block, and determine each of the obtained local temperature maxima points as candidate pixel block points; S203. Starting from the candidate pixel block, check whether the gradient direction of the adjacent pixel block on the current heat diffusion region boundary points to the candidate pixel block according to the gradient direction map, and check whether the temperature value of the adjacent pixel block is greater than the background temperature threshold according to the infrared thermal image. If both conditions are met, the adjacent pixel block candidate point is included in the thermal diffusion region and continues to grow outward until no new pixel meets the conditions, thus obtaining the thermal diffusion region corresponding to the pixel block candidate point. S204. Obtain the region compactness according to each of the heat diffusion regions; calculate the thermal center offset of the heat diffusion region according to the geometric center position of each heat diffusion region and the position of the highest temperature pixel in the region; obtain the gradient direction consistency of the heat diffusion region according to the gradient direction of each pixel in each heat diffusion region and the reference direction of each pixel pointing to the candidate point of the pixel block. For example, region compactness is used to quantify how close the shape of a heat-affected region is to an ideal circle. The heat diffusion of a point heat source in a homogeneous medium is isotropic; therefore, its corresponding heat-affected region should exhibit a compact, approximately circular shape on an infrared thermal image. The formula for calculating region compactness is as follows: ; Where C represents the region compactness, S represents the area of ​​the heat diffusion region R, that is, the total number of pixel blocks contained in region R; P represents the perimeter of the boundary of the heat diffusion region R, that is, the total length of the connecting edges between all adjacent pixel blocks on the boundary of region R. Thermal center offset is used to quantify the degree to which the highest temperature point in a heat diffusion region deviates from its geometric center. For an ideal point heat source, its temperature peak position should roughly coincide with the geometric center of the heat-affected zone. A large offset may indicate an irregular shape of the heat source or a composite thermal field formed by multiple superimposed heat sources. The geometric center position G of the heat diffusion region R is calculated. Let the heat diffusion region R consist of N pixel blocks, and the coordinates of the i-th pixel block in the infrared thermal image be (x...). i ,y i If the geometric center position G is G, then the coordinates of G(G) are... x G y )for: ; ; Where i takes values ​​from 1 to N; then find the position H of the pixel with the highest temperature within the heat diffusion region R. By traversing the temperature values ​​of all pixel blocks within region R, find the pixel block with the highest temperature value and record its coordinates as (H). x H y ); Calculate the Euclidean distance between the geometric center position G and the highest temperature pixel position H, as the thermal center offset D of this heat diffusion region: ; Where D represents the thermal center offset, a non-negative real value, measured in pixels. The closer the D value is to 0, the closer the highest temperature point is to the geometric center of the region, and the more its shape conforms to the characteristics of an ideal point heat source.

[0034] Gradient direction consistency is used to quantify the degree to which the thermal field gradient directions of each pixel block within a thermal diffusion region commonly point towards the center of the regional heat source. In a thermal field diffusing outward from a central heat source, the temperature gradient direction should be away from the center and outward, while its opposite direction should be highly consistent, pointing towards the center of the heat source. This invention measures this consistency by calculating the difference between the gradient direction of each pixel within the region and the direction vector of each pixel pointing towards the candidate point of that pixel block.

[0035] For each pixel block j within the thermal diffusion region R, perform the following consistency metric calculation: Obtain the gradient direction vector g of pixel block j. j This vector is obtained directly from the gradient direction pattern generated in step S201, and its direction angle is denoted as θ. j .

[0036] Next, the direction vector d from pixel j to the candidate pixel point O corresponding to the heat diffusion region is calculated. j Let the coordinates of pixel block j be (x...). i ,y i The coordinates of the candidate pixel point O are (x...o ,y o If ), then vector d j For (x) o -x i y o -y i Its direction angle is denoted as Φ. j .

[0037] Then, calculate the gradient direction vector g. j With the direction vector d pointing to the center j The absolute value of the included angle between them |Δθ j |: |Δθ j ∣=∣θ j -Φ j |; If |Δθ j If | is greater than 180°, then subtract this value from 360° for correction to ensure that the included angle is within the range of [0°, 180°]. Finally, the included angle |Δθ| corresponding to all pixel blocks within the heat diffusion region R is calculated. j | Perform statistical summarization to obtain a quantitative value characterizing the overall consistency. A typical method is to calculate the mean μ or standard deviation σ of all included angles. For example, the average angular deviation can be used as a measure of gradient direction consistency U: Where M is the total number of pixel blocks within region R; Under this definition, the value of U ranges from [0,1]. When the gradient directions of all pixel blocks within a region strictly point to the candidate point of that pixel block, all |Δθ j When | is 0°, U is 1, indicating that the gradient directions are completely consistent. When the gradient directions of each pixel are disordered and generally deviate greatly from the direction pointing to the center, the U value approaches 0, indicating that the gradient direction consistency is extremely poor.

[0038] S205. Based on the compactness, thermal center offset, and gradient direction consistency of each heat diffusion region, compare them with a preset point heat source morphology threshold, and select regions that meet the point heat source morphology threshold from all heat diffusion regions as the suspected heat hazard regions.

[0039] As described in steps S201-S205 above, after obtaining the horizontal and vertical temperature gradients of each pixel block, this invention generates a gradient direction map based on the gradient components in these two orthogonal directions. The value recorded at each pixel position in the gradient direction map corresponds to the angle between the gradient direction vector of that pixel block and a certain reference axis (e.g., the horizontal X-axis of the image). This gradient direction vector points in the direction of the fastest increase in temperature. Therefore, for pixel blocks around the heat source, their gradient direction should theoretically be away from the center of the heat source and outwards. Conversely, depending on the analysis needs, we will utilize the characteristic that its opposite direction points towards the center of the heat source.

[0040] By traversing each pixel block in the infrared thermal image, its corresponding temperature value is read. For any pixel block, if its temperature value is higher than the temperature values ​​of all its directly adjacent pixel blocks, then the pixel block is marked as a local temperature maxima. All identified local temperature maxima points are potential cores of thermal hazards because they are the centers of local thermal radiation. This invention uniformly determines all acquired local temperature maxima points as candidate pixel blocks. Subsequent region growing processes will unfold one by one using these candidate pixel blocks as seed points.

[0041] This invention uses a candidate pixel block obtained in S202 as a seed starting point to initialize a thermal diffusion region containing only that seed point. Then, iteratively checks each adjacent pixel block on the boundary of this thermal diffusion region to determine whether it should be included in the current region. The inclusion determination must simultaneously satisfy the following two conditions: The directional consistency condition is determined by checking whether the gradient direction of the adjacent pixel block points to the candidate point of the pixel block. Specifically, firstly, based on the coordinates of the adjacent pixel block and the candidate point of the pixel block in the infrared thermal image, a reference direction vector is calculated, starting from the adjacent pixel block and ending at the candidate point of the pixel block. Secondly, the gradient direction vector of the adjacent pixel block is obtained from the gradient direction map generated in S201. Then, the angle between the reference direction vector and the gradient direction vector is calculated. When the angle is less than a preset angle threshold, it indicates that the heat diffusion direction of the pixel block is basically aligned with the center of the heat source, that is, the gradient direction of the adjacent pixel block is determined to point to the candidate point of the pixel block, thus satisfying the directional consistency condition.

[0042] For the temperature significance condition, the criterion is to check whether the temperature value of the adjacent pixel block is greater than a preset background temperature threshold. This preset background temperature threshold is used to distinguish the heat source-affected area from the normal background area with no significant temperature difference. Its value can be dynamically set according to the average ambient temperature of the entire monitoring scene, or set as a fixed absolute temperature value according to safety management regulations. The average ambient temperature is obtained by: statistically analyzing the temperature values ​​of all pixels in the current frame of the infrared thermal image and calculating their arithmetic mean; or, statistically analyzing the average temperature of pixels in the current frame of the infrared thermal image whose temperature values ​​are located in the middle range of the global temperature distribution (for example, calculating the mean after removing pixels whose temperature values ​​are in the highest 5% and lowest 5% ranges) to eliminate interference from extreme high-temperature heat sources and low-temperature noise; or, using the statistical value of historical temperature data of the monitoring area as the estimated average ambient temperature value of the current frame.

[0043] If both of the above conditions are met, the adjacent pixel block is included in the current heat diffusion region. This newly included pixel block then becomes part of the region boundary, available for use in the next round of growth checks. This growth process is repeated, expanding outwards by one layer of pixels in each iteration, until, in a certain iteration, no adjacent pixel block on the boundary can simultaneously satisfy both of the above conditions. At this point, the complete heat diffusion region corresponding to the candidate pixel block is obtained. This process ensures that the grown region geometrically conforms to the natural heat diffusion path. This invention predefines a set of point heat source shape thresholds, which define the lower or upper limits of various indicators required to conform to the point heat source shape. For example, a compactness lower limit threshold (e.g., 0.6), a heat center offset upper limit threshold (e.g., a distance value in pixels), and a gradient direction consistency lower limit threshold (e.g., an angle value measured by the standard deviation of the angle).

[0044] During the subsequent filtering process, each heat diffusion region obtained in step S203 is traversed sequentially. For the current region, its three feature values—compactness, thermal center offset, and gradient direction consistency—calculated in step S204, are compared with the corresponding three thresholds in the preset point heat source morphology threshold. Only when the region's compactness is greater than or equal to the lower limit threshold, its thermal center offset is less than or equal to the upper limit threshold, and its gradient direction consistency is greater than or equal to the lower limit threshold—that is, when all three conditions are met—is the heat diffusion region determined to satisfy the point heat source morphology threshold. All heat diffusion regions that pass this comprehensive determination will form a set, which will be the suspected heat hazard regions output in the final embodiment. These regions will be passed to the subsequent step S3 for position mapping and cropping of bimodal image blocks.

[0045] In one embodiment, the step of checking whether the gradient directions of adjacent pixel blocks on the current heat diffusion region boundary point to the pixel block candidate point according to the gradient direction map, starting from the pixel block candidate point, includes: S2031. Based on the coordinates of the adjacent pixels in the infrared thermal image and the coordinates of the candidate point of the pixel block in the infrared thermal image, calculate a reference direction vector with the adjacent pixels as the starting point and the candidate point of the pixel block as the ending point. S2032. Obtain the gradient direction vector of adjacent pixel blocks according to the gradient direction map; S2033. Calculate the angle between the reference direction vector and the gradient direction vector; S2034. When the included angle is less than a preset angle threshold, it is determined that the gradient direction of the adjacent pixel points to the candidate point of the pixel block.

[0046] As described in steps S2031-S2034 above, when the growth process progresses to the boundary of the thermal diffusion region corresponding to a candidate pixel block, the system checks each adjacent pixel block located on that boundary one by one. For one of the adjacent pixel blocks to be determined, the system first obtains the coordinate position of the adjacent pixel block and the coordinate position of the candidate pixel block corresponding to the currently growing region in the global coordinate system of the infrared thermal image. Both of these coordinate positions are known and definite values.

[0047] Subsequently, using the coordinates of the adjacent pixel block as the starting point of the vector and the coordinates of the candidate point of the pixel block as the ending point of the vector, a directed line segment connecting these two points is calculated. This directed line segment is defined as the reference direction vector. Geometrically, this reference direction vector explicitly expresses the shortest path direction from the current adjacent pixel block directly towards the candidate point of the heat source core.

[0048] In step S201, the system has calculated and generated a gradient direction map covering the entire infrared thermal image based on the infrared thermal image. This map corresponds to the infrared thermal image pixel by pixel in space, and each location in the map stores the gradient direction vector corresponding to that pixel block. This gradient direction vector points to the direction in which the temperature rises the fastest at the location of that pixel block.

[0049] For the currently being evaluated boundary-adjacent pixel block, the system directly uses its coordinate position as an index to perform a query and read operation in the gradient direction map, thereby obtaining the gradient direction vector corresponding to the adjacent pixel block. This vector reflects the objectively existing heat flow direction at that location based on the actual temperature distribution. The angle between the reference direction vector and the gradient direction vector is calculated by comparing the numerical values ​​of their directions in the same coordinate system. The result is an angle value ranging from 0° to 180°. The smaller the angle value, the closer the actual heat flow direction is to the theoretical direction pointing towards the core of the heat source; the larger the angle value, the farther the actual heat flow direction deviates from the direction pointing towards the core.

[0050] Internally, there is a strict standard for defining directional consistency, called the preset angle threshold. This preset angle threshold is a specific angle value, such as 15 degrees or 20 degrees, and its function is to define an acceptable range of deviation.

[0051] The system compares the angle value calculated in step S2033 with this preset angle threshold. If the calculated angle value is less than the preset angle threshold, it means that the deviation between the actual gradient direction of the adjacent pixel block and the ideal direction pointing to the candidate core of the heat source falls within an acceptable range. At this time, the system makes a positive determination, that is, it determines that the gradient direction of the adjacent pixel points to the candidate point of the pixel block, and the adjacent pixel block meets the direction consistency condition for being included in the heat diffusion region.

[0052] Conversely, if the calculated angle value is equal to or greater than the preset angle threshold, it means that the actual heat flow direction of the adjacent pixel block deviates too much from the expected direction. The system makes a negative judgment, and the adjacent pixel block does not meet the direction consistency condition and will not be included in the current heat diffusion area.

[0053] Through the above four sub-steps, the system calculates from a quantitative perspective and compares explicit thresholds to determine the directional attributes of each boundary adjacent pixel block, ensuring that the region growth process strictly follows the physical diffusion path of the thermal field.

[0054] In one embodiment, the step of obtaining the hazard identification result of the current frame based on the visible light image patch and the infrared image patch, and obtaining the hazard category based on the hazard identification result, includes: S401. Obtain multiple pixel values ​​based on the infrared image block, and obtain the temperature distribution feature map and the highest temperature value of the infrared image block based on the pixel values; S402. Obtain the color channel value corresponding to each pixel value according to the visible light image block, and obtain the texture feature map of the visible light image block according to the color channel value; S403. Based on the temperature distribution feature map of the infrared image block and the texture feature map of the visible light image block, the feature response intensity of each local region corresponding to the spatial position in the two feature maps is compared to obtain a cross-modal consistency feature map that characterizes the consistency of the feature response of the two modes at the corresponding spatial positions. S404. Obtain a temperature margin feature value that characterizes the degree to which the current temperature is close to the danger temperature based on the highest temperature value, and perform feature aggregation on the temperature margin feature value and the cross-modal consistency feature map to obtain a fused feature set; S405. Based on the fusion feature set, obtain the confidence scores of the regions corresponding to the visible light image block and the infrared image block belonging to each preset hazard category; S406. Generate a hazard category based on the confidence score.

[0055] As described in steps S401-S406 above, for the infrared image block obtained by cropping in step S3, the present invention first reads the pixel value corresponding to each pixel position in the image block. This pixel value directly reflects the thermal radiation intensity of the object at that position, and after calibration and mapping, it becomes the temperature value. After obtaining all pixel values, the infrared image block is input into a pre-trained convolutional neural network branch. This branch performs feature encoding on the spatial distribution pattern of the temperature values ​​through filtering operations of multiple convolutional kernels, and finally outputs a temperature distribution feature map. This temperature distribution feature map can capture the local thermal field structure in the original temperature data that does not possess translation invariance, such as the shape of the high-temperature core region and the temperature decay pattern from the center to the edge.

[0056] On the other hand, by iterating through all pixel values ​​in the infrared image block and comparing them pixel by pixel, the maximum pixel value is found and recorded as the highest temperature value of the infrared image block. This highest temperature value, as a scalar value, provides the most direct upper temperature limit reference for assessing the hazard level of the area.

[0057] For the visible light image block obtained by cropping in step S3, this image block is usually a color image, and each pixel location stores multiple color channel values, such as red component values, green component values, and blue component values. First, read the color channel values ​​corresponding to each pixel value in the visible light image block to obtain complete color information.

[0058] Subsequently, the visible light image patch is input into another pre-trained convolutional neural network branch. This branch, like the infrared branch, uses multi-layer convolutional kernel filtering operations to encode the color channel values ​​and their spatial variations. Unlike the infrared branch, the features extracted by this branch primarily respond to the texture patterns of the object's surface, such as material roughness, edge orientation, and the repetition period of regular patterns. After processing by this branch, a texture feature map of the visible light image patch is output, which can characterize the visual texture attributes and spatial distribution of objects within that region.

[0059] The temperature distribution feature map obtained in step S401 and the texture feature map obtained in step S402 are spatially pixel-by-pixel corresponding because they originate from spatially registered infrared and visible light image blocks, respectively. This invention compares the feature response intensity of each local region corresponding to a spatial location in these two feature maps. The local region can be a neighborhood window centered on each pixel, or a grid cell on the feature map.

[0060] The specific comparison method is as follows: for a local region corresponding to a certain spatial location, the feature response intensity value of that region is obtained from the temperature distribution feature map and recorded as the first response value; the feature response intensity value of that region is obtained from the texture feature map and recorded as the second response value. Then, the similarity or correlation between these two response values ​​is calculated, for example, by calculating their product, dot product, or normalized correlation coefficient. This calculation result serves as the cross-modal consistency metric at that spatial location. When a local region exhibits a strong response on the temperature distribution feature map (e.g., the presence of a high-temperature anomaly pattern) and also exhibits a strong response on the texture feature map (e.g., the presence of a texture pattern related to an overheated object), the cross-modal consistency metric at that location is high, indicating that the feature responses of the two modes at that location are highly consistent; conversely, if one mode's response is strong while the other's response is weak, the consistency metric is low. Arranging the consistency metrics of all spatial locations in the entire map according to the original spatial layout forms a cross-modal consistency feature map. This cross-modal consistency feature map characterizes the degree of consistency of the feature responses of the two modes at corresponding spatial locations.

[0061] Obtain the highest temperature value of the infrared image block extracted in step S401. Compare this highest temperature value with a preset danger temperature threshold to calculate a temperature margin feature value. This temperature margin feature value characterizes how much margin there is between the highest temperature of the current area and the danger temperature. For example, the highest temperature value can be subtracted from the preset danger temperature threshold to obtain a temperature difference value. This difference value is then normalized so that when the highest temperature value is much lower than the danger temperature threshold, the temperature margin feature value is large, indicating sufficient safety margin; when the highest temperature value is close to or exceeds the danger temperature threshold, the temperature margin feature value approaches zero or becomes negative, indicating extremely low safety margin or a dangerous state has been entered.

[0062] Then, the scalar temperature margin feature value is aggregated with the cross-modal consistency feature map obtained in step S403. The feature aggregation operation can be performed by appending the temperature margin feature value as a global descriptor to the feature vector obtained after global pooling of the cross-modal consistency feature map; or by broadcasting the temperature margin feature value as an additional channel in space and concatenating it with the cross-modal consistency feature map.

[0063] The fused feature set obtained in step S404 is input into a pre-trained classifier, such as a classification network composed of fully connected layers; the classifier is pre-trained in the following manner: A training sample set is constructed. Each sample in the training sample set includes: the fusion feature set extracted in the manner described in step S404, and the corresponding hazard category label. The hazard categories include smoldering ignition point, overheating of electrical circuits, overheating of flammable material accumulation, open flame, high-temperature pipeline, and normal without hazards. The hazard category label of each training sample is determined by manual annotation. An initial classification network consisting of three fully connected layers is constructed, with 512, 256, and N neurons in each layer, respectively, where N is the number of preset hazard categories. The ReLU activation function is used between each fully connected layer, and the Softmax activation function is used in the output layer to output the probability distribution of each category. The cross-entropy loss function is used as the objective function, and the Adam optimizer is employed with an initial learning rate of 1×10⁻⁶. -3 The batch size is set to 64. The training samples are input into the initial classification network for forward propagation to calculate the loss value. The network weight parameters are iteratively updated through backpropagation until the loss function value tends to stabilize and no longer decreases significantly, thus obtaining the trained classifier. The classifier internally stores the feature weight parameters of each category learned through the above training.

[0064] The classifier performs forward computation on the input fused feature set, mapping through layer-by-layer weighted summation and non-linear activation functions, ultimately outputting a set of values. Each value in this set corresponds to a confidence score indicating that the region corresponding to the visible light image patch and the infrared image patch belongs to a certain preset hazard category. The preset hazard categories are a set of category labels predefined according to the actual types of fire hazards, exemplarily including smoldering points, overheating of electrical wiring, overheating of flammable material accumulation, open flames, operation of high-temperature equipment, and normal no-hazard state, etc.

[0065] After obtaining the confidence scores of the corresponding area of ​​the image patch belonging to each preset hazard category (output in step S405), the final hazard category is determined according to preset decision rules. The most commonly used decision rule is the maximum value rule, which involves iterating through all confidence scores, finding the category with the highest score, and using it as the initial hazard category for the area in the current frame.

[0066] Additionally, a confidence threshold can be set as a fallback mechanism. If the highest confidence score is still lower than the preset confidence threshold, it indicates that the classifier lacks confidence in judging all preset hazard categories, and the area may contain unknown heat source types not covered by training or complex interference. In this case, the hazard category for the area can be output as "unknown anomaly" or "pending verification," indicating that manual intervention is required.

[0067] If the highest confidence score reaches or exceeds the confidence threshold, the preset hazard category corresponding to the highest score is directly output as the hazard category in the current frame's hazard identification result. For example, if the confidence score of "overheating electrical circuits" is the highest and greater than the threshold, then the hazard category of the area corresponding to this image patch is determined to be "overheating electrical circuits".

[0068] In one embodiment, the step of generating a hazard category based on the confidence score includes: S4061. Based on the confidence scores of each preset hazard category obtained by the classification processing of the fused feature set, determine the category with the highest confidence score as the initial hazard category. S4062. Determine whether the preliminary hazard category belongs to the preset open flame related category set; If it does not belong to the category, the preliminary hazard category will be directly output as the hazard category in the hazard identification result; If it is found, the open flame verification process is triggered. In the open flame verification process, the pixel positions in the visible light image block that are determined to be flame texture regions by the texture feature map are obtained according to the cross-modal consistency feature map. Based on the pixel positions of the flame texture regions, the feature value statistics of the corresponding positions in the cross-modal consistency feature map are obtained. The verification result is obtained by comparing the statistics with the preset open flame verification threshold. S4063. When the review result is passed, the initial hazard category is output as the hazard category in the hazard identification result; S4064. When the review result is not passed, the initial hazard category is excluded from the current candidate categories, and the hazard category is re-determined and output based on the confidence score of the remaining categories.

[0069] As described in steps S4061-S4064 above, after obtaining the confidence scores of the regions corresponding to the visible light image block and the infrared image block belonging to each preset hazard category in step S405, the present invention iterates through these scores and sorts them by size. The score with the largest value is then identified, and the preset hazard category corresponding to this score is determined as the preliminary hazard category. This preliminary hazard category represents the direct classification result based on the fused feature set, but this result still needs further verification when it involves open flame categories.

[0070] Next, a predefined set of open flame-related categories is defined, which contains all hazard category labels related to open flame conditions, such as "open flame" and "flaming combustion". The preliminary hazard categories determined in step S4061 are compared one by one with each category in the predefined set of open flame-related categories.

[0071] If the initially identified hazard category does not belong to any of the sets, meaning it is not related to open flames, then the classification result does not require open flame verification. In this case, the initially identified hazard category is directly used as the final result and output as the hazard category in the hazard identification result.

[0072] If the initial hazard category belongs to one of the categories in this set, meaning it is initially identified as related to open flames, this assessment cannot be directly accepted; instead, an open flame review process must be triggered. The specific steps of this open flame review process are as follows: First, the pixel locations identified as flame texture regions by the texture feature map need to be obtained from the visible light image patch. The texture feature map, calculated in step S402, contains high-level feature information about the object's surface texture pattern. Based on the feature response patterns of each spatial location in the texture feature map, a matching determination is made with the unique texture feature patterns of a flame (such as irregular flickering edges, stripe structures of specific frequencies, etc.) to identify which pixel locations in the visible light image patch exhibit flame texture features. The set of connected locations formed by these pixels identified as having flame texture features is marked as the flame texture region.

[0073] Next, based on the pixel positions within the flame texture region, the feature values ​​corresponding to these pixel positions are searched one by one in the cross-modal consistency feature map obtained in step S403. The feature values ​​in this cross-modal consistency feature map characterize the degree of consistency between the infrared temperature distribution features and the visible light texture features at their corresponding spatial locations. These feature values ​​are statistically summarized to obtain the statistical quantity. This statistical quantity can be the average, maximum, sum, or median of these feature values. The physical meaning of this statistical quantity is: how consistent are the infrared thermal features and visible light texture features at those pixel positions determined by the visible light texture features to "look like a flame." If the region is indeed an open flame, the visual texture of the flame should spatially overlap with its high temperature distribution, thus generating a high cross-modal consistency feature value within that region.

[0074] Then, this statistic is compared with a preset open flame verification threshold. This preset open flame verification threshold is an empirical value used to define the degree of cross-modal consistency required to confirm a genuine open flame. After the comparison, the verification result is obtained. The verification result has only two possibilities: pass or fail.

[0075] In the open flame verification process, if the calculated statistic reaches a preset open flame verification threshold (i.e., the statistic is greater than or equal to the threshold), it indicates that the infrared thermal features and visible light texture features exhibit a high degree of consistency at the pixel locations identified as flame textures by the texture features. This means that the high-temperature areas in the infrared image and the flame texture areas in the visible light image are spatially highly correlated, and the evidence from both corroborates each other. In this case, the verification result is passed, and the initial hazard category determination receives strong support from bimodal consistency evidence. At this point, the initial judgment result is trusted, and the initial hazard category is output as the hazard category in the hazard identification result.

[0076] In the open flame verification process, if the calculated statistic fails to reach the preset open flame verification threshold (i.e., the statistic is less than the threshold), it indicates that the feature values ​​of the cross-modal consistency feature map are generally low at the pixel locations identified as flame textures by texture features. This means that although some areas in the visible light image appear to have flame texture features, these locations do not exhibit a corresponding high-temperature distribution pattern in the infrared thermal image. This contradictory phenomenon of "texture present but no corresponding high temperature" suggests that the area may not be a real open flame, but rather a misjudgment caused by interference with flame textures (such as red fabric being blown by the wind, sunlight reflection, screen images, etc.).

[0077] In this case, the review result is "failed". At this point, the initial hazard category judgment is overturned, and this initial hazard category needs to be excluded from the currently available candidate category set. Subsequently, the confidence scores of each preset hazard category obtained in step S405 are re-examined. After excluding the open flame category that was rejected during the review, the category with the highest confidence score is selected again from the remaining candidate categories and used as the re-determined hazard category, which is then output as the hazard category in the hazard identification result.

[0078] Through the above-mentioned open flame verification mechanism, this invention effectively integrates the fine judgment capability of visible light texture with the direct physical evidence of infrared temperature distribution, ensuring that the determination of open flame categories with high risk levels has extremely high accuracy and avoiding false alarms caused by surface features of single modal information.

[0079] In one embodiment, the step of tracking the hazard targets in the hazard identification results of multiple consecutive frames and obtaining the temporal feature sequence of each hazard instance includes: S501. Obtain the preset hazard trajectory, and obtain the position overlap value based on the position area of ​​each hazard target in the hazard identification result of the current frame and the historical position area of ​​the most recent frame in the preset hazard trajectory; S502. Obtain feature similarity values ​​based on the visible light image patch of each of the hidden danger targets and the texture feature distance of the visible light image patch corresponding to the most recent frame of each hidden danger trajectory; S503. Based on the location overlap value and feature similarity value, the hidden danger targets and hidden danger trajectories are associated and matched. The successfully matched hidden danger targets are merged into the corresponding hidden danger trajectories, and new hidden danger trajectories are created for the unmatched hidden danger targets to obtain an updated set of hidden danger trajectories. S504. Based on the set of hidden danger trajectories, obtain multiple updated hidden danger trajectories. According to the infrared image blocks associated with the hidden danger targets in each historical frame, extract the highest temperature value in each frame's infrared image block one by one, and arrange them in frame order to form a temperature evolution sequence. S505. Based on the updated hazard trajectory, extract the total number of pixels contained in the suspected thermal hazard area of ​​the corresponding hazard target in each historical frame, and arrange them in frame order to form an area evolution sequence. S506. Based on the confidence scores of the hazard categories of the corresponding hazard targets in each historical frame, the updated hazard trajectories are arranged in frame order to form a confidence evolution sequence, and the temperature evolution sequence, area evolution sequence, and confidence evolution sequence are combined to form a temporal feature sequence of the corresponding hazard instance.

[0080] As described in steps S501-S506 above, during the frame-by-frame processing of a continuous video stream, the present invention maintains a set of preset hazard trajectories after processing each previous frame. Each preset hazard trajectory in this set corresponds to a hazard instance that has been continuously tracked for a certain number of frames, and records information such as the location region, image block, and recognition result of the hazard instance in each historical frame. Among these, the historical location region of the most recent frame contained in each preset hazard trajectory is the most important spatial reference information for performing current frame association matching.

[0081] When step S4 outputs the hazard identification result for the current frame, the result contains several hazard targets identified in the current frame. Each hazard target has its corresponding location area, which is usually represented by the coordinates of the bounding box.

[0082] For each hazard target in the current frame's hazard identification results, its location region is compared with the historical location region of the most recent frame in each preset hazard trajectory to calculate the overlap. A common method for calculating the location overlap value is the intersection-union ratio (IUGR), which is the ratio of the intersection area to the union area of ​​two bounding box regions. This ratio ranges from zero to one; the larger the value, the higher the degree of spatial overlap between the two regions, and the greater the spatial probability that the hazard target and the preset hazard trajectory belong to the same object.

[0083] For each hazard target in the current frame's hazard identification result, its corresponding visible light image patch can be obtained from step S3. Simultaneously, for each preset hazard trajectory, the visible light image patch corresponding to its most recent frame can also be obtained.

[0084] Subsequently, texture features were extracted from the two visible light image patches respectively. Texture feature extraction can be accomplished using a pre-trained convolutional neural network. The image patch is input into the network, and a fixed-dimensional feature vector is output from an intermediate layer of the network. This feature vector encodes the texture, shape, and structural information of the objects in the image patch.

[0085] After obtaining the texture feature vectors of two visible light image patches, the distance between these two vectors is calculated as the texture feature distance. Commonly used distance metrics include Euclidean distance or cosine distance. To maintain directional consistency with the positional overlap value (i.e., a larger value indicates greater similarity), the texture feature distance can be converted into a feature similarity value through a mapping function. For example, taking the reciprocal of the texture feature distance, or subtracting the normalized cosine distance from one, etc. The larger the feature similarity value, the more similar the two visible light image patches are visually, and the higher the probability that they belong to the same hazard instance.

[0086] For all potential hazards in the current frame, a correlation cost matrix is ​​constructed with all potential hazard trajectories in the preset potential hazard trajectory set. Each element in this matrix combines the positional overlap value obtained in step S501 and the feature similarity value obtained in step S502 to calculate a comprehensive matching score. The comprehensive matching score can be a weighted sum of the two values, or it can be the smaller of the two values ​​or their product, etc.

[0087] Then, a data association algorithm, such as the Hungarian algorithm, is used to solve for the globally optimal matching scheme based on the association cost matrix. This algorithm outputs a set of pairing results, where each pair specifies the correspondence between a potential hazard target and a potential hazard trajectory.

[0088] For a successfully matched potential hazard, it means that the hazard is a continuation of the hazard instance tracked by its corresponding hazard trajectory in the current frame. At this time, the current frame identification result, location region, and image patch data of the hazard are appended to the historical record of the corresponding hazard trajectory to complete the trajectory update.

[0089] For any remaining hazard target that fails to match any preset hazard trajectory after association matching, it means that it is a new hazard instance appearing for the first time in the current frame. In this case, a new hazard trajectory is created for the hazard target. The new hazard trajectory takes the hazard target as its starting point, records its initial location area and identification result, and will be continuously updated in the processing of subsequent frames.

[0090] For hazard trajectories in the preset hazard trajectory set that fail to match any hazard target in the current frame, it may mean that the corresponding hazard instance has disappeared or been occluded. These trajectories can be marked as lost and temporarily stored in the updated trajectory set. If no match is found in subsequent frames, they can be deleted.

[0091] After the above processing, all updated hazard trajectories, newly created hazard trajectories, and temporarily stored lost trajectories are aggregated to obtain an updated hazard trajectory set. This updated hazard trajectory set will serve as the preset hazard trajectory set for processing the next frame, achieving iterative iteration.

[0092] After completing the trajectory update for the current frame, iterate through each updated hazard trajectory in the updated hazard trajectory set. For each updated hazard trajectory, trace its entire historical record from the starting frame to the current frame. For the historical record of each frame, find the hazard target corresponding to the updated hazard trajectory in that frame, and obtain the infrared image patch of the hazard target cropped in step S3. Compare all pixel values ​​in the infrared image patch and extract the highest temperature value in the infrared image patch of that frame.

[0093] Arrange these highest temperature values ​​in chronological order of the frames to form a one-dimensional numerical sequence, which is the temperature evolution sequence of the hazard instance corresponding to the updated hazard trajectory. This sequence depicts the trend of the core temperature of the hazard changing over time.

[0094] For each updated hazard trajectory in the updated hazard trajectory set, its records in each historical frame are also traced. For each frame's historical record, the hazard target corresponding to the updated hazard trajectory in that frame is found, and the suspected thermal hazard area determined in step S2 is obtained. This suspected thermal hazard area appears as a group of connected pixels in the infrared thermal image. The total number of pixels contained within this suspected thermal hazard area is counted, and this value directly reflects the size of the abnormally high temperature area.

[0095] Arrange these total pixels in chronological order of the frames to form a one-dimensional numerical sequence, which is the area evolution sequence of the hazard instance corresponding to the updated hazard trajectory. This sequence depicts the trend of the expansion or contraction of the hazard's thermal impact range over time.

[0096] For each updated hazard trajectory in the updated hazard trajectory set, its records in each historical frame are traced. For the historical records of each frame, the confidence score corresponding to the hazard category finally determined for that hazard target when outputting the hazard identification result in step S4 is obtained. This confidence score represents the degree of certainty in determining the hazard category in that frame, and its value changes can reflect the trend of hazard characteristics becoming more typical or more ambiguous as they develop.

[0097] Arrange these confidence scores according to the chronological order of the frames to form a one-dimensional numerical sequence, which is the confidence evolution sequence of the hidden danger instance corresponding to the updated hidden danger trajectory.

[0098] Finally, the temperature evolution sequence, area evolution sequence, and confidence evolution sequence extracted from the same hazard instance are packaged and combined as a whole to form the complete time-series feature sequence corresponding to that hazard instance. This time-series feature sequence is a multi-dimensional time series data structure, with each frame corresponding to a time point, and each time point simultaneously containing feature values ​​of the three dimensions of temperature, area, and confidence. This time-series feature sequence will be passed to subsequent steps as the direct data basis for analyzing the development trend of hazard and generating graded early warning signals.

[0099] In one embodiment, the step of generating a graded early warning signal based on the development trend of potential hazards reflected by the time-series feature sequence includes: S507. Take the temperature rise rate between each adjacent frame according to the temperature evolution sequence, and generate a temperature rise rate sequence according to the temperature rise rate between each adjacent frame. S508. Obtain the area change rate between each adjacent frame according to the area evolution sequence, and generate an area expansion rate sequence according to the area expansion rate between each adjacent frame. S509. Obtain the temperature rise acceleration between each adjacent frame according to the temperature rise rate sequence; S510. Obtain the area expansion acceleration between each adjacent frame according to the area expansion rate sequence; S511. Normalize the temperature rise acceleration and the area expansion acceleration to obtain the temperature acceleration index and the area acceleration index respectively. Obtain a classification judgment value based on the temperature acceleration index and the area acceleration index, and determine whether the classification judgment value is greater than a preset threshold. If it is less than or equal to, the hidden danger instance remains unchanged at its current development stage; If the severity is greater than the threshold, the hazard instance is determined to have entered the next development stage, wherein the development stage includes a latent period, a development period, and a danger period, and the severity of the three stages increases sequentially. S512. Based on the development stage and the hidden danger category corresponding to the hidden danger instance, generate a graded early warning signal of the corresponding level according to the preset early warning mapping rules.

[0100] As described in steps S507-S512 above, after obtaining the temperature evolution sequence of a potential hazard instance in step S506, the sequence records the highest temperature value of the potential hazard instance in each historical frame. Since the video frames are continuously acquired at fixed time intervals, the time interval between adjacent frames is known and constant.

[0101] For each pair of adjacent frames in the temperature evolution sequence, the rate of temperature rise between them is calculated. Specifically, the highest temperature value of the preceding frame is subtracted from the highest temperature value of the later frame in the time sequence to obtain the absolute temperature change over the time interval between the adjacent frames. This absolute change is then divided by the time interval between the adjacent frames to obtain the rate of temperature rise between them. The unit of this rate of temperature rise can be degrees Celsius per second or degrees Celsius per frame. If the temperature of the later frame is lower than that of the earlier frame, the rate of temperature rise is negative.

[0102] The calculated temperature rise rates between all adjacent frames are arranged in chronological order to form a one-dimensional numerical sequence with a length similar to the original temperature evolution sequence; this is the temperature rise rate sequence. This sequence reflects whether the core temperature of the potential hazard rises rapidly, slowly, or tends to stabilize within each time period.

[0103] After obtaining the area evolution sequence of a potential hazard instance in step S506, the sequence records the total number of pixels contained in the suspected thermal hazard area of ​​the hazard instance in each historical frame. Similarly, for each pair of adjacent frames in the area evolution sequence, the area expansion rate between them is calculated.

[0104] The absolute change in area between adjacent frames is obtained by subtracting the total number of pixels in the preceding frame from the total number of pixels in the later frame. This absolute change is then divided by the time interval between the adjacent frames to obtain the area expansion rate between them. The unit of this area expansion rate can be pixels per second or pixels per frame. If the area of ​​the later frame is smaller than that of the earlier frame, the area expansion rate is negative, indicating that the high-temperature region has shrunk.

[0105] The calculated area expansion rates between all adjacent frames are arranged in chronological order to form a one-dimensional numerical sequence, which is the area expansion rate sequence. This sequence reflects whether the impact range of the potential hazard expands rapidly, slowly, or remains stable within each time period.

[0106] After obtaining the temperature rise rate sequence in step S507, this sequence itself is also a numerical sequence that changes over time. The inter-frame difference operation is then performed again on this sequence.

[0107] The specific calculation method is as follows: for each pair of adjacent rate values ​​in the temperature rise rate sequence, subtract the preceding rate value from the later rate value in the time sequence to obtain the absolute change in rate. Then, divide this absolute change by the time interval between adjacent frames to obtain the temperature rise acceleration between those adjacent frames. The unit of this temperature rise acceleration can be degrees Celsius per square second or degrees Celsius per square frame.

[0108] The acceleration of temperature rise is a key early warning indicator. When the acceleration of temperature rise is positive and large, it indicates that the temperature is not only rising, but the rate of increase is also accelerating, which is a strong signal that potential hazards are worsening rapidly. When the acceleration of temperature rise is close to zero, it indicates that the temperature is rising at a relatively stable rate. When the acceleration of temperature rise is negative, it indicates that the rate of temperature increase is slowing down.

[0109] After obtaining the area expansion rate sequence in step S508, the adjacent frame difference operation is performed on the sequence in a manner similar to that in step S509.

[0110] The specific calculation method is as follows: for each pair of adjacent rate values ​​in the area expansion rate sequence, subtract the preceding rate value from the later rate value in time sequence to obtain the absolute change in rate. Then, divide this absolute change by the time interval between adjacent frames to obtain the area expansion acceleration between those adjacent frames. The unit of this area expansion acceleration can be pixels per square second or pixels per square frame.

[0111] The area expansion acceleration reflects the strength of the expansion momentum of the potential thermal impact range. When the area expansion acceleration is positive and large, it indicates that the high-temperature area is spreading outward at an increasingly faster rate; when the area expansion acceleration is negative, it indicates that the expansion momentum is being suppressed or the area is beginning to shrink.

[0112] After obtaining the acceleration due to temperature rise and the acceleration due to area expansion, since the two have different dimensions (the dimension of acceleration due to temperature rise is ℃ / s), 2 The area expansion acceleration metric is in pixels per second. 2 Since these cannot be directly used for numerical calculations, we first normalize them to map them to a unified 0,1 interval, thus obtaining dimensionless temperature acceleration and area acceleration indices.

[0113] The normalization method can be one of the following: Method 1, perform linear normalization based on the maximum possible acceleration value; Method 2, perform standardization based on the mean and standard deviation of the historical statistical data and then map it to the 0,1 interval; Method 3, map the acceleration value to the 0,1 interval through a preset sigmoid function.

[0114] After obtaining the temperature acceleration index and area acceleration index with unified dimensions, the graded judgment value is calculated by weighted fusion.

[0115] The calculated grading judgment value is compared with a preset threshold. This preset threshold is a critical value for determining whether a hidden danger has entered a more advanced stage of development.

[0116] If the classification judgment value is less than or equal to a preset threshold, it indicates that the current rate of deterioration of the hazard has not yet reached a level sufficient to change its development stage. In this case, the hazard instance maintains its current development stage. For example, if the hazard instance was previously determined to be in the latent stage, and the classification judgment value does not exceed the preset threshold, it remains in the latent stage; if the hazard instance is a newly created trajectory, and its development stage is initialized to the latent stage upon creation, then under this comparison result, its development stage remains unchanged as the latent stage.

[0117] If the grading judgment value exceeds a preset threshold, it indicates that the hazard's deterioration has accelerated significantly, reaching the threshold for altering its developmental state. At this point, the current developmental stage of the hazard is determined based on the specific extent to which the grading judgment value exceeds the preset threshold. The developmental stages are divided into three progressive levels: latent, development, and danger. The latent stage corresponds to the hazard's initial appearance, with the temperature slightly higher than the background and changing slowly; the development stage corresponds to the hazard's temperature continuously rising and its area beginning to expand; the danger stage corresponds to the hazard's temperature and area both rapidly deteriorating, potentially turning into an open flame or igniting surrounding objects at any time. The greater the grading judgment value exceeds the preset threshold, the higher the corresponding developmental stage level.

[0118] A set of early warning mapping rules is pre-defined, which takes the development stage and the category of potential hazard as input conditions and a graded early warning signal as the output result. This rule can be a two-dimensional mapping table or a set of conditional decision logic.

[0119] Based on the current development stage of the hazard instance determined in step S511, and the hazard category determined and recorded in the trajectory of the hazard instance in step S4, these are used as input parameters for querying or logical matching within the preset early warning mapping rules. The output of the rules is the corresponding level of graded early warning signal. For example, the graded early warning signals can be divided into three levels: Level 1 warning corresponds to the dangerous period and the hazard category is the highest level of urgency related to open flames, requiring immediate activation of the fire extinguishing system and notification of the fire department; Level 2 warning corresponds to the development or dangerous period and the hazard category is electrical overheating, flammable material accumulation, etc., requiring immediate on-site verification and handling; Level 3 warning corresponds to the incubation period, requiring continuous close monitoring and recording at the monitoring center. Different levels of early warning signals correspond to different response procedures and resource allocation schemes.

[0120] Through the above steps, a complete early warning process is completed, from trend analysis of the temporal characteristic sequence of hidden dangers to determination of development stage and generation of graded early warning signals, realizing the intelligent management goal of fire hazard management from passive monitoring to proactive early warning.

[0121] like Figure 2 As shown, the present invention also provides a real-time monitoring and early warning system for fire hazards that combines image recognition, comprising: The image pair acquisition module is used to acquire visible light images and infrared thermal images of the monitored area based on image recognition, and to perform spatial registration and temporal alignment on the visible light images and the infrared thermal images to obtain pixel-by-pixel corresponding dual-modal image pairs and the registration relationship between the visible light images and the infrared thermal images; The hidden danger area acquisition module is used to acquire the heat diffusion area of ​​the infrared thermal image based on the dual-modal image pair, and to filter out the areas that meet the point heat source morphology conditions based on the heat diffusion area, and to regard them as suspected heat hidden danger areas. The image patch acquisition module is used to map the suspected thermal hazard area to the corresponding position in the visible light image according to the registration relationship, and to crop the corresponding visible light image patch and infrared image patch from the dual-modal image pair; The hazard category acquisition module is used to obtain the hazard identification result of the current frame based on the visible light image block and the infrared image block, and to obtain the hazard category based on the hazard identification result; The graded early warning signal acquisition module is used to track the hidden danger targets in the hidden danger identification results of multiple consecutive frames, acquire the temporal feature sequence of each hidden danger instance, and generate a graded early warning signal based on the hidden danger development trend reflected by the temporal feature sequence.

[0122] like Figure 3 As shown, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of a method for real-time monitoring and early warning of fire hazards combined with image recognition.

[0123] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of a method for real-time monitoring and early warning of fire hazards combined with image recognition.

[0124] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of a method for real-time monitoring and early warning of fire hazards combined with image recognition. The above descriptions are merely preferred embodiments of this invention and do not limit the scope of the patent. Any equivalent results or procedural transformations made based on the description and drawings of this invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this invention.

Claims

1. A method for real-time monitoring and early warning of fire hazards combining image recognition, characterized in that, include: Visible light images and infrared thermal images of the monitored area are obtained based on image recognition, and spatial registration and temporal alignment are performed on the visible light images and the infrared thermal images to obtain pixel-by-pixel corresponding dual-modal image pairs and the registration relationship between the visible light images and the infrared thermal images; The thermal diffusion area of ​​the infrared thermal image is obtained based on the dual-modal image pair, and the area that meets the point heat source morphology condition is selected based on the thermal diffusion area and regarded as the suspected thermal hazard area. The suspected thermal hazard area is mapped to the corresponding position in the visible light image according to the registration relationship, and the corresponding visible light image block and infrared image block are cropped from the dual-modal image pair; The hazard identification result of the current frame is obtained based on the visible light image block and the infrared image block, and the hazard category is obtained based on the hazard identification result; The hidden danger targets in the hidden danger identification results of multiple consecutive frames are tracked to obtain the temporal feature sequence of each hidden danger instance, and a graded early warning signal is generated based on the hidden danger development trend reflected by the temporal feature sequence.

2. The method for real-time monitoring and early warning of fire hazards combined with image recognition according to claim 1, characterized in that, The step of obtaining the heat diffusion region of the infrared thermal image based on the dual-modal image pair, and filtering out regions that meet the point heat source morphology conditions based on the heat diffusion region, and identifying these regions as suspected heat hazard areas, includes: The horizontal and vertical temperature gradients of each pixel block in the infrared thermal image are obtained based on the dual-modal image, and a gradient direction pattern is generated based on the horizontal and vertical temperature gradients. The temperature value of each pixel block is obtained based on the infrared thermal image, and all local temperature maxima are obtained based on the temperature value of each pixel block. The obtained local temperature maxima are determined as candidate pixel blocks. Obtain the background temperature threshold. Starting from the candidate pixel block, check whether the gradient direction of the adjacent pixel blocks on the current heat diffusion region boundary points to the candidate pixel block according to the gradient direction map. Also, check whether the temperature value of the adjacent pixel block is greater than the background temperature threshold according to the infrared thermal image. If both conditions are met, the adjacent pixel block candidate point is included in the thermal diffusion region and continues to grow outward until no new pixel meets the conditions, thus obtaining the thermal diffusion region corresponding to the pixel block candidate point. The region compactness is obtained based on each heat diffusion region, and the thermal center offset of the heat diffusion region is calculated based on the geometric center position of each heat diffusion region and the position of the highest temperature pixel within the region. The consistency of the gradient direction of the heat diffusion region is obtained by comparing the gradient direction of each pixel within each heat diffusion region with the reference direction of each pixel pointing to the candidate point of its pixel block. Based on the compactness, thermal center offset, and gradient direction consistency of each heat diffusion region, a comparison is made with a preset point heat source morphology threshold. Regions that meet the point heat source morphology threshold are selected from all heat diffusion regions as suspected heat hazard regions.

3. The method for real-time monitoring and early warning of fire hazards combined with image recognition according to claim 2, characterized in that, The step of checking whether the gradient direction of adjacent pixel blocks on the current heat diffusion region boundary points to the pixel block candidate point according to the gradient direction map, starting from the pixel block candidate point, includes: Based on the coordinates of the adjacent pixels in the infrared thermal image and the coordinates of the candidate point of the pixel block in the infrared thermal image, a reference direction vector is calculated with the adjacent pixels as the starting point and the candidate point of the pixel block as the ending point. The gradient direction vectors of adjacent pixel blocks are obtained based on the gradient direction map. Calculate the angle between the reference direction vector and the gradient direction vector; When the included angle is less than a preset angle threshold, it is determined that the gradient direction of the adjacent pixel points to the candidate point of the pixel block.

4. The method for real-time monitoring and early warning of fire hazards combined with image recognition according to claim 1, characterized in that, The step of obtaining the hazard identification result of the current frame based on the visible light image block and the infrared image block, and obtaining the hazard category based on the hazard identification result, includes: Multiple pixel values ​​are obtained from the infrared image block, and the temperature distribution feature map and the highest temperature value of the infrared image block are obtained from the pixel values. The color channel value corresponding to each pixel value is obtained from the visible light image block, and the texture feature map of the visible light image block is obtained from the color channel value. Based on the temperature distribution feature map of the infrared image block and the texture feature map of the visible light image block, the feature response intensity of each local region corresponding to the spatial position in the two feature maps is compared to obtain a cross-modal consistency feature map that characterizes the consistency of the feature response of the two modes at corresponding spatial positions. The temperature margin feature value, which characterizes the degree to which the current temperature is close to the danger temperature, is obtained based on the highest temperature value. The temperature margin feature value is then aggregated with the cross-modal consistency feature map to obtain a fused feature set. Based on the fused feature set, the confidence scores of the regions corresponding to the visible light image patch and the infrared image patch belonging to each preset hazard category are obtained respectively; Hazard categories are generated based on the confidence scores.

5. A method for real-time monitoring and early warning of fire hazards combined with image recognition according to claim 4, characterized in that, The step of generating hazard categories based on the confidence score includes: Based on the confidence scores of each preset hazard category obtained by the classification processing of the fused feature set, the category with the highest confidence score is determined as the initial hazard category; Determine whether the preliminary hazard category belongs to a preset set of open flame-related categories; If it does not belong to the category, the preliminary hazard category will be directly output as the hazard category in the hazard identification result; If it is found, the open flame verification process is triggered. In the open flame verification process, the pixel positions in the visible light image block that are determined to be flame texture regions by the texture feature map are obtained according to the cross-modal consistency feature map. Based on the pixel positions of the flame texture regions, the feature value statistics of the corresponding positions in the cross-modal consistency feature map are obtained. The verification result is obtained by comparing the statistics with the preset open flame verification threshold. When the review result is passed, the initial hazard category is output as the hazard category in the hazard identification result; When the review result is not passed, the initial hazard category is excluded from the current candidate categories, and the hazard category is re-determined and output based on the confidence score of the remaining categories.

6. The method for real-time monitoring and early warning of fire hazards combined with image recognition according to claim 1, characterized in that, The step of tracking the hazard targets in the hazard identification results of multiple consecutive frames and obtaining the temporal feature sequence of each hazard instance includes: Obtain the preset hazard trajectory, and obtain the position overlap value based on the position area of ​​each hazard target in the hazard identification result of the current frame and the historical position area of ​​the most recent frame in the preset hazard trajectory; The feature similarity value is obtained based on the texture feature distance between the visible light image patch of each of the potential hazards and the visible light image patch corresponding to the most recent frame of each potential hazard trajectory; The hazard targets and hazard trajectories are associated and matched based on the location overlap value and feature similarity value. Successfully matched hazard targets are incorporated into the corresponding hazard trajectories, and new hazard trajectories are created for unmatched hazard targets to obtain an updated hazard trajectory set. Based on the set of hidden danger trajectories, multiple updated hidden danger trajectories are obtained. According to the infrared image blocks associated with the hidden danger targets in each historical frame, the highest temperature value in each frame's infrared image block is extracted one by one and arranged in frame order to form a temperature evolution sequence. Based on the updated hazard trajectory, the suspected thermal hazard area corresponding to the hazard target in each historical frame is extracted one by one, and the total number of pixels contained in the suspected thermal hazard area of ​​each frame is arranged in frame order to form an area evolution sequence. Based on the confidence scores of the hazard categories of the corresponding hazard targets in each historical frame, the updated hazard trajectories are arranged in frame order to form a confidence evolution sequence. The temperature evolution sequence, area evolution sequence, and confidence evolution sequence are then combined to form a temporal feature sequence of the corresponding hazard instance.

7. A method for real-time monitoring and early warning of fire hazards combined with image recognition according to claim 6, characterized in that, The step of generating a graded early warning signal based on the development trend of hidden dangers reflected by the time-series feature sequence includes: The temperature rise rate between each adjacent frame is obtained based on the temperature evolution sequence, and a temperature rise rate sequence is generated based on the temperature rise rate between each adjacent frame. The area change rate between each adjacent frame is obtained based on the area evolution sequence, and an area expansion rate sequence is generated based on the area expansion rate between each adjacent frame. The temperature rise acceleration between each adjacent frame is obtained based on the temperature rise rate sequence. The area expansion acceleration between each adjacent frame is obtained based on the area expansion rate sequence; The temperature rise acceleration and the area expansion acceleration are normalized to obtain temperature acceleration index and area acceleration index respectively. A grading judgment value is obtained based on the temperature acceleration index and the area acceleration index, and it is determined whether the grading judgment value is greater than a preset threshold. If it is less than or equal to, the hidden danger instance remains unchanged at its current development stage; If the value is greater than the specified value, the potential hazard instance is determined to have entered the next development stage, wherein the development stage includes the incubation period, the development period, and the danger period. Based on the development stage and the hazard category corresponding to the hazard instance, a corresponding level of graded early warning signal is generated according to the preset early warning mapping rules.

8. A real-time monitoring and early warning system for fire hazards combining image recognition, characterized in that, include: The image pair acquisition module is used to acquire visible light images and infrared thermal images of the monitored area based on image recognition, and to perform spatial registration and temporal alignment on the visible light images and the infrared thermal images to obtain pixel-by-pixel corresponding dual-modal image pairs and the registration relationship between the visible light images and the infrared thermal images; The hidden danger area acquisition module is used to acquire the heat diffusion area of ​​the infrared thermal image based on the dual-modal image pair, and to filter out the areas that meet the point heat source morphology conditions based on the heat diffusion area, and to regard them as suspected heat hidden danger areas. The image patch acquisition module is used to map the suspected thermal hazard area to the corresponding position in the visible light image according to the registration relationship, and to crop the corresponding visible light image patch and infrared image patch from the dual-modal image pair; The hazard category acquisition module is used to obtain the hazard identification result of the current frame based on the visible light image block and the infrared image block, and to obtain the hazard category based on the hazard identification result; The graded early warning signal acquisition module is used to track the hidden danger targets in the hidden danger identification results of multiple consecutive frames, acquire the temporal feature sequence of each hidden danger instance, and generate a graded early warning signal based on the hidden danger development trend reflected by the temporal feature sequence.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images