Multi-modal sensing intelligent fire-fighting monitoring system based on deep learning

Abstract

The invention relates to the technical field of fire-fighting monitoring, in particular to a multi-modal sensing intelligent fire-fighting monitoring system based on deep learning, which comprises an environment detection module, a detection analysis module, a fire-fighting judgment module, a fire-fighting early warning module and a deep learning module, and is characterized in that the environment detection module acquires various information of a monitoring area and divides fire-fighting monitoring sub-areas; the detection analysis module determines a fire-fighting analysis representation image of the fire-fighting monitoring sub-region, and determines a fire-fighting abnormal tendency parameter to determine a fire-fighting abnormal tendency category; the fire-fighting judgment module selects a corresponding fire-fighting analysis mode and determines whether fire-fighting abnormity early warning is carried out or not according to a fire-fighting analysis result; the fire-fighting early warning module sends fire-fighting early warning to the corresponding fire-fighting monitoring sub-area according to the early warning judgment result; the deep learning module adjusts the fire-fighting monitoring subareas and determines a fire-fighting traversal inspection mode according to the fire-fighting early warning record; according to the invention, the accuracy of fire-fighting hidden danger detection is improved, and the occupation of redundant monitoring resources is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of fire monitoring, and particularly to a multi-modal sensing intelligent fire monitoring system based on deep learning. Background Art

[0002] In the prior art, multi-modal fire detection mainly relies on the combination of various sensors. For example, image sensors are used for visual monitoring of flames and smoke, gas sensors detect the concentration of harmful gases, and temperature sensors monitor environmental temperature changes, etc. These multi-modal sensors work together to provide more comprehensive fire monitoring data and improve the detection ability for abnormal situations such as fires. At the same time, the intelligent decision-making system fuses multi-source information such as video images, gas concentration, temperature and humidity, etc., combines expert knowledge bases and deep learning algorithms to realize intelligent monitoring and risk assessment of the fire environment, and can automatically generate emergency decision-making plans. This technology significantly improves the efficiency and reliability of fire monitoring.

[0003] However, there are still some deficiencies in the prior art in terms of multi-modal data fusion and analysis. Firstly, the data formats and frequencies collected by different sensors are inconsistent, resulting in greater difficulty in data fusion and affecting the real-time performance and accuracy of the system. Secondly, when the deep learning model processes complex multi-modal data, problems such as overfitting or insufficient generalization ability may occur, affecting its detection effect in actual scenarios. Summary of the Invention

[0004] Therefore, the present invention provides a multi-modal sensing intelligent fire monitoring system based on deep learning to overcome the problems of large difficulty in multi-modal fire monitoring data fusion and insufficient generalization ability of the deep learning model in the prior art.

[0005] To achieve the above object, the present invention provides a multi-modal sensing intelligent fire monitoring system based on deep learning, including:

[0006] An environment detection module, which is used to continuously obtain video information and infrared information of the monitoring area, obtain the environmental information of the monitoring area, and divide the monitoring area into several fire monitoring sub-areas;

[0007] A detection and analysis module, which is connected to the environment detection module, used to determine the fire analysis characterization images of all fire monitoring sub-areas within each monitoring time period, parse each fire analysis characterization image, determine the fire abnormal tendency parameters of the corresponding fire monitoring sub-areas in combination with the environmental information, and determine the fire abnormal tendency categories according to the fire abnormal tendency parameters;

[0008] A fire determination module, which is respectively connected to the environment detection module and the detection and analysis module, is used to select a corresponding fire analysis method according to the fire abnormal tendency categories of each fire monitoring sub-region, including obtaining the peak value of environmental information in the current monitoring time period, or analyzing the flow information in the current monitoring time period, and determining whether to issue a fire abnormal warning according to the fire analysis result;

[0009] A fire warning module, which is respectively connected to the environment detection module and the environment detection module, is used to issue a fire warning to the corresponding fire monitoring sub-region according to the determination result of the fire abnormal warning;

[0010] A deep learning module, which is respectively connected to the environment detection module, the detection and analysis module and the fire determination module, is used to adjust the fire monitoring sub-region according to the fire abnormal tendency parameters for several consecutive times, and determine the fire traversal inspection method according to the fire warning records.

[0011] As a preferred technical solution of the multi-modal sensing intelligent fire monitoring system based on deep learning, the environment detection module includes:

[0012] A video acquisition unit, including several camera devices, is used to acquire video information and infrared information of the shootable area in the monitoring area;

[0013] A sensing detection unit, including several multi-mode sensor groups, is used to acquire the environmental information of each fire monitoring sub-region;

[0014] An information transmission unit, which is respectively connected to the video acquisition unit and the sensing detection unit, is used to collect the video information, infrared information and all environmental information of the monitoring area, and divide the monitoring area into several fire monitoring sub-regions according to the distribution positions of the multi-mode sensor groups;

[0015] Wherein, the environmental information includes temperature information, audio information and smoke information.

[0016] As a preferred technical solution of the multi-modal sensing intelligent fire monitoring system based on deep learning, the detection and analysis module selects a fire analysis characterization image according to the video information and infrared information in each fire monitoring sub-region during the monitoring time period;

[0017] The detection and analysis module selects the image in the video information corresponding to the moment with the least area of the red edge and its internal area in the infrared information during the monitoring time period as the fire analysis characterization image of the monitoring time period.

[0018] As an optimal technical solution of the multi-modal sensing intelligent fire monitoring system based on deep learning, the detection and analysis module determines the fire abnormal tendency parameter of the corresponding fire monitoring sub-region according to the parsing result of the fire analysis characterization image and in combination with the temperature information in the environmental information;

[0019] The detection and analysis module determines the proportion of the area of the red edge and its internal region in the fire analysis characterization image, which is recorded as the parsing result of the characterization image, and determines the abnormal tendency coefficient;

[0020] The detection and analysis module calculates the average value of the temperature information and its average deviation, and obtains the abnormal tendency factor by dividing the temperature average deviation by the temperature average value;

[0021] The detection and analysis module determines the fire abnormal tendency parameter according to the product of the abnormal tendency coefficient and the abnormal tendency factor.

[0022] As an optimal technical solution of the multi-modal sensing intelligent fire monitoring system based on deep learning, the detection and analysis module determines the fire abnormal tendency category of the corresponding fire monitoring sub-region according to the fire abnormal tendency parameter, including:

[0023] If the fire abnormal tendency parameter is greater than the standard abnormal tendency parameter threshold, the detection and analysis module determines that the corresponding fire monitoring sub-region has an obvious fire abnormal tendency;

[0024] If the fire abnormal tendency parameter is less than or equal to the standard abnormal tendency parameter threshold, the detection and analysis module determines that the corresponding fire monitoring sub-region has a hidden fire abnormal tendency.

[0025] As an optimal technical solution of the multi-modal sensing intelligent fire monitoring system based on deep learning, the fire determination module selects the corresponding fire analysis method according to the fire abnormal tendency category of the fire monitoring sub-region, including:

[0026] In response to the fire abnormal tendency category of the fire monitoring sub-region being an obvious fire abnormal tendency, the fire analysis method is to analyze the flow information in the current monitoring time period according to the position of the camera device;

[0027] In response to the fire abnormal tendency category of the fire monitoring sub-region being a hidden fire abnormal tendency, the fire analysis method is to obtain the peak value of the environmental information in the current monitoring time period.

[0028] As an optimal technical solution of the multi-modal sensing intelligent fire monitoring system based on deep learning, the fire warning module determines whether to issue a fire abnormal warning according to the fire analysis result, including:

[0029] The fire warning module analyzes the traffic information during the current monitoring period based on the position of the camera device, determines the rgb representation value of the moving target during the current period, and if the absolute value of the representation value difference in the rgb representation value is greater than the traffic law threshold, it determines to issue a fire anomaly warning;

[0030] The fire warning module obtains the environmental information peak value and the duration corresponding to the peak value during the current monitoring time. If there is no peak value in the temperature information, and / or the pitch of the audio information exceeds the environmental threshold, and / or the smoke information exceeds the allowable threshold, it determines to issue a fire anomaly warning.

[0031] As an optimal technical solution of the multi-modal sensing intelligent fire monitoring system based on deep learning, the deep learning module adjusts the fire monitoring sub-region according to the fire anomaly tendency parameters for several consecutive times,

[0032] If the number of consecutive times that the fire anomaly tendency parameters of adjacent fire monitoring sub-regions are within the stable interval exceeds the stable time, the adjacent fire monitoring sub-regions are integrated into a new fire monitoring sub-region.

[0033] As an optimal technical solution of the multi-modal sensing intelligent fire monitoring system based on deep learning, the deep learning module determines the fire traversal inspection method according to the fire warning records, including:

[0034] The deep learning module determines the inspection selection time period and the areas that must be passed during the inspection according to the occurrence time of the fire warning records;

[0035] The deep learning module selects the time period with the largest coverage range in all inspection selection time periods as the traversal inspection time according to the length of the traversal inspection time.

[0036] The beneficial effects of the present invention are:

[0037] This technical solution realizes intelligent fire monitoring through multi-module collaboration. The environmental detection module integrates video, infrared, and multi-modal sensor data, divides the fire monitoring sub-areas, and dynamically collects multi-modal information such as temperature, audio, and smoke. The detection and analysis module calculates the fire anomaly tendency parameters by combining the proportion of the infrared image area and the temperature deviation, and distinguishes the dominant / recessive risk categories. The fire determination module matches the flow analysis or environmental peak monitoring strategy for different risk categories. The early warning module accurately triggers early warnings based on multi-dimensional data such as the difference in RGB characterization values and the peaks of multi-source information. The deep learning module dynamically integrates adjacent low-risk sub-areas based on continuous risk parameters to optimize the monitoring unit, and intelligently plans the inspection time period and path according to historical early warning records. Through multi-modal data fusion analysis, dynamic threshold calibration, and adaptive area management, the present invention significantly improves the accuracy and efficiency of fire hazard detection, reduces the occupancy of redundant monitoring resources. At the same time, through risk grading early warning and intelligent inspection strategies, it enhances the adaptability to complex environments and the intelligent level of fire prevention and control, providing efficient and reliable technical support for early fire detection and emergency response.

[0038] In particular, in the present invention, by selecting the image in the video information corresponding to the moment with the smallest sum of the areas of the red edge and its internal area in the infrared information during the monitoring time period as the fire analysis characterization image, the problem of occlusion of the fire anomaly area by moving heat sources in the monitoring area can be effectively addressed. The key area in the infrared information is determined using vision technology or image processing technology to ensure that the selected characterization image can best reflect whether there are fire hazards in the current area, not only improving the accuracy of fire hazard detection, but also enhancing the adaptability of the system to complex environments through the fusion analysis of multi-modal data.

[0039] In particular, through multi-modal data fusion and dynamic threshold adjustment, the present invention significantly improves the calculation accuracy and response efficiency of fire anomaly tendency parameters. First, by combining image analysis and temperature data analysis, the complementary verification of visual and thermal data is realized, avoiding misjudgment by a single sensor. Second, the product model of the anomaly coefficient and factor is introduced to dynamically correlate the image anomaly features (such as the flame spread range) with the environmental temperature change trend (such as the local temperature rise rate), enhancing the sensitivity to early fires. When there is no camera or the image anomaly is small, the system automatically degrades to the temperature-dominated mode to ensure the basic monitoring ability; while when the image anomaly is significant, the coefficient is set to ∞ and it is directly determined as an anomaly to ensure timely response to unrecognizable anomalies. By dynamically calibrating the area threshold with historical data, the system can adapt to the risk characteristics of different scenarios, taking into account both sensitivity and false alarm control. This solution is particularly suitable for complex industrial environments and can achieve accurate and graded fire early warnings under resource-constrained conditions, effectively improving the prevention and control efficiency and reliability.

[0040] In particular, in the present invention, the fire determination module flexibly selects corresponding fire analysis methods according to the categories of fire abnormal tendency, taking into account both the traffic information analysis under obvious fire abnormal tendency and the environmental information peak monitoring under hidden fire abnormal tendency. At the same time, the fire warning module comprehensively analyzes multi-dimensional data such as the rgb characterization value difference, environmental information peak, and duration in the traffic information to accurately judge fire abnormal situations. The analysis of the rgb characterization value difference utilizes the feature of the consistency of personnel clothing colors to effectively identify potential fire hazards such as the intrusion of foreign personnel or abnormal aggregation of personnel; the environmental information monitoring captures early signs of fire occurrence in a timely manner through the real-time monitoring of key indicators such as temperature, audio, and smoke. In addition, the system also has the ability of adaptive adjustment, dynamically calibrating the threshold according to historical data to ensure accurate and hierarchical fire warning in different scenarios, effectively improving the intelligent level and reliability of fire prevention and control, and providing strong support for the early detection and emergency disposal of fires.

[0041] In particular, in the present invention, the deep learning module continuously analyzes the fire abnormal tendency parameters of each fire monitoring sub-region, reflecting the regional fire risk trend. When the abnormal tendency parameters of adjacent sub-regions continuously remain stable within a preset stable interval (such as the low-risk threshold), it indicates that their environmental risk characteristics are convergent and controllable. At this time, the system integrates adjacent sub-regions into a new fire monitoring sub-region to reduce the occupancy of redundant analysis resources. This dynamic adjustment not only improves the system efficiency but also ensures that the merged region can still maintain the safety monitoring ability through long-term stability verification, reflecting the adaptive management ability of the intelligent fire protection system to complex environments.

[0042] In particular, in the present invention, the inspection time period and the necessary areas are set according to the occurrence time of the warning records. These necessary areas are the places where fire warnings have occurred in history. The module will select a time period with the largest coverage range from all possible inspection time periods as the actual inspection time, which can ensure that the inspection is concentrated in the areas and time periods where fire problems are most likely to occur, rather than conducting non-discriminatory inspections on all monitoring areas, thereby improving the efficiency and pertinence of the inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic structural diagram of the multi-modal sensing intelligent fire monitoring system based on deep learning according to an embodiment of the present invention;

[0044] Figure 2 It is a schematic structural diagram of the environmental detection module according to an embodiment of the present invention;

[0045] Figure 3 It is a logic diagram for determining the categories of fire abnormal tendency of corresponding fire monitoring sub-regions according to an embodiment of the present invention;

[0046] Figure 4Logic diagram for selecting corresponding fire analysis methods for embodiments of the present invention. Detailed implementation manners

[0047] In order to make the objectives and advantages of the present invention clearer and more understandable, the present invention will be further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0048] The preferred implementation manners of the present invention will be described below with reference to the accompanying drawings. Those skilled in the art should understand that these implementation manners are only used to explain the technical principles of the present invention and do not limit the protection scope of the present invention.

[0049] It should be noted that in the description of the present invention, terms indicating directions or positional relationships such as "upper", "lower", "left", "right", "inner", "outer", etc. are based on the directions or positional relationships shown in the drawings. This is only for the convenience of description and does not indicate or imply that the device or component must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention.

[0050] In addition, it should also be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installation", "connection", and "connection" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0051] Please refer to Figure 1 As shown, it is a structural schematic diagram of a multi-modal sensing intelligent fire monitoring system based on deep learning according to an embodiment of the present invention. The present invention provides a multi-modal sensing intelligent fire monitoring system based on deep learning, including:

[0052] An environment detection module, which is used to continuously obtain video information and infrared information of the monitoring area, obtain the environmental information of the monitoring area, and divide the monitoring area into several fire monitoring sub-areas;

[0053] A detection and analysis module, which is connected to the environment detection module, used to determine the fire analysis characterization images of all fire monitoring sub-areas within each monitoring time period, analyze each fire analysis characterization image, combine the environmental information to determine the fire anomaly tendency parameters of the corresponding fire monitoring sub-areas, and determine the fire anomaly tendency categories according to the fire anomaly tendency parameters;

[0054] A fire determination module, which is respectively connected to the environment detection module and the detection and analysis module, and is used to select a corresponding fire analysis method according to the fire abnormal tendency categories of each fire monitoring sub-region, including obtaining the peak value of environmental information in the current monitoring time period, or analyzing the flow information in the current monitoring time period, and determining whether to issue a fire abnormal warning according to the fire analysis result;

[0055] A fire warning module, which is respectively connected to the environment detection module and the environment detection module, and is used to issue a fire warning to the corresponding fire monitoring sub-region according to the determination result of the fire abnormal warning;

[0056] A deep learning module, which is respectively connected to the environment detection module, the detection and analysis module and the fire determination module, and is used to adjust the fire monitoring sub-region according to continuous fire abnormal tendency parameters for several times, and determine the fire traversal inspection method according to the fire warning record.

[0057] In implementation, the specific structures of the detection and analysis module, the fire determination module and the deep learning module of the present invention are not limited, and they can be composed of logic components, and the logic components include a field programmable processor, a computer and a microprocessor in the computer.

[0058] The monitoring time period is selected within 5 - 30 minutes, the initial monitoring time period is 5 minutes, and the duration of the actual monitoring time period can be selected according to the change situation of the monitoring area.

[0059] This technical solution realizes intelligent fire monitoring through multi-module collaboration. The environment detection module integrates video, infrared and multi-mode sensor data, divides the fire monitoring sub-region and dynamically collects multi-modal information such as temperature, audio, and smoke; the detection and analysis module combines the infrared image area ratio and temperature deviation to calculate the fire abnormal tendency parameter and distinguish the dominant / hidden risk categories; the fire determination module matches the flow analysis or environmental peak monitoring strategy for different risk categories, and the warning module accurately triggers the warning by integrating multi-dimensional data such as the rgb characterization value difference and multi-source information peak; the deep learning module dynamically integrates adjacent low-risk sub-regions based on continuous risk parameters to optimize the monitoring unit, and intelligently plans the inspection time period and path according to the historical warning record. The present invention significantly improves the accuracy and efficiency of fire hazard detection through multi-modal data fusion analysis, dynamic threshold calibration and adaptive area management, reduces the occupation of redundant monitoring resources, and at the same time enhances the adaptability to complex environments and the intelligent level of fire prevention and control through risk classification warning and intelligent inspection strategies, providing efficient and reliable technical support for early fire detection and emergency disposal.

[0060] Specifically, the environment detection module includes:

[0061] The video acquisition unit includes a number of camera devices for acquiring video information and infrared information of the shootable areas within the monitoring area;

[0062] The sensing detection unit includes a number of multimode sensor groups for acquiring the environmental information of each of the fire monitoring sub-areas;

[0063] The information transmission unit is respectively connected to the video acquisition unit and the sensing detection unit for collecting the video information, infrared information and all environmental information of the monitoring area, and dividing the monitoring area into a number of fire monitoring sub-areas according to the distribution positions of the multimode sensor groups;

[0064] Wherein, the environmental information includes temperature information, audio information and smoke information.

[0065] In implementation, the video acquisition unit is composed of a number of cameras. The ways for the video acquisition unit to acquire monitoring videos and for the multimode sensor groups to collect environmental information can be any one in the prior art and are not specifically limited herein. A single sensor module includes a temperature sensor, a sound sensor and a smoke sensor. The types of sensors and the ways for the sensors to acquire information are any one in the prior art and are not specifically limited herein.

[0066] Each fire monitoring sub-area contains at least one group of multimode sensor groups and does not necessarily contain camera devices. The number of initial fire monitoring sub-areas is the same as the number of the multimode sensor groups. The monitoring range of a single camera device can cover multiple fire monitoring sub-areas. The sensors are arranged according to local fire codes (such as GB 50116 in China) while ensuring that the total detection area covers the entire monitoring area.

[0067] It can be understood that since there are blind spots for camera monitoring and non-shootable areas (such as blocked spaces, bathrooms, etc.) in the monitoring area, the shootable areas are the camera coverage areas of all monitoring devices in the permitted shooting areas within each monitoring area.

[0068] Specifically, the detection and analysis module selects a fire analysis characterization image according to the video information and infrared information within each fire monitoring sub-area during the monitoring time period;

[0069] The detection and analysis module selects the image in the video information corresponding to the moment with the smallest sum of the areas of the red edges and their internal areas in the infrared information during the monitoring time period as the fire analysis characterization image of the monitoring time period.

[0070] In implementation, determining the red edge and the area of its internal region in the infrared information can be achieved by any one of vision technologies (object detection algorithms based on deep learning (such as YOLO series, Faster R-CNN, etc.), or traditional computer vision feature extraction and analysis (such as color feature extraction, edge detection, and region segmentation, etc.)) or image processing technologies (threshold methods based on image segmentation (such as global threshold method, adaptive threshold method, etc.)), and no specific limitation is made here.

[0071] It can be understood that since there are movable heat sources (such as animals, people) in the monitoring area, there will be occlusion of the fire abnormal area during the movement. Therefore, the infrared image with the smallest area is selected as the representation image of the fire monitoring sub-region in the current monitoring time period for analysis, and the analysis result can most comprehensively represent whether there are fire hazards in the current area.

[0072] In the present invention, by selecting the image in the video information corresponding to the moment with the smallest area of the red edge and its internal region in the infrared information during the monitoring time period as the fire analysis representation image, the problem of occlusion of the fire abnormal area by the movable heat source in the monitoring area can be effectively addressed. The key area in the infrared information is determined by vision technology or image processing technology to ensure that the selected representation image can best reflect whether there are fire hazards in the current area. This not only improves the accuracy of fire hazard detection but also enhances the adaptability of the system to complex environments through the fusion analysis of multi-modal data.

[0073] Specifically, the detection and analysis module determines the fire abnormal tendency parameter of the corresponding fire monitoring sub-region according to the analysis result of the fire analysis representation image and in combination with the temperature information in the environmental information;

[0074] The detection and analysis module determines the proportion of the area of the red edge and its internal region in the fire analysis representation image, which is recorded as the analysis result of the representation image, and determines the abnormal tendency coefficient;

[0075] The detection and analysis module calculates the average value of the temperature information and its average deviation, and obtains the abnormal tendency factor by dividing the temperature average deviation by the temperature average value;

[0076] The detection and analysis module determines the fire abnormal tendency parameter according to the product of the abnormal tendency coefficient and the abnormal tendency factor.

[0077] In implementation, if the fire monitoring sub-region does not contain a camera device, or the analysis result of the representation image is that the area proportion is less than or equal to the area threshold, the abnormal tendency coefficient is recorded as 1 at this time, and the fire abnormal tendency parameter is determined by the abnormal tendency factor. If the analysis result of the representation image is that the area proportion is greater than the area threshold, the abnormal coefficient is recorded as ∞;

[0078] The area threshold is determined according to the minimum value of the proportion of the area of the red edge and its internal area in the historical fire warning records.

[0079] Please refer to Figure 3 As shown, it is a logic diagram for determining the fire abnormal tendency category of the corresponding fire monitoring sub-region in the embodiment of the present invention. The detection and analysis module determines the fire abnormal tendency category of the corresponding fire monitoring sub-region according to the fire abnormal tendency parameter, including:

[0080] If the fire abnormal tendency parameter is greater than the standard abnormal tendency parameter threshold, the detection and analysis module determines that the corresponding fire monitoring sub-region has an obvious fire abnormal tendency;

[0081] If the fire abnormal tendency parameter is less than or equal to the standard abnormal tendency parameter threshold, the detection and analysis module determines that the corresponding fire monitoring sub-region has a hidden fire abnormal tendency.

[0082] In implementation, the standard abnormal tendency parameter threshold is determined by subtracting twice the standard deviation from the average value of the abnormal tendency parameter thresholds in historical fire warnings.

[0083] Please refer to Figure 4 As shown, it is a logic diagram for selecting the corresponding fire analysis method in the embodiment of the present invention. The fire determination module selects the corresponding fire analysis method according to the fire abnormal tendency category of the fire monitoring sub-region, including:

[0084] In response to the fire abnormal tendency category of the fire monitoring sub-region being an obvious fire abnormal tendency, the fire analysis method is to analyze the traffic information in the current monitoring time period according to the position of the camera device;

[0085] In response to the fire abnormal tendency category of the fire monitoring sub-region being a hidden fire abnormal tendency, the fire analysis method is to obtain the peak value of the environmental information in the current monitoring time period.

[0086] In implementation, if the current fire monitoring sub-region contains a camera device, the video information of the camera device in the fire monitoring sub-region is selected to analyze the traffic information therein; if the current fire monitoring sub-region does not contain a camera device, the video information of the camera device closest to the position of the current fire monitoring sub-region is selected to analyze the traffic information therein.

[0087] Through multi-modal data fusion and dynamic threshold adjustment, the present invention significantly improves the calculation accuracy and response efficiency of fire abnormal tendency parameters. First, by combining image analysis and temperature data analysis, complementary verification of visual and thermal data is achieved, avoiding misjudgment by a single sensor. Second, a product model of abnormal coefficients and factors is introduced to dynamically correlate image abnormal features (such as the flame spread range) with the environmental temperature change trend (such as the local temperature rise rate), enhancing the sensitivity to early fires. When there is no camera or the image abnormality is small, the system automatically degrades to a temperature-dominated mode to ensure basic monitoring capabilities; while when the image abnormality is significant, the coefficient is set to ∞ and directly determined as abnormal to ensure timely response to unrecognizable abnormalities. By dynamically calibrating the area threshold with historical data, the system can adapt to the risk characteristics of different scenarios, taking into account both sensitivity and false alarm control. This solution is particularly applicable to complex industrial environments and can achieve accurate and hierarchical fire warnings under resource-constrained conditions, effectively improving the prevention and control efficiency and reliability.

[0088] Specifically, the fire warning module determines whether to issue a fire abnormal warning according to the fire analysis result, including:

[0089] The fire warning module analyzes the traffic information during the current monitoring period according to the position of the camera device, determines the rgb representation value of the moving target during the current period. If the absolute value of the difference in the representation values in the rgb representation value is greater than the traffic law threshold, it is determined to issue a fire abnormal warning;

[0090] The fire warning module obtains the peak value of the environmental information and the duration corresponding to the peak value during the current monitoring time. If there is no peak value in the temperature information, and / or the pitch of the audio information exceeds the environmental threshold, and / or the smoke information exceeds the allowable threshold, it is determined to issue a fire abnormal warning.

[0091] In implementation, the traffic law threshold is determined according to the minimum value of the absolute value of the difference in the representation values for which warnings are issued in the historical abnormal records.

[0092] In the surveillance video, the fire warning module analyzes the monitoring area of the camera device, extracts and sums the color features of the personnel's clothes, that is, the rgb representation value.

[0093] It can be understood that the consistency of the colors of personnel's clothing can be used as a relatively stable feature in some scenarios. For example, in a specific work area, the staff usually wears uniform work clothes. In this case, the rgb representation value of the clothes color is relatively stable and the difference is small. When there are outsiders in the monitoring area or the color of the personnel's clothing changes abnormally (such as the clothes color changing due to a person entering the fire source area, etc.), the rgb representation value of their clothes will have a large difference from the normal value.

[0094] The fire warning module calculates the difference between the RGB representation value of a moving target and a reference value (such as the RGB representation value of the color of normal work clothes) within the current time period. If the difference exceeds the preset traffic pattern threshold, this may indicate an abnormal situation, such as an outsider breaking in, a person staying in a dangerous area, etc. The system then determines to issue a fire anomaly warning accordingly.

[0095] For example, in a factory workshop, all employees wear blue work clothes (the RGB representation value is approximately R = 0, G = 0, B = 255). When the monitoring system detects a person wearing red clothes (the RGB representation value is approximately R = 255, G = 0, B = 0) in a certain area, the difference in the RGB representation value significantly exceeds the preset traffic pattern threshold. At this time, the system will determine it as a fire anomaly and issue a warning. Because this person in red clothes may be an unauthorized outsider, or the clothes of the people in that area have changed color due to being burned by the fire during a fire in that area, etc. These may all pose potential fire safety hazards.

[0096] Similarly, for example, in the normal working environment of a hospital, medical staff usually wear unified work clothes, such as white or blue uniforms, and the colors of these clothes are highly consistent. In the corridors of the hospital or at the entrance of a specific department, the personnel flow may be relatively scattered usually, and the difference in the RGB representation values of the clothes colors is small because there are different groups of people such as patients and family members in addition to medical staff. But when an emergency occurs, such as a fire or a sudden medical event, a large number of medical staff may gather in this area. At this time, the color consistency of the clothes of the people in this area will increase, and the difference in the RGB representation values of the moving targets will increase, indicating an abnormal situation, such as medical staff evacuating patients urgently or dealing with emergencies such as fires. At this time, the system will determine to issue a fire anomaly warning.

[0097] It can be understood that in normal fire environment monitoring, the temperature usually has a certain fluctuation range but will not show extreme peaks. For example, in an indoor environment, the temperature will change due to factors such as personnel activities and equipment operation, but these changes are usually relatively gentle and regular. When there is no peak in the temperature information, this may indicate that there are some potential anomalies in the fire environment. For example, there may be a device failure resulting in abnormal heat dissipation, or there is an undetected fire source slowly heating up but not yet reaching the stage where it can cause a sharp rise in temperature.

[0098] For the pitch of the audio information exceeding the environmental threshold, in the fire monitoring area, there is usually a relatively stable environmental pitch. For example, in an indoor environment, there may be some background noises, such as the sound of air conditioners and the slight movement of people. The pitches of these sounds are usually within a certain range. When the pitch of the audio information exceeds the environmental threshold, it indicates that there is an abnormal sharp sound. This may be the sharp sound generated by the burning of items, equipment failures, or people's calls for help during a fire. At this time, it is necessary to issue a warning in a timely manner so as to take corresponding measures.

[0099] For the smoke information exceeding the allowable threshold, when there is no abnormal situation such as a fire, the smoke concentration in the environment is usually at a relatively low level and will not exceed the allowable threshold; when the smoke information exceeds the allowable threshold, this is very likely to be a sign of a fire. Smoke is a common product in a fire, and the increase in its concentration usually means that the fire is spreading or a large amount of combustion products have been released into the air. At this time, it is necessary to immediately issue a fire prevention anomaly warning so as to carry out emergency operations such as extinguishing fires and evacuating personnel in a timely manner.

[0100] When any one or more of the above three situations occur simultaneously, the system determines to issue a fire prevention anomaly warning; by comprehensively considering multiple information sources (temperature, audio, smoke), it is possible to more accurately judge whether the fire prevention environment is normal. The anomaly of a single condition may have a certain degree of contingency, but the simultaneous anomaly of multiple conditions greatly increases the possibility of a fire prevention anomaly. The present invention discovers and takes measures in a timely manner at the early stage of fire prevention anomalies such as fires to reduce the loss of life and property caused by fires. In this way, corresponding actions can be taken at the initial stage of a fire, or even at the potential stage before a fire occurs, such as inspecting equipment, evacuating personnel, and starting a fire extinguishing system.

[0101] In the present invention, the fire prevention determination module flexibly selects the corresponding fire prevention analysis method according to the fire prevention anomaly tendency category, taking into account both the analysis of traffic information under the obvious fire prevention anomaly tendency and the monitoring of the environmental information peak under the hidden fire prevention anomaly tendency. At the same time, the fire prevention warning module comprehensively analyzes multi-dimensional data such as the rgb characterization value difference, environmental information peak, and duration in the traffic information to accurately judge the fire prevention anomaly situation. The analysis of the rgb characterization value difference uses the feature of the consistency of the colors of people's clothing to effectively identify potential fire prevention hazards such as the intrusion of outsiders or the abnormal aggregation of people; the environmental information monitoring timely captures the early signs of a fire by real-time monitoring of key indicators such as temperature, audio, and smoke. In addition, the system also has an adaptive adjustment ability to dynamically calibrate the threshold according to historical data to ensure accurate and hierarchical fire warnings can be achieved in different scenarios, effectively improving the intelligent level and reliability of fire prevention and control, and providing strong support for the early discovery and emergency disposal of fires.

[0102] Specifically, the deep learning module adjusts the fire monitoring sub-region according to the fire abnormal tendency parameters for several consecutive times.

[0103] If the number of consecutive times that the fire abnormal tendency parameters of adjacent fire monitoring sub-regions are within the stable interval exceeds the stable time, the adjacent fire monitoring sub-regions are integrated into a new fire monitoring sub-region.

[0104] In implementation, the stable time is selected within 4 - 12 h, preferably 6 h. The stable interval is determined according to all the fire abnormal tendency parameters of one of the fire monitoring sub-regions and using the normal distribution.

[0105] In the present invention, the deep learning module continuously analyzes the fire abnormal tendency parameters of each fire monitoring sub-region to reflect the regional fire risk trend. When the abnormal tendency parameters of adjacent sub-regions are continuously stable within the preset stable interval (such as the low-risk threshold), it indicates that their environmental risk characteristics are convergent and controllable. At this time, the system integrates the adjacent sub-regions into a new fire monitoring sub-region to reduce the occupancy of redundant analysis resources. This dynamic adjustment not only improves the system efficiency but also ensures that the merged region can still maintain the safety monitoring ability through long-term stability verification, reflecting the adaptive management ability of the intelligent fire protection system to complex environments.

[0106] Specifically, the deep learning module determines the fire traversal inspection method according to the fire warning records, including:

[0107] The deep learning module determines the inspection selection time period and the necessary inspection areas according to the occurrence time of the fire warning records.

[0108] The deep learning module selects the time period with the largest coverage range from all the inspection selection time periods according to the length of the traversal inspection time as the traversal inspection time.

[0109] In implementation, the traversal inspection includes all the necessary inspection areas, and the necessary inspection areas are the areas where fire warnings have occurred in the historical records. It can be understood that the traversal inspection does not perform inspections on all the monitoring areas.

[0110] In the present invention, the inspection time period and the necessary inspection areas are set according to the occurrence time of the warning records. These necessary inspection areas are the places where fire warnings have occurred in history. The module will select a time period with the largest coverage range from all possible inspection time periods as the actual inspection time, which can ensure that the inspection is concentrated in the areas and time periods where fire problems are most likely to occur, rather than performing undifferentiated inspections on all the monitoring areas, thereby improving the efficiency and pertinence of the inspection.

[0111] So far, the technical solution of the present invention has been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easily understood by those skilled in the art that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0112] The above are only the preferred embodiments of the present invention and are not used to limit the present invention; for those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-modal sensing intelligent fire monitoring system based on deep learning, characterized in that, Including: An environmental detection module, which is used to continuously obtain video information and infrared information of a monitoring area, obtain the environmental information of the monitoring area, and divide the monitoring area into several fire monitoring sub-areas; A detection and analysis module, which is connected to the environmental detection module, used to determine the fire analysis characterization images of all fire monitoring sub-areas within each monitoring time period, analyze each fire analysis characterization image, combine the environmental information to determine the fire abnormal tendency parameters of the corresponding fire monitoring sub-areas, and determine the fire abnormal tendency categories according to the fire abnormal tendency parameters; A fire determination module, which is respectively connected to the environmental detection module and the detection and analysis module, used to select the corresponding fire analysis method according to the fire abnormal tendency categories of each fire monitoring sub-area, including obtaining the peak value of environmental information in the current monitoring time period, or analyzing the flow information in the current monitoring time period, and determining whether to issue a fire abnormal warning according to the fire analysis result; A fire warning module, which is respectively connected to the environmental detection module and the environmental detection module, used to issue a fire warning to the corresponding fire monitoring sub-area according to the determination result of the fire abnormal warning; A deep learning module, which is respectively connected to the environmental detection module, the detection and analysis module and the fire determination module, used to adjust the fire monitoring sub-areas according to the fire abnormal tendency parameters for several consecutive times, and determine the fire traversal inspection method according to the fire warning records.

2. The multi-modal sensing intelligent fire monitoring system based on deep learning according to claim 1, characterized in that, The environmental detection module includes: A video acquisition unit, including several camera devices, used to acquire video information and infrared information of the shootable area within the monitoring area; A sensing detection unit, including several multi-mode sensor groups, used to acquire the environmental information of each fire monitoring sub-area; An information transmission unit, which is respectively connected to the video acquisition unit and the sensing detection unit, used to collect the video information, infrared information and all environmental information of the monitoring area, and divide the monitoring area into several fire monitoring sub-areas according to the distribution positions of the multi-mode sensor groups; Among them, the environmental information includes temperature information, audio information and smoke information.

3. The multi-modal sensing intelligent fire monitoring system based on deep learning according to claim 2, wherein, The detection and analysis module selects the fire analysis characterization image according to the video information and infrared information within each fire monitoring sub-area during the monitoring time period; The detection and analysis module selects the image in the video information corresponding to the moment with the least area of the red edge and its internal area in the infrared information during the monitoring time period as the fire analysis characterization image of the monitoring time period.

4. The multi-modal sensing intelligent fire monitoring system based on deep learning according to claim 3, wherein The detection and analysis module determines the fire abnormal tendency parameters of the corresponding fire monitoring sub-area according to the analysis result of the fire analysis characterization image and combines the temperature information in the environmental information; The detection and analysis module determines the proportion of the area of the red edge and its internal area in the fire analysis characterization image, which is recorded as the analysis result of the characterization image, and determines the abnormal tendency coefficient; The detection and analysis module calculates the average value of the temperature information and its average deviation, and obtains the abnormal tendency factor by comparing the temperature average deviation with the temperature average value; The detection and analysis module determines the fire abnormal tendency parameters according to the product of the abnormal tendency coefficient and the abnormal tendency factor.

5. The multi-modal sensing intelligent fire monitoring system based on deep learning according to claim 4, wherein, The detection and analysis module determines the fire abnormal tendency category corresponding to the fire monitoring sub-region according to the fire abnormal tendency parameter, including: If the fire abnormal tendency parameter is greater than the standard abnormal tendency parameter threshold, the detection and analysis module determines that the corresponding fire monitoring sub-region has an obvious fire abnormal tendency; If the fire abnormal tendency parameter is less than or equal to the standard abnormal tendency parameter threshold, the detection and analysis module determines that the corresponding fire monitoring sub-region has a hidden fire abnormal tendency.

6. The multi-modal sensing intelligent fire monitoring system based on deep learning according to claim 5, wherein The fire determination module selects the corresponding fire analysis method according to the fire abnormal tendency category of the fire monitoring sub-region, including: In response to the fire abnormal tendency category of the fire monitoring sub-region being an obvious fire abnormal tendency, the fire analysis method is to analyze the traffic information in the current monitoring time period according to the position of the camera device; In response to the fire abnormal tendency category of the fire monitoring sub-region being a hidden fire abnormal tendency, the fire analysis method is to obtain the peak value of the environmental information in the current monitoring time period.

7. The multimodal sensing intelligent fire monitoring system based on deep learning according to claim 6, characterized in that, The fire warning module determines whether to issue a fire abnormal warning according to the fire analysis result, including: The fire warning module analyzes the traffic information in the current monitoring time period according to the position of the camera device, determines the rgb representation value of the moving target in the current time period. If the absolute value of the representation value difference in the rgb representation value is greater than the traffic law threshold, it is determined to issue a fire abnormal warning; The fire warning module obtains the peak value of the environmental information and the duration corresponding to the peak value in the current monitoring time. If there is no peak value in the temperature information, and / or the pitch of the audio information exceeds the environmental threshold, and / or the smoke information exceeds the allowable threshold, it is determined to issue a fire abnormal warning.

8. The multi-modal sensing intelligent fire monitoring system based on deep learning according to claim 7, characterized in that The deep learning module adjusts the fire monitoring sub-region according to the fire abnormal tendency parameters for several consecutive times, If the number of consecutive times that the fire abnormal tendency parameters of adjacent fire monitoring sub-regions are within the stable range exceeds the stable time, the adjacent fire monitoring sub-regions are integrated into a new fire monitoring sub-region.

9. The multi-modal sensing intelligent fire monitoring system based on deep learning according to claim 8, characterized in that The deep learning module determines the fire traversal inspection method according to the fire warning record, including: The deep learning module determines the inspection selection time period and the necessary inspection area according to the occurrence time of the fire warning record; The deep learning module selects the time period with the largest coverage area among all the inspection selection time periods as the traversal inspection time according to the length of the traversal inspection time.

Images