A method and system for intelligent monitoring and early warning of fire hazards based on multimodal data

Abstract

This invention discloses a method and system for intelligent monitoring and early warning of fire hazards based on multimodal data, relating to the field of fire alarms. The method includes: acquiring electromagnetic radiation signals in a target area; determining abnormal radiation events and classifying radiation sources by calculating the Mahalanobis distance with an electromagnetic fingerprint baseline to obtain abnormal detection data; grouping detection data using at least three electromagnetic receivers, performing cross-verification of abnormal detection data, calculating the detection distance based on pulse arrival time, and solving for radiation source coordinates using multiple hyperboloid intersections; acquiring heat map data and image frame data, performing spatial registration based on the radiation source coordinates, performing multi-source joint judgment, and managing fire hazard alarms. This application addresses the problem in existing electrical fire hazard early warning systems that struggle to achieve low false alarm rates, timely and reliable judgment during the early stages of fault development, thus improving the timeliness, reliability, and low false alarm rate of early warnings during the early stages of fault evolution.

Description

Technical Field

[0001] This application relates to the field of fire alarms, and in particular to a method and system for intelligent monitoring and early warning of fire hazards based on multimodal data. Background Technology

[0002] Early detection and warning of electrical fire hazards are crucial for protecting life and property. Currently, addressing this issue primarily relies on stand-alone smoke detectors, point-type heat detectors, and online monitoring devices based on single physical quantities. These devices are typically installed around electrical equipment or within power distribution lines, detecting smoke particle concentration or changes in ambient temperature to determine the presence of a fire. However, among these methods, smoke detectors only respond after significant smoke has been produced, resulting in a noticeable response lag; point-type heat detectors are only sensitive to localized temperatures, making it difficult to cover large-scale or concealed electrical faults; and monitoring of single electrical parameters, such as current and voltage, is susceptible to load fluctuations, leading to a high false alarm rate and an inability to effectively identify the warning stages before a fault occurs.

[0003] At present, the relevant technologies for early warning of electrical fire hazards have the technical problem of being unable to achieve low false alarms, timely and reliable judgment in the early stages of fault development. Summary of the Invention

[0004] This application provides a method and system for intelligent monitoring and early warning of fire hazards based on multimodal data. It collects electromagnetic radiation signals in the target area, uses an anomaly detector with an embedded normal electromagnetic fingerprint baseline to calculate Mahalanobis distance to determine the presence of abnormal radiation events and classify radiation sources. Using at least three electromagnetic receivers as a group, it calculates distance based on the time difference of signal pulse arrival and solves for radiation source coordinates using hyperboloid intersections. It acquires infrared thermal images and visible light images, performs spatial registration and multi-source joint judgment based on the radiation source coordinates, and triggers graded alarms based on the results. These technical means solve the technical problem of existing electrical fire hazard early warning systems, which struggle to achieve low false alarm rates, timely and reliable judgment during the early stages of fault development. This achieves the technical effect of improving the timeliness, reliability, and low false alarm rate of early warnings during the early stages of fire hazard development.

[0005] This application provides a method for intelligent monitoring and early warning of fire hazards based on multimodal data, comprising: collecting electromagnetic radiation signals in a target area; determining abnormal radiation events and classifying radiation sources by calculating the Mahalanobis distance with an electromagnetic fingerprint baseline based on an anomaly detector, thereby obtaining anomaly detection data; wherein the anomaly detector is embedded with an electromagnetic fingerprint baseline under normal conditions; grouping detection data into at least three electromagnetic receivers, performing mutual verification of anomaly detection data, calculating the detection distance based on the pulse arrival time of the electromagnetic radiation signal, and solving for the coordinates of the radiation source through multiple hyperboloid intersections; acquiring thermal image data from an infrared thermal imager and image frame data from a visible light camera, performing spatial registration based on the coordinates of the radiation source, performing multi-source joint judgment, and responding to a graded alarm module for fire hazard alarm management.

[0006] In a possible implementation, anomaly detection data is obtained, and the following processing is performed: a broadband electromagnetic receiver is deployed in the target area to collect electromagnetic radiation signals in space through an antenna, wherein the electromagnetic radiation signals are the superposition of electromagnetic radiation generated by electromagnetic interference sources and electrical fault arcs; an electromagnetic fingerprint baseline under normal conditions is constructed; anomaly detectors are deployed based on the electromagnetic fingerprint baseline to receive the electromagnetic radiation signals for anomaly detection and radiation source classification, and anomaly detection data is determined.

[0007] In a possible implementation, the following processing is performed: by collecting continuous electromagnetic background noise, the power spectral density distribution at each frequency point is statistically analyzed to form the electromagnetic fingerprint baseline, wherein the power spectral density distribution is determined by the mean, variance, and higher-order moments; wherein the arc discharge has a first electromagnetic radiation characteristic, and the normal state has a second electromagnetic radiation characteristic.

[0008] In a possible implementation, the electromagnetic radiation signal is received for anomaly detection, and the following processing is performed: Fourier transform is performed on the electromagnetic radiation signal to obtain the frame signal power spectral density of each frame signal; the frame signal power spectral density is compared with the electromagnetic fingerprint baseline to calculate the Mahalanobis distance, wherein the Mahalanobis distance measures the degree of deviation from the normal within the statistical distribution framework of environmental noise; it is determined whether the Mahalanobis distance is greater than a preset threshold, and if it is greater, it is determined that there is an abnormal radiation event based on arc discharge.

[0009] In a possible implementation, the following processing is performed: if an abnormal radiation event exists, radiation source classification is triggered; wherein the radiation source classification step includes: extracting the time-domain features and frequency-domain features of the electromagnetic radiation signal, wherein the time-domain features include pulse width, repetition frequency, and rise time, and the frequency-domain features include center frequency, bandwidth, and power spectral flatness; concatenating the time-domain features and frequency-domain features to obtain a feature vector; performing random forest classification on the feature vector to obtain interference labels, and adding the interference labels to the anomaly detection data, wherein the interference labels are marked with confidence levels.

[0010] In a possible implementation, the coordinates of the radiation source are solved by performing the following processes: obtaining the arrival times of signal pulses from a distributed array of multiple electromagnetic receivers and calculating the detection distance; and using a hyperboloid positioning method to solve for the coordinates of the radiation source based on the detection distances of at least three electromagnetic receivers.

[0011] In a possible implementation, the following processing is performed: a first hyperboloid is drawn in three-dimensional space based on the distance difference between the first detection distance and the second detection distance; a second hyperboloid is drawn in three-dimensional space based on the distance difference between the first detection distance and the third detection distance; a third hyperboloid is drawn in three-dimensional space based on the distance difference between the second detection distance and the third detection distance; the intersection point of the first hyperboloid, the second hyperboloid, and the third hyperboloid is taken as the radiation source position, and the radiation source coordinates are generated.

[0012] In a possible implementation, multi-source joint judgment is performed, and the following processes are executed: acquiring thermal image data from an infrared thermal imager and image frame data from a visible light camera; performing spatial registration based on radiation source coordinates on the thermal image data and image frame data to obtain multi-source detection data; and generating a graded alarm strategy by performing joint judgment on the multi-source detection data.

[0013] In a possible implementation, the joint judgment of the multi-source detection data is performed to generate a graded alarm strategy, and the following processing is performed: spatial registration is performed between the coordinates of the radiation source and the coordinates of the high-temperature point in the heat map data, and the Euclidean distance is calculated to obtain a first judgment result, wherein if the Euclidean distance is less than the spatial tolerance, it is judged to be the same heat source; spatial registration is performed between the coordinates of the radiation source and the image frame data to obtain the smoke visual features of the coordinate region, which is used as a second judgment result; the graded alarm strategy is generated based on the first judgment result and the second judgment result.

[0014] This application also provides a fire hazard intelligent monitoring and early warning system based on multimodal data, comprising: an anomaly detection module, used to collect electromagnetic radiation signals in a target area, and based on the anomaly detector, perform anomalous radiation event judgment and radiation source classification by solving the Mahalanobis distance with the electromagnetic fingerprint baseline to obtain anomaly detection data, wherein the anomaly detector is embedded with the electromagnetic fingerprint baseline under normal conditions; a radiation source coordinate solving module, used to group detections with at least three electromagnetic receivers, perform anomaly detection data mutual verification, calculate the detection distance based on the pulse arrival time of the electromagnetic radiation signal, and solve the radiation source coordinates through multiple hyperboloid intersections; and a fire hazard alarm management module, used to acquire thermal image data from an infrared thermal imager and image frame data from a visible light camera, perform spatial registration based on the radiation source coordinates, perform multi-source joint judgment, and respond to the hierarchical alarm module for fire hazard alarm management.

[0015] This application proposes a method and system for intelligent monitoring and early warning of fire hazards based on multimodal data. First, electromagnetic radiation signals are collected in the target area. Based on an anomaly detector, the Mahalanobis distance is calculated using an electromagnetic fingerprint baseline to determine anomaly radiation events and classify radiation sources, resulting in anomaly detection data. The anomaly detector embeds a normal electromagnetic fingerprint baseline. Then, at least three electromagnetic receivers are used to group the detection data, and cross-verification of the anomaly detection data is performed. The detection distance is calculated based on the pulse arrival time of the electromagnetic radiation signal. The coordinates of the radiation source are determined using multiple hyperboloid intersections. Finally, thermal image data from an infrared thermal imager and image frame data from a visible light camera are acquired. Spatial registration is performed using the radiation source coordinates as a reference, and multi-source joint judgment is performed. The system then responds to a graded alarm module for fire hazard alarm management. Through this process, the method and system proposed in this application achieve the technical effect of improving the timeliness, reliability, and low false alarm rate of early warnings during the early stages of fire hazard development. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments of the present invention will be briefly described below. Flowcharts are used in this application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from these processes.

[0017] Figure 1 This is a flowchart illustrating an intelligent monitoring and early warning method for fire hazards based on multimodal data, provided in an embodiment of this application.

[0018] Figure 2This is a schematic diagram of the structure of an intelligent fire hazard monitoring and early warning system based on multimodal data, provided in an embodiment of this application.

[0019] Figure labeling: Anomaly detection module 10, Radiation source coordinate solving module 20, Fire hazard alarm management module 30. Detailed Implementation

[0020] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0021] This application provides an intelligent monitoring and early warning method for fire hazards based on multimodal data, such as... Figure 1 As shown, the method includes: Step S100: Electromagnetic radiation signal acquisition is performed in the target area. Based on the anomaly detector, the Mahalanobis distance is calculated with the electromagnetic fingerprint baseline to determine the abnormal radiation event and classify the radiation source, thereby obtaining anomaly detection data. The anomaly detector is embedded with the electromagnetic fingerprint baseline under normal conditions.

[0022] Specifically, an electromagnetic signal acquisition front-end, such as a broadband electromagnetic receiver and its matching antenna, is deployed within the target area. This system continuously acquires electromagnetic radiation signals in the space, which are the superposition of electromagnetic radiation generated by various electromagnetic interference sources in the environment and potential electrical fault arc discharges. The acquired raw analog signals are converted into digital signal streams through analog-to-digital conversion. This digital signal stream is input to a pre-configured anomaly detector, which is a software module or hardware logic unit with an embedded electromagnetic fingerprint baseline. The electromagnetic fingerprint baseline refers to the baseline characteristics formed by statistically analyzing the power spectral density distribution at each frequency point under normal conditions where there are no abnormal radiation events such as electrical fault arcs in the target area. This baseline is determined by the mean, variance, and higher-order moments of the power spectral density distribution at each frequency point, and is used to characterize the normal electromagnetic environment state of the target area. The anomaly detector performs frame processing on the input signal stream. For each frame, it calculates the Mahalanobis distance between it and the electromagnetic fingerprint baseline. In this application, the Mahalanobis distance refers to a quantitative indicator used to measure the deviation between the power spectral density of the real-time acquired frame signal and the electromagnetic fingerprint baseline within the framework of the statistical distribution of environmental noise. Based on the calculated Mahalanobis distance, the anomaly detector determines whether the signal in the frame represents an anomalous radiation event. If it is determined to be anomalous, the internal radiation source classification process is further triggered to classify and identify the source of the anomalous radiation signal. Finally, the anomaly detector integrates the determination result of whether an anomalous event has occurred and the classified radiation source type label into anomaly detection data output.

[0023] In one possible implementation, after obtaining anomaly detection data, step S100 further includes step S110, deploying a broadband electromagnetic receiver in the target area to collect electromagnetic radiation signals in space via an antenna. The electromagnetic radiation signals are the superposition of electromagnetic radiation generated by electromagnetic interference sources and electrical fault arc discharges. Specifically, to achieve broadband electromagnetic signal capture, a broadband electromagnetic receiver is installed around key electrical equipment or in open spaces within the target area. As an example, the receiver's operating frequency range can be set from 10kHz to 100MHz to cover the frequency band of electromagnetic radiation generated by common electrical fault arc discharges. The receiver is connected to a broadband active antenna, such as a biconical antenna or a log-periodic antenna, to convert electromagnetic waves in space into voltage signals. The electromagnetic radiation signals continuously collected by the antenna are the instantaneous superposition result of various electromagnetic interference sources in the environment, such as switching power supplies and wireless communication devices, and the electromagnetic radiation generated by electrical fault arc discharges.

[0024] Step S120: Constructing the electromagnetic fingerprint baseline under normal conditions. This involves collecting continuous electromagnetic background noise and statistically analyzing the power spectral density distribution at each frequency point to form the electromagnetic fingerprint baseline. The power spectral density distribution is determined by the mean, variance, and higher-order moments. Arc discharge exhibits a first electromagnetic radiation characteristic, while normal conditions exhibit a second electromagnetic radiation characteristic. Specifically, the baseline construction process is initiated during a period when the target area is operating normally without any known electrical faults or abnormal discharge events. Electromagnetic background noise is continuously collected over a period of time using the aforementioned broadband electromagnetic receiver, for example, continuously for 24 hours to cover electromagnetic environment changes during the operating cycles of different electrical equipment. The collected time-domain signal is subjected to a Fast Fourier Transform to convert it to the frequency domain, obtaining the power spectral density at each frequency point. The power spectral density values ​​at all sampling times for each frequency point are statistically analyzed, and their mean, variance, skewness, and kurtosis (i.e., higher-order moments) are calculated. This complete set of distribution characteristics, composed of the mean, variance, and higher-order moments at each frequency point, constitutes the electromagnetic fingerprint baseline for the target area. This baseline characterizes the electromagnetic features of normal operation, such as the narrow-band, continuous, and periodic radiation generated by electrical equipment during normal operation; that is, normal operation exhibits the second electromagnetic radiation characteristic. In contrast, during arc discharge, the air breakdown between contacts generates a pulse current with a rise time on the order of seconds, radiating a wide-spectrum electromagnetic wave with sudden, wide-bandwidth, and pulse cluster characteristics; that is, arc discharge exhibits the first electromagnetic radiation characteristic. The electromagnetic fingerprint baseline is used as a discrimination criterion in subsequent detection, classifying signals with the first electromagnetic radiation characteristic as abnormal, and signals with the second electromagnetic radiation characteristic as normal.

[0025] Step S130: Deploy an anomaly detector based on the electromagnetic fingerprint baseline, receive the electromagnetic radiation signal for anomaly detection and radiation source classification, and determine anomaly detection data. Specifically, an anomaly detector software service is instantiated on a server or embedded edge computing device. Upon startup, this service loads the electromagnetic fingerprint baseline data constructed in step S120. The anomaly detector receives the electromagnetic radiation signal stream collected and digitized by the electromagnetic receiver in step S110 in real time. For each received signal segment, the anomaly detector first executes anomaly detection logic to determine whether there is anomalous radiation with sudden, wide-bandwidth, pulse cluster characteristics, such as arc discharge. Once an anomaly is detected, radiation source classification logic is executed to distinguish whether the anomalous radiation originates from an electrical fault arc or other non-faulty electromagnetic interference. Finally, the detection and classification results are encapsulated into structured anomaly detection data, for example, using JavaScript object spectrum format, including fields such as timestamp, anomaly flag, and classification label.

[0026] In one possible implementation, receiving the electromagnetic radiation signal for anomaly detection, step S130 further includes step S131, performing a Fourier transform on the electromagnetic radiation signal to obtain the frame signal power spectral density of each frame. Specifically, the anomaly detector first performs frame processing on the input continuous time-domain electromagnetic radiation signal, for example, setting the frame length to 1024 sampling points and the frame shift to 512 sampling points to ensure time resolution. A Hamming window is applied to each frame of the time-domain signal to suppress spectral leakage, and then a fast Fourier transform is performed to convert the time-domain signal into a frequency-domain representation. The square of the amplitude at each frequency point of the frequency-domain signal is calculated and divided by the product of the frame length and the receiver's equivalent noise bandwidth to obtain the power spectral density estimate of that frame signal, i.e., the frame signal power spectral density.

[0027] Step S132: Compare the frame signal power spectral density with the electromagnetic fingerprint baseline to calculate the Mahalanobis distance, where the Mahalanobis distance measures the degree of deviation from the norm within the statistical distribution framework of environmental noise. Specifically, the frame signal power spectral density of the current frame calculated in step S131 is considered as a multi-dimensional vector, where the dimensions correspond to each frequency point. Simultaneously, the mean vector in the electromagnetic fingerprint baseline constructed in step S120, i.e., the mean of the power spectral density at each frequency point, is used as the reference vector, and the covariance matrix in the electromagnetic fingerprint baseline, i.e., the matrix composed of the variance of the power spectral density at each frequency point and the covariance between different frequency points, is used as the metric. The Mahalanobis distance calculation steps are: 1. Subtract the overall mean vector from the current frame signal power spectral density vector; 2. Transpose the result of the subtraction; 3. Multiply the transposed result by the inverse of the covariance matrix; 4. Multiply the product by the difference vector calculated in step 1; 5. Take the square root of the final multiplication result to obtain the Mahalanobis distance value. This distance value quantifies the degree of deviation of the electromagnetic radiation distribution of the current frame from the normal baseline in a statistical sense.

[0028] Step S133: Determine whether the Mahalanobis distance is greater than a preset threshold. If it is, determine that an abnormal radiation event based on arc discharge exists. Specifically, a Mahalanobis distance threshold is preset based on the acceptable false alarm rate of the target area. For example, by analyzing multiple sets of normal data offline, the 99.9th percentile of all Mahalanobis distance values ​​is taken as the threshold. The anomaly detector compares the Mahalanobis distance calculated in step S132 with the preset threshold. If the Mahalanobis distance of the current frame is greater than the preset threshold, it is determined that an abnormal radiation event occurred in the target area at the time corresponding to the signal of that frame. Given that arc discharge has electromagnetic radiation characteristics such as suddenness, wide bandwidth, and pulse clusters, this abnormal event is initially identified as possibly caused by an electrical fault arc.

[0029] In one possible implementation, step S130 further includes step S134, triggering radiation source classification if an abnormal radiation event exists; wherein the radiation source classification step includes: extracting time-domain features and frequency-domain features of the electromagnetic radiation signal, wherein the time-domain features include pulse width, repetition frequency, and rise time, and the frequency-domain features include center frequency, bandwidth, and power spectral flatness; concatenating the time-domain features and frequency-domain features to obtain a feature vector; performing random forest classification on the feature vector to obtain interference labels, and adding the interference labels to the anomaly detection data, wherein the interference labels are marked with confidence levels.

[0030] Specifically, when step S133 determines that an abnormal radiation event exists, the anomaly detector automatically triggers the classification process. First, time-domain and frequency-domain features are extracted from the electromagnetic radiation signal frame identified as abnormal. Time-domain features include, but are not limited to, pulse width, repetition frequency, and rise time. Pulse width refers to the duration for which the signal amplitude exceeds a preset threshold; repetition frequency refers to the number of pulse occurrences per unit time; and rise time refers to the time required for the pulse leading edge to rise from 10% amplitude to 90% amplitude. Frequency-domain features include, but are not limited to, center frequency, bandwidth, and power spectral flatness. Center frequency refers to the frequency corresponding to the peak of the power spectral density; bandwidth refers to the frequency range corresponding to a 3dB drop in power spectral density from the peak; and power spectral flatness refers to the ratio of the geometric mean to the arithmetic mean of the power spectral density. Then, all extracted time-domain and frequency-domain features are concatenated into a one-dimensional feature vector, which is then input into a pre-trained random forest classifier.

[0031] The random forest classifier is trained offline, and the training process is as follows: First, a training dataset is constructed by collecting electromagnetic radiation signal samples from multiple known radiation sources, including at least three categories: electrical fault arc samples, power electronic equipment switching interference samples, and electrostatic discharge samples. Each category has at least 1000 samples. Each sample generates a corresponding feature vector using the aforementioned time-domain and frequency-domain feature extraction methods, and its category label is manually assigned. Then, the hyperparameters of the random forest classifier are set, including the number of decision trees, the maximum depth of each tree, and the number of features randomly selected when splitting each tree. Next, a bootstrap aggregation sampling method is used to randomly select N samples with replacement from the original training dataset, where N is equal to the size of the original dataset, to construct a training subset for each decision tree. For each node of each decision tree, a specified number of feature subsets are randomly selected from all features. Within this subset, the optimal splitting feature and splitting threshold are selected, using the Gini impurity minimization criterion as the splitting basis. The above process is repeated to grow each decision tree until the preset maximum depth is reached or the number of node samples is less than the minimum number of splitting samples. After training, all decision trees are combined into a random forest classifier. Classification uses majority voting: each decision tree outputs a predicted class, and the class with the most votes is selected as the final classification result. The ratio of the number of votes to the total number of trees is used as the confidence score for that classification result. The classifier outputs the class with the highest probability as a distractor label, along with its confidence score. Possible distractor labels include electrical fault arcs, power electronic equipment switching interference, or electrostatic discharge. Finally, the anomaly detector adds the distractor label and its confidence score to the anomaly detection data.

[0032] Step S200: Detection groups are formed using at least three electromagnetic receivers, anomaly detection data mutual verification is performed, the detection distance is calculated based on the pulse arrival time of the electromagnetic radiation signal, and the coordinates of the radiation source are solved by multiple hyperboloid intersections.

[0033] Specifically, to locate the anomalous radiation source, at least three receivers with known spatial locations and not collinear are selected from the broadband electromagnetic receiver group used in step S100 to form a positioning group. When the anomaly detection data output in step S100 indicates that an anomalous radiation event has occurred, each receiver in this positioning group records the arrival time of the same anomalous pulse signal. By calculating the difference between the arrival times recorded by different receivers pairwise and multiplying it by the speed of light (the speed of electromagnetic waves in air), the distance difference from the radiation source to each pair of receivers is calculated. Using the principle of three-dimensional spatial positioning, each distance difference corresponds to a hyperboloid. The unique intersection point of the hyperboloids corresponding to at least three independent distance differences is solved, and the coordinates of this intersection point are the position of the radiation source in space. Simultaneously, the detection data from multiple receivers can be cross-checked, for example, by comparing the time of occurrence of the anomalous event detected by each receiver with the signal characteristics, to confirm that the detected event is from the same radiation source.

[0034] In one possible implementation, after solving for the coordinates of the radiation source, step S200 further includes step S210, obtaining the arrival time of the signal pulse based on multiple distributed electromagnetic receivers, and calculating the detection distance. Specifically, in the distributed electromagnetic receiver system, each receiver is equipped with a high-precision global navigation satellite system clock synchronization module to ensure that the time reference error of all receivers is less than, for example, 10 ns. When an abnormal radiation event occurs, each receiver records the arrival time of the pulse by detecting the trigger point of the rising edge of the received signal envelope. Specifically, the envelope is extracted by performing a Hilbert transform on the received signal, and the moment when the envelope amplitude first exceeds 10 times the root mean square value of the background noise is recorded as the pulse arrival time. Let the arrival time recorded by receiver P1 be T1, and the arrival time recorded by receiver P2 be T2, then the difference in detection distance from the radiation source to receivers P1 and P2 is K12 = c × (T1 - T2), where c is the speed of light. The absolute detection distance from the radiation source to a single receiver cannot be directly determined by a single arrival time, but the positioning calculation is based on the distance difference.

[0035] Step S220: Based on the detection distances of at least three electromagnetic receivers, the coordinates of the radiation source are determined using a hyperboloid positioning method. Specifically, let the coordinates of the three receivers involved in the positioning be known points P1, P2, and P3. According to step S210, the distance difference K12 from the radiation source to P1 and P2, and the distance difference K13 to P1 and P3, are calculated. According to the definition of a hyperboloid, the locus of all points whose distance difference to two fixed points is constant is a hyperboloid of revolution. Therefore, the radiation source must be located on the first hyperboloid with foci P1 and P2 and a distance difference of K12, and also on the second hyperboloid with foci P1 and P3 and a distance difference of K13. The intersection of the two hyperboloids is generally a curve. Introducing the third hyperboloid corresponding to the third distance difference K23 (obtained from P2 and P3) and with foci P2 and P3, the unique common intersection point of the three hyperboloids in space is the location of the radiation source. By solving the system of equations consisting of three hyperboloid equations, the three-dimensional coordinates (x, y, z) of the intersection point can be obtained, which are the coordinates of the radiation source.

[0036] In one possible implementation, step S220 further includes step S221, which involves drawing a first hyperboloid in three-dimensional space based on the distance difference between the first detection distance and the second detection distance. Specifically, let the coordinate point of the first electromagnetic receiver be P1, and the coordinate point of the second electromagnetic receiver be P2. The difference in detection distances from the radiation source to P1 and P2, calculated in step S210, is a constant K12. In three-dimensional space, the set of all points that satisfy the condition that the distance to P1 minus the distance to P2 equals K12 constitutes a hyperboloid of revolution, i.e., the first hyperboloid. The principal axis of this hyperboloid is a straight line passing through P1 and P2, and its shape is determined by the ratio of K12 to the distances between P1 and P2.

[0037] Step S222: Based on the distance difference between the first detection distance and the third detection distance, draw the second hyperboloid in three-dimensional space. Specifically, let the coordinate point of the first electromagnetic receiver be P1 and the coordinate point of the third electromagnetic receiver be P3. The detection distance difference from the radiation source to P1 and P3, calculated in step S210, is a constant K13. In three-dimensional space, the set of all points that satisfy the condition that the distance to P1 minus the distance to P3 equals K13 constitutes another hyperboloid of revolution, namely the second hyperboloid.

[0038] Step S223: Based on the distance difference between the second and third detection distances, a third hyperboloid is drawn in three-dimensional space. Specifically, let the coordinate point of the second electromagnetic receiver be P2, and the coordinate point of the third electromagnetic receiver be P3. The detection distance difference from the radiation source to P2 and P3, calculated in step S210, is a constant K23. In three-dimensional space, the set of all points that satisfy the condition that the distance to P2 minus the distance to P3 equals K23 constitutes another hyperboloid of revolution, namely the third hyperboloid. These three hyperboloids intersect each other.

[0039] Step S224: The intersection point of the first, second, and third hyperboloids is taken as the location of the radiation source, generating the radiation source coordinates. Specifically, mathematically, three equations representing the first, second, and third hyperboloids are combined to form a nonlinear system of equations. A numerical solution method, such as the Newton-Raphson iterative method, is used to iteratively solve this system of equations starting from the initial estimated point. The numerical solution (x, y, z) obtained after iterative convergence is the unique common intersection point of the three hyperboloids. The coordinates of this intersection point are determined as the precise location of the anomalous electromagnetic radiation source in the three-dimensional space of the target area, and the radiation source coordinate data is generated accordingly.

[0040] Step S300: Acquire thermal image data from the infrared thermal imager and image frame data from the visible light camera, perform spatial registration based on the coordinates of the radiation source, perform multi-source joint judgment, and respond to the hierarchical alarm module for fire hazard alarm management.

[0041] Specifically, the system simultaneously invokes an infrared thermal imager and a visible light camera deployed in the target area and whose spatial coordinates have been calibrated. The infrared thermal imager outputs real-time thermal image data representing the temperature distribution, while the visible light camera outputs real-time image frame data containing visual details. Using the radiation source coordinates obtained in step S200 as a reference point, the three-dimensional spatial coordinates of this point are projected onto the imaging planes of the infrared thermal imager and the visible light camera, respectively, to achieve spatial registration. After registration, the temperature information of the point corresponding to the coordinates and its neighborhood is extracted from the thermal image data, and visual features such as smoke and flames in the area corresponding to the coordinates are extracted from the image frame data. Information from the three dimensions of electromagnetic radiation, thermal imaging, and visible light is combined for joint judgment. For example, if abnormal electromagnetic radiation, local high temperature, and smoke visual features are present at the radiation source coordinates simultaneously, it is determined to be a high-confidence fire hazard. Based on the confidence level of the joint judgment result and the preset severity level, a corresponding alarm command is generated and sent to the graded alarm module. The graded alarm module executes different alarm actions according to the command level. For example, a level 1 alarm only logs the alarm, a level 2 alarm activates the on-site audible and visual alarm, and a level 3 alarm automatically dials the fire alarm number and activates the fire extinguishing device.

[0042] In one possible implementation, multi-source joint judgment is performed, and step S300 further includes step S310, acquiring thermal image data from an infrared thermal imager and image frame data from a visible light camera. Specifically, the digital thermal image data stream output by the infrared thermal imager is read in real time via an Ethernet or Universal Serial Bus interface. This thermal image data is a two-dimensional matrix, where the value of each element represents the temperature value or radiation intensity of the corresponding spatial point. Simultaneously, the video stream output by the visible light camera is read in real time via a separate interface, and continuous image frame data is decoded from it, with each image frame being a three-channel (red, green, blue) color digital image.

[0043] Step S320 involves performing spatial registration based on radiation source coordinates on the thermal image data and image frame data to obtain multi-source detection data. Specifically, firstly, a projection mapping relationship is established between the world coordinate system containing the radiation source coordinates and the image coordinate systems of the infrared thermal imager and the visible light camera. This mapping relationship is obtained through joint calibration of the two sensors, with calibration parameters including intrinsic and extrinsic parameters. Then, the radiation source coordinates obtained in step S200 are substituted into the two projection equations to calculate the pixel coordinates of the point in the thermal image data and the pixel coordinates in the image frame data. Regions of interest are delineated in the thermal image data and the image frame data, centered on these two pixel coordinates. Image blocks of regions of interest containing temperature information and image blocks of regions of interest containing visual information, along with the original thermal image and image frame, are packaged together into multi-source detection data.

[0044] Step S330: A graded alarm strategy is generated by jointly judging the multi-source detection data. Specifically, the multi-source detection data generated in step S320 is analyzed. First, the thermal image data corresponding to the radiation source coordinates is parsed to obtain the absolute temperature value of that point and its average temperature rise compared to the surrounding area. Second, image frame data at the same location is parsed, and a pre-deployed smoke and flame recognition model based on a convolutional neural network, such as a lightweight mobile network single-step multi-frame detector model, is used to determine whether there are visual features of smoke or flame in the area. Combining evidence from three aspects—abnormal electromagnetic radiation, abnormal local temperature, and visual smoke and flame features—and based on preset decision logic, such as weighted voting or threshold judgment, the final fire hazard level is determined. According to the determined level, the corresponding alarm action command is retrieved from the preset graded alarm strategy table and output to the alarm management module for execution.

[0045] In one possible implementation, the joint judgment of the multi-source detection data is performed to generate a graded alarm strategy. Step S330 further includes step S331, which involves spatially registering the coordinates of the radiation source with the coordinates of the high-temperature points in the heat map data, calculating the Euclidean distance, and obtaining a first judgment result. If the Euclidean distance is less than the spatial tolerance, the source is considered to be the same heat source. Specifically, from the heat map data obtained in step S320, the coordinates of the high-temperature points are extracted by finding the maximum value of the matrix, i.e., the position of the pixel with the highest temperature value. These high-temperature point coordinates are coordinates in the image coordinate system. The radiation source coordinates obtained in step S200 are transformed to the same heat map image coordinate system through the projection mapping established in step S320 to obtain the coordinates of the radiation source projection points. The Euclidean distance between the high-temperature point coordinates and the radiation source projection point coordinates is calculated using the formula: Euclidean distance = √[(high-temperature point row coordinates - radiation source projection point row coordinates)]. 2 +(Coordinates of high-temperature points - Coordinates of radiation source projection points) 2The Euclidean distance is compared with a preset spatial tolerance, such as 10 pixels. If the Euclidean distance is less than the spatial tolerance, the radiation source location and the high-temperature point location are determined to be the same heat source, and the first judgment result is yes; otherwise, they are determined to be different heat sources, and the first judgment result is no.

[0046] Step S332 involves spatially registering the radiation source coordinates with the image frame data to obtain the smoke visual features of the coordinate region, which serves as the second judgment result. Specifically, the radiation source coordinates obtained in step S200 are transformed into the pixel coordinate system of the image frame data through the visible light camera projection mapping established in step S320, resulting in projection point coordinates. A fixed-size image block, such as 100×100 pixels, is cropped from the image frame data centered on these projection point coordinates. This image block is then input into a pre-trained convolutional neural network classifier, specifically designed to identify visual patterns of smoke and flame. The classifier outputs the probability values ​​for the image block to contain smoke features, flame features, and no smoke or flame features. The category corresponding to the highest probability value is taken as the smoke visual feature recognition result for the coordinate region, such as smoke, fire, or normal. This recognition result is the second judgment result.

[0047] Step S333: Generate the hierarchical alarm strategy based on the first and second judgment results. Specifically, the system has a built-in hierarchical decision table. For example: if the first judgment result is yes and the second judgment result is fire, a level 3 alarm strategy is triggered, with the instruction to immediately sound an alarm and initiate automatic fire suppression; if the first judgment result is yes and the second judgment result is smoke, a level 2 alarm strategy is triggered, with the instruction to issue an audible and visual warning and notify the safety officer to confirm on-site; if the first judgment result is no but the second judgment result is smoke or fire, a level 2 alarm strategy is triggered, with an additional prompt indicating a location deviation requiring manual verification; if the first judgment result is yes and the second judgment result is normal, a level 1 alarm strategy is triggered, with the instruction to record a hazard log and continuously monitor; if both the first and second judgment results are negative, no alarm is triggered, or the lowest level of system self-check normal information is triggered. Finally, the generated alarm levels and their corresponding execution instructions are output to the alarm module as a hierarchical alarm strategy.

[0048] This application's embodiments solve the technical problem of existing electrical fire hazard early warning systems, which struggle to achieve low false alarm rates, timely and reliable judgment during the early stages of fault development. This is achieved by collecting electromagnetic radiation signals in a target area, using an anomaly detector embedded with a normal electromagnetic fingerprint baseline to calculate Mahalanobis distance to determine the presence of abnormal radiation events and classify radiation sources. At least three electromagnetic receivers are grouped together, and distances are calculated based on the signal pulse arrival time difference. Radiation source coordinates are then used to solve for the coordinates of the radiation sources. Spatial registration and multi-source joint judgment are performed based on the radiation source coordinates, and graded alarms are triggered according to the results.

[0049] In the above text, refer to Figure 1 This paper describes in detail a method for intelligent monitoring and early warning of fire hazards based on multimodal data according to an embodiment of the present invention. Next, we will refer to... Figure 2 This invention describes an intelligent fire hazard monitoring and early warning system based on multimodal data according to an embodiment of the present invention.

[0050] An intelligent fire hazard monitoring and early warning system based on multimodal data, according to an embodiment of the present invention, addresses the technical problem of existing electrical fire hazard early warning systems failing to achieve low false alarm rates, timely and reliable judgment during the early stages of fault development. This system improves the timeliness, reliability, and low false alarm rate of early warnings during the early stages of fire hazard development. The intelligent fire hazard monitoring and early warning system based on multimodal data includes: an anomaly detection module 10, a radiation source coordinate solving module 20, and a fire hazard alarm management module 30.

[0051] Anomaly detection module 10 is used to collect electromagnetic radiation signals in the target area. Based on the anomaly detector, it calculates the Mahalanobis distance with the electromagnetic fingerprint baseline, judges abnormal radiation events, and classifies radiation sources to obtain anomaly detection data. The anomaly detector is embedded with the electromagnetic fingerprint baseline under normal conditions. Radiation source coordinate solving module 20 is used to group detections with at least three electromagnetic receivers, perform cross-verification of anomaly detection data, calculate the detection distance based on the pulse arrival time of the electromagnetic radiation signal, and solve the radiation source coordinates through multiple hyperboloid intersections. Fire hazard alarm management module 30 is used to acquire thermal image data from an infrared thermal imager and image frame data from a visible light camera, perform spatial registration based on the radiation source coordinates, perform multi-source joint judgment, and respond to the hierarchical alarm module for fire hazard alarm management.

[0052] The specific configuration of the anomaly detection module 10 is described in detail below: As mentioned above, to obtain anomaly detection data, the anomaly detection module 10 may further include: an electromagnetic radiation signal acquisition unit for deploying a broadband electromagnetic receiver in the target area to acquire electromagnetic radiation signals in space through an antenna, wherein the electromagnetic radiation signals are the superposition of electromagnetic radiation generated by electromagnetic interference sources and electrical fault arcs; an electromagnetic fingerprint baseline construction unit for constructing an electromagnetic fingerprint baseline under normal conditions; and an anomaly detection unit for deploying an anomaly detector based on the electromagnetic fingerprint baseline, receiving the electromagnetic radiation signals to perform anomaly detection and radiation source classification, and determining the anomaly detection data.

[0053] The electromagnetic fingerprint baseline construction unit may further include: collecting continuous electromagnetic background noise and statistically analyzing the power spectral density distribution at each frequency point to form the electromagnetic fingerprint baseline, wherein the power spectral density distribution is determined by the mean, variance, and higher-order moments; wherein the arc discharge has a first electromagnetic radiation characteristic and the normal state has a second electromagnetic radiation characteristic.

[0054] The anomaly detection unit, which receives the electromagnetic radiation signal and performs anomaly detection, may further include: a Fourier transform subunit for performing a Fourier transform on the electromagnetic radiation signal to obtain the frame signal power spectral density of each frame; a Mahalanobis distance calculation subunit for comparing the frame signal power spectral density with the electromagnetic fingerprint baseline to calculate the Mahalanobis distance, wherein the Mahalanobis distance measures the degree of deviation from the normal within the statistical distribution framework of environmental noise; and a determination subunit for determining whether the Mahalanobis distance is greater than a preset threshold. If it is greater, it is determined that there is an abnormal radiation event based on arc discharge.

[0055] The anomaly detection unit may further include: a radiation source classification triggering subunit for triggering radiation source classification if an abnormal radiation event exists; wherein the radiation source classification steps include: extracting the time-domain and frequency-domain features of the electromagnetic radiation signal, wherein the time-domain features include pulse width, repetition frequency, and rise time, and the frequency-domain features include center frequency, bandwidth, and power spectral flatness; concatenating the time-domain and frequency-domain features to obtain a feature vector; performing random forest classification on the feature vector to obtain interference labels, and adding the interference labels to the anomaly detection data, wherein the interference labels are marked with confidence levels.

[0056] The detailed description of the specific configuration of the radiation source coordinate solving module 20 is explained as follows: As mentioned above, the radiation source coordinate solving module 20 can further include: a detection distance calculation unit for obtaining the arrival time of signal pulses based on multiple distributed electromagnetic receivers and calculating the detection distance; and a radiation source coordinate solving unit for solving the radiation source coordinates based on the detection distances of at least three electromagnetic receivers using a hyperbolic surface positioning method.

[0057] The radiation source coordinate solving unit may further include: a first hyperboloid drawing subunit for drawing a first hyperboloid in three-dimensional space based on the distance difference between a first detection distance and a second detection distance; a second hyperboloid drawing subunit for drawing a second hyperboloid in three-dimensional space based on the distance difference between a first detection distance and a third detection distance; a third hyperboloid drawing subunit for drawing a third hyperboloid in three-dimensional space based on the distance difference between a second detection distance and a third detection distance; and a radiation source coordinate generation subunit for taking the intersection point of the first hyperboloid, the second hyperboloid, and the third hyperboloid as the radiation source position to generate the radiation source coordinates.

[0058] The detailed description of the specific configuration of the fire hazard alarm management module 30 is explained as follows: As mentioned above, to perform multi-source joint judgment, the fire hazard alarm management module 30 may further include: a data acquisition unit for acquiring thermal image data from an infrared thermal imager and image frame data from a visible light camera; a spatial registration unit for performing spatial registration of the thermal image data and image frame data based on radiation source coordinates to obtain multi-source detection data; and a joint judgment unit for generating a graded alarm strategy by performing joint judgment of the multi-source detection data.

[0059] The joint judgment unit, which performs the joint judgment of the multi-source detection data to generate a graded alarm strategy, may further include: an Euclidean distance calculation subunit, which calculates the Euclidean distance by spatially registering the coordinates of the radiation source with the coordinates of the high-temperature point in the heat map data to obtain a first judgment result, wherein if the Euclidean distance is less than the spatial tolerance, it is judged to be the same heat source; a spatial registration subunit, which performs spatial registration with the coordinates of the radiation source and the image frame data to obtain the smoke visual features of the coordinate region as a second judgment result; and a graded alarm strategy generation subunit, which generates the graded alarm strategy based on the first judgment result and the second judgment result.

[0060] The fire hazard intelligent monitoring and early warning system based on multimodal data provided in this embodiment of the invention can execute the fire hazard intelligent monitoring and early warning method based on multimodal data provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0061] Although this application makes various references to certain modules in the system according to the embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy distinction between each other and are not used to limit the scope of protection of this invention.

[0062] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for intelligent monitoring and early warning of fire hazards based on multimodal data, characterized in that, The method includes: Electromagnetic radiation signal acquisition is performed in the target area. Based on the anomaly detector, the Mahalanobis distance is calculated with the electromagnetic fingerprint baseline to determine the abnormal radiation event and classify the radiation source, thereby obtaining anomaly detection data. The anomaly detector is embedded with the electromagnetic fingerprint baseline under normal conditions. The detection group is formed by using at least three electromagnetic receivers, and the abnormal detection data is cross-verified. The detection distance is calculated based on the pulse arrival time of the electromagnetic radiation signal, and the coordinates of the radiation source are solved by multiple hyperboloid intersections. Acquire thermal image data from an infrared thermal imager and image frame data from a visible light camera, perform spatial registration based on the coordinates of the radiation source, execute multi-source joint judgment, and respond to the hierarchical alarm module for fire hazard alarm management. Obtain anomaly detection data, including: A broadband electromagnetic receiver is deployed in the target area to collect electromagnetic radiation signals in space through an antenna. The electromagnetic radiation signals are the superposition of electromagnetic radiation generated by electromagnetic interference sources and electrical fault arcs. Establish an electromagnetic fingerprint baseline under normal conditions; An anomaly detector is deployed based on the electromagnetic fingerprint baseline to receive the electromagnetic radiation signal for anomaly detection and radiation source classification, and to determine the anomaly detection data. By collecting continuous electromagnetic background noise and statistically analyzing the power spectral density distribution at each frequency point, the electromagnetic fingerprint baseline is constructed, wherein the power spectral density distribution is determined by the mean, variance, and higher-order moments. Among them, electric arc discharge has the first electromagnetic radiation characteristic, and normal state has the second electromagnetic radiation characteristic; Perform multi-source joint judgment, including: Acquire thermal image data from an infrared thermal imager and image frame data from a visible light camera; Spatial registration based on radiation source coordinates is performed on the heat map data and image frame data to obtain multi-source detection data; A tiered alarm strategy is generated by performing a joint judgment on the multi-source detection data; Perform joint judgment on the multi-source detection data to generate a graded alarm strategy, including: By spatially registering the coordinates of the radiation source with the coordinates of the high-temperature points in the heat map data, calculating the Euclidean distance, and obtaining the first judgment result, wherein if the Euclidean distance is less than the spatial tolerance, it is judged to be the same heat source; Spatial registration is performed between the radiation source coordinates and image frame data to obtain the smoke visual features of the coordinate region, which is used as the second judgment result. The hierarchical alarm strategy is generated based on the first and second judgment results.

2. The intelligent monitoring and early warning method for fire hazards based on multimodal data as described in claim 1, characterized in that, Receiving the electromagnetic radiation signal for anomaly detection includes: Perform a Fourier transform on the electromagnetic radiation signal to obtain the frame signal power spectral density of each frame. The power spectral density of the frame signal is compared with the electromagnetic fingerprint baseline to calculate the Mahalanobis distance, wherein the Mahalanobis distance measures the degree of deviation from the normal within the statistical distribution framework of environmental noise; Determine whether the Mahalanobis distance is greater than a preset threshold. If it is, determine that there is an abnormal radiation event based on arc discharge.

3. The intelligent monitoring and early warning method for fire hazards based on multimodal data as described in claim 2, characterized in that, If an abnormal radiation event is detected, radiation source classification will be triggered. The steps for classifying radiation sources include: Extract the time-domain and frequency-domain features of the electromagnetic radiation signal, wherein the time-domain features include pulse width, repetition frequency and rise time, and the frequency-domain features include center frequency, bandwidth and power spectrum flatness; The time-domain features and frequency-domain features are concatenated to obtain a feature vector; Random forest classification is performed on the feature vector to obtain interference labels, and the interference labels are added to the anomaly detection data, wherein the interference labels are marked with confidence levels.

4. The intelligent monitoring and early warning method for fire hazards based on multimodal data as described in claim 3, characterized in that, Solving for the coordinates of the radiation source includes: Obtain the arrival time of signal pulses from a distributed array of electromagnetic receivers and calculate the detection distance. Based on the detection distances of at least three electromagnetic receivers, the coordinates of the radiation source are determined using a hyperboloid positioning method.

5. The intelligent monitoring and early warning method for fire hazards based on multimodal data as described in claim 4, characterized in that, Based on the distance difference between the first detection distance and the second detection distance, draw the first hyperboloid in three-dimensional space; Based on the distance difference between the first detection distance and the third detection distance, draw the second hyperboloid in three-dimensional space; Based on the distance difference between the second and third detection distances, a third hyperboloid is drawn in three-dimensional space; The intersection points of the first hyperboloid, the second hyperboloid, and the third hyperboloid are taken as the locations of the radiation source, and the coordinates of the radiation source are generated.

6. A fire hazard intelligent monitoring and early warning system based on multimodal data, characterized in that, The system is used to implement the intelligent monitoring and early warning method for fire hazards based on multimodal data as described in any one of claims 1-5, and the system includes: An anomaly detection module is used to collect electromagnetic radiation signals in the target area. Based on the anomaly detector, the module calculates the Mahalanobis distance with the electromagnetic fingerprint baseline, judges the abnormal radiation events, classifies the radiation sources, and obtains anomaly detection data. The anomaly detector is embedded with the electromagnetic fingerprint baseline under normal conditions. The radiation source coordinate solving module is used to group detections with at least three electromagnetic receivers, perform cross-verification of anomaly detection data, calculate the detection distance based on the pulse arrival time of the electromagnetic radiation signal, and solve the radiation source coordinates through multiple hyperboloid intersections. The fire hazard alarm management module is used to acquire thermal image data from infrared thermal imagers and image frame data from visible light cameras, perform spatial registration based on the coordinates of the radiation sources, perform multi-source joint judgment, and respond to the hierarchical alarm module for fire hazard alarm management.

Images