Fire real-time monitoring method and system based on multi-modal data fusion

Abstract

The invention relates to the technical field of fire monitoring, in particular to a fire real-time monitoring method and system based on multi-modal data fusion, and the method specifically comprises the steps: extracting smoke sound wave historical development data in a historical fire occurrence region, and collecting real-time environment sound wave data in a monitoring unit region; constructing a smoke sound wave self-learning model, and outputting smoke sound wave similarity in real time; comparing the smoke sound wave similarity output in real time with the standard similarity; real-time environment sound wave data and similar smoke sound wave historical development data in a current monitoring unit area are imported into a fire initial position prediction strategy to obtain a fire initial prediction position, a fire transition trend value is synchronously calculated, and a classification alarm strategy is executed; according to the invention, the problem that the early fire features in the early stage of the fire are difficult to dynamically and comprehensively identify in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of fire monitoring, and is a real-time fire monitoring method and system based on multi-modal data fusion. Background Art

[0002] Existing fire monitoring methods mainly rely on single sensor data, such as smoke detectors, thermal sensors, and flame detectors. Although these methods can detect fires in a timely manner to a certain extent, they have certain limitations in complex environments. Traditional smoke alarms rely on the concentration of smoke particles when detecting fires. However, in the early stages of a fire, the concentration of smoke particles may be low, especially in large open areas or environments with strong ventilation. Smoke alarms may not be able to detect fires in a timely manner. At the same time, smoke alarms are easily interfered by non-fire factors, such as cooking smoke, steam, dust, etc. These non-fire factors may cause false alarms, thereby reducing the reliability of the alarm system and the user's trust in it. In addition, in the early stages of a fire, early fire features, such as smoke sound waves, ambient temperature, and gas concentration changes, are often relatively weak. Traditional feature recognition methods have difficulty capturing these subtle changes, which may lead to insufficient sensitivity and accuracy in early fire monitoring, shortening precious fire rescue time. Summary of the invention

[0003] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0004] The technical problem to be solved by the present invention is that it is difficult to dynamically and comprehensively identify the early fire characteristics in the early stage of the fire in the prior art, and a real-time fire monitoring method and system based on multimodal data fusion are proposed.

[0005] In order to achieve the above object, the technical solution of the real-time fire monitoring method based on multimodal data fusion of the present invention comprises the following steps:

[0006] S1: Divide the fire monitoring area into monitoring unit areas, extract the historical development data of smoke sound waves in the historical fire occurrence area, and collect real-time environmental sound wave data in the monitoring unit area;

[0007] S2: Build a smoke sound wave self-learning model, use the trained smoke sound wave self-learning model in the monitoring unit area to identify the real-time environmental sound wave data in the current monitoring unit area, and output the smoke sound wave similarity in real time;

[0008] S3: preset standard similarity, compare the real-time output smoke sound wave similarity with the standard similarity, when the real-time output smoke sound wave similarity meets the similarity condition, execute S5, otherwise execute S4;

[0009] S4: extracting multimodal fire characterization data within the monitoring unit area, and repeatedly verifying the fire risk within the monitoring unit area based on the multimodal fire characterization data;

[0010] S5: Import the real-time environmental sound wave data and the historical development data of similar smoke sound waves in the current monitoring unit area into the fire start position prediction strategy to obtain the fire start prediction position and simultaneously calculate the fire change trend value;

[0011] S6: Execute the classification alarm strategy based on the predicted fire starting location and fire change trend value.

[0012] Specifically, S1 includes the following specific steps:

[0013] S11: According to the geographic information system, the entire fire monitoring area is divided into a plurality of monitoring unit areas, wherein the area of ​​each monitoring unit area is Q;

[0014] S12: query the historical fire database to extract the fire start location, fire start timestamp and fire cause type of the fire events that occurred in the monitoring unit area;

[0015] S13: For each historical fire event, extract the historical development data of smoke sound waves recorded in the monitoring unit area at that time, wherein the historical development data of smoke sound waves include: sound wave time spectrum, sound wave frequency, intensity, and duration, wherein the data duration of each historical development data of smoke sound waves is 1 second, and the data acquisition frequency of the sound wave sensor is 1000 Hz; each historical development data of smoke sound waves constitutes an original historical development data sample of smoke sound waves , , n is the index of the data collection point, These are the historical development data of smoke sound waves collected at the first and second data sampling points respectively.

[0016] Specifically, S2 includes the following specific steps:

[0017] S21: performing normalization processing on the extracted historical development data of smoke sound waves, and using a filter to remove noise in the historical development data of smoke sound waves, wherein the normalization formula is:

[0018] ;

[0019] Develop data samples for the original smoke sound wave history; It is the normalized historical development data sample of smoke sound waves; are the mean and standard deviation of the original smoke sound wave historical development data samples;

[0020] S22: using fast Fourier transform to convert the original smoke sound wave historical development data into spectrum data, and extracting spectrum features to obtain a feature vector of the original smoke sound wave historical development data;

[0021] S23: Divide the feature vectors of all original smoke sound wave historical development data into a training set and a test set with a ratio of 8:2, use the training set to train the smoke sound wave self-learning model in the monitoring area, and use the test set to test the trained smoke sound wave self-learning model;

[0022] S24: collecting real-time environmental sound wave data in the current monitoring unit area from the sound wave sensor, preprocessing the collected real-time environmental sound wave data, including normalization and denoising, and returning to execute S22 to obtain a feature vector of the real-time environmental sound wave data;

[0023] S25: Input the feature vector of the real-time environmental sound wave data into the trained smoke sound wave self-learning model, and calculate the similarity between the real-time environmental sound wave data and the smoke sound wave historical development data through a similarity function, wherein the similarity function formula is: , are the feature vectors of real-time environmental sound wave data and historical development data of smoke sound wave, are the modulus of the feature vectors of the real-time environmental sound wave data and the historical development data of the smoke sound wave, and SIM is the similarity of the smoke sound wave;

[0024] S26: Output smoke sound wave similarity in real time.

[0025] Specifically, S22 includes the following specific steps:

[0026] S221: The strategy for extracting the spectrum features is: divide the historical development data of smoke sound waves into frames with a length of 25 ms and a frame shift of 10 ms, and select The corresponding data sampling point is the first frame sampling point of the original smoke sound wave historical development data sample. The smoke sound wave data frame after frame processing is: , They are the sampling values ​​corresponding to the 1st, 2nd and 25th frame sampling points respectively. A smoke sound wave data frame containing sampling values ​​corresponding to 25 frame sampling points;

[0027] S222: Apply a Hamming window to each frame of data at the same time, specifically: ;in, is the Hamming window function, is the windowed smoke sound wave data frame;

[0028] S223: Apply fast Fourier transform to the windowed smoke sound wave data frame to convert it from the time domain to the frequency domain, specifically: ,in, is the smoke sound wave data frame after fast Fourier transformation, k is the frequency sample number, is an imaginary unit;

[0029] S224: intercept the first five frequency components, including: ; At the same time, the first five frequency components are normalized to obtain the characteristic vector X of the sound wave data, specifically: .

[0030] Specifically, S3 includes the following specific steps:

[0031] S31: preset a standard similarity, and compare the real-time output smoke sound wave similarity with the standard similarity;

[0032] S32: When the smoke sound wave similarities in three consecutive data collection periods all exceed the standard similarity, execute S5, otherwise execute S4.

[0033] Specifically, in S4, the multi-modal fire characterization data includes: ambient temperature data and ambient gas concentration data;

[0034] In S4, the repeated verification includes:

[0035] ;

[0036] in, is the repeated verification value; H is the total duration of a data collection period; h is the index of the time point;

[0037] They are respectively the derivative curves of the ambient temperature data function curve and the ambient gas concentration data function curve;

[0038] are the standard temperature change rate and the standard gas concentration change rate respectively;

[0039] They are the temperature change rate range and the gas concentration change rate range respectively;

[0040] In S4, the repeated verification also includes: presetting a repeated verification threshold. When the repeated verification value is greater than the repeated verification threshold, verifying that there is a fire risk of a new type of fire cause in the monitoring unit area, classifying the entire monitoring unit area as a major risk area, and issuing an early warning. At the same time, returning to S2, the real-time environmental sound wave data of the monitoring unit area is updated to the training set of the smoke sound wave self-learning model.

[0041] Specifically, in S5, obtaining the predicted fire start position includes:

[0042] S51: extracting the fire starting position of the smoke sound wave historical development data with the top five smoke sound wave similarities to the real-time environmental sound wave data output by the smoke sound wave self-learning model;

[0043] S52: Import the data extracted in S51 into the fire start position prediction strategy to obtain the fire start position prediction. The fire start position prediction strategy is specifically:

[0044] ;

[0045] Among them, i is the serial index of 5 groups of data, The straight-line distance between the acoustic wave sensor for collecting the i-th group of smoke acoustic wave historical development data and the fire starting position corresponding to the i-th group of smoke acoustic wave historical development data;

[0046] is the average of the straight-line distances between 5 groups of acoustic wave sensors that collect historical development data of smoke acoustic waves and the fire starting positions corresponding to the historical development data of smoke acoustic waves;

[0047] is the smoke sound wave similarity between the real-time environmental sound wave data and the i-th group of smoke sound wave historical development data; is the mean of the similarity of the five groups of smoke sound waves;

[0048] It is the straight-line distance between the acoustic sensor that collects the current real-time environmental acoustic data and the predicted fire start location.

[0049] Specifically, S5 also includes the following specific steps:

[0050] S53: When the real-time output smoke sound wave similarity meets the similarity condition for the first time, a timed evaluation is performed through the fire change trend evaluation module, wherein the timed evaluation includes: triggering a timed evaluation once in three data collection periods after the real-time output smoke sound wave similarity meets the similarity condition for the first time;

[0051] S54: extract the real-time environmental sound wave data of the t-th, t+1-th and t+2-th data collection periods respectively, and select the sound wave fluctuation periods in which the amplitude difference between adjacent peaks and valleys in the sound wave spectrum is greater than 30 decibels, wherein the number of sound wave fluctuation periods in the real-time environmental sound wave data of the t-th, t+1-th and t+2-th data collection periods is , the frequency of the sound wave fluctuation is , the root mean square of the amplitude difference during the acoustic wave fluctuation is ;

[0052] S55: Calculate and obtain the fire transition trend value. The calculation strategy of the fire transition trend value is:

[0053] ;

[0054] in, is the fire change trend value obtained from the jth timed assessment;

[0055] are proportional weight factors respectively;

[0056] Specifically, in S6, the classification alarm strategy includes: within the monitoring area, with the acoustic wave sensor as the center, A fire alarm area is established for the radius, the broadcasting equipment in the fire alarm area is turned on to broadcast the fire, and the color of the escape sign in the fire alarm area is gradiently adjusted according to the fire change trend value.

[0057] In addition, the real-time fire monitoring system based on multimodal data fusion of the present invention includes the following modules:

[0058] Smoke acoustic wave data acquisition module, smoke model building module, similarity comparison module, duplicate verification module, fire location positioning module and classification alarm module;

[0059] The smoke acoustic wave data acquisition module is used to divide the fire monitoring area into monitoring unit areas, extract the historical development data of smoke acoustic waves in the historical fire occurrence area, and collect real-time environmental acoustic wave data in the monitoring unit area;

[0060] The smoke model building module is used to build a smoke sound wave self-learning model, use the trained smoke sound wave self-learning model in the monitoring unit area to identify the real-time environmental sound wave data in the current monitoring unit area, and output the smoke sound wave similarity in real time;

[0061] The similarity comparison module is used to preset a standard similarity and compare the real-time output smoke sound wave similarity with the standard similarity;

[0062] The repeated verification module is used to extract multi-modal fire characterization data within the monitoring unit area, and to repeatedly verify the fire risk within the monitoring unit area according to the multi-modal fire characterization data;

[0063] The fire location positioning module is used to import the real-time environmental sound wave data and the historical development data of similar smoke sound waves in the current monitoring unit area into the fire starting position prediction strategy, obtain the fire starting prediction position, and simultaneously calculate the fire change trend value;

[0064] The classification alarm module is used to execute the classification alarm strategy according to the predicted fire starting position and the fire change trend value.

[0065] Compared with the prior art, the technical effects of the present invention are as follows:

[0066] 1. The present invention realizes accurate detection of early fire by analyzing the spectrum and amplitude of smoke sound waves. Compared with traditional smoke alarms, this technology can capture early fire signals several minutes in advance, significantly improving the timeliness and reliability of fire warning.

[0067] 2. The present invention integrates multiple sensor data such as smoke sound waves, ambient temperature and gas concentration to construct a multimodal data fusion model. This system can provide comprehensive and accurate fire monitoring information, effectively make up for the shortcomings of single sensor data, and significantly improve the accuracy of fire identification and the reliability of the system. The present invention also conducts joint analysis of smoke sound waves with ambient temperature and gas concentration. The system can effectively identify and eliminate false alarms caused by non-fire factors, reduce the false alarm rate, and enhance users' trust in the alarm system and the practicality of the system.

[0068] 3. By analyzing the dynamic changes of smoke sound waves under different environmental conditions, the present invention can adjust the monitoring parameters in real time to adapt to scenes with strong winds or other dynamic changes. This dynamic adaptability enables the system to maintain high sensitivity and high accuracy in complex environments, significantly improving the robustness of the system.

[0069] 4. Through the analysis of historical data and real-time data of smoke sound waves, the system of the present invention can predict the expansion trend and development direction of fire, provide important decision-making support for emergency response, and help firefighters deploy resources and formulate response strategies more effectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0071] Figure 1 It is a flow chart of a real-time fire monitoring method based on multimodal data fusion of the present invention;

[0072] Figure 2 It is a structural schematic diagram of the real-time fire monitoring system based on multimodal data fusion of the present invention. DETAILED DESCRIPTION

[0073] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0074] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0075] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0076] Embodiment 1:

[0077] like Figure 1 As shown, the real-time fire monitoring method based on multimodal data fusion according to an embodiment of the present invention includes the following specific steps:

[0078] S1: Divide the fire monitoring area into monitoring unit areas, extract the historical development data of smoke sound waves in the historical fire occurrence area, and collect real-time environmental sound wave data in the monitoring unit area;

[0079] S1 includes the following specific steps:

[0080] S11: According to the geographic information system, the entire fire monitoring area is divided into a plurality of monitoring unit areas, wherein the area of ​​each monitoring unit area is Q;

[0081] S12: query the historical fire database to extract the fire start location, fire start timestamp and fire cause type of the fire events that occurred in the monitoring unit area;

[0082] S13: For each historical fire event, extract the historical development data of smoke sound waves recorded in the monitoring unit area at that time, wherein the historical development data of smoke sound waves include: sound wave time spectrum, sound wave frequency, intensity, and duration, wherein the data duration of each historical development data of smoke sound waves is 1 second, and the data acquisition frequency of the sound wave sensor is 1000 Hz; each historical development data of smoke sound waves constitutes an original historical development data sample of smoke sound waves , , n is the index of the data collection point, These are the historical development data of smoke sound waves collected at the first and second data sampling points respectively.

[0083] S2: Build a smoke sound wave self-learning model, use the trained smoke sound wave self-learning model in the monitoring unit area to identify the real-time environmental sound wave data in the current monitoring unit area, and output the smoke sound wave similarity in real time;

[0084] S2 includes the following specific steps:

[0085] S21: performing normalization processing on the extracted historical development data of smoke sound waves, and using a filter to remove noise in the historical development data of smoke sound waves, wherein the normalization formula is:

[0086] ;

[0087] Develop data samples for the original smoke sound wave history; It is the normalized historical development data sample of smoke sound waves; are the mean and standard deviation of the original smoke sound wave historical development data samples;

[0088] S22: using fast Fourier transform to convert the original smoke sound wave historical development data into spectrum data, and extracting spectrum features to obtain a feature vector of the original smoke sound wave historical development data;

[0089] S23: Divide the feature vectors of all original smoke sound wave historical development data into a training set and a test set with a ratio of 8:2, use the training set to train the smoke sound wave self-learning model in the monitoring area, and use the test set to test the trained smoke sound wave self-learning model;

[0090] S24: collecting real-time environmental sound wave data in the current monitoring unit area from the sound wave sensor, preprocessing the collected real-time environmental sound wave data, including normalization and denoising, and returning to execute S22 to obtain a feature vector of the real-time environmental sound wave data;

[0091] S25: Input the feature vector of the real-time environmental sound wave data into the trained smoke sound wave self-learning model, and calculate the similarity between the real-time environmental sound wave data and the smoke sound wave historical development data through a similarity function, wherein the similarity function formula is: , are the feature vectors of real-time environmental sound wave data and historical development data of smoke sound wave, are the modulus of the feature vectors of the real-time environmental sound wave data and the historical development data of the smoke sound wave, and SIM is the similarity of the smoke sound wave;

[0092] S26: Output smoke sound wave similarity in real time.

[0093] S22 includes the following specific steps:

[0094] S221: The strategy for extracting the spectrum features is: divide the historical development data of smoke sound waves into frames with a length of 25 ms and a frame shift of 10 ms, and select The corresponding data sampling point is the first frame sampling point of the original smoke sound wave historical development data sample. The smoke sound wave data frame after frame processing is: , They are the sampling values ​​corresponding to the 1st, 2nd and 25th frame sampling points respectively. A smoke sound wave data frame containing sampling values ​​corresponding to 25 frame sampling points;

[0095] S222: Apply a Hamming window to each frame of data at the same time, specifically: ;in, is the Hamming window function, is the windowed smoke sound wave data frame;

[0096] S223: Apply fast Fourier transform to the windowed smoke sound wave data frame to convert it from the time domain to the frequency domain, specifically: ,in, is the smoke sound wave data frame after fast Fourier transformation, k is the frequency sample number, is an imaginary unit;

[0097] S224: intercept the first five frequency components, including: ; At the same time, the first five frequency components are normalized to obtain the characteristic vector X of the sound wave data, specifically: .

[0098] S3: preset standard similarity, compare the real-time output smoke sound wave similarity with the standard similarity, when the real-time output smoke sound wave similarity meets the similarity condition, execute S5, otherwise execute S4;

[0099] S3 includes the following specific steps:

[0100] S31: preset a standard similarity, and compare the real-time output smoke sound wave similarity with the standard similarity;

[0101] S32: When the smoke sound wave similarities in three consecutive data collection periods all exceed the standard similarity, execute S5, otherwise execute S4.

[0102] S4: extracting multimodal fire characterization data within the monitoring unit area, and repeatedly verifying the fire risk within the monitoring unit area based on the multimodal fire characterization data;

[0103] In S4, the multi-modal fire characterization data includes: ambient temperature data and ambient gas concentration data;

[0104] In S4, the repeated verification includes:

[0105] ;

[0106] in, is the repeated verification value; H is the total duration of a data collection period; h is the index of the time point;

[0107] They are respectively the derivative curves of the ambient temperature data function curve and the ambient gas concentration data function curve;

[0108] are the standard temperature change rate and the standard gas concentration change rate respectively;

[0109] They are the temperature change rate range and the gas concentration change rate range respectively;

[0110] In S4, the repeated verification also includes: presetting a repeated verification threshold. When the repeated verification value is greater than the repeated verification threshold, verifying that there is a fire risk of a new type of fire cause in the monitoring unit area, classifying the entire monitoring unit area as a major risk area, and issuing an early warning. At the same time, returning to S2, the real-time environmental sound wave data of the monitoring unit area is updated to the training set of the smoke sound wave self-learning model.

[0111] S5: Import the real-time environmental sound wave data and the historical development data of similar smoke sound waves in the current monitoring unit area into the fire start position prediction strategy to obtain the fire start prediction position and simultaneously calculate the fire change trend value;

[0112] In S5, obtaining the predicted fire start position includes:

[0113] S51: extracting the fire starting position of the smoke sound wave historical development data with the top five smoke sound wave similarities to the real-time environmental sound wave data output by the smoke sound wave self-learning model;

[0114] S52: Import the data extracted in S51 into the fire start position prediction strategy to obtain the fire start position prediction. The fire start position prediction strategy is specifically:

[0115] ;

[0116] Among them, i is the serial index of 5 groups of data, The straight-line distance between the acoustic wave sensor for collecting the i-th group of smoke acoustic wave historical development data and the fire starting position corresponding to the i-th group of smoke acoustic wave historical development data;

[0117] is the average of the straight-line distances between 5 groups of acoustic wave sensors that collect historical development data of smoke acoustic waves and the fire starting positions corresponding to the historical development data of smoke acoustic waves;

[0118] is the smoke sound wave similarity between the real-time environmental sound wave data and the i-th group of smoke sound wave historical development data; is the mean of the similarity of the five groups of smoke sound waves;

[0119] It is the straight-line distance between the acoustic sensor that collects the current real-time environmental acoustic data and the predicted fire start location.

[0120] S5 also includes the following specific steps:

[0121] S53: When the real-time output smoke sound wave similarity meets the similarity condition for the first time, a timed evaluation is performed through the fire change trend evaluation module, wherein the timed evaluation includes: triggering a timed evaluation once in three data collection periods after the real-time output smoke sound wave similarity meets the similarity condition for the first time;

[0122] S54: extract the real-time environmental sound wave data of the t-th, t+1-th and t+2-th data collection periods respectively, and select the sound wave fluctuation periods in which the amplitude difference between adjacent peaks and valleys in the sound wave spectrum is greater than 30 decibels, wherein the number of sound wave fluctuation periods in the real-time environmental sound wave data of the t-th, t+1-th and t+2-th data collection periods is , the frequency of the sound wave fluctuation is , the root mean square of the amplitude difference during the acoustic wave fluctuation is ;

[0123] S55: Calculate and obtain the fire transition trend value. The calculation strategy of the fire transition trend value is:

[0124] ;

[0125] in, is the fire change trend value obtained from the jth timed assessment;

[0126] are proportional weight factors respectively;

[0127] ,

[0128] Exemplarily, a method for determining a proportional weight factor is provided in this embodiment, specifically: ;

[0129] ;

[0130] is the average of the smoke sound wave similarity between the real-time environmental sound wave data in the t+1th data collection period and the five groups of smoke sound wave historical development data with the top five similarities; It is the average of the smoke sound wave similarities between the real-time environmental sound wave data in the t+2th data collection period and the five groups of smoke sound wave historical development data with the top five similarities.

[0131] S6: Execute the classification alarm strategy based on the predicted fire starting location and fire change trend value.

[0132] In S6, the classification alarm strategy includes: within the monitoring area, with the acoustic wave sensor as the center, A fire alarm area is established for the radius, the broadcasting equipment in the fire alarm area is turned on to broadcast the fire, and the color of the escape sign in the fire alarm area is gradiently adjusted according to the fire change trend value.

[0133] Embodiment 2:

[0134] like Figure 2 As shown, the real-time fire monitoring system based on multimodal data fusion according to the embodiment of the present invention includes the following modules:

[0135] Smoke acoustic wave data acquisition module, smoke model building module, similarity comparison module, duplicate verification module, fire location positioning module and classification alarm module;

[0136] The smoke acoustic wave data acquisition module is used to divide the fire monitoring area into monitoring unit areas, extract the historical development data of smoke acoustic waves in the historical fire occurrence area, and collect real-time environmental acoustic wave data in the monitoring unit area;

[0137] The smoke model building module is used to build a smoke sound wave self-learning model, use the trained smoke sound wave self-learning model in the monitoring unit area to identify the real-time environmental sound wave data in the current monitoring unit area, and output the smoke sound wave similarity in real time;

[0138] The similarity comparison module is used to preset a standard similarity and compare the real-time output smoke sound wave similarity with the standard similarity;

[0139] The repeated verification module is used to extract multi-modal fire characterization data within the monitoring unit area, and to repeatedly verify the fire risk within the monitoring unit area according to the multi-modal fire characterization data;

[0140] The fire location positioning module is used to import the real-time environmental sound wave data and the historical development data of similar smoke sound waves in the current monitoring unit area into the fire starting position prediction strategy, obtain the fire starting prediction position, and simultaneously calculate the fire change trend value;

[0141] The classification alarm module is used to execute the classification alarm strategy according to the predicted fire starting position and the fire change trend value.

[0142] In summary, compared with the prior art, the technical effects of the present invention are as follows:

[0143] 1. The present invention realizes accurate detection of early fire by analyzing the spectrum and amplitude of smoke sound waves. Compared with traditional smoke alarms, this technology can capture early fire signals several minutes in advance, significantly improving the timeliness and reliability of fire warning.

[0144] 2. The present invention integrates multiple sensor data such as smoke sound waves, ambient temperature and gas concentration to construct a multimodal data fusion model. This system can provide comprehensive and accurate fire monitoring information, effectively make up for the shortcomings of single sensor data, and significantly improve the accuracy of fire identification and the reliability of the system. The present invention also conducts joint analysis of smoke sound waves with ambient temperature and gas concentration. The system can effectively identify and eliminate false alarms caused by non-fire factors, reduce the false alarm rate, and enhance users' trust in the alarm system and the practicality of the system.

[0145] 3. By analyzing the dynamic changes of smoke sound waves under different environmental conditions, the present invention can adjust the monitoring parameters in real time to adapt to scenes with strong winds or other dynamic changes. This dynamic adaptability enables the system to maintain high sensitivity and high accuracy in complex environments, significantly improving the robustness of the system.

[0146] 4. Through the analysis of historical data and real-time data of smoke sound waves, the system of the present invention can predict the expansion trend and development direction of fire, provide important decision-making support for emergency response, and help firefighters deploy resources and formulate response strategies more effectively.

[0147] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A real-time fire monitoring method based on multimodal data fusion, characterized in that: The method comprises the following specific steps: S1: Divide the fire monitoring area into monitoring unit areas, extract the historical development data of smoke sound waves in the historical fire occurrence area, and collect real-time environmental sound wave data in the monitoring unit area; S2: Build a smoke sound wave self-learning model, use the trained smoke sound wave self-learning model in the monitoring unit area to identify the real-time environmental sound wave data in the current monitoring unit area, and output the smoke sound wave similarity in real time; S3: preset standard similarity, compare the real-time output smoke sound wave similarity with the standard similarity, when the real-time output smoke sound wave similarity meets the similarity condition, execute S5, otherwise execute S4; S4: extracting multimodal fire characterization data within the monitoring unit area, and repeatedly verifying the fire risk within the monitoring unit area based on the multimodal fire characterization data; S5: Import the real-time environmental sound wave data and the historical development data of similar smoke sound waves in the current monitoring unit area into the fire start position prediction strategy to obtain the fire start prediction position and simultaneously calculate the fire change trend value; S6: Execute the classification alarm strategy based on the predicted fire starting location and fire change trend value.

2. The real-time fire monitoring method based on multimodal data fusion according to claim 1 is characterized in that: S1 includes the following specific steps: S11: According to the geographic information system, the entire fire monitoring area is divided into a plurality of monitoring unit areas, wherein the area of ​​each monitoring unit area is Q; S12: query the historical fire database to extract the fire start location, fire start timestamp and fire cause type of the fire events that occurred in the monitoring unit area; S13: For each historical fire event, extract the historical development data of smoke sound waves recorded in the monitoring unit area at that time, wherein the historical development data of smoke sound waves include: sound wave time spectrum, sound wave frequency, intensity, and duration, wherein the data duration of each historical development data of smoke sound waves is 1 second, and the data acquisition frequency of the sound wave sensor is 1000 Hz; each historical development data of smoke sound waves constitutes an original historical development data sample of smoke sound waves , , n is the index of the data collection point, These are the historical development data of smoke sound waves collected at the first and second data sampling points respectively.

3. The real-time fire monitoring method based on multimodal data fusion according to claim 2 is characterized in that: S2 includes the following specific steps: S21: performing normalization processing on the extracted historical development data of smoke sound waves, and using a filter to remove noise in the historical development data of smoke sound waves, wherein the normalization formula is: ； Develop data samples for the original smoke sound wave history; It is the normalized historical development data sample of smoke sound waves; are the mean and standard deviation of the original smoke sound wave historical development data samples; S22: using fast Fourier transform to convert the original smoke sound wave historical development data into spectrum data, and extracting spectrum features to obtain a feature vector of the original smoke sound wave historical development data; S23: Divide the feature vectors of all original smoke sound wave historical development data into a training set and a test set with a ratio of 8:2, use the training set to train the smoke sound wave self-learning model in the monitoring area, and use the test set to test the trained smoke sound wave self-learning model; S24: collecting real-time environmental sound wave data in the current monitoring unit area from the sound wave sensor, preprocessing the collected real-time environmental sound wave data, including normalization processing and denoising processing, and returning to execute S22 to obtain a feature vector of the real-time environmental sound wave data; S25: Input the feature vector of the real-time environmental sound wave data into the trained smoke sound wave self-learning model, and calculate the similarity between the real-time environmental sound wave data and the smoke sound wave historical development data through a similarity function, wherein the similarity function formula is: , are the feature vectors of real-time environmental sound wave data and historical development data of smoke sound wave, are the modulus of the feature vectors of the real-time environmental sound wave data and the historical development data of the smoke sound wave, and SIM is the similarity of the smoke sound wave; S26: Output smoke sound wave similarity in real time.

4. The real-time fire monitoring method based on multimodal data fusion according to claim 3 is characterized in that: S22 includes the following specific steps: S221: The strategy for extracting the spectrum features is: divide the historical development data of smoke sound waves into frames with a length of 25 ms and a frame shift of 10 ms, and select The corresponding data sampling point is the first frame sampling point of the original smoke sound wave historical development data sample. The smoke sound wave data frame after frame processing is: , They are the sampling values ​​corresponding to the 1st, 2nd and 25th frame sampling points respectively. A smoke sound wave data frame containing sampling values ​​corresponding to 25 frame sampling points; S222: Apply a Hamming window to each frame of data at the same time, specifically: ;in, is the Hamming window function, is the windowed smoke sound wave data frame; S223: Apply fast Fourier transform to the windowed smoke sound wave data frame to convert it from the time domain to the frequency domain, specifically: ,in, is the smoke sound wave data frame after fast Fourier transformation, k is the frequency sample number, is an imaginary unit; S224: intercept the first five frequency components, including: ; At the same time, the first five frequency components are normalized to obtain the characteristic vector X of the sound wave data, specifically: .

5. The real-time fire monitoring method based on multimodal data fusion according to claim 4 is characterized in that: S3 includes the following specific steps: S31: preset a standard similarity, and compare the real-time output smoke sound wave similarity with the standard similarity; S32: When the smoke sound wave similarities in three consecutive data collection periods all exceed the standard similarity, execute S5, otherwise execute S4.

6. The real-time fire monitoring method based on multimodal data fusion according to claim 5 is characterized in that: In S4, the multi-modal fire characterization data includes: ambient temperature data and ambient gas concentration data; In S4, the repeated verification includes: ； in, is the repeated verification value; H is the total duration of a data collection period; h is the index of the time point; They are respectively the derivative curves of the ambient temperature data function curve and the ambient gas concentration data function curve; are the standard temperature change rate and the standard gas concentration change rate respectively; They are the temperature change rate range and the gas concentration change rate range respectively; In S4, the repeated verification also includes: presetting a repeated verification threshold. When the repeated verification value is greater than the repeated verification threshold, verifying that there is a fire risk of a new type of fire cause in the monitoring unit area, classifying the entire monitoring unit area as a major risk area, and issuing an early warning. At the same time, returning to S2, the real-time environmental sound wave data of the monitoring unit area is updated to the training set of the smoke sound wave self-learning model.

7. The real-time fire monitoring method based on multimodal data fusion according to claim 6 is characterized in that: In S5, obtaining the predicted fire start position includes: S51: extracting the fire starting position of the smoke sound wave historical development data with the top five smoke sound wave similarities to the real-time environmental sound wave data output by the smoke sound wave self-learning model; S52: Import the data extracted in S51 into the fire start position prediction strategy to obtain the fire start position prediction. The fire start position prediction strategy is specifically: ； Among them, i is the serial index of 5 groups of data, The straight-line distance between the acoustic wave sensor for collecting the i-th group of smoke acoustic wave historical development data and the fire starting position corresponding to the i-th group of smoke acoustic wave historical development data; is the average of the straight-line distances between 5 groups of acoustic wave sensors that collect historical development data of smoke acoustic waves and the fire starting positions corresponding to the historical development data of smoke acoustic waves; is the smoke sound wave similarity between the real-time environmental sound wave data and the i-th group of smoke sound wave historical development data; is the mean of the similarity of the five groups of smoke sound waves; It is the straight-line distance between the acoustic sensor that collects the current real-time environmental acoustic data and the predicted fire start location.

8. The real-time fire monitoring method based on multimodal data fusion according to claim 7 is characterized in that: S5 also includes the following specific steps: S53: When the real-time output smoke sound wave similarity meets the similarity condition for the first time, a timed evaluation is performed through the fire change trend evaluation module, wherein the timed evaluation includes: triggering a timed evaluation once in three data collection periods after the real-time output smoke sound wave similarity meets the similarity condition for the first time; S54: extract the real-time environmental sound wave data of the t-th, t+1-th and t+2-th data collection periods respectively, and select the sound wave fluctuation periods in which the amplitude difference between adjacent peaks and valleys in the sound wave spectrum is greater than 30 decibels, wherein the number of sound wave fluctuation periods in the real-time environmental sound wave data of the t-th, t+1-th and t+2-th data collection periods is , the frequency of the sound wave fluctuation is , the root mean square of the amplitude difference during the acoustic wave fluctuation is ; S55: Calculate and obtain the fire transition trend value. The calculation strategy of the fire transition trend value is: ； in, is the fire change trend value obtained from the jth timed assessment; are proportional weight factors respectively; .

9. The real-time fire monitoring method based on multimodal data fusion according to claim 8 is characterized in that: In S6, the classification alarm strategy includes: within the monitoring area, with the acoustic wave sensor as the center, A fire alarm area is established for the radius, the broadcasting equipment in the fire alarm area is turned on to broadcast the fire, and the color of the escape sign in the fire alarm area is gradiently adjusted according to the fire change trend value.

10. A real-time fire monitoring system based on multimodal data fusion, used to implement the real-time fire monitoring method based on multimodal data fusion as claimed in any one of claims 1 to 9, characterized in that: The system includes the following modules: Smoke acoustic wave data acquisition module, smoke model building module, similarity comparison module, duplicate verification module, fire location positioning module and classification alarm module; The smoke acoustic wave data acquisition module is used to divide the fire monitoring area into monitoring unit areas, extract the historical development data of smoke acoustic waves in the historical fire occurrence area, and collect real-time environmental acoustic wave data in the monitoring unit area; The smoke model building module is used to build a smoke sound wave self-learning model, use the trained smoke sound wave self-learning model in the monitoring unit area to identify the real-time environmental sound wave data in the current monitoring unit area, and output the smoke sound wave similarity in real time; The similarity comparison module is used to preset a standard similarity and compare the real-time output smoke sound wave similarity with the standard similarity; The repeated verification module is used to extract multi-modal fire characterization data within the monitoring unit area, and to repeatedly verify the fire risk within the monitoring unit area according to the multi-modal fire characterization data; The fire location positioning module is used to import the real-time environmental sound wave data and the historical development data of similar smoke sound waves in the current monitoring unit area into the fire starting position prediction strategy, obtain the fire starting prediction position, and simultaneously calculate the fire change trend value; The classification alarm module is used to execute the classification alarm strategy according to the predicted fire starting position and the fire change trend value.

Images