A smart campus fire alarm method and system based on multimodal event detection
Through multimodal event detection technology, combined with audio and image data, using convolutional neural network and MobieNetV2 network model, the problems of high false alarm rate and high false alarm rate in the existing technology are solved, and the fire on campus is realized The rapid and accurate identification and timely alarm have improved the efficiency of fire response and campus safety management.
Patent Information
- Application Number
- CN202410531278.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-29
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2044-04-29
AI Technical Summary
The existing technology has high false alarm rates and high false alarm rates in fire detection, and cannot monitor dynamic changes on campus in real time, and cannot meet the needs of complex campus environments.
A smart campus fire alarm method based on multimodal event detection is adopted, and a convolutional neural network and MobieNetV2 network model are established through the comprehensive analysis of audio and image data to achieve rapid and accurate identification of fires and timely alarm signals are issued.
It improves the accuracy and reliability of fire detection, reduces the false alarm rate, realizes real-time monitoring and timely alarm of fires on campus, and improves the speed of fire response and campus safety management efficiency.
Smart Images

Figure CN118351644B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of campus environment fire risk assessment, and in particular to a smart campus fire alarm method and system based on multimodal event detection. Background Art
[0002] Fire detection and alarm is an extremely important part of smart campus safety management. Existing technologies are mostly based on a single sensor or sensor network, and can only rely on a single indicator such as smoke and temperature to detect fires, which has problems such as high false alarm rate and high missed alarm rate. At the same time, existing technologies usually only provide static information and cannot monitor and analyze dynamic changes in the campus in real time, such as personnel activities and environmental changes, and cannot meet the needs of complex campus environments.
[0003] In view of the problems of the prior art, the present invention proposes the need to introduce multimodal data, such as audio and image data, to comprehensively analyze multiple data sources to improve the accuracy and comprehensiveness of risk assessment. At the same time, it is also necessary to monitor various dynamic changes in the campus in real time and issue early warnings in time so that timely emergency measures can be taken. Therefore, the present invention proposes a smart campus fire alarm method and system based on multimodal event detection to solve the shortcomings of the prior art. Summary of the invention
[0004] In view of the deficiencies in the prior art, the present invention develops a smart campus fire alarm method and system based on multimodal event detection, which can realize rapid and accurate identification of emergency events such as fires on campus, and send out alarm signals in time, thereby improving the efficiency and reliability of fire emergency response in smart campuses.
[0005] The technical solution to the technical problem of the present invention is: a smart campus fire alarm method based on multimodal event detection, comprising the following steps:
[0006] S1. Prepare data set: simulate a fire situation in a campus environment and obtain the data therein as a data set, which includes an audio data set and an image data set;
[0007] S2. Establish a fire audio classification model and train the model: establish a fire audio classification model based on a convolutional neural network, the fire audio classification model includes an audio feature extraction unit and a convolutional neural network classification unit, obtain the final classification result through convolution of the convolutional neural network classification unit, and train the fire audio classification model;
[0008] S3. Establish a fire image classification model and train the model: The fire image classification model includes a DW convolution unit and a MobieNetV2 network unit. The final prediction result is obtained through the MobieNetV2 network unit, and the fire image classification model is trained;
[0009] S4. Establish a fire detection alarm model: including a fire condition definition unit and a fire monitoring alarm unit, and determine whether to trigger the fire alarm in the fire monitoring alarm unit according to the fire condition in the fire condition definition unit.
[0010] In a specific implementation manner, S1 is as follows:
[0011] S1.1, Audio Dataset:
[0012] The audio data set is recorded in the simulated campus fire scene. Specifically, the audio is recorded by the Audio-Technica AT2020 microphone driven by Behringer UM2 sound at a distance of 2 to 10 meters from the fire source. The data length of each audio segment is 5 seconds. The collected data is synthesized into an audio data set , ,in , , A represents the set of audio data points, a represents the set of audio category labels, T represents the number of audio data points and audio category labels, represents the i-th audio data point, Represents the i-th audio category label, each audio data point Corresponding to an audio category label ;
[0013] S1.2. Image dataset: The image dataset consists of two parts. One part is the images of the same scene when the audio data was collected, and the other part is the images obtained by fire simulation at different time periods under different weather conditions. The images of the two parts are adjusted to a uniform size and then synthesized into an image dataset. , ,in , , P represents the set of image data points, p represents the set of image category labels, T represents the number of image data points and image category labels, represents the i-th image data point, Represents the i-th image category label, each data point Corresponding to a category label .
[0014] In a specific implementation, the audio feature extraction unit:
[0015] The audio dataset The set A of audio data points in is input to the audio feature extraction unit, , respectively for the audio data points in the set A of audio data points Perform short-time Fourier transform to obtain the corresponding time-frequency matrix vector , audio data points The discrete representation in the time domain is , There are K discrete data information in total, Represents the kth discrete data information in the ith audio data point, and the short-time Fourier transform formula is defined as follows:
[0016] ,
[0017] in, Represents the time-frequency matrix vector obtained after short-time Fourier transform , j represents an imaginary number, Indicates frequency, h represents the window function, T represents the number of audio data points, Represents the sliding position of the window function, and the window function is defined as follows:
[0018] ,
[0019] Where n represents the input of the window function, represents the number of points of the Fourier transform;
[0020] Finally, the time-frequency matrix vector Combined into a time-frequency matrix ,in represents the i-th time-frequency matrix vector, the i-th time-frequency matrix vector Corresponding to the i-th audio data point ;
[0021] Then use the Mel filter to filter the time-frequency matrix Filter and get the Mel time-frequency matrix M , where r represents the number of Mel filter banks;
[0022] Then take the logarithm of M and get Q= , and do discrete cosine transform, the discrete cosine transform of the sth vector in Q is defined as follows:
[0023] ,
[0024] in, represents the sth vector in the Mel time-frequency matrix after taking the logarithm, j represents an imaginary number, Represents the number of points of discrete cosine transform. The result of discrete cosine transform is length The vector of , takes the 2nd to 13th values in the vector as features, and the final feature representation is ,in for The feature vectors, The length of the inner feature vector is 13.
[0025] In a specific implementation, the convolutional neural network classification unit in the fire audio classification model is composed of It consists of layers of convolutional layers and fully connected layers. The convolutional layers include convolution kernels, activation functions, maximum pooling layers, and Dropout layers.
[0026] No. The input feature map of the convolutional layer is , whose dimensions are , the convolution kernel size is , the step length is , the output feature map dimension is , whose dimensions are ,in, Respectively represent The height, width and depth of the layer features, , and Respectively represent l The height, width, and depth of the layer feature are calculated as follows:
[0027] ,
[0028] ,
[0029] ,
[0030] in, Indicates l The size of the convolution kernel of the layer, Indicates l The number of convolution kernels in the layer, represents the step size, in particular, when =1, = , = ;
[0031] for Each value in is activated using ReLU as the activation function to obtain the activated features , the specific process is defined as follows:
[0032] ,
[0033] in, express The value in Indicates Layer dimension, max If the input is greater than 0, the original value is output; if the input is less than or equal to 0, 0 is output;
[0034] No. l The pooling window size of the maximum pooling layer is , the step length is ;
[0035] Then the output of the activation function After being sent to the maximum pooling layer, the dimension of the output feature after the maximum pooling layer is:
[0036] ,
[0037] ,
[0038] in, Indicates l The height of the max pooling layer, Indicates l The width of the max pooling layer, Indicates l The step size of the maximum pooling layer is used to obtain the output of the pooling layer. ;
[0039] Then the output of the maximum pooling layer The Dropout layer randomly discards some neurons in the network during training to reduce the risk of overfitting. The The value is calculated as follows:
[0040] ,
[0041] represents random probability, Indicates the probability of setting the discard, and finally the Dropout layer outputs the feature map ;
[0042] The output of the Dropout layer Send it to the fully connected layer, the fully connected layer will have the dimension of Flatten to a vector of dimension 1 , the flattened feature vector and the weight matrix of the fully connected layer Multiply and add the bias , the operation of the fully connected layer is defined as follows:
[0043] ,
[0044] The output of the fully connected layer Input into the Softmax activation function, and finally get the predicted result. The predicted probability of each category is defined as follows:
[0045]
[0046] in, represents the number of categories, Indicates categories.
[0047] In a specific implementation, the fire audio classification model is trained:
[0048] Input the data in the image data point set A into the fire audio classification model for training and calculate the prediction results , using the cross entropy loss function Calculate the prediction results with the actual audio category labels The difference is defined as follows:
[0049] ,
[0050] The gradient is calculated according to the loss function, and the gradient is passed back to each layer of the network through the back-propagation algorithm. The optimization algorithm Adam is used to update the weights and bias parameters of the model according to the calculated gradient.
[0051] In a specific implementation, the fire image classification model is constructed as follows:
[0052] The MobieNetV2 network is used as the backbone structure of the fire image classification network unit, which includes a Bottleneck layer and a fully connected layer. The Bottleneck layer is defined as a convolutional block. The convolutional block includes a total of B layers. The input feature dimension of the bth layer is , the number of convolutional neurons is , the convolution kernel size is , the step length is , the output feature map dimension is ;
[0053] in, Respectively represent -1 layer features height, width and depth, , and Respectively represent Height, width, and depth of layer features;
[0054] After each convolution layer except the last one, the ReLU activation function will be used to process the output features. After the ReLU activation function, a maximum pooling layer will follow. The first and third layers use ordinary convolution, and the second layer uses DW convolution. After the third convolution layer, the residual of the convolution result and the input is used as the final output of the Bottleneck convolution block.
[0055] The pooling window size of the b-th pooling layer is , the step length is ;
[0056] The MobieNetV2 network is composed of multiple Bottleneck convolution blocks and other stacked structures. The MobieNetV2 network has a total of M layers, and the input feature dimension of the mth layer is , the number of convolutional neurons is , the convolution kernel size is , the step length is , the number of module repetitions is , the output feature map dimension is ;
[0057] The last layer in the MobieNetV2 network is a fully connected layer, which receives the output of the previous layer's Bottleneck convolutional block. As input. The fully connected layer takes the dimension of Flattened to a vector of dimension 1 , the flattened feature vector and the weight matrix of the fully connected layer 'Multiply and add the bias term ', the operation of the fully connected layer is defined as follows:
[0058] ',
[0059] The output of the fully connected layer Input into the Softmax activation function, and finally get the predicted result. The predicted probability of the λth category is defined as follows:
[0060]
[0061] in, represents the number of categories, Indicates categories.
[0062] In a specific implementation, the fire image classification model is trained:
[0063] Input the data in the image data point set P into the fire image classification model for training and calculate the prediction results , using the cross entropy loss function calculate The difference between the actual image category label p,
[0064] Then its definition is as follows:
[0065] ,
[0066] The gradient is calculated according to the loss function, and the gradient is passed back to each layer of the network through the back-propagation algorithm. The optimization algorithm Adam is used to update the weights and bias parameters of the model according to the calculated gradient.
[0067] In a specific implementation manner, the fire condition definition unit:
[0068] The convolutional neural network classification unit will judge the input audio and output whether it meets the audio fire alarm condition. The MobieNetV2 network unit will judge the input image and output whether it meets the image fire alarm condition. The situation that meets the audio fire alarm condition is recorded as FcA, and the situation that meets the image fire alarm condition is recorded as FcP. Multiple continuous inferences are set as the basis for judgment at the same time, and the number of continuous inferences is recorded as , represents the input audio sample, represents the probability that the input audio sample belongs to fire, The minimum value among the audio samples that indicates the model's output belongs to fire , TP represents true positive, FP represents false positive, For a given The minimum number of consecutive TPs in For a given The maximum number of consecutive FPs in * Determined maximum ,but and Satisfies the following formula:
[0069] ,
[0070] ,
[0071] ,
[0072] If the above situation occurs, FP will not be generated, avoiding the false negative situation of FN. False negative means that the predicted sample is a negative sample, but it is actually a positive sample.
[0073] In the fire detection scenario, the most dangerous situations are:
[0074] ,
[0075] *Accurate range:
[0076] ,
[0077] FcA / FcP * / *、 * / * indicates that the model needs to be at least continuous * / * Output fire is higher than or equal to * / *Probability, the specific process of calculating FcA / FcP is:
[0078] Setting parameters or The range and step size of the change are or , according to the calculation, the corresponding and , and meet FcA / FcP * / *and * / *.
[0079] In a specific implementation, the fire detection and alarm unit:
[0080] Run the fire audio classification model to infer the received audio signal and determine whether the model output meets FcA. If it does not meet FcA, the model continues to monitor; otherwise, calculate and store the average value of the model output that meets FcA audio. The fire audio classification model is paused, and then the fire picture classification model is run to calculate whether the current picture result meets FcP. If not, the fire picture classification model stops running and the fire audio classification model is awakened; otherwise, the average value of the model output that meets FcP is calculated and stored. And sounded a fire alarm.
[0081] The present invention also provides a smart campus fire alarm system based on multimodal event detection, which performs a smart campus fire alarm method based on multimodal event detection, specifically including the following units:
[0082] Data acquisition unit: simulate the fire situation in the campus environment and obtain the data as a data set;
[0083] Audio classification unit: A fire audio classification model based on a convolutional neural network is established. The fire audio classification model includes an audio feature extraction unit and a convolutional neural network classification unit. The final classification result is obtained through convolution of the convolutional neural network classification unit, and the fire audio classification model is trained.
[0084] Image classification unit: Establish a fire image classification model, which includes a DW convolution unit and a MobieNetV2 network unit. The final prediction result is obtained through the MobieNetV2 network unit, and the fire image classification model is trained;
[0085] Fire detection alarm unit: The fire detection alarm unit includes a fire condition definition unit and a fire monitoring alarm unit. It determines whether to trigger the fire alarm in the fire monitoring alarm unit according to the fire conditions in the fire condition definition unit.
[0086] The effects provided in the content of the invention are only the effects of the embodiments, rather than all the effects of the invention. The above technical solution has the following advantages or beneficial effects:
[0087] The present invention proposes a smart campus fire alarm method and system based on multimodal event detection. The present invention combines multimodal audio and image data, and can detect fire events from different angles and different perception methods, greatly improving the accuracy and reliability of fire detection. By comprehensively analyzing audio and image data, signs of fire can be discovered more quickly and the false alarm rate can be reduced. At the same time, the present invention can monitor the sound and image data in the campus in real time. Once signs of fire are detected, the system can immediately trigger an alarm and notify relevant personnel and departments in time, thereby improving the speed of fire response. As part of a smart campus, the implementation of the present invention will help improve the management level of the campus. By collecting and analyzing audio and image data, the system can provide more data support for campus management departments, helping them to better monitor campus safety and prevent the occurrence of fire accidents, and can respond to emergencies in a timely and effective manner, thereby improving the overall safety and management efficiency of the campus. BRIEF DESCRIPTION OF THE DRAWINGS
[0088] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.
[0089] Figure 1 Schematic diagram of the process of the present invention. DETAILED DESCRIPTION
[0090] In order to clearly illustrate the technical features of the present invention, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing different structures of the present invention. In order to simplify the disclosure of the present invention, the components and settings of specific examples are described below.
[0091] Example 1
[0092] A smart campus fire alarm system method based on multimodal event detection includes the following steps:
[0093] S1. Prepare data set: simulate a fire situation in a campus environment and obtain the data therein as a data set, which includes an audio data set and an image data set;
[0094] S2. Establish a fire audio classification model and train the model: establish a fire audio classification model based on a convolutional neural network, the fire audio classification model includes an audio feature extraction unit and a convolutional neural network classification unit, obtain the final classification result through convolution of the convolutional neural network classification unit, and train the fire audio classification model;
[0095] S3. Establish a fire image classification model and train the model: The fire image classification model includes a DW convolution unit and a MobieNetV2 network unit. The final prediction result is obtained through the MobieNetV2 network unit, and the fire image classification model is trained;
[0096] S4. Establish a fire detection alarm model: including a fire condition definition unit and a fire monitoring alarm unit, and determine whether to trigger the fire alarm in the fire monitoring alarm unit according to the fire condition in the fire condition definition unit.
[0097] In a specific implementation manner, S1 is as follows:
[0098] S1.1, Audio Dataset:
[0099] The audio data set is recorded in the simulated campus fire scene. Specifically, the audio is recorded by the Audio-Technica AT2020 microphone driven by Behringer UM2 sound at a distance of 2 to 10 meters from the fire source. The data length of each audio segment is 5 seconds. The collected data is synthesized into an audio data set , ,in , , A represents the set of audio data points, a represents the set of audio category labels, T represents the number of audio data points and audio category labels, represents the i-th audio data point, Represents the i-th audio category label, each audio data point Corresponding to an audio category label ;
[0100] S1.2. Image dataset: The image dataset consists of two parts. One part is the images of the same scene when the audio data was collected, and the other part is the images obtained by fire simulation at different time periods under different weather conditions. The images of the two parts are adjusted to a uniform size and then synthesized into an image dataset. , ,in , , P represents the set of image data points, p represents the set of image category labels, T represents the number of image data points and image category labels, represents the i-th image data point, Represents the i-th image category label, each data point Corresponding to a category label .
[0101] In a specific implementation, the audio feature extraction unit:
[0102] The audio dataset The set A of audio data points in is input to the audio feature extraction unit, , respectively for the audio data points in the set A of audio data points Perform short-time Fourier transform to obtain the corresponding time-frequency matrix vector , audio data points The discrete representation in the time domain is , There are K discrete data information in total, Represents the kth discrete data information in the ith audio data point, and the short-time Fourier transform formula is defined as follows:
[0103] ,
[0104] in, Represents the time-frequency matrix vector obtained after short-time Fourier transform , j represents an imaginary number, Indicates frequency, h represents the window function, T represents the number of audio data points, Represents the sliding position of the window function, and the window function is defined as follows:
[0105] ,
[0106] Where n represents the input of the window function, represents the number of points of the Fourier transform;
[0107] Finally, the time-frequency matrix vector Combined into a time-frequency matrix ,in represents the i-th time-frequency matrix vector, the i-th time-frequency matrix vector Corresponding to the i-th audio data point ;
[0108] Then use the Mel filter to filter the time-frequency matrix Filter and get the Mel time-frequency matrix M , where r represents the number of Mel filter banks;
[0109] Then take the logarithm of M and get Q= , and do discrete cosine transform, the discrete cosine transform of the sth vector in Q is defined as follows:
[0110] ,
[0111] in, represents the sth vector in the Mel time-frequency matrix after taking the logarithm, j represents an imaginary number, Represents the number of points of discrete cosine transform. The result of discrete cosine transform is length The vector of , takes the 2nd to 13th values in the vector as features, and the final feature representation is ,in for The feature vectors, The length of the inner feature vector is 13.
[0112] In a specific implementation, the convolutional neural network classification unit in the fire audio classification model is composed of It consists of layers of convolutional layers and fully connected layers. The convolutional layers include convolution kernels, activation functions, maximum pooling layers, and Dropout layers.
[0113] No. The input feature map of the convolutional layer is , whose dimensions are , the convolution kernel size is , the step length is , the output feature map dimension is , whose dimensions are ,in, Respectively represent The height, width and depth of the layer features, , and Respectively represent l The height, width, and depth of the layer feature are calculated as follows:
[0114] ,
[0115] ,
[0116] ,
[0117] in, Indicates l The size of the convolution kernel of the layer, Indicates l The number of convolution kernels in the layer, represents the step size, in particular, when =1, = , = ;
[0118] for Each value in is activated using ReLU as the activation function to obtain the activated features , the specific process is defined as follows:
[0119] ,
[0120] in, express The value in Indicates Layer dimension, max If the input is greater than 0, the original value is output; if the input is less than or equal to 0, 0 is output;
[0121] No. l The pooling window size of the maximum pooling layer is , the step length is ;
[0122] Then the output of the activation function After being sent to the maximum pooling layer, the dimension of the output feature after the maximum pooling layer is:
[0123] ,
[0124] ,
[0125] in, Indicates l The height of the max pooling layer, Indicates l The width of the max pooling layer, Indicates l The step size of the maximum pooling layer is used to obtain the output of the pooling layer. ;
[0126] Then the output of the maximum pooling layer The Dropout layer randomly discards some neurons in the network during training to reduce the risk of overfitting. The The value is calculated as follows:
[0127] ,
[0128] represents random probability, Indicates the probability of setting the discard, and finally the Dropout layer outputs the feature map ;
[0129] The output of the Dropout layer Send it to the fully connected layer, the fully connected layer will have the dimension of Flatten to a vector of dimension 1 , the flattened feature vector and the weight matrix of the fully connected layer Multiply and add the bias , the operation of the fully connected layer is defined as follows:
[0130] ,
[0131] The output of the fully connected layer Input into the Softmax activation function, and finally get the predicted result. The predicted probability of each category is defined as follows:
[0132]
[0133] in, represents the number of categories, Indicates categories.
[0134] In a specific implementation, the fire audio classification model is trained:
[0135] Input the data in the image data point set A into the fire audio classification model for training and calculate the prediction results , using the cross entropy loss function Calculate the prediction results with the actual audio category labels The difference is defined as follows:
[0136] ,
[0137] The gradient is calculated according to the loss function, and the gradient is passed back to each layer of the network through the back-propagation algorithm. The optimization algorithm Adam is used to update the weights and bias parameters of the model according to the calculated gradient.
[0138] In a specific implementation, the fire image classification model is constructed as follows:
[0139] The MobieNetV2 network is used as the backbone structure of the fire image classification network unit, which includes a Bottleneck layer and a fully connected layer. The Bottleneck layer is defined as a convolutional block. The convolutional block includes a total of B layers. The input feature dimension of the bth layer is , the number of convolutional neurons is , the convolution kernel size is , the step length is , the output feature map dimension is ;
[0140] in, Respectively represent -1 layer features height, width and depth, , and Respectively represent Height, width, and depth of layer features;
[0141] After each convolution layer except the last one, the ReLU activation function will be used to process the output features. After the ReLU activation function, a maximum pooling layer will follow. The first and third layers use ordinary convolution, and the second layer uses DW convolution. After the third convolution layer, the residual of the convolution result and the input is used as the final output of the Bottleneck convolution block.
[0142] The pooling window size of the b-th pooling layer is , the step length is ;
[0143] The MobieNetV2 network is composed of multiple Bottleneck convolution blocks and other stacked structures. The MobieNetV2 network has a total of M layers, and the input feature dimension of the mth layer is , the number of convolutional neurons is , the convolution kernel size is , the step length is , the number of module repetitions is , the output feature map dimension is ;
[0144] The last layer in the MobieNetV2 network is a fully connected layer, which receives the output of the previous layer's Bottleneck convolutional block. As input. The fully connected layer takes the dimension of Flatten to a vector of dimension 1 , the flattened feature vector and the weight matrix of the fully connected layer 'Multiply and add the bias term ', the operation of the fully connected layer is defined as follows:
[0145] ',
[0146] The output of the fully connected layer Input into the Softmax activation function, and finally get the predicted result. The predicted probability of the λth category is defined as follows:
[0147]
[0148] in, represents the number of categories, Indicates categories.
[0149] In a specific implementation, the fire image classification model is trained:
[0150] Input the data in the image data point set P into the fire image classification model for training and calculate the prediction results , using the cross entropy loss function calculate The difference between the actual image category label p,
[0151] Then its definition is as follows:
[0152] ,
[0153] The gradient is calculated according to the loss function, and the gradient is passed back to each layer of the network through the back-propagation algorithm. The optimization algorithm Adam is used to update the weights and bias parameters of the model according to the calculated gradient.
[0154] In a specific implementation manner, the fire condition definition unit:
[0155] The convolutional neural network classification unit will judge the input audio and output whether it meets the audio fire alarm condition. The MobieNetV2 network unit will judge the input image and output whether it meets the image fire alarm condition. The situation that meets the audio fire alarm condition is recorded as FcA, and the situation that meets the image fire alarm condition is recorded as FcP. Multiple continuous inferences are set as the basis for judgment at the same time, and the number of continuous inferences is recorded as , represents the input audio sample, represents the probability that the input audio sample belongs to fire, The minimum value among the audio samples that indicates the model's output belongs to fire , TP represents true positive, FP represents false positive, For a given The minimum number of consecutive TPs in For a given The maximum number of consecutive FPs in * Determined maximum ,but and Satisfies the following formula:
[0156] ,
[0157] ,
[0158] ,
[0159] If the above situation occurs, FP will not be generated, avoiding the false negative situation of FN. False negative means that the predicted sample is a negative sample, but it is actually a positive sample.
[0160] In the fire detection scenario, the most dangerous situations are:
[0161] ,
[0162] *Accurate range:
[0163] ,
[0164] FcA / FcP * / *、 * / * indicates that the model needs to be at least continuous * / * Output fire is higher than or equal to * / *Probability, the specific process of calculating FcA / FcP is:
[0165] Setting parameters or The range and step size of the change are or , according to the calculation, the corresponding and , and meet FcA / FcP * / *and * / *.
[0166] In a specific implementation, the fire detection and alarm unit:
[0167] Run the fire audio classification model to infer the received audio signal and determine whether the model output meets FcA. If it does not meet FcA, the model continues to monitor; otherwise, calculate and store the average value of the model output that meets FcA audio. The fire audio classification model is paused, and then the fire picture classification model is run to calculate whether the current picture result meets FcP. If not, the fire picture classification model stops running and the fire audio classification model is awakened; otherwise, the average value of the model output that meets FcP is calculated and stored. And sounded a fire alarm.
[0168] The specific implementation operations are:
[0169] Step 1, such as Figure 1 The audio fire classification model shown in FIG. Specifically, the audio is first processed with a sliding window length of 4s and a step length of 2s; then, a short-time Fourier transform is performed to obtain the transformed data. The specific parameters of the short-time Fourier transform are ; After that, the transformed data is filtered using a Mel filter; the filtered data is subjected to cepstrum analysis, which first takes the inverse of the filtered data and then performs a discrete cosine transform, and its specific parameters are , the 2nd to 13th coefficients of the transformed result are taken as audio features and input into the convolutional neural network. The number of convolutional layers of the convolutional neural network is 4. The number of neurons in the first convolutional layer is , convolution kernel size , step length ; The number of neurons in the second convolutional layer is , convolution kernel size , step length ; The number of neurons in the third convolutional layer is 32, convolution kernel size , step length ; The number of neurons in the 4th convolutional layer is , convolution kernel size , step length ; The pooling layer window size is , step length ; The probability p of the Dropout layer is set to 0.5, the number of neurons in the fully connected layer is set to 64, and the output of the fully connected layer is input into the Softmax activation function to output the predicted probability of fire.
[0170] Step 2: Use the audio dataset D1 to train the audio fire classification model. 1 Divide into audio training set, audio test set and audio verification set in the ratio of 6:2:2. Set the learning rate of the back propagation algorithm to 0.005. The training can be terminated after reaching the preset number of iterations (Epochs). At the same time, in order to prevent overfitting, set the early stopping mechanism: if the performance of the model on the verification set does not improve significantly in several consecutive Epochs, the training will be terminated early. The preset number of iterations is 1000, and the number of consecutive iterations for early termination of training is 20.
[0171] Step 3: Image fire classification model. The image fire classification model is mainly composed of Bottleneck convolution blocks. The number of Bottleneck convolution layers is 3. The number of neurons in the first convolutional layer is , convolution kernel size , step length ; The number of neurons in the second convolutional layer is , convolution kernel size , step length ; The number of neurons in the third convolutional layer is 64, convolution kernel size , step length ;
[0172] Specifically, in the image fire classification model, M=12 and the number of neurons in the first convolutional layer is , convolution kernel size , step length , module repetition times = 1; the second layer of Bottleneck convolution block, stride 1. Module repetition times = 1; the third layer of Bottleneck convolution block, stride 2. Module repetition times = 2; 4th layer Bottleneck convolution block, stride 2. Module repetition times = 3; 5th layer Bottleneck convolution block, stride 2. Module repetition times = 4; the sixth layer of Bottleneck convolution block, stride 1. Module repetition times = 1; 7th layer Bottleneck convolution block, stride 2. Module repetition times = 3; 8th layer Bottleneck convolution block, stride 1. Module repetition times = 1; the number of neurons in the 9th convolutional layer is , convolution kernel size , step length , module repetition times = 1; the pooling window size of the 10th average pooling layer is 7. Step length ; The number of neurons in the 11th convolutional layer is , convolution kernel size , step length , module repetition times = 1; the 12th layer is a fully connected layer, and the number of neurons is set to 128.
[0173] Step 4: Use the image dataset Train the image fire classification model. Specifically, the image dataset 2 The data is divided into audio training set, audio test set and audio verification set in the ratio of 6:2:2. The learning rate of the back propagation algorithm is set to 0.000045. The training can be terminated after reaching the preset number of iterations. At the same time, in order to prevent overfitting, an early stopping mechanism is set: if the performance of the model on the verification set does not improve significantly in several consecutive Epochs, the training is terminated early. The preset number of iterations is 3000, and the number of consecutive iterations for early termination of training is 100.
[0174] Step 5, the smart campus fire alarm system based on multimodal event detection, after starting the system, first runs the audio fire classification model, monitors the current audio in real time and determines whether it satisfies FcA. If it satisfies FcA, the average output of the model is calculated and stored, and the image fire classification model is awakened at the same time; otherwise, continue to monitor, and after waking up the image fire classification model, take the current image as input and determine whether it satisfies FcP. If it satisfies FcP, the average output of the model is calculated and stored, and a fire alarm is issued; otherwise, wake up the audio fire classification model and repeat the above process.
[0175] Specifically, the process of calculating FcA / FcP is to / Change from 0.6 to 1 in steps of 0.001 and calculate the corresponding and , the result is that the condition that satisfies FcA is *=0.808 and *=2, which means that the model needs to output the fire probability higher than or equal to 0.808 for at least 2 consecutive times; the condition for satisfying FcP is *=0.808 and *=4.
[0176] Example 2
[0177] A smart campus fire alarm system based on multimodal event detection, which performs a smart campus fire alarm method based on multimodal event detection, specifically includes the following units:
[0178] Data acquisition unit: simulate the fire situation in the campus environment and obtain the data as a data set;
[0179] Audio classification unit: A fire audio classification model based on a convolutional neural network is established. The fire audio classification model includes an audio feature extraction unit and a convolutional neural network classification unit. The final classification result is obtained through convolution of the convolutional neural network classification unit, and the fire audio classification model is trained.
[0180] Image classification unit: Establish a fire image classification model, which includes a DW convolution unit and a MobieNetV2 network unit. The final prediction result is obtained through the MobieNetV2 network unit, and the fire image classification model is trained;
[0181] Fire detection alarm unit: The fire detection alarm unit includes a fire condition definition unit and a fire monitoring alarm unit. It determines whether to trigger the fire alarm in the fire monitoring alarm unit according to the fire conditions in the fire condition definition unit.
[0182] The specific operation process is:
[0183] The data collection unit will involve the synthesis of corresponding audio and image data sets that simulate fire situations in campus environments. The data sets include audio data sets and image data sets. The audio data sets record the audio in the fire scene within a predetermined distance through a specific device, and the image data sets include images of the same scene and fire simulation images under different weather conditions. Each sample in the data set has a corresponding label indicating whether the sample is a fire.
[0184] The audio classification unit is equipped with an audio data acquisition module and a fire audio classification model. The audio data acquisition module consists of a single microphone and can capture the audio in the environment in real time. The fire audio classification model uses a convolutional neural network as the main structure to classify the input audio to output the probability that the audio is related to the fire. The model training process uses the audio samples and corresponding labels in the audio data set for supervised learning, so that the model can accurately distinguish between fire audio and non-fire audio.
[0185] The image classification unit is provided with an image data acquisition module and a fire image classification model. The image data acquisition module is composed of a single camera and can capture images in the environment in real time. The fire image classification model adopts the MobileNetV2 network structure to classify the input image to output the probability of the image being related to the fire. During the model training process, supervised learning is performed using image samples and corresponding labels in the image data set so that the model can accurately distinguish between fire audio and non-fire images. Before inputting the model, the image will be resized to a uniform size to facilitate model training.
[0186] The fire detection alarm unit is provided with a fire condition definition unit and a fire monitoring alarm unit. The fire condition definition unit is divided into two parts, namely, an audio fire alarm condition and an image fire alarm condition. The fire detection alarm unit operates 24 hours a day and includes a fire audio monitoring system and a fire image monitoring system. The audio data acquisition module in the fire audio monitoring system first continuously collects audio signals through a microphone, and sends the received audio signals to the fire audio classification model for inference at regular intervals. The output of the fire audio classification model is used to determine whether the audio fire alarm condition in the fire condition definition unit is met. If the condition is met, the fire audio monitoring system is paused, the fire image monitoring system is started, and the output of the fire audio classification model is saved. Otherwise, the fire audio monitoring system continues to run. After the fire image monitoring system is awakened, the camera of the image data acquisition module is first used to collect image data and sent to the fire image classification model. The fire image classification model infers the received image data, and determines whether the image fire alarm condition in the fire condition definition unit is met according to the output of the fire image classification model. If the condition is met, an alarm signal is issued and relevant personnel are notified to respond to the emergency. The output of the fire image classification model is saved. Otherwise, the fire image monitoring system is paused and the fire audio monitoring system is started.
[0187] Although the above describes the specific implementation mode of the invention in conjunction with the drawings, it is not intended to limit the scope of protection of the invention. Based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.
Claims
1. A smart campus fire alarm method based on multimodal event detection, characterized in that: The following steps are involved: S1. Prepare data set: simulate a fire situation in a campus environment and obtain the data therein as a data set, which includes an audio data set and an image data set; Audio Dataset: The audio data set is recorded in the simulated campus fire scene. Specifically, the audio is recorded by the Audio-Technica AT2020 microphone driven by Behringer UM2 sound at a distance of 2 to 10 meters from the fire source. The data length of each audio segment is 5 seconds. The collected data is synthesized into an audio data set , ,in , , A represents the set of audio data points, a represents the set of audio category labels, T represents the number of audio data points and audio category labels, represents the i-th audio data point, Represents the i-th audio category label, each audio data point Corresponding to an audio category label ; S2. Establish a fire audio classification model and train the model: establish a fire audio classification model based on a convolutional neural network, the fire audio classification model includes an audio feature extraction unit and a convolutional neural network classification unit, obtain the final classification result through convolution of the convolutional neural network classification unit, and train the fire audio classification model; Audio feature extraction unit: The audio dataset The set A of audio data points in is input to the audio feature extraction unit, , respectively for the audio data points in the set A of audio data points Perform short-time Fourier transform to obtain the corresponding time-frequency matrix vector , audio data points The discrete representation in the time domain is , There are K discrete data information in total, Represents the kth discrete data information in the ith audio data point, and the short-time Fourier transform formula is defined as follows: , in, Represents the time-frequency matrix vector obtained after short-time Fourier transform , j represents an imaginary number, Indicates frequency, h represents the window function, T represents the number of audio data points, Represents the sliding position of the window function, and the window function is defined as follows: , Where n represents the input of the window function, represents the number of points of the Fourier transform; Finally, the time-frequency matrix vector Combined into a time-frequency matrix ,in represents the i-th time-frequency matrix vector, the i-th time-frequency matrix vector Corresponding to the i-th audio data point ; Then use the Mel filter to filter the time-frequency matrix Filter and get the Mel time-frequency matrix M , where r represents the number of Mel filter banks; Then take the logarithm of M and get Q= , and do discrete cosine transform, the discrete cosine transform of the sth vector in Q is defined as follows: , in, represents the sth vector in the Mel time-frequency matrix after taking the logarithm, j represents an imaginary number, Represents the number of points of discrete cosine transform. The result of discrete cosine transform is length The vector of , takes the 2nd to 13th values in the vector as features, and the final feature representation is ,in for The feature vectors, The length of the internal feature vector is 13; The convolutional neural network classification unit in the fire audio classification model consists of It consists of layers of convolutional layers and fully connected layers. The convolutional layers include convolution kernels, activation functions, maximum pooling layers, and Dropout layers. No. The input feature map of the convolutional layer is , whose dimensions are , the convolution kernel size is , the step length is , the output feature map dimension is , whose dimensions are ,in, Respectively represent The height, width and depth of the layer features, , and Respectively represent l The height, width, and depth of the layer feature are calculated as follows: , , , in, Indicates l The size of the convolution kernel of the layer, Indicates l The number of convolution kernels in the layer, represents the step size, in particular, when =1, = , = ; for Each value in is activated using ReLU as the activation function to obtain the activated features , the specific process is defined as follows: , in, express The value in Indicates Layer dimension, max If the input is greater than 0, the original value is output; if the input is less than or equal to 0, 0 is output; No. l The pooling window size of the maximum pooling layer is , the step length is ; Then the output of the activation function After being sent to the maximum pooling layer, the dimension of the output feature after the maximum pooling layer is: , , in, Indicates l The height of the max pooling layer, Indicates l The width of the max pooling layer, Indicates l The step size of the maximum pooling layer is used to obtain the output of the pooling layer. ; Then the output of the maximum pooling layer The Dropout layer randomly discards some neurons in the network during training to reduce the risk of overfitting. The The value is calculated as follows: , represents random probability, Indicates the probability of setting the discard, and finally the Dropout layer outputs the feature map ; The output of the Dropout layer Send it to the fully connected layer, the fully connected layer will have the dimension of Flattened to a vector of dimension 1 , the flattened feature vector and the weight matrix of the fully connected layer Multiply and add the bias , the operation of the fully connected layer is defined as follows: , The output of the fully connected layer Input into the Softmax activation function, and finally get the predicted result. The predicted probability of each category is defined as follows: in, represents the number of categories, Indicates categories; S3. Establish a fire image classification model and train the model: The fire image classification model includes a DW convolution unit and a MobieNetV2 network unit. The final prediction result is obtained through the MobieNetV2 network unit, and the fire image classification model is trained; S4. Establish a fire detection alarm model: including a fire condition definition unit and a fire monitoring alarm unit, and determine whether to trigger a fire alarm in the fire monitoring alarm unit according to the fire condition in the fire condition definition unit; Fire condition definition unit: The convolutional neural network classification unit will judge the input audio and output whether it meets the audio fire alarm condition. The MobieNetV2 network unit will judge the input image and output whether it meets the image fire alarm condition. The situation that meets the audio fire alarm condition is recorded as FcA, and the situation that meets the image fire alarm condition is recorded as FcP. Multiple continuous inferences are set as the basis for judgment at the same time, and the number of continuous inferences is recorded as , represents the input audio sample, represents the probability that the input audio sample belongs to fire, The minimum value among the audio samples that indicates the model's output belongs to fire , TP represents true positive, FP represents false positive, For a given The minimum number of consecutive TPs in For a given The maximum number of consecutive FPs in * Determined maximum ,but and Satisfies the following formula: , , , If the above situation occurs, FP will not be generated, avoiding the false negative situation of FN. False negative means that the predicted sample is a negative sample, but it is actually a positive sample. In the fire detection scenario, the most dangerous situations are: , *Accurate range: , FcA / FcP * / *、 * / * indicates that the model needs to be at least continuous * / * Output fire is higher than or equal to * / *Probability, the specific process of calculating FcA / FcP is: Setting parameters or The range and step size of the change are or , according to the calculation, the corresponding and , and meet FcA / FcP * / *and * / *; Fire detection and alarm unit: Run the fire audio classification model to infer the received audio signal and determine whether the model output meets FcA. If it does not meet FcA, the model continues to monitor; otherwise, calculate and store the average value of the model output that meets FcA audio The fire audio classification model is paused, and then the fire picture classification model is run to calculate whether the current picture result meets FcP. If not, the fire picture classification model stops running and the fire audio classification model is awakened; otherwise, the average value of the model output that meets FcP is calculated and stored. And sounded a fire alarm.
2. According to the method of intelligent campus fire alarm based on multimodal event detection according to claim 1, it is characterized in that: S1 is as follows: Image dataset: The image dataset consists of two parts. One part is the images of the same scene when the audio data was collected, and the other part is the images obtained by fire simulation at different time periods under different weather conditions. The images of the two parts are adjusted to a uniform size and then synthesized into an image dataset. , ,in , , P represents the set of image data points, p represents the set of image category labels, T represents the number of image data points and image category labels, represents the i-th image data point, Represents the i-th image category label, each data point Corresponding to a category label .
3. According to the method of intelligent campus fire alarm based on multimodal event detection according to claim 2, it is characterized in that: Training the fire audio classification model: Input the data in the audio data point set A into the fire audio classification model for training and calculate the prediction results , using the cross entropy loss function Calculate the prediction results with the actual audio category labels The difference is defined as follows: , The gradient is calculated according to the loss function, and the gradient is passed back to each layer of the network through the back-propagation algorithm. The optimization algorithm Adam is used to update the weights and bias parameters of the model according to the calculated gradient.
4. According to the method of intelligent campus fire alarm based on multimodal event detection according to claim 3, it is characterized in that: Construction of fire image classification model: The MobieNetV2 network is used as the backbone structure of the fire image classification network unit, which includes a Bottleneck layer and a fully connected layer. The Bottleneck layer is defined as a convolutional block. The convolutional block includes a total of B layers. The input feature dimension of the bth layer is , the number of convolutional neurons is , the convolution kernel size is , the step length is , the output feature map dimension is ; in, Respectively represent -1 layer features height, width and depth, , and Respectively represent Height, width, and depth of layer features; After each convolution layer except the last one, the ReLU activation function will be used to process the output features. After the ReLU activation function, a maximum pooling layer will follow. The first and third layers use ordinary convolution, and the second layer uses DW convolution. After the third convolution layer, the residual of the convolution result and the input is used as the final output of the Bottleneck convolution block. The pooling window size of the b-th pooling layer is , the step length is ; The MobieNetV2 network is composed of multiple Bottleneck convolution blocks and other stacked structures. The MobieNetV2 network has a total of M layers, and the input feature dimension of the mth layer is , the number of convolutional neurons is , the convolution kernel size is , the step length is , the number of module repetitions is , the output feature map dimension is ; The last layer in the MobieNetV2 network is a fully connected layer, which receives the output of the previous layer's Bottleneck convolutional block. As input, the fully connected layer takes the dimension of Flatten to a vector of dimension 1 , the flattened feature vector and the weight matrix of the fully connected layer 'Multiply and add the bias term ', the operation of the fully connected layer is defined as follows: ', The output of the fully connected layer Input into the Softmax activation function, and finally get the predicted result. The predicted probability of the λth category is defined as follows: in, represents the number of categories, Indicates categories.
5. According to the method of intelligent campus fire alarm based on multimodal event detection according to claim 4, it is characterized in that: Train the fire image classification model: Input the data in the image data point set P into the fire image classification model for training and calculate the prediction results , using the cross entropy loss function calculate The difference between the actual image category label p, Then its definition is as follows: , The gradient is calculated according to the loss function, and the gradient is passed back to each layer of the network through the back-propagation algorithm. The optimization algorithm Adam is used to update the weights and bias parameters of the model according to the calculated gradient.
6. A smart campus fire alarm system based on multimodal event detection, characterized by: A smart campus fire alarm method based on multimodal event detection according to any one of claims 1 to 5 is implemented, specifically comprising the following units: Data acquisition unit: simulate the fire situation in the campus environment and obtain the data as a data set; Audio classification unit: A fire audio classification model based on a convolutional neural network is established. The fire audio classification model includes an audio feature extraction unit and a convolutional neural network classification unit. The final classification result is obtained through convolution of the convolutional neural network classification unit, and the fire audio classification model is trained. Image classification unit: Establish a fire image classification model, which includes a DW convolution unit and a MobieNetV2 network unit. The final prediction result is obtained through the MobieNetV2 network unit, and the fire image classification model is trained; Fire detection alarm unit: The fire detection alarm unit includes a fire condition definition unit and a fire monitoring alarm unit. It determines whether to trigger the fire alarm in the fire monitoring alarm unit according to the fire conditions in the fire condition definition unit.
Citation Information
Patent Citations
Method, software, and device for training an alarm system to classify audio of an event
CN113314146A
Cable fire early-stage heating sound signal identification method based on deep learning
CN117116278A
Multi-mode fire monitoring method for intelligent fire fighting
CN117373190A
Fire video image recognition method and system, computer device, and storage medium
WO2023184350A1
