Fire early warning method based on multi-source information fusion, system, and medium

Abstract

A fire early warning method based on multi-source information fusion, a system, and a medium. The method comprises single vehicle model training, model deployment, fire risk assessment, BIM display and anomaly information notification. In the fire early warning method based on multi-source information fusion, the system, and the medium, a vehicle temperature fault mode can be quickly and accurately identified by performing deep fusion and model training on image and location and temperature information of the vehicle, thereby improving the accuracy of fire early warning; and multi-source information is acquired in real time, such that when over temperature occurs in the vehicle, workers can quickly localize the vehicle and the vicinity thereof, so that measures can be taken promptly for the over-temperature vehicle, thereby preventing the occurrence of fires or reducing the severity of fire damage, and providing strong support for fire prevention and control work. By establishing a bridge BIM model, a bridge fire risk map is displayed on a display device or a user terminal, and a command department, workers and drivers can quickly ascertain information regarding the location, cause, severity, and vehicle density of a current bridge fire by means of the bridge fire risk map; the system has high environment adaptability, fire risk assessment can be carried out in different environments, the decision-making efficiency of the command department is improved, the workers can quickly arrive at a rescue site to carry out firefighting and rescue operations, and drivers in low-risk areas can quickly evacuate from the bridge or assist the workers in evacuating others, thereby reducing casualties and economic losses caused by fires as much as possible.

Description

Fire warning method and system based on multi-source information fusion and medium TECHNICAL FIELD

[0001] The present application relates to the technical field of fire warning, in particular to a fire warning method and system based on multi-source information fusion of image recognition technology, radar technology, deep learning, thermal imaging technology and BIM modeling technology and a medium. BACKGROUND

[0002] With the accelerated progress of urbanization, bridges, as important transportation hubs connecting cities and regions, play an increasingly important role in the transportation network. However, the risk of fire accidents on bridges also increases. Due to the structure of the bridge, when a fire occurs, personnel cannot timely obtain disaster information and it is extremely difficult to escape. Fire can also damage the bridge structure, affecting the carrying capacity and stability of the bridge, and bringing great challenges to maintaining public safety and the stability of the bridge structure.

[0003] Therefore, it is necessary to monitor and warn of bridge fires. Traditional bridge fire detection methods mainly rely on manual inspection and regular maintenance inspection. Although this method can to some extent find potential fire hazards, manual inspection and regular maintenance inspection are limited by factors such as the number of personnel, personnel professional competence and weather conditions, and have problems such as slow response, low detection efficiency and incomplete detection, which cannot achieve real-time monitoring of bridges in all-weather and all directions. Moreover, the traditional bridge fire detection method has strong hysteresis. When manual inspection detects abnormal conditions, the fire has often accumulated to a certain extent, and early warning cannot be performed. When a fire occurs, due to the lack of information sharing, it is difficult for staff to notify each person on the bridge in real time, increasing the difficulty of fire evacuation and rescue. SUMMARY

[0004] The present application aims to overcome at least one of the above-mentioned defects of the prior art, and provides a fire warning method, system and medium based on multi-source information fusion, which solves the technical problem that the prior art cannot monitor bridge fires in real time, provide early warning and notify personnel on the bridge in real time.

[0005] The present application provides a fire warning method based on multi-source information fusion, which comprises single vehicle model training, model deployment, fire risk assessment, BIM modeling display and abnormal information notification.

[0006] The single vehicle model training comprises:

[0007] S1, obtaining image data of license plates and dangerous chemical product identification of a single vehicle, point cloud data of real-time positioning and speed of the vehicle and temperature thermal imaging data of the vehicle to form vehicle data;

[0008] S2, vehicle data preprocessing and preliminary data fusion are performed on the vehicle data to generate fused vehicle monitoring data;

[0009] S3, the fused vehicle monitoring data is uploaded and stored;

[0010] S4, high-level feature extraction and deep fusion are performed on the vehicle monitoring data by a deep learning convolutional neural network, and model training is performed on the deep fusion convolutional neural network to identify fire failure modes.

[0011] Since it is necessary to monitor the fire risk of the bridge in real time and give early warning to suppress the fire in the early stage, personnel casualties and economic losses can be reduced, vehicle accidents and external environment are the main causes of bridge fire, therefore, by collecting vehicle data, a deep learning convolutional neural network is used to train a model for vehicles on the bridge, and the trained model is used to monitor the bridge vehicle situation in real time, so as to realize the early warning of bridge fire.

[0012] High-definition camera technology is used to collect and identify the license plate of a single vehicle and the dangerous chemical product identification, which is used for subsequent vehicle matching and failure mode identification. Laser radar technology is used to accurately position and measure the speed of the vehicle, so as to realize real-time acquisition of vehicle positioning and speed data in the radar measurement field area. Infrared thermal imaging technology is used for vehicle temperature measurement and pseudo-color image generation to find vehicle over-temperature abnormal conditions. The image data, point cloud data and thermal imaging data form vehicle data.

[0013] The initially obtained vehicle data may have unclear images, blurred identification and other situations, and needs to be preprocessed. The vehicle data preprocessing includes: image feature extraction is performed on the image data to identify the type of vehicle, point cloud data is filtered and point trace clustering is performed to optimize point cloud data and accurately identify the three-dimensional position of the vehicle, and temperature calibration and image enhancement are performed on the thermal imaging data to obtain clear temperature images.

[0014] The preprocessed vehicle data still has the problem of scattered data, which cannot be directly input into the deep learning convolutional neural network for model training, and needs to be preliminarily fused to generate fused vehicle monitoring data for uploading and storage.

[0015] The vehicle monitoring data is input into the deep learning neural network for high-level feature extraction, including: using a deep convolutional neural network to extract high-level visual features such as vehicle contours and vehicle information from video frames, using a Point Net point cloud processing algorithm to extract three-dimensional shape and position features of the vehicle from radar point cloud data, and using a deep convolutional neural network to extract temperature distribution features from thermal imaging data. The extracted high-level features are deeply fused, and a model is trained through a deep convolutional neural network to identify fire failure modes, and a cross-entropy loss function is used for model optimization, Adam optimization algorithm is used to update model parameters, and validation set data is used to evaluate the performance of the trained model, including accuracy, recall rate, F1 score and other indicators.

[0016] After training a single vehicle model, the model can accurately identify fire failure modes in real time, and can be deployed, including:

[0017] S5, deploying the trained model in the model in the deep learning processing unit for real-time fire failure mode identification of newly collected vehicle data.

[0018] The single vehicle model can identify the fire failure mode and obtain information and real-time positioning of the vehicle, but it does not have predictive ability for the overall bridge fire situation and cannot obtain the disaster degree of each position of the bridge, so it is necessary to perform fire risk assessment on the entire bridge system, including:

[0019] S6, collecting system data of the current bridge environment, and performing fire risk assessment and fire risk level classification through the system data and the fire failure mode identification result.

[0020] Through the fire risk assessment, the disaster degree of each position of the bridge can be obtained, in order to further facilitate the command department, fire personnel and personnel on the bridge to grasp the bridge fire situation at the first time, the BIM modeling can be used to display the bridge fire risk map. The BIM modeling display includes:

[0021] S7, according to the system data and the fire failure mode result, a bridge BIM model is modeled, the fire risk assessment result is mapped to the bridge BIM model, and a bridge fire risk map is generated.

[0022] The bridge fire risk map can be displayed on a display device and a user terminal device of a monitoring center, a command department can make quick and effective decisions according to the bridge fire risk map, and it is convenient for fire workers and bridge personnel to understand the fire situation, reduce the human cost of fire situation notification, and quickly launch fire rescue work to control the disaster in the early stage. The bridge fire risk map is displayed through the BIM model, and the over-temperature and fire information are also broadcasted through the bridge information board and the loudspeaker again to ensure that the abnormal information is notified to the driver.

[0023] Therefore, the abnormal information notification includes:

[0024] S8, according to the bridge fire risk map, the over-temperature information is notified to the driver through the bridge information board and the loudspeaker.

[0025] Since the vehicle monitoring data is based on multi-source information fusion, the image data, point cloud data and thermal imaging data in the vehicle monitoring data need to be extracted through deep learning convolutional neural network respectively, and the extracted feature vectors are still in a separated state, and deep fusion of feature vectors is needed. Further extract high-level features through convolution and pooling, and then classify and identify the results of the fault mode through the full connection layer.

[0026] Therefore, S4, the high-level feature extraction and deep fusion of the vehicle monitoring data through the deep learning convolutional neural network, and the model training of the deep fusion convolutional neural network to identify the fire fault mode further include:

[0027] S41, the vehicle monitoring data is subjected to convolution operation to extract feature vectors: F I = CNN I (I) F P = CNN P (P) F T = CNN T (T)

[0028] Among them, CNN I , CNN P , CNN T respectively represent the first convolutional neural network for processing the image data, the point cloud data and the thermal imaging data, F I , F P , F T are the extracted feature vectors.

[0029] S42, the extracted feature vectors are subjected to the deep fusion: F fused = [F I , F P , F T ]

[0030] wherein F fused is the deep fused feature vector;

[0031] S43, further convolution and pooling operations are performed on the deep fused feature vector to extract high-level features: F high = CNN high (F fused )

[0032] wherein CNN high represents a second convolutional neural network of the deep fused feature, F high is the extracted high-level features;

[0033] The high-level features are input into a fully connected layer for fully connected layer classification to obtain the fire fault mode recognition result: y = FC(high)

[0034] wherein FC is a fully connected layer, and y is the output fire fault mode recognition result.

[0035] When the single vehicle model is trained, the vehicle data is multi-source information from a high-definition camera, a laser radar and an infrared thermal imaging camera respectively, and it cannot be guaranteed that the high-definition camera and the laser radar obtain the same target. If the vehicle data is directly subjected to preliminary data fusion in step S3, the subsequent model training result may be inaccurate or errors may occur. Therefore, the targets obtained by the high-definition camera and the laser radar need to be associated for matching to determine whether the targets obtained by different devices are the same target. Therefore, the preliminary fusion of the vehicle data in S2 further includes:

[0036] The vehicle data is subjected to preliminary data fusion by an association degree-based fusion algorithm, a target association degree threshold is set, and the association degrees between past vehicle targets are calculated:

[0037] wherein S R is a radar detection target area, O R is a center point of the area; S P is a video detection target area, O P is a center point of the area; O τ , S τ is a target association degree threshold; ΔO is a center distance of the radar and the video target area, and IOU is an intersection over union of the target areas;

[0038] When the center distance and the area intersection ratio of the target region exceed the target correlation threshold, the preliminary data fusion is performed on the vehicle data to generate the preliminary fused vehicle monitoring data, which can ensure that the preliminary data fusion vehicle data comes from the same target;

[0039] When the target correlation does not exceed the target correlation threshold, it indicates that the obtained vehicle data is not the vehicle data of the same target or is uncertain, and the thermal imaging data is output alone or the thermal imaging data and the point cloud data, the thermal imaging data and the image data are output as the vehicle monitoring data, thereby ensuring the accuracy of the single vehicle model training.

[0040] Since the main cause of the bridge fire is a vehicle accident, in the single vehicle model training, the real-time temperature of the vehicle is collected, and when the temperature of the vehicle exceeds a certain temperature value, it is determined that there is a fire risk in the region, and an over-temperature warning is performed, so that the temperature thermal imaging data obtained in S1 further includes:

[0041] The temperature of the single vehicle is monitored by an infrared thermal imaging camera, the temperature value obtained by the temperature monitoring is converted into a pseudo-color image, and the temperature value and the pseudo-color image are used as the temperature thermal imaging data of the vehicle. When it is monitored that the measured temperature exceeds the set threshold, an over-temperature warning is prompted, at this time, the region where the vehicle is located has not yet appeared a fire, but has the possibility of a fire, the staff can take timely measures through the license plate information, positioning information and the over-temperature warning of the vehicle, and the occurrence of the fire can be avoided. The temperature monitoring determines whether there is a temperature anomaly through a temperature fluctuation degree and a temperature anomaly positioning coefficient.

[0042] The temperature fluctuation degree is the temperature change in any time period in the radar region, which is used to measure the size of the temperature change, and the temperature fluctuation degree is calculated as follows:

[0043] Where, tf i is the temperature fluctuation degree in the i-th time period; t i,j is the temperature value of the j-th detection sub-region in the i-th time period; is the average temperature in the detection region in the i-th time period; n is the total number of sub-regions in the detection region;

[0044] The temperature anomaly positioning coefficient is used to measure the temperature anomaly and locate the over-temperature region, and the temperature anomaly positioning coefficient is calculated as follows:

[0045] When the temperature anomaly positioning coefficient Z i,j exceeds the threshold, the radar region has a temperature anomaly, and an over-temperature warning is prompted.

[0046] In order to make the bridge fire risk map generated in the bridge BIM model further show the fire situation of the bridge, so that the command department, staff and drivers can clearly understand the current fire situation, S7 further comprises: high-risk area and over-temperature vehicle labeling on the bridge fire risk map, providing visual fire risk distribution, and real-time display of early warning state and emergency measures through the monitoring center or user terminal.

[0047] Since the single vehicle model has the license plate, dangerous chemical type, vehicle speed, position and temperature information of each vehicle, the information of each vehicle can only be regarded as an independent individual, cannot reflect the overall fire situation of the bridge, and cannot grasp the vehicle situation near each vehicle, and the fire warning and decision-making need to be judged through the overall situation of the bridge at this time, therefore, the system data collected by the system for the current bridge environment in S6 at least includes one or more of bridge environment data such as vehicle speed, number of vehicles in transit, vehicle type, vehicle density, weather, ambient temperature and vehicle temperature, and the system data and the fire fault mode recognition result are used for fire risk assessment and fire risk level division, and the system data has comprehensive and guiding significance for fire warning.

[0048] In the single vehicle model training, the fault mode of the vehicle temperature is finally obtained through the deep learning convolutional neural network, and the fire fault mode at least includes one or more of data-labeled vehicle abnormal heating, non-dangerous goods vehicle abnormal high temperature, dangerous goods vehicle abnormal high temperature, non-dangerous goods vehicle fire and dangerous goods vehicle fire, facilitating model training and management, and the fire risk level is divided into one or more of low, medium and high fire risk levels according to historical records or expert experience, and the fire risk level is set in the bridge fire risk map, and the fire situation of each position of the bridge is clear.

[0049] The application also provides a fire warning system based on multi-source information fusion, which comprises a single vehicle model training module, a model deployment module, a fire risk assessment module, a BIM modeling display module and an abnormal information notification module, and is characterized in that:

[0050] The single vehicle model training module comprises:

[0051] The acquisition module acquires image data of the license plate and dangerous chemical identification of a single vehicle, point cloud data of real-time positioning and speed of the vehicle and temperature thermal imaging data of the vehicle, and forms vehicle data;

[0052] The fusion module performs vehicle data preprocessing and preliminary data fusion on the vehicle data to generate fused vehicle monitoring data;

[0053] The storage module uploads and stores the fused vehicle monitoring data.

[0054] The deep learning module: through deep learning convolutional neural network, high-level feature extraction and deep fusion are carried out on the vehicle monitoring data, the deep fusion convolutional neural network is model trained, and a fire fault mode is identified;

[0055] The model deployment module: the trained model is deployed in the model in the deep learning processing unit, and is used for real-time fire fault mode identification on newly collected vehicle data;

[0056] The fire risk assessment module: system data of the current bridge environment are collected, fire risk assessment and fire risk level division are carried out through the system data and the fire fault mode identification result;

[0057] The BIM modeling display module: through the system data and the fire fault mode result, a bridge BIM model is modeled, the fire risk assessment result is mapped to the bridge BIM model, a bridge fire risk map is generated, and a high-risk area is labeled;

[0058] The abnormal information notification module: according to the bridge fire risk map, over-temperature information is notified to the driver through a bridge bulletin board and a loudspeaker.

[0059] The application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is characterized in that the computer program is executed by a processor to implement the steps of the fire warning method based on multi-source information fusion according to any one of claims 1-8.

[0060] The fire warning method, system and medium based on multi-source information fusion have the advantages that through deep fusion and model training on the image, positioning and temperature information of the vehicle, the temperature fault mode of the vehicle can be quickly and accurately identified, the accuracy of fire warning is improved, multi-source information real-time acquisition can quickly locate the vehicle and the surrounding area when the vehicle has an over-temperature phenomenon, measures can be taken in time to prevent fire or reduce the disaster degree caused by fire, and strong support is provided for fire prevention and control work. By establishing a bridge BIM model, a bridge fire risk map is displayed on a display device or a user terminal, the command department, the staff and the driver can quickly grasp the disaster site, the fire cause, the disaster degree, the vehicle density and other information of the current bridge fire through the bridge fire risk map, the environmental adaptability of the system is strong, the fire risk can be evaluated in different environments, the decision-making efficiency of the command department is improved, the staff can quickly arrive at the rescue site to carry out fire extinguishing and rescue work, the drivers in the low-risk area can quickly evacuate from the bridge or assist the staff in evacuating people, and the personnel casualties and economic losses caused by fire are reduced as much as possible. BRIEF DESCRIPTION OF DRAWINGS

[0061] In order to more clearly illustrate the technical solutions of the present application or the prior art, the drawings required to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0062] Figure 1 is a step diagram of a fire warning method based on multi-source information fusion.

[0063] Figure 2 is a step diagram of model training of the vehicle monitoring data by a deep learning convolutional neural network.

[0064] Figure 3 is a working schematic diagram of a fire warning system based on multi-source information fusion.

[0065] Figure 4 is a working flowchart of a fire warning system based on multi-source information fusion.

[0066] The reference signs in the specification include: high-definition camera 1, stroboscopic fill light 2, laser radar 3, infrared thermal imaging camera 4, edge computing module 5, network switch 6, blockchain server 7, deep learning processing unit 8, BIM modeling workstation 9, monitoring center display device 10, user terminal device 11, gantry information board 12, directional loudspeaker 13. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solutions and advantages of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the drawings:

[0068] The embodiments of the present application provide an embodiment of a fire warning method and system based on multi-source information fusion. Although a logical sequence is shown in the flowchart, under certain data, the steps shown or described can be completed in an order different from that here.

[0069] Embodiment 1: Refer to Figure 1

[0070] As shown in the step diagram of the fire warning method based on multi-source information fusion of Figure 1, the present application provides a fire warning method based on multi-source information fusion. The present application provides a fire warning method based on multi-source information fusion, which includes single vehicle model training, model deployment, fire risk assessment, BIM modeling display and abnormal information notification.

[0071] The single vehicle model training includes:

[0072] S1, obtaining image data of license plate and dangerous chemical identification of a single vehicle, point cloud data of real-time positioning and speed of the vehicle, and temperature thermal imaging data of the vehicle, forming vehicle data;

[0073] S2, vehicle data preprocessing and preliminary data fusion are performed on the vehicle data to generate fused vehicle monitoring data;

[0074] S3, the fused vehicle monitoring data is uploaded and stored;

[0075] S4, high-level feature extraction and deep fusion are performed on the vehicle monitoring data by a deep learning convolutional neural network, and model training is performed on the deep fusion convolutional neural network to identify fire failure modes.

[0076] Since it is necessary to monitor the fire hazard of the bridge in real time and give early warning to suppress the fire in the early stage, the personnel casualties and economic losses can be reduced, and the vehicle accident and the external environment are the main causes of the bridge fire. Therefore, by collecting vehicle data and using a deep learning convolutional neural network to train a model for vehicles on the bridge, the trained model is used to monitor the bridge vehicle situation in real time, thereby realizing the early warning of the bridge fire.

[0077] High-definition camera technology is used to collect and identify the license plate of a single vehicle and the dangerous chemical product identification for subsequent vehicle matching and failure mode identification. Laser radar technology is used to accurately position and measure the speed of the vehicle to realize real-time acquisition of vehicle positioning and speed data within the radar measurement field area. Infrared thermal imaging technology is used for vehicle temperature measurement and pseudo-color image generation to find vehicle over-temperature anomalies. The image data, point cloud data and thermal imaging data form the vehicle data.

[0078] The initially obtained vehicle data may have unclear images and blurred identification, and needs to be preprocessed. The vehicle data preprocessing includes: image feature extraction on the image data to identify the type of vehicle, filtering and point cluster on the point cloud data to optimize the point cloud data and accurately identify the three-dimensional position of the vehicle, temperature calibration and image enhancement on the thermal imaging data to obtain clear temperature images.

[0079] The preprocessed vehicle data still has the problem of scattered data, which cannot be directly input into the deep learning convolutional neural network for model training. Therefore, preliminary data fusion is performed on the vehicle data to generate fused vehicle monitoring data for uploading and storage.

[0080] The vehicle monitoring data is input into the deep learning neural network for high-level feature extraction, including: using a deep convolutional neural network to extract high-level visual features such as vehicle contours and vehicle information from video frames, using a Point Net point cloud processing algorithm to extract three-dimensional shape and position features of the vehicle from radar point cloud data, and using a deep convolutional neural network to extract temperature distribution features from thermal imaging data. The extracted high-level features are deeply fused, and a model is trained through a deep convolutional neural network to identify fire failure modes, and a cross-entropy loss function is used for model optimization, Adam optimization algorithm is used to update model parameters, and validation set data is used to evaluate the performance of the trained model, including accuracy, recall rate, F1 score and other indicators.

[0081] After training a single vehicle model, the model can accurately identify fire failure modes in real time, and the model can be deployed, including:

[0082] S5, deploy the trained model in the model in the deep learning processing unit for real-time fire failure mode identification of newly collected vehicle data.

[0083] The single vehicle model can identify the fire failure mode and obtain information and real-time positioning of the vehicle, but it does not have predictive value for the overall bridge fire situation and cannot obtain the disaster degree of each position of the bridge, so it is necessary to perform fire risk assessment on the entire bridge system, including:

[0084] S6, collect system data of the current bridge environment, and perform fire risk assessment and fire risk level classification based on the system data and the fire failure mode identification result.

[0085] Through the fire risk assessment, the disaster degree of each position of the bridge can be obtained, in order to further facilitate the command department, fire personnel and personnel on the bridge to grasp the bridge fire situation at the first time, the bridge fire risk map can be displayed through the BIM modeling, including:

[0086] S7, according to the system data and the fire failure mode result, a bridge BIM model is modeled, the fire risk assessment result is mapped to the bridge BIM model, and a bridge fire risk map is generated.

[0087] The bridge fire risk map can be displayed on the monitoring center's display equipment and user terminal equipment. The command department can make quick and effective decisions based on the bridge fire risk map, and it is convenient for firefighters and people on the bridge to understand the fire situation, reducing the manpower cost of fire situation notification, and enabling rapid fire rescue work to control the disaster in the early stage. While displaying the bridge fire risk map through the BIM model, it is also necessary to broadcast the overheating and fire information again through bridge information boards and loudspeakers to ensure that abnormal information is notified to the drivers.

[0088] Therefore, the abnormal information notification includes:

[0089] S8. Based on the bridge fire risk map, notify the driver of the over-temperature information through the bridge information board and loudspeaker.

[0090] Example 2: Refer to Figure 2

[0091] Since the vehicle monitoring data is based on multi-source information fusion, the image data, point cloud data, and thermal imaging data in the vehicle monitoring data need to be extracted as feature vectors through deep learning convolutional neural networks. However, the extracted feature vectors are still separate and need to be deeply fused. Further high-level features can be extracted through convolution and pooling before the fault mode can be identified through a fully connected layer.

[0092] Therefore, the step S4, which involves performing advanced feature extraction and deep fusion of the vehicle monitoring data using a deep learning convolutional neural network, and training the deep fusion convolutional neural network to identify fire fault modes, further includes:

[0093] S41. Perform a convolution operation on the vehicle monitoring data to extract the feature vector: F I =CNN I (I) F P =CNN P (P) F T =CNN T (T)

[0094] Among them, CNN I CNN P CNN T F represents the first convolutional neural network used to process the image data, point cloud data, and thermal imaging data, respectively. I F P F T It is the extracted feature vector;

[0095] S42. Perform deep fusion on the extracted feature vectors: F fused =[F I ,FP F T ]

[0096] wherein, F fused is the feature vector after deep fusion;

[0097] S43, further convolution and pooling operations are performed on the deep fusion feature vector, and high-level features are extracted: F high =CNN high (F fused )

[0098] wherein, CNN high represents the second convolutional neural network of the deep fusion feature, F high is the extracted high-level feature;

[0099] The high-level feature is input into a fully connected layer for fully connected layer classification to obtain the fire fault mode recognition result: y=FC(high)

[0100] wherein, FC is a fully connected layer, and y is the output fire fault mode recognition result.

[0101] In order to improve the accuracy of the single vehicle model in identifying the fire fault mode, it is necessary to extract useful features from the vehicle data, such as temperature change rate, vehicle driving speed, environment temperature and other features most useful for fire fault mode recognition, which can effectively improve the recognition speed and accuracy of the model.

[0102] When training the single vehicle model, the preliminary fused vehicle monitoring data set is divided into a training set and a test set, and the ratio is 8:2, which ensures that the model can still effectively predict when encountering data outside the training set. Define the LSTM layer, input layer, output layer and hidden layer, set the hyperparameters of LSTM, including the number of hidden units, time steps, learning rate, etc. Train the model using the training set, adjust the model parameters to minimize the loss function, and validate the trained model on the validation set to prevent overfitting. Use accuracy, precision, recall and F1-score to evaluate the performance of the model on the test set.

[0103] Embodiment 3: refer to FIGS. 3-4

[0104] The application also provides a fire warning system based on multi-source information fusion, comprising a single vehicle model training module, a model deployment module, a fire risk assessment module, a BIM modeling display module and an abnormal information notification module, characterized in that:

[0105] The single vehicle model training module comprises:

[0106] Acquisition module: Acquires image data of license plate and hazardous chemical identification of a single vehicle, point cloud data of real-time vehicle positioning and speed, and thermal imaging data of vehicle temperature to form vehicle data;

[0107] Fusion module: performs vehicle data preprocessing and preliminary data fusion on the vehicle data to generate fused vehicle monitoring data;

[0108] Storage module: Uploads and stores the fused vehicle monitoring data;

[0109] Deep learning module: Performs advanced feature extraction and deep fusion on the vehicle monitoring data through deep learning convolutional neural networks, trains the model on the deep fusion convolutional neural network, and identifies fire fault modes;

[0110] The model deployment module: deploys the trained model in the model of the deep learning processing unit for real-time fire fault mode identification of the newly collected vehicle data;

[0111] The fire risk assessment module collects system data of the current bridge environment and performs fire risk assessment and fire risk level classification based on the system data and the fire failure mode identification results.

[0112] The BIM modeling and display module: uses the system data and the fire failure mode results to model the bridge BIM model, maps the fire risk assessment results onto the bridge BIM model, generates a bridge fire risk map, and marks high-risk areas.

[0113] The abnormal information notification module: Based on the bridge fire risk map, it notifies the driver of overheating information through the bridge information board and loudspeaker.

[0114] As shown in Figures 3 and 4, the acquisition module includes a high-definition camera 1, a strobe-illuminated camera 2, a lidar 3, and an infrared thermal imaging camera mounted on the bridge gantry. When a vehicle travels to the capture trigger line 14, the acquisition module acquires image data of the license plate and hazardous chemical identification of a single vehicle, point cloud data of the vehicle's real-time positioning and speed, and thermal imaging data of the vehicle's temperature, forming vehicle data. The fusion module performs vehicle data preprocessing and preliminary data fusion on the vehicle data through the edge computing module 4 to generate fused vehicle monitoring data. The storage module stores the vehicle monitoring data to the network switch 6 and uploads it to the blockchain server 7 via the network.

[0115] The deep learning module obtains the vehicle monitoring data from the blockchain server 7, performs high-level feature extraction and deep fusion on the vehicle monitoring data through a deep learning convolutional neural network in the deep learning processing unit 8, trains the deep fusion convolutional neural network, and identifies the fire failure mode.

[0116] The model deployment module deploys the trained model in the model in the deep learning processing unit, for real-time fire failure mode identification on newly collected vehicle data.

[0117] The fire risk assessment module collects system data of the current bridge environment, and performs fire risk assessment and fire risk level division through the system data and the fire failure mode identification result.

[0118] The BIM modeling display module inputs the system data and the fire failure mode result into the BIM modeling workstation 9 to model the bridge BIM model, maps the fire risk assessment result to the bridge BIM model, generates a bridge fire risk map, labels high-risk areas, and visually displays the bridge fire risk map through the monitoring center display device 10 and the user terminal device 11 for fire warning.

[0119] Finally, the abnormal information notification module notifies the driver of the over-temperature information through the gantry information board 12 and the directional loudspeaker 13.

Claims

1. A fire warning method based on multi-source information fusion, comprising single vehicle model training, model deployment, fire risk assessment, BIM modeling display and abnormal information notification, characterized in that: the single vehicle model training comprises: S1, obtaining image data of license plates and dangerous chemical product identification of a single vehicle, point cloud data of real-time positioning and speed of the vehicle, and temperature thermal imaging data of the vehicle, forming vehicle data; S2, performing vehicle data preprocessing and preliminary data fusion on the vehicle data to generate fused vehicle monitoring data; S3, uploading and storing the fused vehicle monitoring data; S4, performing advanced feature extraction and deep fusion on the vehicle monitoring data through a deep learning convolutional neural network, training the deep fusion convolutional neural network, and identifying a fire failure mode; the model deployment comprises: S5, deploying the trained model in a model in a deep learning processing unit to identify a real-time fire failure mode for newly collected vehicle data; the fire risk assessment comprises: S6, collecting system data of a current bridge environment, performing fire risk assessment and fire risk level division based on the system data and the fire failure mode identification result; the BIM modeling display comprises: S7, modeling a bridge BIM model according to the system data and the fire failure mode result, mapping the fire risk assessment result to the bridge BIM model, and generating a bridge fire risk map; the abnormal information notification comprises: S8, notifying the driver of the over-temperature information through a bridge information board and a loudspeaker according to the bridge fire risk map. S4 further comprises: S41, performing convolution operation on the vehicle monitoring data to extract a feature vector; S42, performing deep fusion on the extracted feature vector; S43, performing further convolution and pooling operation on the deep fused feature vector to extract high-level features; S44, inputting the high-level features into a full connection layer for full connection layer classification to obtain the fire failure mode identification result: y=FC(high) wherein, FC is a full connection layer, and y is an output fire failure mode identification result. In S2, the preliminary data fusion of the vehicle data further comprises: when the center distance and area intersection ratio of the target area exceed the target correlation threshold, performing preliminary data fusion on the vehicle data to generate the preliminary fused vehicle monitoring data; when the target correlation does not exceed the target correlation threshold, outputting the thermal imaging data alone, or outputting thermal imaging data and point cloud data, thermal imaging and image data as vehicle monitoring data. In S1, the temperature thermal imaging data further comprises: monitoring the temperature of the single vehicle through an infrared thermal imaging camera, converting the temperature value obtained by the temperature monitoring into a pseudo-color image, and taking the temperature value and the pseudo-color image as the temperature thermal imaging data of the vehicle. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ 2. The fire warning method based on multi-source information fusion according to claim 1, characterized in that, ​ ​ F I = CNN I (I) F P = CNN P (P) F T = CNN T (T) wherein CNN I , CNN P , CNN T respectively represent a first convolutional neural network for processing the image data, the point cloud data and the thermal imaging data, F I , F P , F T is the extracted feature vector; ​ F fused = [F I , F P , F T ] wherein F fused is the feature vector after deep fusion; ​ F high = CNN high (F fused ) wherein CNN high represents a second convolutional neural network for representing the fused features, F high is the extracted high-level features; ​ ​ ​ 3. The fire warning method based on multi-source information fusion according to claim 1, characterized in that, ​ The vehicle data is preliminarily fused by a fusion algorithm based on the correlation degree, a target correlation degree threshold is set, and the correlation degree between the past vehicle targets is calculated: Wherein, S R is the area of the radar detected target region, O R is the center point of the region; S P is the area of the video detected target region, O P is the center point of the region; O τ , S τ is the target correlation threshold; ΔO is the distance between the centers of the radar and video target regions, and IOU is the intersection over union of the areas of the two target regions. ​ ​ 4. The fire warning method based on multi-source information fusion according to claim 1, characterized in that, ​ ​ When the measured temperature exceeds the set threshold, an over-temperature early warning prompt is given.

5. The fire warning method based on multi-source information fusion according to claim 4, characterized in that, The temperature monitoring determines whether there is a temperature anomaly through temperature fluctuation degree and temperature anomaly positioning coefficient; The temperature fluctuation degree is the temperature change size of any time period in the radar area, used to measure the size of temperature change, and the temperature fluctuation degree is calculated: wherein tf i is the temperature fluctuation degree in the i-th time period; t i,j is the temperature value in the j-th detection sub-region in the i-th time period; The average temperature in the detection area in the i time period is referred to as the average temperature in the detection area in the i time period; n refers to the total number of sub-areas in the detection area; The temperature anomaly positioning coefficient is used to measure the temperature anomaly, position the area location of over-temperature, and calculate the temperature anomaly positioning coefficient: When the temperature anomaly positioning coefficient Z i,j When the temperature anomaly positioning coefficient Z When the temperature anomaly positioning coefficient Z When the temperature anomaly positioning coefficient Z When the temperature anomaly positioning coefficient Z When the temperature anomaly positioning coefficient Z When the temperature anomaly positioning coefficient Z When the temperature anomaly positioning coefficient Z When the temperature anomaly positioning coefficient Z When 6. The fire warning method based on multi-source information fusion according to claim 1, characterized in that, S7 further comprises: The high-risk area and over-temperature vehicle are marked on the bridge fire risk map to provide visual fire risk distribution, and the early warning state and emergency measures are displayed in real time through the monitoring center or the user terminal.

7. The fire warning method based on multi-source information fusion according to claim 1, characterized in that, The system data of the current bridge environment in S6 at least includes one or more of bridge environment data such as vehicle speed, number of vehicles on the way, vehicle type, vehicle density, weather, ambient temperature, and vehicle temperature.

8. The fire warning method based on multi-source information fusion according to claim 1, characterized in that, The fire failure mode at least includes one or more of data-labeled vehicle abnormal heating, non-dangerous goods vehicle abnormal high temperature, dangerous goods vehicle abnormal high temperature, non-dangerous goods vehicle fire, and dangerous goods vehicle fire. The fire risk level is at least divided into one or more of low, medium, and high fire risk levels according to historical records or expert experience.

9. A wireless IoT communication system for liquid level meters, comprising a plurality of distributed liquid level meters, a wireless hotspot and a back office management system, characterized in that, Further comprising the wireless Internet of Things communication circuit board for acquiring liquid level instrument data according to any one of claims 1-8, each wireless Internet of Things communication circuit board for acquiring liquid level instrument data corresponds to one liquid level instrument, the liquid level instrument is connected with the interface communication module through a cable, all wireless Internet of Things communication circuit boards for acquiring liquid level instrument data are wirelessly connected with the wireless hotspot through a wireless module, and liquid level instrument data is uploaded to the background management system through a wireless network. 10.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is processed and executed to realize the steps of the fire warning method based on multi-source information fusion according to any one of claims 1-8. The computer program is processed and executed to realize the steps of the fire warning method based on multi-source information fusion according to any one of claims 1-8.

Images