Fire early warning method and system based on multi-source information fusion and medium

Abstract

The invention discloses a fire early warning method and system based on multi-source information fusion and a medium. The method comprises single vehicle model training, model deployment, fire risk assessment, BIM modeling display and abnormal information notification. According to the fire early warning method and system based on multi-source information fusion and the medium, deep fusion and model training are carried out on the image, positioning and temperature information of the vehicle, the vehicle temperature fault mode can be rapidly and accurately recognized, the accuracy of fire early warning is improved, multi-source information is obtained in real time, and when the vehicle has an overtemperature phenomenon, the fire early warning efficiency is improved. A worker can quickly locate the vehicle and nearby the vehicle, measures can be taken for the overtemperature vehicle in time, fire disasters are prevented or the fire disaster damage degree is reduced, and powerful support is provided for fire prevention and control work. A bridge fire risk map is displayed on a display device or a user terminal by establishing a bridge BIM model, a command department, workers and drivers can quickly grasp information such as a disaster site, a fire reason, a disaster situation degree and a vehicle density degree of a current bridge fire through the bridge fire risk map, and the environment adaptability of the system is high; according to the invention, the fire risk can be evaluated in different environments, the decision-making efficiency of a command department is improved, a worker can quickly arrive at a rescue site to carry out fire extinguishing and rescue work, and a driver in a low-risk area can quickly evacuate from a bridge or assist the worker in personnel evacuation. And casualties and economic losses caused by fire disasters can be reduced as much as possible.

Description

Technical Field

[0001] The present invention relates to the field of fire warning technology, and specifically to a fire warning method, system and medium that integrates multi-source information of image recognition technology, radar technology, deep learning, thermal imaging technology and BIM modeling technology. Background Art

[0002] With the acceleration of urbanization, bridges, as important transportation hubs connecting cities and regions, play an increasingly important role in the transportation network. However, the risk of fire accidents on bridges has also increased. Due to the structure of the bridge, personnel cannot obtain disaster information in time when a fire occurs, making it extremely difficult to escape. Fire can also damage the bridge structure, affecting the bridge's bearing capacity and stability, bringing great challenges to maintaining public safety and the stability of the bridge structure.

[0003] Therefore, it is necessary to monitor and warn of bridge fires. The traditional bridge fire detection method mainly relies on manual inspections and regular maintenance inspections. Although this method can detect potential fire hazards to a certain extent, manual inspections and regular maintenance inspections are limited by factors such as the number of personnel, professional quality of personnel, and weather conditions. There are problems such as slow response speed, low detection efficiency, and incomplete detection. It is impossible to achieve all-weather, all-round real-time monitoring of bridges. In addition, the traditional bridge fire detection method has a strong lag. When manual inspections find abnormal conditions, the fire has often accumulated to a certain extent, and it is impossible to provide early warning of the fire. When a fire occurs, due to the lack of information exchange, it is difficult for the staff to notify everyone on the bridge of the fire situation in real time, which increases the difficulty of fire evacuation and rescue. Summary of the invention

[0004] The present invention aims to overcome at least one defect of the above-mentioned prior art and provide a fire warning method, system and medium based on multi-source information fusion, which is used to solve the technical problem that the prior art cannot perform real-time monitoring, early warning and real-time notification of bridge fire personnel.

[0005] The present invention provides a fire warning method based on multi-source information fusion, including single vehicle model training, model deployment, fire risk assessment, BIM modeling display and abnormal information notification.

[0006] The single vehicle model training includes:

[0007] S1. Obtain image data of a single vehicle's license plate and hazardous chemicals logo, point cloud data of the vehicle's real-time location and speed, and thermal imaging data of the vehicle's temperature to form vehicle data;

[0008] S2, performing vehicle data preprocessing and preliminary data fusion on the vehicle data to generate fused vehicle monitoring data;

[0009] S3, uploading and storing the fused vehicle monitoring data;

[0010] S4. Performing advanced feature extraction and deep fusion on the vehicle monitoring data through a deep learning convolutional neural network, performing model training on the deep fusion convolutional neural network, and identifying fire fault modes.

[0011] Since it is necessary to conduct real-time monitoring and early warning of fire risks on bridges and suppress fires in the early stages of their occurrence, only in this way can casualties and economic losses be reduced. Vehicle accidents and external environment are the main causes of bridge fires. Therefore, by collecting vehicle data and using deep learning convolutional neural networks to train models of vehicles on bridges, the trained models are used to monitor the conditions of bridge vehicles in real time, thereby achieving early warning of bridge fires.

[0012] High-definition camera technology is used to collect and identify the license plate and hazardous chemical logo of a single vehicle for subsequent vehicle matching and fault mode identification. LiDAR technology is used to accurately locate and measure the speed of the vehicle, enabling real-time acquisition of vehicle positioning and speed data within the radar measurement area. Infrared thermal imaging technology is used to measure vehicle temperature and generate pseudo-color images to detect abnormal vehicle over-temperature conditions. The acquired image data, point cloud data, and thermal imaging data form vehicle data.

[0013] The initially acquired vehicle data may have unclear images and fuzzy recognition, and vehicle data preprocessing is required. The preprocessing of vehicle data includes: extracting image features from the image data, identifying the type of vehicle, filtering and clustering point cloud data, optimizing point cloud data, accurately identifying the three-dimensional position of the vehicle, and performing temperature calibration and image enhancement on thermal imaging data to obtain a clear temperature image.

[0014] The pre-processed vehicle data still has the problem of scattered data. This type of data cannot be directly input into the deep learning convolutional neural network for model training. It is necessary to perform preliminary data fusion on the vehicle data to generate fused vehicle monitoring data for uploading and storage.

[0015] The vehicle monitoring data is input into the deep learning neural network to extract advanced features, including: using a deep convolutional neural network to extract advanced visual features such as vehicle outlines and vehicle information from video frames, using the Point Net point cloud processing algorithm to extract the three-dimensional shape and position features of the vehicle from radar point cloud data, and using a deep convolutional neural network to extract temperature distribution features from thermal imaging data. The extracted advanced features are deeply fused, the model is trained through a deep convolutional neural network, the fire failure mode is identified, and the cross entropy loss function is used to optimize the model, the Adam optimization algorithm is used to update the model parameters, and the validation set data is used to evaluate the performance of the trained model, including indicators such as accuracy, recall rate, and F1 score.

[0016] After a single vehicle model is trained, the model can accurately identify fire failure modes in real time and can be deployed, including:

[0017] S5. Deploy the trained model in a model in a deep learning processing unit to perform real-time fire fault pattern recognition on the newly collected vehicle data.

[0018] The single vehicle model can identify the fire failure mode and obtain the information and real-time location of the vehicle, but it has no predictive value for the fire situation of the entire bridge and cannot obtain the damage degree of each location of the bridge. Therefore, it is necessary to conduct a fire risk assessment of the entire bridge system. The fire risk assessment includes:

[0019] S6. Collect system data of the current bridge environment, and perform fire risk assessment and fire risk level classification based on the system data and the fire failure pattern recognition results.

[0020] Through the fire risk assessment, the degree of damage at each location of the bridge can be obtained. In order to further facilitate the command department, firefighters and personnel on the bridge to grasp the bridge fire situation in the first place, the bridge fire risk map can be displayed through the BIM modeling. The BIM modeling display includes:

[0021] S7. Modeling a bridge BIM model according to the system data and the fire failure mode results, mapping the fire risk assessment results to the bridge BIM model, and generating a bridge fire risk map.

[0022] The bridge fire risk map can be displayed on the display equipment and user terminal equipment of the monitoring center. The command department can make decisions quickly and effectively based on the bridge fire risk map, and it is convenient for fire fighters and people on the bridge to understand the fire situation, reduce the manpower cost of fire situation notification, and quickly launch fire rescue work to control the disaster in the early stage. While displaying the bridge fire risk map through the BIM model, it is also necessary to broadcast the over-temperature and fire information again through the bridge information board and loudspeaker to ensure that the abnormal information is notified to the driver.

[0023] Therefore, the abnormal information notification includes:

[0024] S8. According to the bridge fire risk map, the over-temperature information is notified to the driver through the bridge information board and the horn.

[0025] Since the vehicle monitoring data is based on multi-source information fusion, the image data, point cloud data and thermal imaging data in the vehicle monitoring data need to be respectively extracted from feature vectors through deep learning convolutional neural networks. However, the extracted feature vectors are still separated, and deep fusion of feature vectors is required. Further high-level features are extracted through convolution and pooling, so that the fault mode result can be obtained through fully connected layer classification and identification.

[0026] Therefore, the step S4 of extracting high-level features and deeply fusing the vehicle monitoring data through a deep learning convolutional neural network, performing model training on the deep fusion convolutional neural network, and identifying the fire fault mode also includes:

[0027] S41, performing a convolution operation on the vehicle monitoring data to extract a feature vector:

[0028] F I =CNN I (I)

[0029] F P =CNN P (P)

[0030] F T =CNN T (T)

[0031] Among them, CNN I ,CNN P ,CNN T denote the first convolutional neural network for processing the image data, the point cloud data and the thermal imaging data, respectively, I 、F P 、F T is the extracted feature vector;

[0032] S42, performing the deep fusion on the extracted feature vectors:

[0033] F fused =[F I ,F P ,F T ]

[0034] Among them, F fused is the feature vector after deep fusion;

[0035] S43, further convolution and pooling operations are performed on the deeply fused feature vector to extract high-level features:

[0036] F high =CNN high (F fused )

[0037] Among them, CNN high The second convolutional neural network representing the deep fused features, F high It is the extracted high-level features;

[0038] The high-level features are input into the fully connected layer for classification, and the fire fault pattern recognition result is obtained:

[0039] y=FC(high)

[0040] Among them, FC is the fully connected layer and y is the output fire fault pattern recognition result.

[0041] When training the single vehicle model, the vehicle data is multi-source information from high-definition cameras, laser radars, and infrared thermal imaging cameras. It cannot be guaranteed that the high-definition camera and the laser radar acquire the same target. If the preliminary data fusion of the vehicle data in step S3 is directly performed, the subsequent model training results may be inaccurate or erroneous. Therefore, it is also necessary to match the correlation between the targets acquired by the high-definition camera and the laser radar to determine whether the targets acquired by different devices are the same target. Therefore, S2 also includes the following steps:

[0042] The vehicle data is initially fused by a fusion algorithm based on correlation, the target correlation threshold is set, and the correlation between past vehicle targets is calculated:

[0043]

[0044] Among them, S R is the target area detected by the radar, O R is the center point of the area; S P is the area of ​​the target area detected in the video, OP is the center point of the area; τ , S τ is the target correlation threshold; ΔO is the center distance between the radar and video target areas, and IOU is the intersection-over-union ratio of the target areas of the two;

[0045] When the center distance and the area intersection-combination ratio exceeding the target area both exceed the target correlation threshold, the vehicle data is initially fused to generate the vehicle monitoring data after the initial fusion, which can ensure that the vehicle data after the initial fusion comes from the same target;

[0046] When the inter-target correlation does not exceed the target correlation threshold, it means that the acquired vehicle data does not belong to or is not determined to be vehicle data of the same target, and the thermal imaging data is output separately, or the thermal imaging data and point cloud data, thermal imaging and image data are output as vehicle monitoring data to ensure the accuracy of single vehicle model training.

[0047] Since the main cause of bridge fires is vehicle accidents, in the single vehicle model training, by collecting the real-time temperature of the vehicle, when the vehicle temperature exceeds a certain temperature value, it is determined that there is a fire risk in the area and an over-temperature warning is issued. Therefore, the acquisition of temperature thermal imaging data in S1 also includes:

[0048] The temperature of the single vehicle is monitored by an infrared thermal imaging camera, and the temperature value obtained by the temperature monitoring is converted into a pseudo-color image. The temperature value and the pseudo-color image are used as temperature thermal imaging data of the vehicle. When the measured temperature exceeds a set threshold, an over-temperature warning prompt is issued. At this time, there is no fire in the area where the vehicle is located, but there is a possibility of fire. The staff can take timely measures based on the vehicle's license plate information, positioning information and the over-temperature warning to avoid the occurrence of fire. The temperature monitoring determines whether there is a temperature anomaly through temperature fluctuation and temperature anomaly positioning coefficient.

[0049] The temperature fluctuation is the temperature change in any time period within the radar area, which is used to measure the size of the temperature change and calculate the temperature fluctuation:

[0050]

[0051] Among them, tf i refers to the temperature fluctuation within the i-th time period; t i,j It refers to the temperature value of the jth detection sub-area in the i-th time period; It refers to the average temperature in the detection area during the i-th time period; n refers to the total number of sub-areas in the detection area;

[0052] The temperature anomaly location coefficient is used to measure temperature anomalies, locate the over-temperature area, and calculate the temperature anomaly location coefficient:

[0053]

[0054] When the temperature anomaly location coefficient Z i,j When the threshold is exceeded, there is a temperature abnormality in the radar area and an over-temperature warning prompt is issued.

[0055] In order to allow the bridge fire risk map generated in the bridge BIM model to further display the fire situation of the bridge, so that the command department, staff and drivers can have a clearer understanding of the current fire situation, S7 also includes: marking high-risk areas and over-temperature vehicles on the bridge fire risk map, providing a visual fire risk distribution, and displaying the warning status and emergency measures in real time through the monitoring center or user terminal.

[0056] Since the single vehicle model contains the license plate, hazardous chemical type, vehicle speed, location and vehicle temperature information of each vehicle, and the information of each vehicle can only be used as an independent individual, it cannot reflect the overall fire situation of the bridge, nor can it grasp the vehicle situation near each vehicle. The early warning and decision-making of the fire need to be judged by the overall situation of the bridge at this time. Therefore, the system data collected in S6 for the current bridge environment includes at least one or more bridge environment data of vehicle speed, number of vehicles on the road, vehicle type, vehicle density, weather, ambient temperature, and vehicle temperature. The fire risk assessment and fire risk level classification are carried out through the system data and the fire fault pattern recognition results. The system data is comprehensive and instructive for fire early warning.

[0057] In the single vehicle model training, the deep learning convolutional neural network is used to finally obtain the failure mode of the vehicle temperature, and the fire failure mode includes at least one or more of data-labeled abnormal heating of vehicles, abnormally high temperature of non-hazardous goods vehicles, abnormally high temperature of hazardous goods vehicles, fire of non-hazardous goods vehicles, and fire of hazardous goods vehicles, so as to facilitate the training and management of the model, and the fire risk level is divided into at least one or more of low, medium, and high fire risk levels based on historical records or expert experience, and the fire risk level is set in the bridge fire risk map, so that the fire conditions at various locations on the bridge are clearly visible.

[0058] The present invention also provides a fire warning system based on multi-source information fusion, including a single vehicle model training module, a model deployment module, a fire risk assessment module, a BIM modeling display module and an abnormal information notification module, characterized in that:

[0059] The single vehicle model training module includes:

[0060] Acquisition module: acquires the image data of the license plate and hazardous chemicals logo of a single vehicle, the point cloud data of the vehicle's real-time positioning and speed, and the vehicle's temperature thermal imaging data to form vehicle data;

[0061] Fusion module: performs vehicle data preprocessing and preliminary data fusion on the vehicle data to generate fused vehicle monitoring data;

[0062] Storage module: uploading and storing the fused vehicle monitoring data;

[0063] Deep learning module: extracting high-level features and deeply fusing the vehicle monitoring data through a deep learning convolutional neural network, training the deep fusion convolutional neural network model, and identifying fire fault modes;

[0064] The model deployment module deploys the trained model in the model in the deep learning processing unit to perform real-time fire fault pattern recognition on the newly collected vehicle data;

[0065] The fire risk assessment module collects system data of the current bridge environment, and performs fire risk assessment and fire risk level classification based on the system data and the fire failure mode recognition result;

[0066] The BIM modeling and display module is used to model a bridge BIM model using the system data and the fire failure mode results, map the fire risk assessment results onto the bridge BIM model, generate a bridge fire risk map, and mark high-risk areas;

[0067] The abnormal information notification module: according to the bridge fire risk map, the over-temperature information is notified to the driver through the bridge information board and the horn.

[0068] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when processed and executed, implements the steps of the fire warning method based on multi-source information fusion as described in any one of claims 1 to 8.

[0069] The beneficial effects of the present invention are as follows: the fire warning method, system and medium based on multi-source information fusion can quickly and accurately identify the vehicle temperature fault mode by deeply fusing the vehicle's image, positioning and temperature information and training the model, thereby improving the accuracy of fire warning. The real-time acquisition of multi-source information can enable the staff to quickly locate the vehicle and its vicinity when the vehicle is overheated, and can take measures against the overheated vehicle in time to prevent the occurrence of fire or reduce the degree of fire damage, thereby providing strong support for fire prevention and control. By establishing a bridge BIM model and displaying a bridge fire risk map on a display device or user terminal, the command department, staff and drivers can quickly grasp the current bridge fire disaster location, fire cause, disaster degree, vehicle density and other information through the bridge fire risk map. The system has strong environmental adaptability and can evaluate the fire risk in different environments, thereby improving the decision-making efficiency of the command department. Staff can quickly arrive at the rescue site to carry out fire extinguishing and rescue work, and drivers in low-risk areas can quickly evacuate the bridge or assist staff in evacuating personnel, thereby minimizing casualties and economic losses caused by the fire. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0071] Figure 1 This is a step-by-step diagram of a fire warning method based on multi-source information fusion.

[0072] Figure 2 It is a step diagram of a deep learning convolutional neural network performing model training on the vehicle monitoring data.

[0073] Figure 3 This is a working diagram of a fire warning system based on multi-source information fusion.

[0074] Figure 4 It is a workflow diagram of the fire warning system based on multi-source information fusion.

[0075] The figure marks in the specification include: high-definition camera 1, strobe fill light 2, laser radar 3, infrared thermal imaging camera 4, edge computing module 5, network switch 6, blockchain server 7, deep learning processing unit 8, BIM modeling workstation 9, monitoring center display device 10, user terminal device 11, gantry information board 12, directional loudspeaker 13. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solution and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings:

[0077] The embodiments of the present invention provide embodiments of a fire warning method and system based on multi-source information fusion. Although a logical sequence is shown in the flow chart, under certain data, the steps shown or described may be completed in an order different from that shown here.

[0078] Example 1: Reference Figure 1

[0079] like Figure 1 As shown in the step diagram of the fire warning method based on multi-source information fusion, the present invention provides a fire warning method based on multi-source information fusion, and the present invention provides a fire warning method based on multi-source information fusion, including single vehicle model training, model deployment, fire risk assessment, BIM modeling display and abnormal information notification.

[0080] The single vehicle model training includes:

[0081] S1. Obtain image data of a single vehicle's license plate and hazardous chemicals logo, point cloud data of the vehicle's real-time location and speed, and thermal imaging data of the vehicle's temperature to form vehicle data;

[0082] S2, performing vehicle data preprocessing and preliminary data fusion on the vehicle data to generate fused vehicle monitoring data;

[0083] S3, uploading and storing the fused vehicle monitoring data;

[0084] S4. Performing advanced feature extraction and deep fusion on the vehicle monitoring data through a deep learning convolutional neural network, performing model training on the deep fusion convolutional neural network, and identifying fire fault modes.

[0085] Since it is necessary to conduct real-time monitoring and early warning of fire risks on bridges and suppress fires in the early stages of their occurrence, only in this way can casualties and economic losses be reduced. Vehicle accidents and external environment are the main causes of bridge fires. Therefore, by collecting vehicle data and using deep learning convolutional neural networks to train models of vehicles on bridges, the trained models are used to monitor the conditions of bridge vehicles in real time, thereby achieving early warning of bridge fires.

[0086] High-definition camera technology is used to collect and identify the license plate and hazardous chemical logo of a single vehicle for subsequent vehicle matching and fault mode identification. LiDAR technology is used to accurately locate and measure the speed of the vehicle, enabling real-time acquisition of vehicle positioning and speed data within the radar measurement area. Infrared thermal imaging technology is used to measure vehicle temperature and generate pseudo-color images to detect abnormal vehicle over-temperature conditions. The acquired image data, point cloud data, and thermal imaging data form vehicle data.

[0087] The initially acquired vehicle data may have unclear images and fuzzy recognition, and vehicle data preprocessing is required. The preprocessing of vehicle data includes: extracting image features from the image data, identifying the type of vehicle, filtering and clustering point cloud data, optimizing point cloud data, accurately identifying the three-dimensional position of the vehicle, and performing temperature calibration and image enhancement on thermal imaging data to obtain a clear temperature image.

[0088] The pre-processed vehicle data still has the problem of scattered data. This type of data cannot be directly input into the deep learning convolutional neural network for model training. It is necessary to perform preliminary data fusion on the vehicle data to generate fused vehicle monitoring data for uploading and storage.

[0089] The vehicle monitoring data is input into the deep learning neural network to extract advanced features, including: using a deep convolutional neural network to extract advanced visual features such as vehicle outlines and vehicle information from video frames, using the Point Net point cloud processing algorithm to extract the three-dimensional shape and position features of the vehicle from radar point cloud data, and using a deep convolutional neural network to extract temperature distribution features from thermal imaging data. The extracted advanced features are deeply fused, the model is trained through a deep convolutional neural network, the fire failure mode is identified, and the cross entropy loss function is used to optimize the model, the Adam optimization algorithm is used to update the model parameters, and the validation set data is used to evaluate the performance of the trained model, including indicators such as accuracy, recall rate, and F1 score.

[0090] After a single vehicle model is trained, the model can accurately identify fire failure modes in real time and can be deployed, including:

[0091] S5. Deploy the trained model in a model in a deep learning processing unit to perform real-time fire fault pattern recognition on the newly collected vehicle data.

[0092] The single vehicle model can identify the fire failure mode and obtain the information and real-time location of the vehicle, but it has no predictive value for the fire situation of the entire bridge and cannot obtain the damage degree of each location of the bridge. Therefore, it is necessary to conduct a fire risk assessment of the entire bridge system. The fire risk assessment includes:

[0093] S6. Collect system data of the current bridge environment, and perform fire risk assessment and fire risk level classification based on the system data and the fire failure pattern recognition results.

[0094] Through the fire risk assessment, the degree of damage at each location of the bridge can be obtained. In order to further facilitate the command department, firefighters and personnel on the bridge to grasp the bridge fire situation in the first place, the bridge fire risk map can be displayed through the BIM modeling. The BIM modeling display includes:

[0095] S7. Modeling a bridge BIM model according to the system data and the fire failure mode results, mapping the fire risk assessment results to the bridge BIM model, and generating a bridge fire risk map.

[0096] The bridge fire risk map can be displayed on the display equipment and user terminal equipment of the monitoring center. The command department can make decisions quickly and effectively based on the bridge fire risk map, and it is convenient for fire fighters and people on the bridge to understand the fire situation, reduce the manpower cost of fire situation notification, and quickly launch fire rescue work to control the disaster in the early stage. While displaying the bridge fire risk map through the BIM model, it is also necessary to broadcast the over-temperature and fire information again through the bridge information board and loudspeaker to ensure that the abnormal information is notified to the driver.

[0097] Therefore, the abnormal information notification includes:

[0098] S8. According to the bridge fire risk map, the over-temperature information is notified to the driver through the bridge information board and the horn.

[0099] Example 2: Reference Figure 2

[0100] Since the vehicle monitoring data is based on multi-source information fusion, the image data, point cloud data and thermal imaging data in the vehicle monitoring data need to be respectively extracted from feature vectors through deep learning convolutional neural networks. However, the extracted feature vectors are still separated, and deep fusion of feature vectors is required. Further high-level features are extracted through convolution and pooling, so that the fault mode result can be obtained through fully connected layer classification and identification.

[0101] Therefore, the step S4 of extracting high-level features and deeply fusing the vehicle monitoring data through a deep learning convolutional neural network, performing model training on the deep fusion convolutional neural network, and identifying the fire fault mode also includes:

[0102] S41, performing a convolution operation on the vehicle monitoring data to extract a feature vector:

[0103] F I =CNN I (I)

[0104] F P =CNN P (P)

[0105] F T =CNN T (T)

[0106] Among them, CNN I ,CNN P ,CNN T denote the first convolutional neural network for processing the image data, the point cloud data and the thermal imaging data, respectively, I 、F P 、F T is the extracted feature vector;

[0107] S42, performing the deep fusion on the extracted feature vectors:

[0108] F fused =[F I ,F P ,F T ]

[0109] Among them, F fused is the feature vector after deep fusion;

[0110] S43, further convolution and pooling operations are performed on the deeply fused feature vector to extract high-level features:

[0111] F high =CNN high (F fused )

[0112] Among them, CNN high The second convolutional neural network representing the deep fused features, F high It is the extracted high-level features;

[0113] The high-level features are input into the fully connected layer for classification, and the fire fault pattern recognition result is obtained:

[0114] y=FC(high)

[0115] Among them, FC is the fully connected layer and y is the output fire fault pattern recognition result.

[0116] In order to improve the accuracy of the single vehicle model in identifying the fire fault pattern, it is necessary to first extract useful features from the vehicle data, such as temperature change rate, vehicle speed, ambient temperature, etc., which are the most useful features for fire fault pattern identification, which can effectively improve the recognition speed and accuracy of the model.

[0117] When training the single vehicle model, the vehicle monitoring data set after preliminary fusion is divided into a training set and a test set, and the ratio is 8:2 to ensure that the model can still make effective predictions when encountering data outside the training set. The LSTM layer, input layer, output layer and hidden layer are defined, and the hyperparameters of the LSTM are set, including the number of hidden units, time step, learning rate, etc. The model is trained using the training set, and the model parameters are adjusted to minimize the loss function. The trained model is verified on the validation set, and the hyperparameters are adjusted to prevent overfitting. The model performance is evaluated on the test set using indicators such as accuracy, precision, recall and F1-score.

[0118] Example 3: Reference Figures 3-4

[0119] The present invention also provides a fire warning system based on multi-source information fusion, including a single vehicle model training module, a model deployment module, a fire risk assessment module, a BIM modeling display module and an abnormal information notification module, characterized in that:

[0120] The single vehicle model training module includes:

[0121] Acquisition module: acquires the image data of the license plate and hazardous chemicals logo of a single vehicle, the point cloud data of the vehicle's real-time positioning and speed, and the vehicle's temperature thermal imaging data to form vehicle data;

[0122] Fusion module: performs vehicle data preprocessing and preliminary data fusion on the vehicle data to generate fused vehicle monitoring data;

[0123] Storage module: uploading and storing the fused vehicle monitoring data;

[0124] Deep learning module: extracting high-level features and deeply fusing the vehicle monitoring data through a deep learning convolutional neural network, training the deep fusion convolutional neural network model, and identifying fire fault modes;

[0125] The model deployment module deploys the trained model in the model in the deep learning processing unit to perform real-time fire fault pattern recognition on the newly collected vehicle data;

[0126] The fire risk assessment module collects system data of the current bridge environment, and performs fire risk assessment and fire risk level classification based on the system data and the fire failure mode recognition result;

[0127] The BIM modeling and display module is used to model a bridge BIM model using the system data and the fire failure mode results, map the fire risk assessment results onto the bridge BIM model, generate a bridge fire risk map, and mark high-risk areas;

[0128] The abnormal information notification module: according to the bridge fire risk map, the over-temperature information is notified to the driver through the bridge information board and the horn.

[0129] like Figure 3 and Figure 4 As shown, the acquisition module includes a high-definition camera 1, a stroboscopic fill light camera 2, a laser radar 3 and an infrared thermal imaging camera arranged on the bridge gantry. When the vehicle travels to the capture trigger line 14, the acquisition module will obtain the image data of the license plate and hazardous chemicals logo of a single vehicle, the point cloud data of the real-time positioning and speed of the vehicle, and the temperature thermal imaging data of the vehicle to form vehicle data. The fusion module performs vehicle data preprocessing and preliminary data fusion on the vehicle data through the edge computing module 4 to generate fused vehicle monitoring data, and the vehicle monitoring data is stored in the network switch 6 through the storage module, and uploaded to the blockchain server 7 through the network.

[0130] The deep learning module obtains the vehicle monitoring data from the blockchain server 7, performs high-level feature extraction and deep fusion on the vehicle monitoring data through the deep learning convolutional neural network in the deep learning processing unit 8, performs model training on the deep fusion convolutional neural network, and identifies fire fault modes.

[0131] The model deployment module deploys the trained model in the model in the deep learning processing unit for real-time fire fault pattern recognition of the newly collected vehicle data.

[0132] The fire risk assessment module collects system data of the current bridge environment, and performs fire risk assessment and fire risk level classification based on the system data and the fire failure mode recognition result.

[0133] The BIM modeling and display module inputs the system data and the fire failure mode results into the BIM modeling workstation 9 to perform bridge BIM modeling, maps the fire risk assessment results to the bridge BIM model, generates a bridge fire risk map, marks high-risk areas, and visualizes the bridge fire risk map and provides fire warnings through the monitoring center display device 10 and the user terminal device 11.

[0134] Finally, the abnormal information notification module notifies the driver of the over-temperature information through the gantry information board 12 and the directional loudspeaker 13.

Claims

1. A fire warning method based on multi-source information fusion, including single vehicle model training, model deployment, fire risk assessment, BIM modeling display and abnormal information notification, characterized in that: The single vehicle model training includes: S1. Obtain image data of the license plate and hazardous chemicals logo of a single vehicle, point cloud data of the vehicle's real-time positioning and speed, and thermal imaging data of the vehicle's temperature to form vehicle data; S2, performing vehicle data preprocessing and preliminary data fusion on the vehicle data to generate fused vehicle monitoring data; S3, uploading and storing the fused vehicle monitoring data; S4. Performing advanced feature extraction and deep fusion on the vehicle monitoring data through a deep learning convolutional neural network, performing model training on the deep fusion convolutional neural network, and identifying fire fault modes; The model deployment includes: S5. Deploy the trained model in a model in a deep learning processing unit to perform real-time fire fault pattern recognition on the newly collected vehicle data; The fire risk assessment includes: S6, collecting system data of the current bridge environment, and performing fire risk assessment and fire risk level classification based on the system data and the fire failure pattern recognition result; The BIM modeling display includes: S7, building a bridge BIM model according to the system data and the fire failure mode results, mapping the fire risk assessment results to the bridge BIM model, and generating a bridge fire risk map; The abnormal information notification includes: S8. According to the bridge fire risk map, the over-temperature information is notified to the driver through the bridge information board and the horn.

2. A fire warning method based on multi-source information fusion according to claim 1, characterized in that: S4, extracting high-level features and deeply fusing the vehicle monitoring data through a deep learning convolutional neural network, further includes: S41, performing a convolution operation on the vehicle monitoring data to extract a feature vector: F I =CNN I (I) F P =CNN P (P) F T =CNN T (T) Among them, CNN I ,CNN P ,CNN T denote the first convolutional neural network for processing the image data, the point cloud data and the thermal imaging data, respectively, I 、F P 、F T is the extracted feature vector; S42, performing the deep fusion on the extracted feature vectors: F fused =[F I ,F P ,F T ] Among them, F fused is the feature vector after deep fusion; S43, further convolution and pooling operations are performed on the deeply fused feature vector to extract high-level features: F high =CNN high (F fused ) Among them, CNN high The second convolutional neural network representing the deep fused features, F high It is the extracted high-level features; S44, inputting the high-level features into the fully connected layer for fully connected layer classification to obtain the fire fault pattern recognition result: y=FC(high) Among them, FC is the fully connected layer and y is the output fire fault pattern recognition result.

3. A fire warning method based on multi-source information fusion according to claim 1, characterized in that: The preliminary data fusion of the vehicle data in S2 further includes: The vehicle data is initially fused by a fusion algorithm based on correlation, the target correlation threshold is set, and the correlation between past vehicle targets is calculated: Among them, S R is the target area detected by the radar, O R is the center point of the area; S P is the area of ​​the target area detected in the video, O P is the center point of the area; τ , S τ is the target correlation threshold; ΔO is the center distance between the radar and video target areas, and IOU is the intersection-over-union ratio of the target areas of the two; When the center distance and the area intersection-combination ratio exceeding the target area both exceed the target correlation threshold, preliminary data fusion is performed on the vehicle data to generate the preliminary fused vehicle monitoring data; When the inter-target correlation does not exceed the target correlation threshold, the thermal imaging data is outputted alone, or the thermal imaging data and point cloud data, or the thermal imaging and image data are outputted as vehicle monitoring data.

4. A fire warning method based on multi-source information fusion according to claim 1, characterized in that: Acquiring temperature thermal imaging data in S1 also includes: Performing temperature monitoring on the single vehicle by using an infrared thermal imaging camera, converting a temperature value obtained by the temperature monitoring into a pseudo-color image, and using the temperature value and the pseudo-color image as temperature thermal imaging data of the vehicle; When the measured temperature exceeds the set threshold, an over-temperature warning prompt will be issued.

5. A fire warning method based on multi-source information fusion according to claim 4, characterized in that: The temperature monitoring determines whether there is a temperature anomaly by using the temperature fluctuation and the temperature anomaly location coefficient; The temperature fluctuation is the temperature change in any time period within the radar area, which is used to measure the size of the temperature change and calculate the temperature fluctuation: Among them, tf i refers to the temperature fluctuation within the i-th time period; t i,j It refers to the temperature value of the jth detection sub-area in the i-th time period; It refers to the average temperature in the detection area during the i-th time period; n refers to the total number of sub-areas in the detection area; The temperature anomaly location coefficient is used to measure temperature anomalies, locate the over-temperature area, and calculate the temperature anomaly location coefficient: When the temperature anomaly location coefficient Z i,j When the threshold is exceeded, there is a temperature abnormality in the radar area and an over-temperature warning prompt is issued.

6. A fire warning method based on multi-source information fusion according to claim 1, characterized in that: The S7 also includes: High-risk areas and over-temperature vehicles are marked on the bridge fire risk map to provide a visual fire risk distribution, and the warning status and emergency measures are displayed in real time through a monitoring center or a user terminal.

7. A fire warning method based on multi-source information fusion according to claim 1, characterized in that: The system data of the current bridge environment in S6 at least includes one or more bridge environment data of vehicle speed, number of vehicles on the road, vehicle type, vehicle density, weather, ambient temperature, and vehicle temperature.

8. The fire warning method based on multi-source information fusion according to claim 1 is characterized in that: The fire failure mode includes at least one or more of abnormal temperature rise of the data-marked vehicle, abnormal high temperature of the non-dangerous goods vehicle, abnormal high temperature of the dangerous goods vehicle, fire of the non-dangerous goods vehicle, and fire of the dangerous goods vehicle; The fire risk level is divided into at least one or more of low, medium and high fire risk levels based on historical records or expert experience.

9. A wireless Internet of Things communication system for a liquid level meter, comprising a plurality of distributed liquid level meters, wireless hotspots and a background management system, characterized in that: It also includes a wireless Internet of Things communication circuit board for obtaining liquid level meter data as described in any one of claims 1 to 8, each wireless Internet of Things communication circuit board for obtaining liquid level meter data corresponds to a liquid level meter, the liquid level meter is connected to the interface communication module via a cable, and all wireless Internet of Things communication circuit boards for obtaining liquid level meter data are wirelessly connected to the wireless hotspot via a wireless module, and the obtained liquid level meter data is uploaded to the background management system via the wireless network.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is processed and executed, the steps of the fire warning method based on multi-source information fusion as claimed in any one of claims 1 to 8 are implemented.

Images