An underwater vehicle cabin fire state evaluation method based on multi-source heterogeneous data fusion

Abstract

This invention discloses a method for assessing the fire status of a submersible compartment based on multi-source heterogeneous data fusion, comprising the following steps: 1) acquiring monitoring video streams of the compartment under natural light conditions; 2) dividing a acquired video into K segments, randomly selecting a small segment from each segment; inputting the selected small segment into a spatiotemporal dual-stream convolutional neural network to obtain category scores for different small segments, then fusing them to locate fire and smoke areas in the video stream; 3) acquiring a heat map of the monitored compartment area, and assessing the fire status of the compartment based on the thermal imaging image; 4) combining multiple heterogeneous fire early warning signal data for collaborative analysis and information fusion to provide accurate early warning of fire hazards in the compartment. This invention acquires monitoring video streams of the compartment under natural light conditions and designs a convolutional neural network model based on a temporal attention mechanism to process the video stream and detect and locate fire and smoke areas in the video stream.

Description

Technical Field

[0001] This invention relates to ship compartment fire detection and identification technology, and more particularly to a method for assessing the fire status of submersible compartments based on multi-source heterogeneous data fusion. Background Technology

[0002] For fire detection and identification in submersible compartments, the most commonly used methods are based on photosensitive, heat-sensitive, smoke-sensitive, and composite detection sensors. These methods are highly accurate and widely used. However, these methods rely on sensing flames, temperature, and smoke for fire detection. Due to the limitations of their detection principles, they are only applicable in scenarios where open flames have already occurred. They are difficult to detect early-stage fire hazards, and their detection area is relatively limited, thus having certain limitations.

[0003] Due to the stealth of fires and the complexity of cabin environments, relying on a single sensor is insufficient to effectively identify potential fire hazards and thus fail to achieve risk assessment before a fire occurs. Therefore, multiple fire assessment sensors need to work collaboratively, complementing each other's strengths. However, multi-source fire warning signal data from different sensors are often heterogeneous. How to effectively fuse this heterogeneous data to extract information for fire warning assessment remains a challenging problem. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method for assessing the fire status of a submersible compartment based on multi-source heterogeneous data fusion, which addresses the deficiencies in the existing technology.

[0005] The technical solution adopted by this invention to solve its technical problem is: a method for assessing the fire status of a submersible compartment based on multi-source heterogeneous data fusion, comprising the following steps:

[0006] 1) Acquire video streams of the cabin under natural light conditions;

[0007] 2) For the collected video, divide it into K short video segments, and randomly select a small segment from each short video segment; input the selected small segment into a spatiotemporal dual-stream convolutional neural network to obtain the category scores of different small segments, and then fuse them to locate the fire and smoke areas in the video stream;

[0008] The spatiotemporal dual-stream convolutional neural network includes: a temporal dimension attention mechanism network, a spatial dimension attention mechanism network, and a fire identification classifier;

[0009] 2.1) Perform keyframe detection on short segments of video sequences and use a time-related sequence prediction model to extract video segments of interest;

[0010] The temporal attention mechanism network employs a time-series-based attention selection cognitive mechanism to calculate the weight of each video frame, selecting key frames at the video level for focused attention, as follows:

[0011] Given an input video, extract the audio and target dynamic information (optical flow features, etc.) from the video, and then use a deep neural network model to fuse the two features to obtain multi-frame encoded video segment features (audio and optical flow fusion features);

[0012] Among them, the audio features are obtained by sampling and quantizing the original audio data in the video, and the optical flow features are obtained by using the Farneback dense optical flow algorithm after graying the video image;

[0013] This deep neural network processes audio and optical flow data separately and then fuses them.

[0014] The time-series-based attention selection cognitive mechanism constructs a model of the relationship between frames in a video segment through two-layer function transformation, as shown in the formula below. This model adopts a novel video keyframe recalibration strategy, which automatically obtains the weight W of each video frame through learning, and performs video keyframe recalibration to filter redundant information based on the weight W, as shown in formula (1).

[0015] w=Sigmoid(Func2(Func1(Feature))) (1)

[0016] Among them, the Sigmoid() function is used to normalize the weights W, and Func1() and Func2() are two fully connected neural networks used to compress the time series features to the number of video frames.

[0017] Func1() and Func2() together form a "bottleneck" structure, which is widely used in the current mainstream residual network models. This structure first compresses and then expands the feature dimension, and performs non-linear operations through Sigmoid(). While reducing the complexity of the model, it effectively improves the model's expressive power, enabling the designed attention selection cognitive module to accurately express the importance of each frame of video.

[0018] During network training, the loss function value of the video-level attention module is calculated by combining the video understanding results fed back by the image-level attention selection cognitive module, thus realizing a cross-level attention selection cognitive mechanism.

[0019] 2.2) After locating keyframes in the video through the temporal attention mechanism, a spatial location-based attention selection cognitive mechanism is adopted for each keyframe. At the image level, image features of the region of interest are selected for focused attention, and the image features of the keyframe and key region are encoded into video segment features.

[0020] The target region of interest is obtained by rapidly scanning the global image;

[0021] The deep neural network uses two cascaded sub-networks to process the features of the input image. The suppression module performs multi-level feature downsampling in the spatial dimension, gradually compressing each two-dimensional feature channel into a real feature with a global receptive field. Then, the augmentation module performs multi-level feature upsampling in the spatial dimension, gradually restoring the global features to the spatial dimension size of the original input features.

[0022] The spatial location-based attention selection cognitive mechanism mainly consists of two branches, which are used to extract deep features and generate attention weights W for spatial location, as shown in the following formula: W i The weight of video frame i is learned through the attention selection cognitive mechanism at the video level. The Spatial_Squeeze() and Spatial_Expand() functions represent the suppression module and the augmentation module, respectively, which contain multiple convolutional layers and pooling layers.

[0023] The deep neural network uses two cascaded sub-networks to process the features of the input image. The suppression module performs multi-level feature downsampling in the spatial dimension, gradually compressing each two-dimensional feature channel into a real feature with a global receptive field. Then, the augmentation module performs multi-level feature upsampling in the spatial dimension, gradually restoring the global features to the spatial dimension size of the original input features.

[0024] Then, the importance is normalized using the Sigmoid() function, finally obtaining the spatial position weight W′ of the keyframe. i (x), representing the weight of the i-th frame at position x:

[0025] W′ i (x)=W i *Sigmoid(Spatial_Expand(Spatial_Squeeze(Feature i (x)))) (2)

[0026] The generated features based on the spatial attention selection cognitive mechanism are denoted as Feature′. i (x), the formula is shown below, and the T() function represents multiple cascaded linear or nonlinear transformation operations on the feature extraction branch:

[0027] Feature′ i (x)=T(Feature i (x))*W′ i (x) (3)

[0028] Finally, the key region features within multiple keyframes are encoded into video segment features, which are then input into a classifier to obtain video recognition results. These results are then fed back into a spatiotemporal dual-stream convolutional neural network at the video level to accurately locate fires and smoke in the video and to evaluate the detection probability.

[0029] 3) Obtain thermal maps of the monitored compartment area and assess the fire status of the compartment based on the thermal imaging images;

[0030] 3.1) Divide the monitoring screen into several areas, and set a high temperature threshold and a low temperature threshold for each area;

[0031] 3.2) Detect the highest temperature of each region in each frame. When the highest temperature of a region in the current frame is detected to be greater than the high temperature threshold or lower than the low temperature threshold of that region, the frame is set as the i-th frame and recorded as an abnormal region.

[0032] 3.3) A continuous frame detection processing scheme is adopted. If the i-th frame is an abnormal region, the continuous N frames after the i-th frame are detected, i.e., the i+1, i+2, ..., i+n frames, where N is set according to the application scenario. If the highest temperature in the region is continuously higher than the highest temperature threshold or continuously lower than the lowest temperature threshold, the temperature in the region is determined to be abnormal, and a high temperature alarm message is sent.

[0033] 4) Combine various heterogeneous fire early warning signal data for collaborative analysis and information fusion to provide accurate early warning of fire hazards in the cabin; the various heterogeneous fire early warning signal data include natural light video data and image recognition results from cabin cameras, thermal infrared image recognition results from thermal imaging sensors, and alarm data from smoke detectors.

[0034] According to the above scheme, in step 2), a deep network model is used to process and deeply fuse the optical flow features and audio features of the video. The optical flow and audio features are processed by two sub-networks respectively, and then a separate fusion network is used to deeply fuse the two types of perceptual features. Finally, the multi-frame fused features are encoded into video segment features.

[0035] According to the above scheme, the temperature value of a specific area in step 3) is obtained by drawing a specific area in an infrared thermal image and establishing a correlation between the pixel value of the image and the temperature value.

[0036] f(x,y)=F(T(x,y)) (4)

[0037] Equation (1) is the formula for the direct relationship between point temperature and image pixels, where x,y represent the position of image coordinates, f(x,y) represents the pixel value of the displayed image, T(x,y) represents the temperature value of each frame of the corresponding image, and F(.) represents the conversion function from temperature to image, such as the pseudo-color operation function.

[0038] According to the above scheme, information fusion is performed in step 4) using the fuzzy comprehensive evaluation method.

[0039] The beneficial effects of this invention are:

[0040] 1) A fire status assessment method based on video images is proposed. The natural light camera in the submersible is used to collect the monitoring video stream inside the cabin under natural light conditions. A convolutional neural network model based on temporal attention mechanism is designed to process the video stream and detect and locate the fire and smoke areas in the video stream.

[0041] 2) A method for assessing the fire status of a compartment based on thermal imaging images is proposed. By acquiring a thermal map of the monitored compartment area, the fire status can be assessed, which can effectively improve the probability of detecting a compartment fire and accurately locate the fire hazard.

[0042] 3) A method for assessing the fire status of a submersible compartment based on multi-source heterogeneous data fusion is proposed. This method integrates traditional fire detection methods based on temperature sensors and smoke sensors, fire status assessment methods based on video images, and compartment fire status assessment methods based on thermal imaging transmission images. The fuzzy comprehensive evaluation method is used for information fusion. Attached Figure Description

[0043] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0044] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0045] Figure 2 This is a flowchart of a fire status assessment method based on video images according to an embodiment of the present invention;

[0046] Figure 3 This is a flowchart of the cabin fire status assessment method based on infrared thermal images according to an embodiment of the present invention;

[0047] Figure 4 This is a diagram of the deep neural network structure according to an embodiment of the present invention;

[0048] Figure 5 This is a diagram of the deep neural network structure according to an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0050] like Figure 1 As shown, a method for assessing the fire status of a submersible compartment based on multi-source heterogeneous data fusion includes the following steps:

[0051] (1) Fire Status Assessment Method Based on Submarine Video Images

[0052] This patent designs a Temporal Segment Networks (TSN) model to assess fire status based on underwater vehicle video images. The TSN model has a two-branch structure, consisting of a convolutional neural network that processes the temporal dimension (Temporal Stream) and a convolutional neural network that processes the spatial dimension (Spatial Stream). Its model architecture is as follows: Figure 2 As shown, the model first acquires a long video sequence from the cabin camera, and then extracts short video clips using a sparse sampling method. These video clip samples follow a uniform distribution in the time dimension, so information can be collected from the sampled video clips to capture the long-span time structure.

[0053] Given an input video, it is divided into K segments. A small segment is randomly selected from each segment. The selected segment is then fed into a spatiotemporal dual-stream convolutional neural network to obtain category scores for each segment, and finally, these scores are merged.

[0054] The TSN model extracts shorter video segments from a long video sequence using sparse sampling, thereby acquiring long-span temporal information and classifying and identifying targets within the video. However, when the duration of key targets (such as flames and smoke) in the video is relatively short, the model must process a significant amount of redundant image information. This not only leads to substantial redundant computation but also results in low recognition efficiency, impacting the accuracy of fire detection. Therefore, this solution introduces attention mechanisms in both the temporal and spatial dimensions into the TSN model to filter redundant information.

[0055] The attention mechanism originates from the human brain's attention mechanism. When the human brain receives external information, such as visual or auditory information, it typically does not process and understand all of it, but rather focuses its attention on the most significant or interesting information. This helps filter out unimportant information and improves the efficiency of information processing. Applying the attention mechanism to image recognition allows for the use of a smaller receptive field to process the parts of the image of interest, similar to the human brain's attention mechanism, thereby reducing the computational dimensionality.

[0056] This method draws upon perception and target mechanisms for open environments to study a cross-temporal and spatiotemporal attention selection mechanism. This enables the automatic filtering of redundant information in videos and the rapid localization of key information in visual perception (represented as keyframes at the video level and key regions at the image level). For keyframe detection in video sequences, a time-related sequence prediction model is used to automatically extract video segments of interest. For key region detection in static images, a fully convolutional neural network is used to find regions of interest. By organically combining these two attention selection mechanisms, richer perceptual information is provided for rapid and accurate attention transfer and focusing.

[0057] The overall process of this fire status assessment method is as follows: Figure 2 As shown.

[0058] a. Attention extraction in the temporal dimension

[0059] This method first uses a time-series-based attention selection cognitive mechanism to calculate the weight of each video frame, and then selects key frames at the video level for focused attention.

[0060] Given an input video, audio and target dynamic information (optical flow features, etc.) are extracted. Then, a deep network model is used to effectively extract and fuse these two types of features, resulting in multi-frame encoded video segment features (audio and optical flow fusion features). Optical flow is a crucial element in motion detection, enabling the detection of independently moving targets without prior knowledge of the scene. Similarly, audio information (such as screams, explosions, etc.) can assist machines in quickly and accurately locating key video content in open environments.

[0061] Among them, the audio features are obtained by sampling and quantizing the original audio data in the video, and the optical flow features are obtained by using the Farneback dense optical flow algorithm after graying the video image;

[0062] This deep neural network processes and fuses audio and optical flow data separately. Its network structure is as follows: Figure 4 As shown, the network parameters are listed in Tables 1 and 2.

[0063] Table 1 Audio Processing Network Structure

[0064] Network layer Output feature shape parameter Fully connected layer 1 10×32 32×64 Fully connected layer 2 10×64 64×64

[0065] Note: The input audio data dimension C1 is 32, and the time series is taken as T=10 frames.

[0066] Table 2 Optical Flow Processing Network Structure

[0067]

[0068] Note: The time series is taken as T=10 frames, the video frame size is 224×224, the optical flow field of each video frame is a 2-dimensional directional field, and the optical flow is therefore the shape of the input optical flow is 224×224×20.

[0069] This invention addresses the attention-selection cognitive mechanism based on time series. It employs a deep network model to process and deeply fuse the optical flow and audio features of a video. The optical flow and audio features are processed separately through two sub-networks, and then a separate fusion network is used to deeply fuse the two types of perceptual features. Finally, the multi-frame fused features are encoded into video segment features.

[0070] The time-series-based attention selection cognitive mechanism constructs a model of the relationship between frames in a video segment through two-layer function transformation, as shown in the formula below. This model adopts a novel video keyframe recalibration strategy, which automatically obtains the weight W of each video frame through learning and filters redundant information based on the weight W, as shown in formula (1).

[0071] W=Sigmoid(Func2(Func1(Feature))) (1)

[0072] Among them, the Sigmoid() function is used to normalize the weight W, and Func1() and Func2() are two fully connected neural networks used to compress the time series features to the number of video frames. Their network structure parameters are shown in Tables 3 and 4.

[0073] Func1() and Func2() together form a "bottleneck" structure, which is widely used in the current mainstream residual network models. This structure first compresses and then expands the feature dimension, and performs non-linear operations through Sigmoid(). While reducing the complexity of the model, it effectively improves the model's expressive power, enabling the designed attention selection cognitive module to accurately express the importance of each frame of video.

[0074] During network training, the loss function value of the video-level attention module is calculated by combining the video understanding results fed back by the image-level attention selection cognitive module, thus realizing a cross-level attention selection cognitive mechanism.

[0075] 2.2) After locating keyframes in the video through the temporal attention mechanism, a spatial location-based attention selection cognitive mechanism is adopted for each keyframe. At the image level, image features of the region of interest are selected for focused attention, and the image features of the keyframe and key region are encoded into video segment features.

[0076] The target region of interest is obtained by rapidly scanning the global image;

[0077] The deep neural network uses two cascaded sub-networks to process the features of the input image. The "suppression" module performs multi-level feature downsampling in the spatial dimension, gradually compressing each two-dimensional feature channel into a real feature with a global receptive field. Then, the "augmentation" module performs multi-level feature upsampling in the spatial dimension, gradually restoring the global features to the spatial dimension size of the original input features.

[0078] The spatial location-based attention selection cognitive mechanism mainly consists of two branches, which are used to extract deep features and generate attention weights W for spatial location, as shown in the following formula: W i The weight of video frame i is learned through the attention selection cognitive mechanism at the video level. The Spatial_Squeeze() and Spatial_Expand() functions represent the suppression module and the augmentation module, respectively, which contain multiple convolutional layers and pooling layers.

[0079] The deep neural network processes the input image features using two cascaded subnetworks. An inhibition module performs multi-level feature downsampling in the spatial dimension, progressively compressing each two-dimensional feature channel into a real-valued feature with a global receptive field. Then, an augmentation module performs multi-level feature upsampling in the spatial dimension, progressively restoring the global features to the spatial dimension of the original input features. Its network structure is as follows: Figure 5 The parameters are shown in Tables 5 and 6.

[0080] Table 5. Network structure of the suppression module

[0081] Network layer Output feature shape parameter Convolutional layer 1 56×56×64 Stride = 1 3 × 3.64 Convolutional layer 2 56×56×64 Stride = 1 3 × 3.64 Global pooling layer 1×1×64 /

[0082] Table 6 Augmentation Module Network Structure

[0083] Network layer Output feature shape parameter Convolutional layer 1 1×1×64 Stride = 1 1×1.64 upsampling layer 56×56×64 /

[0084] Then, the importance is normalized using the Sigmoid() function, finally obtaining the spatial position weight W′ of the keyframe. i (x), representing the weight of the i-th frame at position x:

[0085] W′ i (x)=W i *Sigmoid(Spatial_Expand(Spatial_Squeeze(Feature i (x)))) (2)

[0086] The generated features based on the spatial attention selection cognitive mechanism are denoted as Feature′. i (x), the formula is shown below, and the T() function represents multiple cascaded linear or nonlinear transformation operations on the feature extraction branch:

[0087] Feature′ i (x)=T(Feature i (x))*W′ i (x) (3)

[0088] Finally, the key region features within multiple keyframes are encoded into video segment features, which are then input into a classifier to obtain video recognition results. These results are then fed back into a spatiotemporal dual-stream convolutional neural network at the video level to accurately locate fires and smoke in the video and to evaluate the detection probability.

[0089] (2) A method for assessing the fire status of a compartment based on thermal imaging images

[0090] The thermal imaging sensor presents the temperature field data of the monitored compartment area in the form of infrared thermal images. The flowchart of the compartment fire status assessment method based on infrared thermal images proposed in this patent is shown below. Figure 3 Thermal infrared images are a representation of the temperature distribution of an object's thermal radiation. Their key feature is temperature sensitivity, allowing for a direct visual representation of an object's temperature. Furthermore, they map the temperature of each point on the object to individual image pixels, enabling the observer to intuitively perceive the object's temperature.

[0091] f(x,y)=F(T(x,y)) (4)

[0092] Equation (1) is the formula for the direct relationship between point temperature and image pixels, where x,y represent the position of image coordinates, f(x,y) represents the pixel value of the displayed image, T(x,y) represents the temperature value of each frame of the corresponding image, and F(.) represents the conversion function from temperature to image, such as the pseudo-color operation function.

[0093] This calculation directly provides the values ​​of f(x,y) and T(x,y). Therefore, this method utilizes this characteristic of infrared thermal images to establish a correlation between specific regions and temperature. Specifically, by plotting specific regions within the infrared thermal image, and recognizing the direct correlation between image pixel values ​​and temperature values, the temperature value of a specific region can be directly obtained. The highest and lowest temperatures of each selected region can be clearly observed.

[0094] First, the monitoring screen is divided into several regions, or it can be set as a single area. A high-temperature threshold and a low-temperature threshold are set for each region. Second, the highest temperature of each region in each frame is detected. When the highest temperature of a region in the current frame is found to be greater than the high-temperature threshold or lower than the low-temperature threshold for that region, that frame is designated as the i-th frame and recorded as an abnormal region.

[0095] In the enclosed space of the cabin, prolonged equipment operation can cause changes in the surrounding temperature. Therefore, relying solely on the highest temperature of a single frame for early warning is prone to false alarms. This solution employs a continuous frame detection approach. The evaluation algorithm detects the next N consecutive frames after the i-th frame (i+1, i+2, ..., i+n), where N can be selected based on the application scenario. If the highest temperature in the area consistently exceeds the highest temperature threshold or consistently falls below the lowest temperature threshold, the area is deemed to have an abnormal temperature, and a high-temperature alarm is issued.

[0096] The advantages of this strategy are: 1) By selecting a region on the thermal infrared image, the temperature monitoring area inside the cabin can be easily set; 2) By adopting a one-to-one correspondence between the region temperature and the image pixel, additional calculations are avoided; 3) By sampling continuous frames for high temperature alarm monitoring, the false alarm rate is minimized.

[0097] (3) Fire Status Assessment Method for Submarine Compartments Based on Multi-Source Heterogeneous Data Fusion

[0098] By combining and fusion multiple heterogeneous fire early warning signal data, precise early warning of fire hazards within the cabin can be achieved. These heterogeneous fire early warning signal data include natural light video data and image recognition results from cabin cameras, thermal infrared image recognition results from thermal imaging sensors, and alarm data from smoke detectors. In complex cabin environments, fire status assessment data from these multiple sensors needs to be processed and comprehensively analyzed to accurately locate fire hazards within the cabin. However, the signal data transmitted by different sensors are often heterogeneous. Although conventional methods can be used to simply combine these signals, such approaches often fail to allow the signals from different sensors to complement each other, leading to false alarms or missed detections in fire warnings.

[0099] This embodiment proposes a fuzzy comprehensive evaluation method to fuse the aforementioned multi-source heterogeneous fire alarm assessment data. The multi-source input data for this evaluation algorithm includes:

[0100] Thermal infrared images from a thermal imaging sensor;

[0101] Natural light video data and image recognition results from the in-cabin camera;

[0102] Alarm data from flame detectors and smoke detectors in the fire suppression system;

[0103] This method uses fuzzy comprehensive evaluation to comprehensively assess various fire alarm assessment data. It utilizes membership functions from fuzzy mathematics to transform fuzzy sets, essentially performing a quantitative comprehensive assessment of an entity influenced by multiple factors using membership functions. Primarily targeting the fuzziness and uncertainty of data, it effectively assesses fuzzy and difficult-to-quantify issues, thus enabling better fusion of multi-source fire alarm assessment data to obtain a comprehensive cabin fire status assessment result.

[0104] The evaluation method consists of the following steps: establishing a factor set, establishing an evaluation set, determining weights, establishing a fuzzy matrix, and performing fuzzy synthesis operations.

[0105] a. Establishing a factor set

[0106] For the object being evaluated, first analyze what factors influence the evaluation results, and establish the corresponding factor set U = {u1, u2, ..., u...} m This method uses fire alarm assessment data from various sensors as sub-elements in a factor set, as shown in the example below: (The specific factor set elements are determined according to the actual sensors deployed in the compartment).

[0107]

[0108] b. Establish an evaluation set

[0109] To provide a direct assessment of the probability of a cabin fire, this method divides the probability into n evaluation levels and establishes an evaluation set V = {v1, v2, ..., v...}. n The evaluation level classification is crucial; too coarse a classification will affect the accuracy of the assessment, while too fine a classification will increase computation time and may lead to unclear evaluation strategies. This system offers three evaluation levels: Level 3 (Safety, Warning, Fire), Level 4 (Safety, Temperature Warning, High Temperature Alarm, Fire), or Level 5 (Safety, Temperature Warning, Temperature Anomaly, High Temperature Alarm, Fire), allowing users to flexibly adjust the levels.

[0110] c. Determine the weights

[0111] Establish a weight set W = {w1, w2, ..., w m The weighting reflects the position and role of each factor in the assessment process, and is a key aspect of ensuring assessment quality. To ensure the accuracy of the weighting factors, this method employs a matrix comparison approach, where experts compare the importance of various fire condition assessment factors to determine the weight of each factor.

[0112] This method uses matrix comparison to obtain weight factors. Matrix comparison constructs a comparison matrix by comparing factors pairwise, and then calculates the largest eigenvalue and largest eigenvector of the comparison matrix. The largest eigenvector is the weight vector. An m×m comparison matrix P is constructed, with all diagonal elements being 1. The judgment values ​​for the comparison matrix P are shown in the table below.

[0113] <![CDATA[p ij Values]]> definition 1 <![CDATA[u i with u j Comparison, equally important 3 <![CDATA[u i with u j Comparison, u i Slightly important 5 <![CDATA[u i with u j Comparison, u i More important 7 <![CDATA[u i with u j Comparison, u i Very important 9 <![CDATA[u i with u j Comparison, u i Absolutely important 2,4,6,8 intermediate state reciprocal <![CDATA[u j with u i For comparison, take 1 / p. ij ]]>

[0114] Based on the comparison matrix, according to PA = λ max Find its largest eigenvalue λ. max And its corresponding eigenvector A. The value of the eigenvector A is the weight set W of each factor.

[0115] d. Establish a fuzzy matrix

[0116] Establish fuzzy matrices R, r ij Represents the i-th factor u i For the j-th evaluation level v j The degree of membership.

[0117]

[0118] The characteristic function of a fuzzy set is called a membership function. Given a fuzzy union S on a universe of discourse G, for... There exists a number S (x)∈[] corresponds to x, that is, the membership degree of x to S, that is, there exists a mapping g. S (x), this mapping is called the membership function of the fuzzy set S. When x takes a certain value, g S S(x) represents the membership degree corresponding to x.

[0119] Membership degree represents the degree to which x belongs to the fuzzy set S. When S(x)→1, it means that the degree to which x belongs to S is getting higher and higher; when S(x)→0, it means that the degree to which x belongs to S is getting lower and lower; when S(x)=0, it means that x absolutely does not belong to S; when S(x)=1, it means that x absolutely belongs to S.

[0120] The membership function in this method is selected according to the following principle: if the fire assessment data of the elements in the factor set is alarm data, then for the assessment result of "fire" in the evaluation set, when an alarm occurs, it is directly set to S(x) = 1, and the rest of the assessment results are set to S(x) = 0; for the fire status assessment results obtained from various image data, a quantized threshold is set to determine the membership function.

[0121] e. Fuzzy synthesis operation

[0122] The fuzzy synthesis operation is shown in the following equation, which involves matrix multiplication of the weight set vector W and the fuzzy matrix R:

[0123]

[0124] Where B represents the fuzzy comprehensive evaluation vector, and the magnitude of each element in B represents the degree to which it belongs to each evaluation level. This method adopts the maximum membership principle and selects the element b with the largest value in B. i That is, selecting fire assessment level v i As a result of this assessment. i The size indicates that the result of this evaluation is v. i The probability is obtained by combining heterogeneous data from multiple sources to arrive at a comprehensive cabin fire alarm assessment result.

[0125] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for assessing the fire status of a submersible compartment based on multi-source heterogeneous data fusion, characterized in that, Includes the following steps: 1) Acquire video streams of the cabin under natural light conditions; 2) For the collected video, divide it into K short video segments, and randomly select a small segment from each short video segment; input the selected small segment into a spatiotemporal dual-stream convolutional neural network to obtain the category scores of different small segments, and then fuse them to locate the fire and smoke areas in the video stream; The spatiotemporal dual-stream convolutional neural network includes: a temporal dimension attention mechanism network, a spatial dimension attention mechanism network, and a fire identification classifier; 2.1) Perform keyframe detection on short segments of video sequences and use a time-related sequence prediction model to extract video segments of interest; The temporal attention mechanism network employs a time-series-based attention selection cognitive mechanism to calculate the weight of each video frame, selecting key frames at the video level for focused attention, as follows: Given an input video, extract the audio and target dynamic information from the video, and then use a deep neural network model to fuse the two features to obtain multi-frame encoded video segment features; The time-series-based attention selection cognitive mechanism constructs a model of the relationships between frames in a video segment through a two-layer function transformation, as shown in the following formula. (1) in, The function is used to normalize the weights W. and These are two fully connected neural networks, used to compress the time-series features to the number of video frames; During the training of the network, the loss function value of the video-level attention module is calculated by combining the video understanding results fed back by the image-level attention selection cognitive module, thereby realizing a cross-level attention selection cognitive mechanism. 2.2) After locating keyframes in the video through the temporal attention mechanism, a spatial location-based attention selection cognitive mechanism is adopted for each keyframe. At the image level, image features of the region of interest are selected for focused attention, and the image features of the keyframe and key region are encoded into video segment features. The deep neural network uses two cascaded sub-networks to process the features of the input image. The suppression module performs multi-level feature downsampling in the spatial dimension, gradually compressing each two-dimensional feature channel into a real feature with a global receptive field. Then, the augmentation module performs multi-level feature upsampling in the spatial dimension, gradually restoring the global features to the spatial dimension size of the original input features. The spatial location-based attention selection cognitive mechanism mainly consists of two different branches, which are used to extract deep features and generate attention weights for spatial location, respectively. The deep neural network uses two cascaded sub-networks to process the features of the input image. The suppression module performs multi-level feature downsampling in the spatial dimension, gradually compressing each two-dimensional feature channel into a real feature with a global receptive field. Then, the augmentation module performs multi-level feature upsampling in the spatial dimension, gradually restoring the global features to the spatial dimension size of the original input features. Then, using Sigmoid The ( ) function normalizes the importance level, ultimately yielding the spatial location weights of the keyframes. , indicating the first i Frame image at position x Weights on: (2) Represents video frames i The weights are learned through the attention selection cognitive mechanism at the video level. The Spatial_Squeeze() and Spatial_Expand() functions represent the suppression module and the augmentation module, respectively, which contain multiple convolutional layers and pooling layers. The generated features based on the spatial attention selection cognitive mechanism are denoted as The formula is shown below. T The () function represents multiple cascaded linear or nonlinear transformation operations on the feature extraction branch: (3) Finally, the key region features within multiple keyframes are encoded into video segment features, which are then input into a classifier to obtain video recognition results. These results are then fed back into a spatiotemporal dual-stream convolutional neural network at the video level to accurately locate fires and smoke in the video and to evaluate the detection probability. 3) Obtain thermal maps of the monitored compartment area and assess the fire status of the compartment based on the thermal imaging images; The fire status assessment of the compartment is based on thermal imaging images, as detailed below: 3.1) Divide the monitoring screen into several areas, and set a high temperature threshold and a low temperature threshold for each area; 3.2) Detect the highest temperature of each region in each frame. When the highest temperature of a region in the current frame is detected to be greater than the high temperature threshold or lower than the low temperature threshold of that region, the frame is set as the i-th frame and recorded as an abnormal region. 3.3) A continuous frame detection processing scheme is adopted. If the i-th frame is an abnormal region, the continuous N frames after the i-th frame are detected, i.e., the i+1, i+2, ..., i+n frames, where N is set according to the application scenario. If the highest temperature in the region is continuously higher than the highest temperature threshold or continuously lower than the lowest temperature threshold, the temperature in the region is determined to be abnormal, and a high temperature alarm message is sent. 4) Combine various heterogeneous fire early warning signal data for collaborative analysis and information fusion to provide accurate early warning of fire hazards in the cabin; the various heterogeneous fire early warning signal data include natural light video data and image recognition results from cabin cameras, thermal infrared image recognition results from thermal imaging sensors, and alarm data from smoke detectors.

2. The method for assessing the fire status of a submersible compartment based on multi-source heterogeneous data fusion according to claim 1, characterized in that, In step 2), a deep network model is used to process and deeply fuse the optical flow and audio features of the video. The optical flow and audio features are processed by two sub-networks respectively, and then a separate fusion network is used to deeply fuse the two types of perceptual features. Finally, the multi-frame fused features are encoded into video segment features.

3. The method for assessing the fire status of a submersible compartment based on multi-source heterogeneous data fusion according to claim 1, characterized in that, In step 3), the thermal map of the monitoring chamber area is obtained by drawing the specific area in the infrared thermal image based on the temperature value of the specific area, and establishing the correlation between the pixel value of the image and the temperature value. (4) Equation (4) is the formula for the direct relationship between point temperature and image pixels, where x, y represent the position of the image coordinates, and f( x,y () represents the pixel values ​​of the displayed image. T ( x,y ) represents the temperature value of each frame of the corresponding image, where F(.) represents the temperature-to-image conversion function.

4. The method for assessing the fire status of a submersible compartment based on multi-source heterogeneous data fusion according to claim 1, characterized in that, In step 4), information fusion is performed using the fuzzy comprehensive evaluation method.

Images