Fire situation sensing method and system based on multi-modal dynamic weight fusion

Abstract

The invention provides a fire situation sensing method and system based on multi-modal dynamic weight fusion. The method comprises the following steps: carrying out detection and image segmentation on a flame region by utilizing a YOLOv8-seg model; constructing a red channel intensity time sequence; identifying the types of comburent by using a SpectraNet network, and constructing spectral intensity data of a characteristic wave band under a time sequence; calculating a local volatility score, calculating a fusion weight according to the local volatility score, and constructing a fire intensity time sequence of weighted fusion; and performing segmentation fitting by using a least square method, and judging the trend change of the fire behavior in each period of time. According to the method, image segmentation and spectral analysis technologies are combined, and the red channel pixel intensity of the flame image and the spectral intensity of the characteristic wave band are dynamically fused, so that the comprehensiveness and stability of fire perception are improved; segmented fitting and slope analysis are carried out on the fused time sequence, and the rising, falling or stable state of the fire behavior is clearly output.

Description

Technical Field

[0001] This invention relates to the field of spectral and image recognition technology, and in particular to a fire situation perception method and system based on multimodal dynamic weight fusion. Background Technology

[0002] With rapid urbanization and intensified global climate change, the frequency of urban and forest fires is constantly rising, posing a serious threat to public safety and ecosystems. Firefighting robots, as emerging intelligent equipment, possess the ability to autonomously perceive and intervene in high-risk environments, and have become an important direction in intelligent firefighting technology research. Among these, accurately judging the size and evolution trend of a fire is a key issue in improving the efficiency and safety of robot firefighting.

[0003] Currently, commonly used fire intensity assessment techniques primarily rely on flame region recognition based on image segmentation. This involves acquiring images of the fire scene using visual sensors and employing deep learning models to segment the flame regions at the pixel level, thereby obtaining the spatial distribution and size of the flames. With the development of convolutional neural networks (CNNs) and real-time semantic segmentation techniques, the accuracy and speed of flame segmentation have been significantly improved. However, the shape of the flame fluctuates drastically due to factors such as wind, oxygen, and combustible materials, leading to high-frequency jitter in the segmentation results. Furthermore, ambient light interference (such as sunlight and artificial light) can easily cause false positives or false negatives. Additionally, the segmented region only reflects the "geometric" size of the flame and cannot directly measure the fire intensity, such as temperature and thermal radiation energy. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a fire situation perception method and system based on multimodal dynamic weight fusion. This invention integrates image segmentation and spectral intensity information to comprehensively assess the size and development trend of the fire over time, thereby further improving the accuracy and robustness of fire situation perception.

[0005] The technical solution of this invention is: a fire situation awareness method based on multimodal dynamic weight fusion, comprising the following steps:

[0006] S1) Obtain images of the fire scene and use the YOLOv8-seg model to detect and segment the flame area; obtain the red channel mask area and extract the red channel intensity time series;

[0007] S2) Real-time acquisition of full-band spectral data of the flame, using the SpectraNet network to identify the type of combustible material, selecting characteristic bands according to the type of combustible material, and obtaining the spectral intensity data of the characteristic bands in time series.

[0008] S3) Normalize and perform sliding window Z-score analysis on the red channel intensity time series and spectral intensity data respectively to calculate the local volatility score;

[0009] S4) Calculate the fusion weight dynamically based on the local volatility scores of the two types of signals, and construct a weighted fusion fire intensity time series;

[0010] S5) Perform least squares segmented fitting on the fused fire intensity time series to determine the fire trend change in each segment; output the fire trend judgment result to control the response strategy of the fire extinguishing robot.

[0011] Preferably, in step S1), the YOLOv8-seg model performs target detection and pixel-level semantic segmentation on the input fire scene image, and extracts the spatial distribution area of ​​flames in each frame of the fire scene image; the output of the YOLOv8-seg model includes detection boxes, flame categories, and flame region segmentation masks.

[0012] Preferably, in step S1), the red channel intensity time series is extracted, including the following steps:

[0013] S11) Split the RGB image frame sequence into red channel images;

[0014] S12) Filter the red channel image according to the segmentation mask, and only select pixels in the red channel image that correspond to the flame region;

[0015] S13) The sum or average of the non-zero pixels in the red channel within the statistical mask is used as the visual intensity value r of the flame in that frame image. i ;

[0016] S14) Following steps S11)-S13), calculate the visual flame intensity values ​​of all fire scene images to obtain the red channel intensity time series R(t) = {r1, r2, ... r i ...,r T}, where t represents the sampling time; T is the total number of image frames acquired within time t.

[0017] Preferably, in step S2), the SpectraNet network includes an input layer, a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a flattening layer, a first fully connected layer, a random dropout layer, and an output layer.

[0018] Preferably, in step S2), the SpectraNet network is used to identify the type of combustible material, which specifically includes the following steps:

[0019] S21) Input the full-band spectral data of the flame through the input layer;

[0020] S22) The first convolutional layer is used to initially extract the feature relationships between local bands, and local correlation features are extracted through one-dimensional convolution operation. Then, batch normalization and ReLU activation function are used for processing.

[0021] S23) After downsampling each channel using the first pooling layer, the second convolutional layer is used to further extract deeper spectral structure information; then the feature map is reduced in dimensionality using the second pooling layer.

[0022] S24) The multi-channel two-dimensional feature map is unfolded into a one-dimensional vector through a flattening layer, and then the flattened feature vector is compressed and mapped using the first fully connected layer, and a non-linear combination of linear transformation and ReLU activation function is used.

[0023] S25) The Softmax activation function is used to normalize the scores of different combustible categories, and the combustible type is output through the output layer.

[0024] Preferably, in step S2), the characteristic bands corresponding to the type of combustible material are matched, and then the spectral intensity of the characteristic bands of the combustible material is measured by a spectrometer to obtain the spectral intensity data of the characteristic bands in the time series S(t)={s1,s2,...s... i ..,s T}; T is the total number of image frames acquired within time t, s i This represents the spectral intensity value of the flame in the i-th frame of spectral intensity data after frame synchronization. The total acquisition time for spectral and visual data is guaranteed to be the same, and frame synchronization of spectral and visual data is maintained at the same frame interval.

[0025] Preferably, in step S3), for the red channel intensity time series R(t) =

[0026] {r1,r2,...r i ....,r T The expression after normalization is:

[0027]

[0028] In the formula, The normalized visual intensity value of the flame; r i r represents the visual intensity value of the flame in the i-th frame; min r max These are the minimum and maximum visual flame intensity values ​​within R(t), respectively;

[0029] For spectral intensity data S(t)={s1,s2,...s... i ..,s T The expression after normalization is:

[0030]

[0031] In the formula, The normalized spectral intensity value of the flame; s i This represents the spectral intensity value of the flame in the i-th frame of spectral intensity data after frame synchronization; s min s max These are the spectral intensity values ​​of the minimum and maximum flames, respectively.

[0032] Preferably, in step S3), a sliding window Z-score is used to evaluate the normalized red channel intensity time series R. norm (t) and spectral intensity data S norm The degree of anomaly at each time step (t) is analyzed, i.e.:

[0033]

[0034] In the formula, Z spec,i for Z-score of quality score; μ s For spectral intensity data S norm The mean of (t); σ s For spectral intensity data S norm The standard deviation of (t); ε is the minimum value;

[0035]

[0036] In the formula, Z grar,i for Quality score Z-score; μ r For the red channel intensity time series R norm The mean of (t); σ r For the red channel intensity time series R norm The standard deviation of (t); ε is the minimum value;

[0037] Then for Z grar,i Z spec,i After normalization, a dynamic score between 0 and 1 is obtained, i.e.:

[0038]

[0039] In the formula, Score grar,i Normalization Quality rating; Score spec,i for Quality rating; Z grar_min Z grar_max The red channel intensity time series R are respectively normThe minimum and maximum values ​​of the quality score of (t); Z spec_min Z spec_max The spectral intensity data S are respectively norm The minimum and maximum values ​​of the quality score of (t).

[0040] Preferably, in step S4), the fusion weight α(t) is dynamically calculated based on the local volatility scores of the two types of signals; the functional expression for the dynamic fusion weight with an acquisition time length of t is:

[0041] α(t)={α1,α2,..,α i ,..,α T};

[0042] The constructed weighted fusion of fire intensity time series F(t) is expressed as:

[0043] F(t)=α(t·S norm (t)+(1-α(t)·R norm (t));

[0044] in:

[0045]

[0046] In the formula, α i ε represents the dynamic weight corresponding to the data in the i-th frame; ε is the minimum value.

[0047] Preferably, in step S5), the slope sign of each trend segment is used to identify the direction of fire trend change, and the fire trend judgment result includes a fire intensity change trend diagram, trend slope value, and response suggestions.

[0048] Preferably, the present invention also provides a fire situation awareness system based on multimodal dynamic weight fusion, comprising:

[0049] The image acquisition module is used to acquire images of the fire scene;

[0050] The image segmentation module uses the YOLOv8-seg model to detect and segment the flame region.

[0051] The red channel temporal intensity construction module is used to construct the red channel temporal intensity sequence based on the fire scene image and the mask segmented by the image segmentation module;

[0052] The spectral acquisition module is used to acquire full-band spectral data of the flame in real time.

[0053] Combustible material identification module, used to identify the types of combustible materials via the SpectraNet network;

[0054] The spectral intensity construction module is used to select characteristic bands according to the type of combustible material and construct the spectral intensity data of the characteristic bands in a time series.

[0055] The signal fluctuation scoring module is used to perform normalization and sliding window Z-score analysis on the red channel intensity time series and spectral intensity data, respectively, and calculate the local fluctuation score.

[0056] The weighted fusion module is used to dynamically calculate the fusion weight based on the local volatility scores of the two types of signals and to construct a weighted fused fire intensity time series.

[0057] The fire trend fitting module performs least squares segmented fitting on the fused fire intensity time series to determine the fire trend change in each time period and outputs the fire trend judgment result.

[0058] The beneficial effects of this invention are as follows:

[0059] 1. This invention combines image segmentation technology with spectral analysis technology, and improves the comprehensiveness and stability of fire perception by dynamically fusing the red channel pixel intensity and the spectral intensity of the characteristic band of the flame image.

[0060] 2. This invention achieves rapid classification of flame spectra based on the SpectraNet network, which can automatically identify the types of combustibles and select the most representative characteristic spectral bands for subsequent analysis, effectively improving the ability to make targeted judgments on specific fire sources.

[0061] 3. This invention, by performing segmented fitting and slope analysis on the fused time series, clearly outputs the rising, falling, or stable state of the fire, and has the ability to identify trends. Attached Figure Description

[0062] Figure 1 This is a flowchart illustrating the method of Embodiment 1 of the present invention;

[0063] Figure 2 This is a structural framework diagram of the YOLOv8-seg model of the present invention;

[0064] Figure 3 These are the spectral diagrams of different combustibles in embodiments of the present invention;

[0065] Figure 4 This is a trend diagram before and after dynamic fusion in an embodiment of the present invention;

[0066] Figure 5 This is a flame trend diagram after fusion in an embodiment of the present invention;

[0067] Figure 6 This is a structural framework diagram of the system in Embodiment 2 of the present invention. Detailed Implementation

[0068] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0069] Example 1

[0070] like Figure 1 As shown, this embodiment provides a fire situation awareness method based on multimodal dynamic weight fusion, including the following steps:

[0071] S1) Obtain images of the fire scene and use the YOLOv8-seg model to detect and segment the flame area; obtain the red channel mask area and extract the red channel intensity time series;

[0072] This embodiment uses a camera or industrial camera to acquire video image streams from the fire scene in real time, and performs frame extraction processing according to a set frame rate to obtain an image frame sequence for flame detection and segmentation; the frame extraction frequency in this embodiment can be set to 10 frames per second; at the same time, the image size is uniformly adjusted to 640×640 pixels; the image format is standardized to RGB three-channel, and saved as PNG or JPEG format; the image can be rotated or cropped as needed to ensure consistent viewing angle.

[0073] This embodiment utilizes the YOLOv8-seg model to perform target detection and pixel-level semantic segmentation on the input fire scene image, extracting the spatial distribution region of flames in each frame of the fire scene image; the output of the YOLOv8-seg model includes detection boxes, flame categories, and flame region segmentation masks, where a value of 1 represents a flame region and 0 represents the background.

[0074] like Figure 2 As shown, the YOLOv8-seg model includes a backbone network, a neck network, and a head network. The backbone network uses CSPDarknet to extract multi-level features. The CSPDarknet includes multiple convolutional layers (Conv), CBS modules, and C2f modules. The C2f modules include C2f_1, C2f_2, C2f_3, and C2f_4 modules.

[0075] The described neck network is used to fuse multi-scale features. The upsampling module Upsample and the concatenation module Concat upsample and concatenate the high-resolution feature map with the low-resolution feature map to output a higher-resolution feature map. The C2f_5 module further fuses feature maps at different levels, and the C2f_6 module further enhances semantic expression. Subsequently, each layer outputs different feature level information X0, X1, and X2 for use in subsequent detection and segmentation heads.

[0076] The head network is used to output the flame target and the corresponding segmentation mask; three anchors are output at each scale, and the bounding box, class probability and confidence score of the target are predicted for each anchor.

[0077] In this embodiment, the extraction of the red channel intensity time series includes the following steps:

[0078] S11) Split the RGB image frame sequence into red channel images;

[0079] S12) Filter the red channel image according to the segmentation mask, and only select pixels in the red channel image that correspond to the flame region;

[0080] S13) The sum or average of the non-zero pixels in the red channel within the statistical mask is used as the visual intensity value r of the flame in that frame image. i ;

[0081] S14) Following steps S11)-S13), calculate the visual flame intensity values ​​of all fire scene images to obtain the red channel intensity time series R(t) = {r1, r2, ... r i ...,r T}, where t represents time; T is the total number of image frames acquired within time t, and r i This represents the visual intensity value of the flame in the i-th frame.

[0082] S2) Real-time acquisition of full-band spectral data of the flame, using the SpectraNet network to identify the type of combustible material, selecting characteristic bands according to the type of combustible material, and obtaining the spectral intensity data of the characteristic bands in time series.

[0083] In this embodiment, high-resolution spectral sensors are used to collect full-band spectral data of the flame in real time.

[0084] In this embodiment, the SpectraNet network includes an input layer, a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a flattening layer, a first fully connected layer, a random dropout layer, and an output layer.

[0085] Among them, the Dropout layer randomly discards some neurons with a probability of 0.5 during training, which effectively prevents network overfitting and improves generalization ability.

[0086] Identifying combustible types using the SpectraNet network specifically includes the following steps:

[0087] S21) Input the full-band spectral data of the flame through the input layer. In this embodiment, its dimension is [1,2048], which represents the 2048 band spectral intensity values ​​collected in the current frame.

[0088] S22) The first convolutional layer is used to initially extract the feature relationships between local bands, and local correlation features are extracted through one-dimensional convolution operation. Then, batch normalization and ReLU activation function are used for processing.

[0089] S23) After downsampling each channel using the first pooling layer, the second convolutional layer is used to further extract deeper spectral structure information; then the feature map is reduced in dimensionality using the second pooling layer.

[0090] S24) The multi-channel two-dimensional feature map is unfolded into a one-dimensional vector through a flattening layer, and then the flattened feature vector is compressed and mapped using the first fully connected layer, and a non-linear combination of linear transformation and ReLU activation function is used.

[0091] S25) The Softmax activation function is used to normalize the scores of different combustible categories, and the combustible type is output through the output layer, such as wood, plastic, alcohol, gasoline, etc.

[0092] Match the characteristic waveband to the type of combustible material, such as... Figure 3 The diagram shows the spectral characteristics of wood-based combustibles and plastic combustion.

[0093] Then, the spectral intensity of the characteristic bands of the combustible material was measured using a spectrometer to obtain the spectral intensity data of the characteristic bands over time, S(t)={s1,s2,..s i ..,s T}; T is the total number of image frames acquired within time t, s i This represents the spectral intensity value of the flame in the i-th frame of spectral intensity data after frame synchronization. The total acquisition time for spectral and visual data is guaranteed to be the same, and frame synchronization of spectral and visual data is maintained at the same frame interval.

[0094] S3) Normalize and perform sliding window Z-score analysis on the red channel intensity time series and spectral intensity data respectively to calculate the local volatility score;

[0095] For the red channel intensity time series R(t) = {r1, r2, ... r... i ....,r T The expression after normalization is:

[0096]

[0097] In the formula, The normalized visual intensity value of the flame; r i r represents the visual intensity value of the flame in the i-th frame; min r max These are the minimum and maximum visual flame intensity values ​​within R(t), respectively;

[0098] For spectral intensity data S(t)={s1,s2,...s... i ..,s T The expression after normalization is:

[0099]

[0100] In the formula, The normalized spectral intensity value of the flame; s i This represents the spectral intensity value of the flame in the i-th frame of spectral intensity data after frame synchronization; s min s max These are the spectral intensity values ​​of the minimum and maximum flames, respectively.

[0101] Preferably, in step S3), a sliding window Z-score is used to evaluate the normalized red channel intensity time series R. norm (t) and spectral intensity data S norm The degree of anomaly at each time step (t) is analyzed, i.e.:

[0102]

[0103] In the formula, Z spec,i for Z-score of quality score; μ s For spectral intensity data S norm The mean of (t); σ s For spectral intensity data S norm The standard deviation of (t); ε is the minimum value;

[0104]

[0105] In the formula, Z grar,i for Quality score Z-score; μ r For the red channel intensity time series R norm The mean of (t); σ r For the red channel intensity time series R norm The standard deviation of (t); ε is the minimum value;

[0106] Then for Z grar,i Z spec,i After normalization, a dynamic score between 0 and 1 is obtained, i.e.:

[0107]

[0108] In the formula, Score grar,i Normalization Quality rating; Scorespec,i for Quality rating; Z grar_min Z grar_max The red channel intensity time series R are respectively norm The minimum and maximum values ​​of the quality score of (t); Z spec_min Z spec_max The spectral intensity data S are respectively norm The minimum and maximum values ​​of the quality score of (t).

[0109] S4) Calculate the fusion weight dynamically based on the local volatility scores of the two types of signals, and construct a weighted fusion fire intensity time series;

[0110] The fusion weight α(t) is dynamically calculated based on the local volatility scores of the two types of signals; the functional expression for the dynamic fusion weight with an acquisition time of t is:

[0111] α(t)={α1,α2,..,α i ,..,α T};

[0112] The constructed weighted fusion of fire intensity time series F(t) is expressed as:

[0113] F(t)=α(t·S norm (t)+(1-α(t)·R norm (t));

[0114] in:

[0115]

[0116] In the formula, α i ε represents the dynamic weight corresponding to the data in the i-th frame; ε is the minimum value.

[0117] Examples of results before and after fusion can be found in [link / reference]. Figure 4 In this graph, the blue line represents the visual trend, the yellow line represents the spectral trend, the black line represents the merged trend, and the purple dashed line represents the dynamic weighted line graph.

[0118] S5) Perform least squares segmented fitting on the fused fire intensity time series to determine the fire trend change in each segment; output the fire trend judgment result to control the response strategy of the fire extinguishing robot.

[0119] In this embodiment, the slope sign of each trend segment is used to identify the direction of fire trend change, and the fire trend judgment result includes a fire intensity change trend graph, trend slope value, and response suggestions.

[0120] The fitted straight line is used to determine the change in fire intensity based on its slope k; a positive value indicates that the fire intensity is rising, and a negative value indicates that the fire intensity is falling. For example... Figure 5 As shown, a red background represents a rising fire, and a green background represents a falling fire.

[0121] Example 2

[0122] like Figure 2 As shown, this embodiment provides a fire situation awareness system based on multimodal dynamic weight fusion, including:

[0123] The image acquisition module is used to acquire images of the fire scene. The image acquisition module is a camera or an industrial camera. The camera or industrial camera acquires video image streams from the fire scene in real time and performs frame extraction processing according to a set frame rate to obtain an image frame sequence for flame detection and segmentation.

[0124] The image segmentation module uses the YOLOv8-seg model to detect and segment the flame region. The YOLOv8-seg model performs target detection and pixel-level semantic segmentation on the input fire scene image, extracting the spatial distribution region of the flame in each frame of the fire scene image. The output of the YOLOv8-seg model includes a detection box, flame category, and flame region segmentation mask, where a value of 1 represents the flame region and 0 represents the background.

[0125] The red channel time series intensity construction module is used to construct red channel time series intensity sequences, and specifically includes the following steps:

[0126] S11) Split the RGB image frame sequence into red channel images;

[0127] S12) Filter the red channel image according to the segmentation mask, and only select pixels in the red channel image that correspond to the flame region;

[0128] S13) The sum or average of the non-zero pixels in the red channel within the statistical mask is used as the visual intensity value r of the flame in that frame image. i ;

[0129] S14) Following steps S11)-S13), calculate the visual flame intensity values ​​of all fire scene images to obtain the red channel intensity time series R(t) = {r1, r2, ... r i ...,r T}, where t represents time; T is the total number of image frames acquired within time t, and r i This represents the visual intensity value of the flame in the i-th frame.

[0130] The spectral acquisition module is used to acquire full-band spectral data of the flame in real time. The spectral acquisition module is a high-resolution spectral sensor that acquires full-band spectral data of the flame in real time.

[0131] The combustible material identification module uses a SpectraNet network to identify the types of combustible materials. The SpectraNet network includes an input layer, a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a flattening layer, a first fully connected layer, a random dropout layer, and an output layer.

[0132] Among them, the Dropout layer randomly discards some neurons with a probability of 0.5 during training, which effectively prevents network overfitting and improves generalization ability.

[0133] Identifying combustible types using the SpectraNet network specifically includes the following steps:

[0134] S21) Input the full-band spectral data of the flame through the input layer. In this embodiment, its dimension is [1,2048], which represents the 2048 band spectral intensity values ​​collected in the current frame.

[0135] S22) The first convolutional layer is used to initially extract the feature relationships between local bands, and local correlation features are extracted through one-dimensional convolution operation. Then, batch normalization and ReLU activation function are used for processing.

[0136] S23) After downsampling each channel using the first pooling layer, the second convolutional layer is used to further extract deeper spectral structure information; then the feature map is reduced in dimensionality using the second pooling layer.

[0137] S24) The multi-channel two-dimensional feature map is unfolded into a one-dimensional vector through a flattening layer, and then the flattened feature vector is compressed and mapped using the first fully connected layer, and a non-linear combination of linear transformation and ReLU activation function is used.

[0138] S25) The Softmax activation function is used to normalize the scores of different combustible categories, and the combustible type is output through the output layer.

[0139] The spectral intensity construction module is used to select characteristic bands according to the type of combustible material and construct the spectral intensity data of the characteristic bands in a time series; the constructed spectral intensity data S(t)={s1,s2,..s i ..,s T}; T is the total number of image frames acquired within time t, s iThis represents the spectral intensity value of the flame in the i-th frame of spectral intensity data after frame synchronization. The total acquisition time for spectral and visual data is guaranteed to be the same, and frame synchronization of spectral and visual data is maintained at the same frame interval.

[0140] The signal fluctuation scoring module is used to perform normalization and sliding window Z-score analysis on the red channel intensity time series and spectral intensity data, respectively, and calculate the local fluctuation score.

[0141] For the red channel intensity time series R(t) = {r1, r2, ... r... i ....,r T The expression after normalization is:

[0142]

[0143] In the formula, The normalized visual intensity value of the flame; r i r represents the visual intensity value of the flame in the i-th frame; min r max These are the minimum and maximum visual flame intensity values ​​within R(t), respectively;

[0144] For spectral intensity data S(t)={s1,s2,...s... i ..,s T The expression after normalization is:

[0145]

[0146] In the formula, The normalized spectral intensity value of the flame; s i This represents the spectral intensity value of the flame in the i-th frame of spectral intensity data after frame synchronization; s min s max These are the spectral intensity values ​​of the minimum and maximum flames, respectively.

[0147] The normalized red channel intensity time series R was analyzed using a sliding window Z-score. norm (t) and spectral intensity data S norm The degree of anomaly at each time step (t) is analyzed, i.e.:

[0148]

[0149] In the formula, Z spec,i for Z-score of quality score; μ s For spectral intensity data S norm The mean of (t); σ s For spectral intensity data S normThe standard deviation of (t); ε is the minimum value;

[0150]

[0151] In the formula, Z grar,i for Quality score Z-score; μ r For the red channel intensity time series R norm The mean of (t); σ r For the red channel intensity time series R norm The standard deviation of (t); ε is the minimum value;

[0152] Then for Z grar,i Z spec,i After normalization, a dynamic score between 0 and 1 is obtained, i.e.:

[0153]

[0154] In the formula, Score grar,i Normalization Quality rating; Score spec,i for Quality rating; Z grar_min Z grar_max The red channel intensity time series R are respectively norm The minimum and maximum values ​​of the quality score of (t); Z spec_min Z spec_max The spectral intensity data S are respectively norm The minimum and maximum values ​​of the quality score of (t).

[0155] The weighted fusion module is used to dynamically calculate the fusion weight based on the local volatility scores of the two types of signals and to construct a weighted fused fire intensity time series.

[0156] The fusion weight α(t) is dynamically calculated based on the local volatility scores of the two types of signals; the functional expression for the dynamic fusion weight with an acquisition time of t is:

[0157] α(t)={α1,α2,..,α i ,..,α T};

[0158] The constructed weighted fusion of fire intensity time series F(t) is expressed as:

[0159] F(t)=α(t·S norm (t)+(1-α(t)·R norm (t));

[0160] in:

[0161]

[0162] In the formula, α i ε represents the dynamic weight corresponding to the data in the i-th frame; ε is the minimum value.

[0163] The fire trend fitting module performs least squares segmented fitting on the fused fire intensity time series, determines the fire trend change in each segment, and outputs the fire trend judgment result.

[0164] The slope sign of each trend segment is used to identify the direction of fire trend change, and the fire trend judgment result includes a fire intensity change trend graph, trend slope value, and response suggestions. The fitted straight line is used to judge the change of fire intensity based on its slope k, with a positive value representing an increase in fire intensity and a negative value representing a decrease in fire intensity.

[0165] The embodiments and descriptions above are merely illustrative of the principles and preferred embodiments of the present invention. Various changes and modifications may be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed.

Claims

1. A fire situation awareness method based on multimodal dynamic weight fusion, characterized in that, Includes the following steps: S1) Obtain images of the fire scene and use the YOLOv8-seg model to detect and segment the flame area; obtain the red channel mask area and construct the red channel intensity time series; S2) Real-time acquisition of full-band spectral data of the flame, using the SpectraNet network to identify the type of combustible material, selecting characteristic bands according to the type of combustible material, and obtaining the spectral intensity data of the characteristic bands in time series. S3) Normalize and perform sliding window Z-score analysis on the red channel intensity time series and spectral intensity data respectively to calculate the local volatility score; S4) Calculate the fusion weight dynamically based on the local volatility scores of the two types of signals, and construct a weighted fusion fire intensity time series; S5) Perform least squares segmented fitting on the fused fire intensity time series to determine the fire trend change in each segment; output the fire trend judgment result to control the response strategy of the fire extinguishing robot.

2. The fire situation awareness method based on multimodal dynamic weight fusion according to claim 1, characterized in that: In step S1), the YOLOv8-seg model performs target detection and pixel-level semantic segmentation on the input fire scene image, and extracts the spatial distribution area of ​​flames in each frame of the fire scene image; the output of the YOLOv8-seg model includes detection boxes, flame categories, and flame region segmentation masks.

3. The fire situation awareness method based on multimodal dynamic weight fusion according to claim 2, characterized in that: In step S1), the red channel intensity time series is constructed, which specifically includes the following steps: S11) Split the RGB image frame sequence into red channel images; S12) Filter the red channel image according to the segmentation mask, and only select pixels in the red channel image that correspond to the flame region; S13) The sum or average of the non-zero pixels in the red channel within the statistical mask is used as the visual intensity value r of the flame in that frame image. i ; S14) Following steps S11)-S13), calculate the visual flame intensity values ​​of all fire scene images to obtain the red channel intensity time series R(t) = {r1, r2, ... r i ...,r T }, t represents the acquisition time; T is the total number of image frames acquired within time t; r i This represents the visual intensity value of the flame in the i-th frame.

4. The fire situation awareness method based on multimodal dynamic weight fusion according to claim 3, characterized in that: In step S2), the SpectraNet network includes an input layer, a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a flattening layer, a first fully connected layer, a random dropout layer, and an output layer.

5. A fire situation awareness method based on multimodal dynamic weight fusion according to claim 4, characterized in that: In step S2), the SpectraNet network is used to identify the types of combustibles, specifically including the following steps: S21) Input the full-band spectral data of the flame through the input layer; S22) The first convolutional layer is used to initially extract the feature relationships between local bands, and local correlation features are extracted through one-dimensional convolution operation. Then, batch normalization and ReLU activation function are used for processing. S23) After downsampling each channel using the first pooling layer, the second convolutional layer is used to further extract deeper spectral structure information; then the feature map is reduced in dimensionality using the second pooling layer. S24) The multi-channel two-dimensional feature map is unfolded into a one-dimensional vector through a flattening layer, and then the flattened feature vector is compressed and mapped using the first fully connected layer, and a non-linear combination of linear transformation and ReLU activation function is used. S25) The Softmax activation function is used to normalize the scores of different combustible categories, and the combustible type is output through the output layer.

6. A fire situation awareness method based on multimodal dynamic weight fusion according to claim 5, characterized in that: In step S2), the characteristic bands corresponding to the type of combustible material are matched, and then the spectral intensity of the characteristic bands of the combustible material is measured by a spectrometer to obtain the spectral intensity data of the characteristic bands under time sequence S(t)={s1,s2,...s... i ..,s T }; T is the total number of image frames acquired within the sampling time t, s i This represents the spectral intensity value of the flame in the i-th frame of spectral intensity data after frame synchronization.

7. A fire situation awareness method based on multimodal dynamic weight fusion according to claim 6, characterized in that: In step S3), for the red channel intensity time series R(t)={r1,r2,...r i ....,r T The expression after normalization is: In the formula, The normalized visual intensity value of the flame; r i r represents the visual intensity value of the flame in the i-th frame; min r max These are the minimum and maximum visual flame intensity values ​​within R(t), respectively; For spectral intensity data S(t)={s1,s2,...s... i ..,s T The expression after normalization is: In the formula, This represents the normalized spectral intensity value of the flame. s i This represents the spectral intensity value of the flame in the i-th frame of spectral intensity data after frame synchronization. s min s max These are the spectral intensity values ​​of the minimum and maximum flames, respectively.

8. A fire situation awareness method based on multimodal dynamic weight fusion according to claim 7, characterized in that: In step S3), a sliding window Z-score is used to analyze the normalized red channel intensity time series R. norm (t) and spectral intensity data S norm The degree of anomaly at each time step (t) is analyzed, i.e.: In the formula, Z spec,i for Z-score of quality score; μ s For spectral intensity data S norm The mean of (t); σ s For spectral intensity data S norm The standard deviation of (t); ε is the minimum value; In the formula, Z grar,i for Quality score Z-score; μ r For the red channel intensity time series R norm The mean of (t); σ r For the red channel intensity time series R norm The standard deviation of (t); ε is the minimum value; Then for Z grar,i Z spec,i After normalization, a dynamic score between 0 and 1 is obtained, i.e.: In the formula, Score grar,i Normalization Quality rating; Score spec,i for Quality rating; Z grar_min Z grar_max The red channel intensity time series R are respectively norm The minimum and maximum values ​​of the quality score of (t); Z spec_min Z spec_max The spectral intensity data S are respectively norm The minimum and maximum values ​​of the quality score of (t).

9. A fire situation awareness method based on multimodal dynamic weight fusion according to claim 8, characterized in that: In step S4), the fusion weight α(t) is dynamically calculated based on the local volatility scores of the two types of signals; the functional expression for the dynamic fusion weight with an acquisition time length of t is: α(t)＝{α1,α2,..,α i ,..,α T}； The constructed weighted fusion of fire intensity time series F(t) is expressed as: F(t)=α(t)·S norm (t)+(1-α(t)·R norm (t)); in: In the formula, α i ε represents the dynamic weight corresponding to the data in the i-th frame; ε is the minimum value.

10. A fire situation awareness system based on multimodal dynamic weight fusion, characterized in that, The system utilizes the method described in any one of claims 1-9 to realize fire situation perception, and the system comprises: The image acquisition module is used to acquire images of the fire scene; The image segmentation module uses the YOLOv8-seg model to detect and segment the flame region. The red channel temporal intensity construction module is used to construct the red channel temporal intensity sequence based on the fire scene image and the segmentation mask of the image segmentation module; The spectral acquisition module is used to acquire full-band spectral data of the flame in real time. Combustible material identification module, used to identify the types of combustible materials via the SpectraNet network; The spectral intensity construction module is used to select characteristic bands according to the type of combustible material and construct the spectral intensity data of the characteristic bands in a time series. The signal fluctuation scoring module is used to perform normalization and sliding window Z-score analysis on the red channel intensity time series and spectral intensity data, respectively, and calculate the local fluctuation score. The weighted fusion module is used to dynamically calculate the fusion weight based on the local volatility scores of the two types of signals and to construct a weighted fused fire intensity time series. The fire trend fitting module performs least squares segmented fitting on the fused fire intensity time series to determine the fire trend change in each time period and outputs the fire trend judgment result.

Images