Highway long tunnel real-time fire source detection method and system based on multi-modal fusion

Abstract

The application discloses a kind of based on multimodal fusion expressway long tunnel real-time fire source detection method and system, it is estimated by introducing smoke temperature airflow dynamic propagation delay under wind speed constraint, visible light, infrared, smoke concentration and environmental temperature and so on heterogeneous data are phase shift compensation and space-time unification, different modal can correspond to the same fire physical time, then respectively extract visual side flame profile, thermal gradient feature and sensing side smoke temperature mutation, time evolution characteristics, and utilize cross-modal interactive fusion to form the fusion feature expression with image representation ability and environmental perception ability, finally, combine target detection and inverse perspective positioning Output fire confidence and fire source coordinates, to solve the problem that multi-source data is difficult to effectively align, early fire source feature weak leads to inaccurate identification, false alarm under complex disturbance is higher and fire source position is difficult to determine in real time accurately, improve the real-time, accuracy and engineering applicability of tunnel fire source detection.

Description

Technical Field

[0001] This application relates to the field of tunnel fire monitoring and early warning technology, and more specifically, to a method and system for real-time fire source detection in extra-long highway tunnels based on multimodal fusion. Background Technology

[0002] With the continuous increase in the number of highways and extra-long tunnels in mountainous areas of my country, a complex operating environment has gradually formed inside these tunnels, characterized by high traffic volume, long depth, and weak ventilation control boundaries. Once a vehicle, electrical system, or flammable material ignites inside a tunnel, the fire can easily spread rapidly in the narrow, semi-enclosed space, accompanied by dangerous factors such as high temperature, dense smoke, and low visibility, severely affecting personnel evacuation, fire response, and the timely activation of fire suppression systems. Especially in the initial stage of a fire, the flame is small, the heat radiation is weak, and the smoke and temperature response is not yet sufficient. If the fire source cannot be detected and accurately located as early as possible, it can easily escalate from a small local fire into a large-scale disaster.

[0003] Existing tunnel fire detection technologies mainly include contact detection methods such as heat and smoke sensing, and non-contact detection methods such as visible light video and infrared imaging. In recent years, composite detection schemes that fuse video and environmental sensor data have also emerged. However, most existing multimodal fusion methods focus on sensor stacking or simple time synchronization, usually assuming that visual signals and smoke and temperature signals can be directly aligned with a unified timestamp, or applying the same delay compensation to the smoke and temperature channels. But in very long tunnels, affected by piston wind, longitudinal airflow, and the propagation conditions of semi-enclosed spaces, flame images are perceived almost instantaneously, while smoke particles and heat signals exhibit significant propagation lag. Moreover, smoke and temperature are not consistent in their propagation rates and transmission mechanisms, and they also have a dynamically changing physical coupling relationship as the fire evolves. If a unified delay and mutually independent compensation method are still used, it is easy to introduce systematic spatiotemporal misalignment, causing the compensated smoke and temperature data to be inconsistent with the physical time of the fire corresponding to the visual frame. This further leads to distortion of cross-modal feature fusion, decreased accuracy of fire source identification and location, and increased risk of false alarms and missed alarms.

[0004] Therefore, an optimized real-time fire source detection scheme based on multimodal fusion for extra-long highway tunnels is desired. Summary of the Invention

[0005] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide a method and system for real-time fire source detection in extra-long highway tunnels based on multimodal fusion.

[0006] According to one aspect of this application, a real-time fire source detection method for extra-long highway tunnels based on multimodal fusion is provided, comprising: Step 1: Based on the collected wind speed data inside the tunnel, determine the dynamic propagation delay of the smoke and temperature airflow. Step 2: Based on the dynamic propagation delay of smoke temperature airflow, phase shift compensation and spatiotemporal alignment processing are performed on visible light video stream, infrared video stream, smoke concentration time series data and ambient temperature time series data to obtain aligned visual tensor and aligned sensor matrix. Step 3: Extract spatial semantic features of flame contour and thermal gradient from the aligned visual tensor to obtain a visual feature map, and extract temporal evolution features of fluctuating smoke temperature gradient from the aligned sensor matrix to obtain an environmental feature vector. Step four involves performing cross-modal feature interaction and deep fusion on the visual feature map and environmental feature vector to obtain a fused feature tensor.

[0007] Step 5: Use the object detection regression network to predict the fire source bounding box and calculate the category probability of the fused feature tensor to obtain the fire confidence level. Combine the pre-set calibration matrix to perform perspective inverse transformation on the centroid coordinates of the bounding box with the highest confidence level to obtain the fire source coordinates.

[0008] Step 6: Input the fire source coordinates and fire confidence along with twin standard data into the joint loss function to comprehensively evaluate spatial bias and classification error. Then, use the backpropagation gradient descent algorithm to adaptively update and optimize the current network parameters to obtain optimized network parameters.

[0009] According to another aspect of this application, a real-time fire source detection system for extra-long highway tunnels based on multimodal fusion is provided, comprising: The dynamic propagation delay determination module is used to determine the dynamic propagation delay of the smoke and temperature airflow based on the collected wind speed data inside the tunnel. The phase shift compensation and spatiotemporal alignment module is used to perform phase shift compensation and spatiotemporal alignment processing on visible light video streams, infrared video streams, smoke concentration time series data and ambient temperature time series data based on the dynamic propagation delay of smoke temperature airflow to obtain aligned visual tensors and aligned sensor matrices. The multimodal feature extraction module is used to extract spatial semantic features of flame contour and thermal gradient from the aligned visual tensor to obtain a visual feature map, and to extract temporal evolution features of fluctuating smoke temperature gradient from the aligned sensor matrix to obtain an environmental feature vector. The cross-modal fusion module is used to perform cross-modal feature interaction and deep fusion of visual feature maps and environmental feature vectors to obtain a fused feature tensor.

[0010] The fire source detection and localization module is used to predict the fire source bounding box and calculate the category probability on the fused feature tensor using the target detection regression network to obtain the fire confidence level. It also uses a pre-set calibration matrix to perform perspective inverse transformation on the centroid coordinates of the bounding box with the highest confidence level to obtain the fire source coordinates.

[0011] The evaluation and optimization module is used to input the fire source coordinates and fire confidence along with twin standard data into the joint loss function to comprehensively evaluate spatial bias and classification error. It also uses the backpropagation gradient descent algorithm to adaptively update and optimize the current network parameters to obtain optimized network parameters.

[0012] Compared with existing technologies, this invention proposes a real-time fire source detection method for extra-long highway tunnels based on multimodal fusion. It integrates visible light video, infrared video, smoke concentration, ambient temperature, and wind speed information into a unified processing chain. First, it determines the dynamic propagation delay of smoke and temperature airflow based on wind speed changes, and then performs phase shift compensation and spatiotemporal alignment on the multi-source heterogeneous data. This allows visual information and environmental perception information to correspond to the same fire event moment, solving the problems of difficulty in synchronous matching of multimodal data, distortion of fusion results, and instability in early fire identification. Based on this, it extracts flame contour and thermal gradient features from the visual side and smoke and temperature abrupt changes and temporal evolution features from the sensor side. Then, through cross-modal interactive fusion, it forms a more discriminative fusion representation, improving the ability to identify weak fire sources, obstructed fire sources, and interference scenes in complex tunnel environments. Finally, it combines detection regression and calibration positioning to achieve fire source confidence determination and spatial coordinate output, thereby simultaneously improving the real-time performance, accuracy, and positioning reliability of fire source detection. Attached Figure Description

[0013] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0014] Figure 1 This is a flowchart of a real-time fire source detection method for extra-long highway tunnels based on multimodal fusion, according to an embodiment of this application. Figure 2 This is a data flow diagram illustrating the real-time fire source detection method for extra-long highway tunnels based on multimodal fusion according to an embodiment of this application. Figure 3The flowchart describes the process of performing phase shift compensation and spatiotemporal alignment on visible light video stream, infrared video stream, smoke concentration time series data and ambient temperature time series data to obtain the aligned visual tensor and aligned sensor matrix based on the dynamic propagation delay of the real-time fire source detection method for extra-long highway tunnels based on multimodal fusion according to the embodiments of this application. Figure 4 This is a flowchart illustrating the real-time fire source detection method for extra-long highway tunnels based on multimodal fusion according to an embodiment of this application, which uses propagation delay as a negative time bias to perform inverse phase shift compensation on time-series data of smoke concentration and time-series data of ambient temperature to obtain compensated smoke data and compensated temperature data. Figure 5 This is a flowchart illustrating the cross-modal feature interaction and deep fusion of visual feature maps and environmental feature vectors to obtain a fused feature tensor, according to the real-time fire source detection method for extra-long highway tunnels based on multimodal fusion according to embodiments of this application. Figure 6 The flowchart of the method for real-time fire source detection in long highway tunnels based on multimodal fusion according to the embodiments of this application is as follows: fire source coordinates and fire confidence along with twin standard data are input into a joint loss function to comprehensively evaluate spatial bias and classification error, and the current network parameters are adaptively updated and optimized by backpropagation gradient descent algorithm to obtain the optimized network parameters. Figure 7 This is a block diagram of a real-time fire source detection system for extra-long highway tunnels based on multimodal fusion, according to an embodiment of this application. Detailed Implementation

[0015] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0016] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0017] While this application makes various references to certain modules in the systems according to embodiments of this application, any number of different modules can be used and run on the central control room monitoring host of a distributed control system, an industrial-grade programmable logic controller in the burner local control cabinet, a field edge computing gateway, and / or a cloud platform. The modules described are merely illustrative, and different aspects of the systems and methods may use different modules.

[0018] Flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0019] Currently, fire source detection technology in long highway tunnels generally suffers from several problems in practical applications. These include the difficulty in mapping multimodal data to the same physical moment of a fire, weak early fire signals that are easily affected by complex airflow disturbances, and the inability to establish stable and consistent criteria for distinguishing between visible light, infrared, smoke, and temperature information. This negatively impacts the accuracy, real-time performance, and reliability of fire identification. Particularly in long tunnel scenarios, visual signals and smoke / temperature signals have inherent differences in propagation mechanisms and arrival times. Using only simple synchronization or conventional fusion methods can easily lead to feature mismatches, resulting in false alarms, missed alarms, and fire source location deviations. Therefore, the technical solution of this application proposes a real-time fire source detection method for extra-long highway tunnels based on multimodal fusion. First, it combines wind speed data within the tunnel to determine the dynamic propagation delay of smoke and temperature airflow. Using this delay as a constraint, phase shift compensation and spatiotemporal alignment processing are performed on visible light video streams, infrared video streams, smoke concentration time-series data, and ambient temperature time-series data, enabling different modal data to be uniformly mapped to the same fire triggering time. Based on this, spatial semantic features such as flame contours and thermal gradients are extracted from the aligned visual data, and abrupt changes in smoke and temperature gradients and their temporal evolution features are extracted from the aligned sensor data. Then, through cross-modal feature interaction and deep fusion, a correlation expression between visual information and environmental information is established, forming a more discriminative fusion feature tensor. Finally, a target detection regression network is used to predict the fire source bounding box and calculate the category probability. Combined with a pre-set calibration matrix, inverse perspective positioning of high-confidence fire source targets is performed to obtain the fire confidence and fire source coordinates, thereby achieving real-time detection and accurate positioning of early-stage fire sources in extra-long tunnels, effectively improving recognition accuracy, anti-interference capability, and engineering application effects under complex working conditions.

[0020] Figure 1 This is a flowchart of a real-time fire source detection method for extra-long highway tunnels based on multimodal fusion, according to an embodiment of this application. Figure 2This is a data flow diagram illustrating the real-time fire source detection method for extra-long highway tunnels based on multimodal fusion, according to an embodiment of this application. Figure 1 and Figure 2 As shown, the real-time fire source detection method for extra-long highway tunnels based on multimodal fusion according to an embodiment of this application includes the following steps: S1, determining the dynamic propagation delay of smoke, temperature and airflow based on collected wind speed data inside the tunnel; S2, performing phase shift compensation and spatiotemporal alignment processing on visible light video stream, infrared video stream, smoke concentration time series data and ambient temperature time series data based on the dynamic propagation delay of smoke, temperature and airflow to obtain an aligned visual tensor and an aligned sensor matrix; S3, extracting spatial semantic features of flame contour and thermal gradient from the aligned visual tensor to obtain a visual feature map, and extracting the temporal evolution features of abrupt changes in smoke and temperature gradient from the aligned sensor matrix to obtain a ring... S4. Perform cross-modal feature interaction and deep fusion on the visual feature map and environmental feature vector to obtain a fused feature tensor; S5. Use an object detection regression network to predict the fire source bounding box and calculate the category probability on the fused feature tensor to obtain the fire confidence score, and combine a pre-set calibration matrix to perform perspective inverse transformation on the centroid coordinates of the bounding box with the highest confidence score to obtain the fire source coordinates; S6. Input the fire source coordinates and fire confidence score together with twin standard data into the joint loss function to comprehensively evaluate spatial bias and classification error, and use the backpropagation gradient descent algorithm to adaptively update and optimize the current network parameters to obtain the optimized network parameters.

[0021] Specifically, in step S1, the dynamic propagation delay of the smoke and heat flow is determined based on the collected wind speed data within the tunnel. It should be noted that, since smoke and heat do not arrive at the corresponding sensor location synchronously with the visible light image after a fire source forms in a long highway tunnel, they are delayed and propagate along the tunnel space due to longitudinal airflow. Therefore, the technical solution of this application further dynamically calculates the propagation delay of the smoke and heat flow from the fire source to the sensor based on wind speed data and a pre-calibrated sensor spatial distance. Through this processing, a consistent time reference can be established between subsequent smoke concentration data, ambient temperature data, and visual data, thereby effectively improving the accuracy of multimodal data alignment.

[0022] More specifically, in a specific example of this application, wind speed data within the tunnel is first collected in real time, and the effective wind speed value at the current moment is extracted according to a time reference consistent with subsequent fire source detection processing. Then, the pre-calibrated sensor spatial distance is retrieved, which characterizes the depth propagation path length between the fire source monitoring area and the corresponding smoke and temperature sensors. Based on this, the propagation process of the smoke and temperature airflow within the tunnel is processed according to the convection diffusion mechanism, assuming that the arrival time of the smoke and temperature signal is jointly determined by the propagation distance and the effective propagation speed, where the effective propagation speed is composed of the current wind speed and the thermal buoyancy diffusion correction amount. The propagation delay is then calculated according to the following formula: in, This indicates the time delay for the smoke and heat flow to travel from the fire source to the target sensor. This indicates the pre-calibrated sensor spatial distance. This represents the longitudinal wind speed of the tunnel currently being collected. This represents the correction rate term corresponding to the smoke temperature rise and diffusion process; subsequently, the calculated propagation delay is output as the time reference for subsequent phase shift compensation processing, so that the smoke concentration time series data and the ambient temperature time series data can be reversed around this propagation delay, thereby completing the determination of the dynamic propagation delay in step one.

[0023] Specifically, in step S2, based on the dynamic propagation delay of smoke, temperature, and airflow, phase shift compensation and spatiotemporal alignment processing are performed on the visible light video stream, infrared video stream, smoke concentration time-series data, and ambient temperature time-series data to obtain an aligned visual tensor and an aligned sensor matrix. It should be noted that, given the instantaneous response of the visible light video stream and infrared video stream to flames and thermal radiation after a fire occurs in a long highway tunnel, while the smoke concentration time-series data and ambient temperature time-series data are affected by piston wind, longitudinal airflow, and thermal diffusion processes, resulting in propagation lag, and the smoke signal and temperature signal are not consistent in propagation rate and arrival time, directly splicing them based solely on a unified timestamp can easily lead to different modes corresponding to different physical fire moments, thus affecting subsequent feature extraction and fusion judgment. Therefore, the technical solution of this application further performs phase shift compensation and spatiotemporal alignment processing on the visible light video stream, infrared video stream, smoke concentration time-series data, and ambient temperature time-series data based on the dynamic propagation delay of smoke, temperature, and airflow to obtain an aligned visual tensor and an aligned sensor matrix. Through the above processing, visual information and smoke temperature information can be established in correspondence under a unified fire event benchmark, thereby effectively reducing feature distortion, false alarms and false alarms caused by multimodal misalignment, and providing a consistent data foundation for the subsequent collaborative extraction and cross-modal fusion of flame contour, thermal gradient and smoke temperature time-series evolution features.

[0024] Figure 3 This document describes a flowchart illustrating the process of phase-shift compensation and spatiotemporal alignment of visible light video streams, infrared video streams, smoke concentration time-series data, and ambient temperature time-series data to obtain aligned visual tensors and aligned sensor matrices, based on the dynamic propagation delay of smoke, temperature, and airflow in a real-time fire source detection method for extra-long highway tunnels based on multimodal fusion, according to embodiments of this application. Figure 3 As shown, step S2 includes: S21, using the propagation delay as a negative time bias, performing inverse phase shift compensation on the smoke concentration time series data and the ambient temperature time series data respectively to obtain compensated smoke data and compensated temperature data; S22, performing affine registration and stacking up-dimensionality on the visible light video stream and the infrared video stream to obtain an aligned visual tensor, and resampling and normalizing the compensated smoke data and the compensated temperature data according to the frame timestamp to obtain an aligned sensor matrix.

[0025] In step S21, using the propagation delay as a negative time bias, inverse phase shift compensation is performed on the time-series data of smoke concentration and ambient temperature to obtain compensated smoke data and compensated temperature data. It should be noted that, given that after a fire occurs in a long highway tunnel, the smoke concentration and ambient temperature are not sensed by the corresponding sensors at the same moment the flame is captured by the video, but rather require the smoke and temperature airflow to propagate along the tunnel's depth before an effective response can be formed. If this propagation lag is not corrected, there will be a time axis misalignment between the time-series data of smoke concentration and ambient temperature and the visual frame, causing subsequent multimodal correlation analysis to deviate from the actual fire evolution process. Based on this, the technical solution of this application further uses the propagation delay as a negative time bias to perform inverse phase shift compensation on the time-series data of smoke concentration and ambient temperature to obtain compensated smoke data and compensated temperature data, so that the originally delayed sensor response is backtracked to a position more consistent with the fire triggering time on the time axis. Through the above processing, the smoke and temperature information can be time-corrected with the fire source event before entering the subsequent resampling, normalization and fusion processing, thereby effectively improving the temporal consistency between the sensor data and the visible light and infrared data, and providing a reliable data foundation for building an aligned sensor matrix.

[0026] Figure 4 This document describes a flowchart illustrating the process of phase-shift compensation and spatiotemporal alignment of visible light video streams, infrared video streams, smoke concentration time-series data, and ambient temperature time-series data to obtain aligned visual tensors and aligned sensor matrices, based on the dynamic propagation delay of smoke, temperature, and airflow in a real-time fire source detection method for extra-long highway tunnels based on multimodal fusion, according to embodiments of this application. Figure 4As shown, step S21 includes: S211, performing differential modal delay decomposition on the propagation delay to obtain smoke mode delay and temperature mode delay; S212, performing dynamic estimation of cross-modal coupling strength based on sliding window on the smoke mode delay and temperature mode delay to obtain cross-modal coupling coefficient; S213, based on the cross-modal coupling coefficient, smoke mode delay and temperature mode delay, performing differential cross-correction phase shift compensation on the time series data of smoke concentration and the time series data of ambient temperature to obtain compensated smoke data and compensated temperature data.

[0027] In step S211, the propagation delay is decomposed using differentiated modal delay to obtain the smoke mode delay and temperature mode delay. It should be noted that existing processing methods apply a uniform delay to both smoke and temperature signals, assuming they arrive at the corresponding sensing location along the same path and at the same rate. However, in long highway tunnels, smoke particles are affected by aerodynamic drag and gravitational settling, while the thermal signal is affected by both convective and infrared radiation heat transfer. Therefore, their propagation mechanisms and arrival times are not consistent, and uniform compensation can easily cause at least one channel's time correction to deviate from the actual fire process. Based on this, the technical solution of this application further decomposes the propagation delay using differentiated modal delay to obtain the smoke mode delay and temperature mode delay. Through this processing, the smoke channel and temperature channel can each obtain a dedicated delay that matches their physical transmission characteristics, effectively avoiding the time alignment deviation introduced by the uniform delay and providing a more accurate time reference for subsequent coupled sensing compensation and multimodal fusion.

[0028] More specifically, in a specific example of this application, the dynamic propagation delay calculated in the preceding steps based on wind speed data and sensor spatial distance is first obtained, and this dynamic propagation delay is used as the reference delay for the two types of sensing channels. Secondly, for the smoke channel, considering the propagation characteristics of smoke particles under longitudinal airflow conditions in the tunnel, including mass inertia, air resistance, and gravitational settling, the reference delay is amplified and corrected to form the smoke mode delay. The corresponding calculation relationship is as follows: in, This represents the smoke modal delay, characterizing the differentiated propagation time of smoke particles from the fire source to the smoke sensor, expressed in seconds. This represents the dynamic propagation delay obtained from the preceding steps, in seconds. The smoke particle blocking factor, ranging from 0 to 1, characterizes the reduction in effective transmission rate of smoke particles due to mass inertia and gravitational settling. A higher value indicates stronger particle blocking, resulting in a longer time for the smoke to reach the sensor. Furthermore, regarding the temperature channel, considering the characteristics of heat transfer under tunnel fire conditions, which are affected by airflow convection and also involve rapid infrared radiation transmission, the baseline delay is reduced and corrected to form the temperature mode delay. The corresponding calculation relationship is as follows: in, This represents the temperature modal delay, characterizing the differentiated propagation time of the thermal signal from the fire source to the temperature sensor, measured in seconds. The thermal radiation acceleration factor, ranging from 0 to 1, characterizes the additional boost to the pure convective heat transfer rate from infrared radiation heat transfer. A higher value indicates a greater contribution from thermal radiation, corresponding to a shorter time for the temperature signal to reach the sensor. Finally, the amplified and corrected smoke mode delay and the reduced and corrected temperature mode delay are output separately as the time basis for subsequent differential phase shift compensation between smoke concentration time-series data and ambient temperature time-series data, thus completing the differential mode delay decomposition process. This processing directly introduces the differences in medium transmission into the time compensation stage, enabling visual information, smoke information, and temperature information to establish a more accurate temporal correspondence around the same fire triggering event.

[0029] For example, when a vehicle catches fire in a tunnel, the visible light video stream and infrared video stream at the front end have captured the flame outline and changes in thermal radiation, while the smoke sensor and temperature sensor located at the downstream monitoring position have not yet formed a synchronous response at the same time. After obtaining the dynamic propagation delay determined in the previous steps, as well as the pre-calibrated smoke particle blocking factor and thermal radiation acceleration factor, the dynamic propagation delay is amplified and reduced respectively. The smoke mode delay is calculated to be a longer delay value, while the temperature mode delay is calculated to be a shorter delay value. This makes the smoke channel and temperature channel correspond to different time compensation benchmarks. Subsequently, the time series data of smoke concentration and ambient temperature are differentially corrected accordingly. Finally, the two types of sensor information and the visual information corresponding to the same fire event are more accurately matched on the time axis, and a consistent data basis is provided for subsequent coupled sensing compensation and cross-modal fusion.

[0030] In step S212, the cross-modal coupling strength of the smoke mode delay and temperature mode delay is dynamically estimated based on a sliding window to obtain the cross-modal coupling coefficient. It should be noted that, given that the physical correlation between changes in smoke concentration and ambient temperature is not constant during the evolution of a fire in a long highway tunnel, existing methods treat the two channels as independent signal links. The compensation process lacks real-time quantification of their coupling degree, which can easily lead to a mismatch between the subsequent mutual correction strength and the actual fire stage. Therefore, the technical solution of this application further performs dynamic estimation of the cross-modal coupling strength of the smoke mode delay and temperature mode delay based on a sliding window to obtain the cross-modal coupling coefficient. Through the above processing, the correlation between the smoke channel and the temperature channel can be dynamically characterized as the fire situation evolves, thereby effectively providing an adjustable physical correlation basis for subsequent coupling sensing compensation and improving the consistency between the compensation results and the actual combustion process.

[0031] More specifically, in a concrete example of this application, after determining the smoke mode delay and temperature mode delay, the current reference time is used as the time anchor point. A sliding time window of a preset number of sampling points is synchronously extracted from the smoke concentration time series data and the ambient temperature time series data. Within this window, the local mean and local standard deviation of the smoke concentration sequence and temperature sequence are calculated respectively to eliminate the interference of dimensional differences and local amplitude fluctuations on the coupling measure. Subsequently, the synchronous change relationship between the two channels within the same window is calculated using normalized cross-correlation, and the cross-modal coupling coefficient at the current time is output. The calculation relationship is expressed as follows: in, This represents the cross-modal coupling coefficient at the reference time, ranging from -1 to +1. A positive value indicates positive coupling; the closer the value is to +1, the stronger the synchronous increase in smoke concentration and ambient temperature; the closer the value is to 0, the weaker the correlation between the two. This indicates the number of sampling points contained in the sliding window. This represents the time series data of smoke concentration at time [time]. The sampled value at that location, This represents the time series data of ambient temperature at time [time]. The sampled value at that location, This represents the arithmetic mean of the smoke concentration within a sliding window starting from the reference time. This represents the arithmetic mean of the ambient temperature within a sliding window starting from the reference time. This represents the standard deviation of the smoke concentration within the corresponding window. This represents the standard deviation of the ambient temperature within the corresponding window. After obtaining this coupling coefficient, it is used as the dynamic weight basis for subsequent dual-channel mutual injection compensation, so that when the smoke-temperature correlation is weak in the initial stage of the fire, a low correction intensity is maintained, while the collaborative correction intensity between the two channels is automatically increased in the stages of combustion development, dense smoke accumulation, and simultaneous increase in temperature rise, thereby completing the dynamic estimation of cross-modal coupling intensity.

[0032] In a specific scenario, when a vehicle fire in a tunnel enters its early stage, the flames already appear as locally bright areas in visible light and infrared images, while the changes in smoke concentration and ambient temperature at the sensor are still in a slow establishment phase. At this time, the estimated cross-modal coupling coefficient calculated within the corresponding sliding window is a low value, indicating a weak synchronous change relationship between the two channels. As combustion continues to develop, the release of high temperature accelerates, and the accumulation of the smoke layer intensifies. The degree of synchronous fluctuation between the smoke concentration curve and the temperature curve within the window increases, and the estimated cross-modal coupling coefficient rises to a higher value. Based on this, the mutual correction between the two channels can be enhanced in subsequent compensation processing, making the compensated smoke and temperature data more consistent with the actual thermodynamic evolution of the fire and maintaining a more stable physical consistency with the visual information at the same moment.

[0033] In step S213, based on the cross-modal coupling coefficient, smoke mode delay, and temperature mode delay, differentially cross-corrected phase shift compensation is performed on the time-series data of smoke concentration and ambient temperature to obtain compensated smoke data and compensated temperature data. It should be noted that, given that after completing the differential delay decomposition of the smoke and temperature channels and the dynamic estimation of cross-modal coupling strength, if only independent time shift compensation is performed on each channel separately, the compensation result can only correct the difference in arrival time, failing to maintain the proper thermodynamic causal consistency between changes in smoke concentration and changes in ambient temperature. This can easily lead to discrepancies or even contradictions between the compensation results of the two channels. Therefore, the technical solution of this application further performs differentially cross-corrected phase shift compensation on the time-series data of smoke concentration and ambient temperature based on the cross-modal coupling coefficient, smoke mode delay, and temperature mode delay to obtain compensated smoke data and compensated temperature data. Through the above processing, the two channels can be aligned with the corresponding fire events on the time axis and maintain a coordinated change relationship with the fire combustion evolution at the signal strength level, thereby effectively improving the physical reliability and temporal consistency of the subsequent multimodal fusion input data.

[0034] More specifically, in a specific example of this application, the smoke concentration time series data is first shifted forward using a differential time axis using a smoke mode delay to obtain a smoke signal with a baseline shift. Then, the ambient temperature time series data is shifted forward using a corresponding differential time axis using a temperature mode delay to obtain a temperature signal with a baseline shift. Subsequently, the ambient baseline values ​​for smoke concentration and ambient temperature are determined from historical data collected over a long period under fire-free conditions. The baseline values ​​are then subtracted from the baseline values ​​of the smoke signal and the temperature signal with the baseline shift to extract the abnormal deviations characterizing fire increment information. The corresponding calculation relationship is expressed as follows: in, Indicates the reference time The abnormal deviation of smoke concentration at a given location reflects the extent of the deviation of smoke concentration from normal environmental levels after the differential treatment time was shifted earlier. Indicates the use of smoke mode delay. After shifting the time axis of the smoke concentration time series data forward, at time... The signal value obtained at that location, This represents the environmental baseline value of smoke concentration, taken from the long-term statistical mean constant of this sensing location under no-fire conditions. Indicates the reference time The abnormal temperature deviation reflects the extent of the deviation of the ambient temperature from the normal ambient level after the differential time was shifted forward. Indicates the use of temperature mode delay. After shifting the time axis of the ambient temperature time series data forward, at time... The signal value obtained at that location, The ambient temperature baseline value is taken from the long-term statistical mean constant of the sensing location under no-fire conditions. After obtaining the above two abnormal deviations, mutual injection correction is performed on the dual channels based on the cross-modal coupling coefficient obtained in the previous steps. That is, the weighted injection value of the temperature abnormal deviation is superimposed on the differential deviation result of the smoke channel itself, and the weighted injection value of the smoke abnormal deviation is superimposed on the differential deviation result of the temperature channel itself. The compensated smoke data and compensated temperature data are output respectively. The calculation relationship is expressed as follows: in, Indicates the reference time The output of the compensated smoke data, Indicates the reference time The compensated temperature data output at the location, Indicates the reference time The cross-modal coupling coefficient at a given point is used to characterize the strength of the dynamic physical correlation between smoke and temperature. This represents the smoke channel coupling injection gain, used to control the scaling ratio of the amplitude when abnormal temperature deviations are injected into the smoke channel. This represents the temperature channel coupling injection gain, used to control the amplitude scaling ratio when abnormal smoke deviations are injected into the temperature channel. This completes the continuous processing of differential delay decomposition, cross-modal coupling estimation, and mutual correction compensation of coupled sensing, ensuring that the compensated dual-channel data both satisfy their respective propagation laws and maintain consistency in cross-channel physical correlation.

[0035] For example, when a vehicle catches fire in a tunnel, the flames and thermal radiation are already reflected in the visible light and infrared video streams, while the time-series data of smoke concentration and ambient temperature are still in the stage of gradually building up the response. Based on the smoke mode delay, temperature mode delay, and cross-modal coupling coefficient obtained from the preprocessing, the differentiated time shift of the smoke channel and temperature channel is first completed, and then the abnormal deviation of the two channels relative to the fire-free background is extracted. When the fire is in the development stage and the estimated cross-modal coupling coefficient is at a high level, the injection intensity of the temperature abnormal deviation on the smoke compensation result and the injection intensity of the smoke abnormal deviation on the temperature compensation result are increased simultaneously, so that the compensated smoke data and the compensated temperature data converge to the physical state of the same fire event. When the tunnel is in a stable operation period and the estimated cross-modal coupling coefficient is at a low level, the mutual injection correction intensity decreases accordingly, and each channel mainly retains its own differentiated time compensation result, thereby avoiding the introduction of additional interference. This processing enables the compensated sensor data to correspond to the same fire triggering time as the visual frame in the time axis dimension, and maintains a correlation with the combustion thermodynamic process of tunnel fire in terms of numerical change trend, providing coordinated and consistent high-quality input data for subsequent cross-modal attention fusion.

[0036] In step S22, the visible light video stream and the infrared video stream are affine registered and stacked to increase their dimensionality to obtain an aligned visual tensor. Then, the compensated smoke data and compensated temperature data are resampled, normalized, and stitched together according to frame timestamps to obtain an aligned sensor matrix. It should be noted that, since the visible light video stream and infrared video stream in the long tunnel of a highway reflect flame appearance information and thermal radiation distribution information respectively, there are spatial differences between them in imaging coordinates, field of view, and pixel response. Furthermore, the compensated smoke data and compensated temperature data are temporal sensing information, and their sampling frequency and time reference differ from the video frame sequence. If they directly enter the subsequent feature extraction and fusion stages, it will cause spatial mismatch within the visual modality and temporal mismatch between the visual modality and the sensing modality. Based on this, the technical solution of this application further performs affine registration and stacking of the visible light video stream and infrared video stream to increase their dimensionality to obtain an aligned visual tensor, and then resamples, normalizes, and stitches together the compensated smoke data and compensated temperature data according to frame timestamps to obtain an aligned sensor matrix. Through the above processing, the visual flame outline and thermal distribution can be aligned in a unified spatial coordinate system, and the changes in smoke and temperature on the sensing side can be aligned with the time reference consistent with the video frame. This effectively ensures that subsequent cross-modal feature extraction and deep fusion are based on a unified and comparable data representation.

[0037] More specifically, in a specific example of this application, the visible light video stream and infrared video stream collected within the tunnel monitoring area are first read at the frame level. The spatial mapping relationship between the infrared image and the visible light image is determined based on the camera installation position, viewing angle deviation, and calibration relationship. Then, according to the spatial mapping relationship, one of the images is processed by translation, rotation, scaling, and cropping to ensure that the position of the same fire source area in both types of images corresponds to a unified pixel coordinate range. After affine registration, the visible light image frames and infrared image frames corresponding to the same moment are stacked according to the channel dimension, thereby forming multi-channel visual data that simultaneously carries visible light texture information and infrared thermal distribution information. This data is further organized into an aligned visual tensor suitable for subsequent convolutional feature extraction. Subsequently, the frame timestamps corresponding to the video frame sequence are extracted and used as a unified time reference for processing the sensor-side data. The compensated smoke data and compensated temperature data obtained in the previous steps are aligned and sampled on the time axis, so that a set of smoke and temperature values ​​is generated corresponding to each frame moment in the sensor sequence. For sampling points that do not directly fall on the frame timestamp, interpolation is used to complete resampling to ensure that the length of the sensor data is consistent with the number of video frames. Subsequently, the resampled and compensated smoke and temperature data were normalized to eliminate the numerical scale inconsistencies caused by the difference between the dimensions of smoke concentration and ambient temperature. Then, the two data were concatenated in a unified order to form a sensor feature sequence, thereby creating an aligned sensor matrix that corresponds frame-by-frame with the aligned visual tensor. After this processing, the flame edge of a vehicle fire location within the tunnel in the visible light image, the thermal anomaly region in the infrared image, and the smoke and temperature responses at the corresponding time are uniformly mapped to the same spatiotemporal reference frame. The subsequently extracted flame contour features, thermal gradient features, and smoke-temperature evolution features can then be correlated around the same fire event, thereby improving the stability of multimodal joint discrimination.

[0038] Specifically, in step S3, spatial semantic features of flame contour and thermal gradient are extracted from the aligned visual tensor to obtain a visual feature map, and temporal evolution features of smoke-temperature gradient abrupt changes are extracted from the aligned sensor matrix to obtain an environmental feature vector. It should be noted that, due to the characteristics of early fire sources in long highway tunnels—small flame area, rapid changes in thermal radiation distribution, delayed smoke-temperature response establishment, and perturbation by piston wind—the aligned visual tensor and aligned sensor matrix obtained after the preceding spatiotemporal alignment still belong to the original multimodal expression and cannot directly form stable discriminative features suitable for subsequent cross-modal interactions. Based on this, the technical solution of this application further extracts spatial semantic features of flame contour and thermal gradient from the aligned visual tensor to obtain a visual feature map, and extracts temporal evolution features of smoke-temperature gradient abrupt changes from the aligned sensor matrix to obtain an environmental feature vector. Through the above processing, the spatial morphological information on the visual side and the dynamic change information on the sensing side can be compressed into discriminative high-level feature expressions, thereby effectively improving the joint recognition capability of early fire sources, thermal anomalies, and smoke-temperature abrupt changes during subsequent cross-modal fusion.

[0039] More specifically, in a concrete example of this application, the aligned visual tensor is first input into a multi-scale spatial feature extraction network. On the same frame of data, convolutional kernels with different receptive fields (3×3 and 5×5) are used to perform parallel scanning of the flame edge, brightness distribution, infrared hotspot contour, and thermal gradient diffusion direction. This allows for the simultaneous extraction of local small-scale flame features and larger-scale thermal diffusion features. The feature responses obtained at different scales are then concatenated along the channel direction and subjected to nonlinear activation processing to obtain a visual feature map representing the spatial semantic information of the flame contour and thermal gradient. The computational relationship can be expressed as follows: in, Represents visual feature maps. Represents the aligned visual tensor. Indicates the first The convolutional kernel weights corresponding to each receptive field scale Indicates the corresponding bias term. Represents a set of different receptive field scales. This represents the convolution operation. This indicates a channel splicing operation. The function represents a nonlinear activation function. Subsequently, the aligned sensor matrix is ​​input sequentially into a temporal feature extraction network. Recursive analysis is performed on the magnitude, direction, and persistence of changes in smoke concentration and ambient temperature at continuous time points. An update gate controls the retention ratio between historical states and current anomalous changes, extracting an environmental feature vector reflecting abrupt changes in smoke temperature gradients and their evolutionary trends. The state update relationship can be expressed as: in, Indicates time The hidden state at the end of the sequence is the environment feature vector. Indicates time The updated gate coefficient, This represents the candidate state generated from the current smoke temperature input. This indicates the hidden state at the previous moment. This represents element-wise multiplication. Finally, the resulting visual feature map and environmental feature vector are output separately, so that the flame and thermal field distribution features of the same fire event in the image domain and the smoke and temperature evolution features in the sensing domain are formed into a structured expression, providing an input basis for subsequent cross-modal feature interaction and deep fusion.

[0040] Specifically, in step S4, cross-modal feature interaction and deep fusion are performed on the visual feature map and environmental feature vector to obtain a fused feature tensor. It should be noted that, given that the aligned visual feature map mainly represents spatial information such as flame outline, brightness distribution, and thermal gradient, while the environmental feature vector mainly represents the temporal evolution information of smoke concentration, ambient temperature, and its gradient abrupt changes, the two differ in expression dimension, information structure, and perceptual focus. If only simple splicing or independent discrimination methods are used, it is difficult to establish an effective correlation between changes in flame appearance and smoke temperature evolution, and it is also not conducive to forming a stable fire source discrimination basis under complex tunnel conditions. Based on this, the technical solution of this application further performs cross-modal feature interaction and deep fusion on the visual feature map and environmental feature vector to obtain a fused feature tensor. By mapping visual features and environmental features to a unified hidden space, and utilizing cross-modal association calculation and weighted aggregation methods, information enhancement and redundancy compression between different modalities are achieved. Through the above processing, the flame and thermal field representation on the visual side and the smoke temperature change representation on the sensor side can form a consistent joint feature expression around the same fire event, thereby effectively improving the discrimination accuracy and anti-interference ability in subsequent fire source detection, fire confidence judgment and spatial positioning processes.

[0041] Figure 5 This is a flowchart illustrating the process of cross-modal feature interaction and deep fusion of visual feature maps and environmental feature vectors to obtain a fused feature tensor, according to an embodiment of this application for a real-time fire source detection method based on multimodal fusion in long highway tunnels. Figure 5As shown, step S4 includes: S41, after spatially flattening the visual feature map, projecting it and the environment feature vector onto the same dimension of the hidden space to obtain the mapped visual vector and the mapped environment vector; S42, generating a query matrix with the mapped visual vector, generating a key matrix and a value matrix with the mapped environment vector, and performing cross-modal association calculation and weighted aggregation on the query matrix, key matrix and value matrix to obtain the enhanced multimodal matrix; S43, performing high-dimensional nonlinear feature projection and activation filtering on the enhanced multimodal matrix, and then performing dimensionality reduction compression and redundancy elimination on the target dimension to obtain the fused feature tensor.

[0042] In step S41, after spatially flattening the visual feature map, it is projected onto a hidden space of the same dimension as the environmental feature vector to obtain mapped visual vectors and mapped environmental vectors. It should be noted that since the visual feature map carries the flame outline, thermal gradient, and spatial distribution relationship, it has a clear two-dimensional spatial structure, while the environmental feature vector carries the temporal evolution results of smoke concentration and ambient temperature, which is a one-dimensional compact expression. The two are inconsistent in data form, feature dimension, and representation space. If intermodal correlation calculations are performed directly, it is difficult to establish a stable correspondence. Based on this, the technical solution of this application further flattens the visual feature map spatially and projects it onto a hidden space of the same dimension as the environmental feature vector to obtain mapped visual vectors and mapped environmental vectors. Through the above processing, spatial semantic features and temporal evolution features can be aligned at a unified feature scale, thereby effectively providing a consistent data entry point for subsequent cross-modal correlation calculations, weighted aggregation, and deep fusion.

[0043] More specifically, in a specific example of this application, the visual feature map obtained in the preceding steps is first spatially flattened, reconstructing the feature data originally organized according to image width, height, and channels into a linear sequence. This transforms spatial semantic information such as flame outline edges, hot spot regions, and thermal gradient directions into vector inputs that can participate in unified mapping calculations while preserving channel response relationships. Subsequently, linear projection processing is applied to the flattened visual feature and environmental feature vectors, respectively. Feature transformation is completed through their corresponding weight matrices and bias terms, compressing or expanding the two types of features to the same hidden dimension, resulting in mapped visual vectors and mapped environmental vectors, whose computational relationship is expressed as follows: in, This represents the mapped visual vector, which is the representation of visual features in a unified latent space. This represents the linear mapping weight matrix corresponding to visual features. Represents visual feature maps The result after spatial flattening This represents the visual feature map obtained from the preceding extraction. Indicates the visual mapping bias term. This represents the mapped environment vector, which is the representation of environment features in a unified latent space. This represents the linear mapping weight matrix corresponding to environmental features. This represents the environmental feature vector obtained from the preceding extraction. This represents the environmental mapping bias term. After the above processing, the flame morphology and thermal field distribution information reflected by the visual side and the smoke temperature change trend reflected by the sensor side are transformed into a unified feature representation space. This eliminates the impact of the original dimensional differences between modalities on subsequent calculations while retaining their respective representation capabilities for fire events. This enables subsequent cross-modal interactions based on queries, keys, and values ​​to establish stable associations around the same fire source target.

[0044] In step S42, a query matrix is ​​generated using mapped visual vectors, and a key matrix and value matrix are generated using mapped environmental vectors. Cross-modal correlation calculations and weighted aggregation are then performed on the query matrix, key matrix, and value matrix to obtain an enhanced multimodal matrix. It should be noted that since the mapped visual vectors mainly retain spatial discrimination information such as flame outlines, hot spot distribution, and thermal gradients, while the mapped environmental vectors mainly retain information on smoke concentration changes, ambient temperature changes, and their temporal evolution, although both are in the same hidden dimension, explicit correlation modeling for the same fire event has not yet been completed. If only direct splicing or linear superposition is performed, it is difficult to accurately identify the correspondence between visually abnormal areas and abnormal smoke and temperature changes. Based on this, the technical solution of this application further generates a query matrix using mapped visual vectors and a key matrix and value matrix using mapped environmental vectors, and performs cross-modal correlation calculations and weighted aggregation on the query matrix, key matrix, and value matrix to obtain an enhanced multimodal matrix. Through the above processing, the visual side can actively retrieve environmental response information with high correlation to suspected fire sources, thereby effectively improving the coupling expression ability between multimodal features and providing more discriminative joint features for subsequent dimensionality reduction fusion and fire source detection.

[0045] More specifically, in a specific example of this application, a linear transformation is first applied to the mapped visual vector obtained in the preceding steps to generate a query matrix representing the visual query intent. Then, two independent linear transformations are applied to the mapped environment vector to generate a key matrix representing the environmental feature matching index and a value matrix carrying the environmental response content. Subsequently, the transpose of the query matrix and the key matrix is ​​multiplied by a dot product to calculate the cross-modal association score between visual features and environmental features. The association score is then scaled according to the feature dimension to suppress the imbalance in weight distribution caused by excessively large numerical amplitudes. A normalization function is then used to convert the scaled association score into a weight matrix. Finally, the weight matrix is ​​multiplied by the value matrix, giving higher aggregation weights to the smoke and temperature evolution information with high correlation to the current visual fire source features, while suppressing the background response with low correlation. The final output is an enhanced multimodal matrix. The corresponding calculation relationship is expressed as follows: in, This represents the enhanced multimodal matrix, which serves as the output of cross-modal correlation calculation and weighted aggregation. Represents the mapping of visual vectors, Represents the mapping environment vector. This represents the weights of the linear mapping used to generate the query matrix. This represents the linear mapping weights used to generate the key matrix. This represents the weights of the linear mapping used to generate the value matrix. This indicates the transpose operation. The characteristic dimension of the key matrix is ​​used to construct the scaling factor. This represents the normalization function, used to convert the association score into a weighted distribution. After this processing, the changes in flame edges in the visible light image, thermal anomaly areas and smoke concentration in the infrared image, and ambient temperature at the same time can form selective aggregation results around the same fire target, so that early fire sources, small-scale heat sources, and anomalous responses under complex disturbance conditions are preserved in the fusion representation.

[0046] In step S43, the enhanced multimodal matrix undergoes high-dimensional nonlinear feature projection and activation filtering, followed by dimensionality reduction and redundancy elimination in the target dimension to obtain a fused feature tensor. It should be noted that while the enhanced multimodal matrix aggregates visual and environmental association information after cross-modal correlation calculation, it still contains high-dimensional mixed responses, repetitive representations, and redundant components with low correlation to fire source discrimination. Directly inputting these components into subsequent detection and regression processes would increase the instability of feature representation and affect fire source boundary prediction and category judgment. Therefore, the technical solution of this application further performs high-dimensional nonlinear feature projection and activation filtering on the enhanced multimodal matrix, followed by dimensionality reduction and redundancy elimination in the target dimension to obtain a fused feature tensor. Through the above processing, the high-dimensional information after cross-modal interaction can be transformed into a more compact and discriminative unified representation, effectively providing suitable feature input for subsequent fire source detection and spatial localization.

[0047] More specifically, in a concrete example of this application, the enhanced multimodal matrix obtained in the preceding steps is first input into the first fully connected transformation unit of the multilayer perceptron processing structure. High-dimensional nonlinear feature projection is then performed on the cross-modal correlation information in the matrix, allowing the established correlations of features such as flame contours, hot spot distribution, smoke abrupt changes, and temperature rise trends to be recombined in a higher-level semantic space. Subsequently, activation filtering is applied to the projection result to suppress negative or low-response components that are weakly correlated with fire events, retaining feature channels that are more sensitive to fire detection. Afterward, the activated features are input into the next fully connected transformation unit, where dimensionality reduction and compression are performed according to the target dimension required by the target detection network. During compression, linear correlation redundancy caused by multimodal superposition is eliminated, ultimately yielding the fused feature tensor. The corresponding computational relationship is expressed as follows: in, This represents the fused feature tensor, which serves as the input feature for subsequent fire source bounding box prediction and class probability calculation. Represents the enhanced multimodal matrix. This represents the weight matrix of the first fully connected layer transformation. Indicates the first-level bias term. This indicates the activation of the filtering function, used for non-linear filtering of high-dimensional projection results. This represents the weight matrix of the dimensionality reduction compression layer. This represents the bias term of the dimensionality reduction and compression layer. After this processing, the flame appearance and thermal distribution information, as well as the dynamic response information of smoke concentration and ambient temperature in the tunnel monitoring image, are compressed into a unified and compact high-level feature representation, enabling subsequent detection processes to make judgments based on the key characteristics of the fire source target and reducing interference from irrelevant background and repetitive features.

[0048] Specifically, in step S5, an object detection regression network is used to predict the fire source bounding box and calculate the category probability of the fused feature tensor to obtain the fire confidence level. Then, a perspective inverse transformation is performed on the centroid coordinates of the highest confidence bounding box using a pre-set calibration matrix to obtain the fire source coordinates. It should be noted that although the fused feature tensor has completed the joint representation of visual and environmental information, it still belongs to the high-level semantic expression within the network and has not yet been transformed into a target location and risk assessment result that can be directly used for tunnel fire response. Furthermore, in long tunnel scenarios, monitoring images have characteristics such as large depth, significant perspective distortion, overlapping targets, and scale variations. Without target-level screening and spatial mapping processing, it is difficult to form stable fire source localization results. Based on this, the technical solution of this application further utilizes an object detection regression network to predict the fire source bounding box and calculate the category probability of the fused feature tensor to obtain the fire confidence level, and combines a pre-set calibration matrix to perform a perspective inverse transformation on the centroid coordinates of the highest confidence bounding box to obtain the fire source coordinates. Through the above processing, the multimodal fusion results can be transformed into the probability of fire occurrence and the actual spatial location, thereby effectively supporting alarm linkage, sprinkler control and emergency response in the tunnel.

[0049] More specifically, in a concrete example of this application, the fused feature tensor obtained in the preceding steps is first input into the target detection head network. In the classification branch, the fire probability is inferred for each candidate region, and in the regression branch, the offset of the center position, width, and height of the candidate regions is predicted. This yields the fire confidence score and the initial bounding box, where the classification prediction relationship is expressed as: The regression prediction relationship is expressed as: in, This indicates the fire confidence level, reflecting the probability that the candidate area is a fire source target. This represents the activation function that maps classification responses to probability intervals. Indicates the weight of the classification branch. Represents the fused feature tensor. Indicates the categorical branch bias term. Represents the initial bounding box. Indicates the regression branch weights. This represents the regression branch bias term. This represents the convolution operation; subsequently, non-maximum suppression filtering is performed on the initial bounding boxes based on fire confidence scores to remove redundant overlapping boxes generated for the same fire source target, retaining only the best box with the highest score. Then, the geometric centroid of this best box is calculated to obtain the pixel centroid. The calculation relationship is expressed as follows: in, The x-coordinate of the pixel centroid. The ordinate representing the centroid of a pixel. and These represent the left and right boundary coordinates of the optimal bounding box, respectively. and Let represent the upper and lower boundary coordinates of the optimal bounding box, respectively. Then, extract the camera intrinsic and extrinsic parameter matrices from the preset calibration matrix, construct the corresponding inverse homography transformation matrix, and combine it with the tunnel road surface height constraint to inversely map the pixel centroid from the image plane to the actual spatial coordinates. The relationship is expressed as: in, and This indicates the horizontal and vertical positions of the fire source in the tunnel's world coordinate system. Indicates the scale factor. Represents the inverse homography transformation matrix. and This indicates the position of the centroid of the optimal bounding box in the image coordinate system. After processing, the flame target in the monitoring screen is mapped to the corresponding fire source spatial coordinates, so that the vehicle ignition point, cargo combustion point, or road surface combustion point in the tunnel can all be output in a unified coordinate form, and together with the fire confidence level, form a detection result that can be directly used for subsequent evaluation and linkage control.

[0050] In step S6, the fire source coordinates and fire confidence scores, along with twin standard data, are input into a joint loss function for a comprehensive evaluation of spatial bias and classification error. The current network parameters are then adaptively updated and optimized using a backpropagation gradient descent algorithm to obtain the optimized network parameters. It should be noted that, given the influence of piston wind disturbance, light and heat obstruction, differences in smoke and temperature propagation, and long-term scene changes during fire source detection in long highway tunnels, the forward-output fire source coordinates and fire confidence scores may still deviate from the actual fire situation under different operating conditions. Without joint feedback correction for the spatial positioning results and classification judgment results, the model parameters will struggle to continuously adapt to the complex environment inside the tunnel. Based on this, the technical solution of this application further incorporates fire source coordinates and fire confidence scores, along with twin standard data, into a joint loss function for comprehensive evaluation of spatial bias and classification error. The current network parameters are then adaptively updated and optimized using a backpropagation gradient descent algorithm. The twin standard data provides reference coordinate labels and category labels corresponding to the actual tunnel structure, sensor distribution, and fire evolution. The joint loss function simultaneously constrains fire source location bias and fire ignition discrimination bias, ensuring that the network update process revolves around the validity of the detection results. Through this processing, the model can form a closed-loop optimization mechanism in the process of tunnel fire source detection, location, and confidence assessment, thereby effectively improving the stability of fire source identification, the consistency of spatial location, and the model's continuous adaptability to environmental changes under complex operating conditions.

[0051] Figure 6 To illustrate the real-time fire source detection method for long highway tunnels based on multimodal fusion according to embodiments of this application, the fire source coordinates and fire confidence level are input together with twin standard data into a joint loss function for comprehensive evaluation of spatial bias and classification error. A flowchart is then used to adaptively update and optimize the current network parameters through backpropagation gradient descent algorithm to obtain the optimized network parameter flowchart. For example... Figure 6 As shown, step S6 includes: S61, calculating regression loss and classification loss by combining the fire source coordinates and fire confidence with the corresponding real coordinate labels and real category labels in the twin standard data, and then weighted summing to obtain the comprehensive loss value; S62, based on the chain rule, solving the partial derivatives of the comprehensive loss value layer by layer along the computational graph path of the network forward propagation, calculating the partial derivatives of the loss with respect to each trainable node in the current network parameters and aggregating them to obtain the gradient matrix; S63, using the gradient descent optimizer to scale the gradient matrix by step size according to the preset learning rate, and applying the scaled gradient to the current network parameters for subtraction update to obtain the optimized network parameters.

[0052] In step S61, regression loss and classification loss are calculated for the fire source coordinates and fire confidence scores against the corresponding real coordinate labels and real category labels in the twin standard data, and then weighted and summed to obtain a comprehensive loss value. It should be noted that since the fire source detection results in long highway tunnels simultaneously include spatial positioning results and fire ignition determination results, evaluating solely based on fire source coordinate deviations or fire confidence score deviations cannot fully reflect the overall deviation between the current output and the actual fire situation. Therefore, the technical solution of this application further calculates regression loss and classification loss for the fire source coordinates and fire confidence scores against the corresponding real coordinate labels and real category labels in the twin standard data, and then weighted and sums them to obtain a comprehensive loss value. Through the above processing, spatial positioning errors and fire determination errors can be unified under the same optimization objective for measurement, thereby effectively providing a consistent error signal for subsequent network parameter updates.

[0053] More specifically, in a specific example of this application, the fire source coordinates and fire confidence scores output from the preceding steps are first matched with twin standard data from the same time and monitoring area. True coordinate labels and true category labels are extracted from the twin standard data. The true coordinate labels characterize the standard location of the fire source in the actual tunnel space, and the true category labels characterize whether the corresponding target belongs to a real fire source. Subsequently, regression loss is calculated for the fire source coordinates and true coordinate labels to quantify the spatial deviation between the predicted location and the standard location. Classification loss is calculated for the fire confidence scores and true category labels to quantify the difference between the fire probability judgment and the true category. The two losses are then weighted and summed according to preset weights to obtain a comprehensive loss value. The calculation relationship is expressed as follows: in, The overall loss value is used as a measure of the total error in spatial localization and classification of the current sample. This represents the regression loss weight, used to adjust the proportion of the spatial bias term in the total loss. This indicates the coordinates of the fire source output in the previous step. This represents the true coordinate labels extracted from twin standard data. This represents the squared Euclidean distance between the predicted and actual coordinates, reflecting the deviation in fire source location. This represents the classification loss weight, used to adjust the proportion of the class error term in the total loss. This represents the fire confidence level output by the preceding steps. This represents the true category label extracted from the twin standard data. The binary cross-entropy loss is used to reflect the difference between the fire probability output and the true category. After the above calculation is completed, the obtained comprehensive loss value is used as the input of the subsequent backpropagation process, so that the spatial bias and classification bias formed under different scenarios such as vehicle fire, cargo smoldering or local thermal anomaly in the tunnel can be uniformly quantified, and a basis for continuous optimization of the model under complex tunnel conditions is provided.

[0054] In step S62, based on the chain rule, the comprehensive loss value is solved by inversely calculating the partial derivatives layer by layer along the computational graph path of the network's forward propagation. The partial derivatives of the loss with respect to each trainable node in the current network parameters are calculated and aggregated to obtain the gradient matrix. It should be noted that since the comprehensive loss value only reflects the overall deviation of the current fire source coordinates and fire confidence relative to the twin standard data, it cannot directly indicate how the network parameters of each layer should be adjusted. If the loss is not decomposed layer by layer to each trainable node along the forward propagation path, the directional information required for subsequent weight updates cannot be formed. Based on this, the technical solution of this application further uses the chain rule to solve the partial derivatives of the comprehensive loss value layer by layer along the computational graph path of the network's forward propagation. The partial derivatives of the loss with respect to each trainable node in the current network parameters are calculated and aggregated to obtain the gradient matrix. Through the above processing, the localization error and classification error at the output end can be mapped back to each processing layer such as visual feature extraction, environmental feature extraction, cross-modal fusion, and detection regression, thereby effectively providing a consistent gradient basis for subsequent parameter iteration updates.

[0055] More specifically, in a specific example of this application, the comprehensive loss value obtained in the preceding steps is first used as the starting point for backpropagation, and partial derivatives are sequentially expanded from the output end to the input end according to the connection order between the layers in the network forward computation graph. Subsequently, the partial derivatives of the comprehensive loss value with respect to the corresponding weights and biases are calculated for the classification branch parameters and regression branch parameters in the target detection regression network, respectively. The partial derivative results are then passed forward to the cross-modal feature interaction layer, the high-dimensional nonlinear projection layer, the visual feature extraction layer, and the temporal feature extraction layer, so that the fire source coordinate deviation and fire confidence deviation form a layer-by-layer traceable error propagation relationship throughout the entire network link. Afterward, according to the chain rule, the derivative of the current local output of each layer with respect to its input and parameters is multiplied with the error signal from the subsequent layers to obtain the partial derivative of the comprehensive loss value with respect to the trainable parameters of the layer. The partial derivative results of all trainable nodes in the same layer are summarized according to the parameter arrangement structure to form a gradient matrix corresponding one-to-one with the current network parameters. The calculation relationship can be expressed as follows: in, The gradient matrix represents the sensitivity of the overall loss value to changes in the current network parameters. This represents the overall loss value, reflecting the total error of the current sample in spatial localization and classification. This represents the partial derivative operator. This represents the current set of network parameters, including the trainable weights and biases in the visual branch, environment branch, cross-modal fusion branch, and detection regression branch. After completing the above solution, the resulting gradient matrix is ​​output to the subsequent optimization process, so that the comprehensive error formed under scenarios such as open flames on vehicles, smoldering of goods, and high-temperature smoke disturbance in tunnels can be specifically allocated to the parameter adjustment of different functional layers, thereby ensuring that the model update process revolves around the actual fire detection deviation.

[0056] In step S63, the gradient descent optimizer is used to scale the gradient matrix by a step size according to a preset learning rate, and the scaled gradient is applied to the current network parameters for subtraction update to obtain optimized network parameters. It should be noted that since the gradient matrix only represents the direction and magnitude of the change in the comprehensive loss value relative to the current network parameters, and has not yet formed new parameter results, if the gradient step size is not controlled according to the preset learning rate and the parameters are not updated, the preceding error evaluation and gradient solution cannot be transformed into model performance improvement. Based on this, the technical solution of this application further utilizes the gradient descent optimizer to scale the gradient matrix by a step size according to a preset learning rate, and applies the scaled gradient to the current network parameters for subtraction update to obtain optimized network parameters. Through the above processing, the network parameters can be iteratively adjusted in the direction of decreasing comprehensive loss, thereby effectively improving the adaptability of the tunnel fire source detection model to complex working conditions.

[0057] More specifically, in a concrete example of this application, the gradient matrix obtained in the previous steps is first input into the gradient descent optimization process along with the current network parameters. The gradient matrix is ​​then scaled according to a preset learning rate to control the update magnitude of different parameter nodes within a predetermined range. Subsequently, the scaled gradient values ​​are subtracted from the current network parameters one by one according to the parameter correspondence. This causes the trainable weights and biases in the visual feature extraction layer, environmental feature extraction layer, cross-modal interaction fusion layer, and object detection regression layer to be simultaneously corrected in the direction of reducing the overall loss value, forming a new set of parameters. The update relationship can be expressed as follows: in, This refers to optimizing network parameters, that is, the new set of parameters after one parameter update. Represents the current set of network parameters. This indicates the preset learning rate, used to control the scaling ratio of the gradient step size. The gradient matrix represents the partial derivative of the comprehensive loss value with respect to the current network parameters. After the above subtraction update is completed, the optimized network parameters are used for forward inference in subsequent rounds or for continued training. This allows the positioning and classification errors formed under different scenarios such as vehicle fires, smoldering cargo, hot smoke accumulation, and light and heat interference in the tunnel to be gradually fed back into the network parameter adjustment, thereby achieving continuous optimization of the fire confidence output and fire source coordinate output.

[0058] In summary, the real-time fire source detection method for extra-long highway tunnels based on multimodal fusion according to the embodiments of this application is elucidated. It introduces a smoke-temperature propagation delay modeling mechanism under wind speed constraints, dynamically compensates for phase shifts and aligns heterogeneous data such as visible light, infrared, smoke concentration, and ambient temperature in a spatiotemporal manner, enabling different modalities to point to the same physical moment of the fire. Based on this, it extracts spatial semantic information such as flame contours and thermal gradients from the visual side, as well as smoke-temperature abrupt changes and evolution information from the sensor side. A unified discriminative representation is formed using cross-modal interactive fusion, and fire source identification and spatial localization are achieved by combining target detection regression and calibration inverse transformation. This concept improves the perception capability of early, weak fires, reduces fusion bias caused by temporal misalignment, and enhances detection accuracy, real-time localization, and system stability under complex airflow disturbances, asynchronous smoke-temperature, and environmental interference conditions, while also considering false alarm suppression and engineering feasibility.

[0059] Furthermore, a real-time fire source detection system for extra-long highway tunnels based on multimodal fusion is also provided.

[0060] Figure 7 This is a block diagram of a real-time fire source detection system for extra-long highway tunnels based on multimodal fusion, according to an embodiment of this application. Figure 7As shown, the real-time fire source detection system 100 for long highway tunnels based on multimodal fusion according to an embodiment of this application includes: a dynamic propagation delay determination module 110, used to determine the dynamic propagation delay of smoke, temperature and airflow based on collected wind speed data inside the tunnel; a phase shift compensation and spatiotemporal alignment module 120, used to perform phase shift compensation and spatiotemporal alignment processing on visible light video stream, infrared video stream, smoke concentration time series data and ambient temperature time series data based on the dynamic propagation delay of smoke, temperature and airflow to obtain an aligned visual tensor and an aligned sensor matrix; a multimodal feature extraction module 130, used to extract spatial semantic features of flame contour and thermal gradient from the aligned visual tensor to obtain a visual feature map, and to extract temporal evolution features of smoke and temperature gradient abrupt changes from the aligned sensor matrix to obtain an environmental feature vector; and a cross-modal fusion module 140, used to perform cross-modal feature interaction and deep fusion on the visual feature map and the environmental feature vector to obtain a fused feature tensor. The fire source detection and localization module 150 uses a target detection regression network to predict fire source bounding boxes and calculate class probabilities on the fused feature tensor to obtain fire confidence. It also uses a pre-set calibration matrix to perform inverse perspective transformation on the centroid coordinates of the highest confidence bounding box to obtain the fire source coordinates. The evaluation and optimization module 160 inputs the fire source coordinates and fire confidence along with twin standard data into a joint loss function for comprehensive evaluation of spatial bias and classification error. It then uses a backpropagation gradient descent algorithm to adaptively update and optimize the current network parameters to obtain optimized network parameters.

[0061] As described above, the real-time fire source detection system 100 for long highway tunnels based on multimodal fusion according to embodiments of this application can be implemented in various types of computing devices or control units. For example, it can be deployed in an edge computing server in a highway tunnel monitoring center or installed in an industrial control computer in a tunnel electromechanical control cabinet. In one possible implementation, the real-time fire source detection system 100 for long highway tunnels based on multimodal fusion according to embodiments of this application can be integrated into the computing device as a software module and / or hardware module. For example, the system 100 can be a resident fire source monitoring service in the operating system of the computing device. The software module is configured to perform dynamic propagation delay calculation of smoke temperature airflow driven by wind speed data in the tunnel, phase shift compensation and spatiotemporal alignment of visible light video streams and infrared video streams, smoke concentration time series data and ambient temperature time series data, spatial semantic feature extraction of flame contour and thermal gradient, temporal evolution feature extraction of smoke temperature gradient abrupt change, cross-modal feature interaction and deep fusion, and fire source location output based on target detection regression and perspective inverse transformation. Alternatively, it can be a dedicated real-time tunnel fire detection and linkage early warning processing program developed for the computing device. Of course, the system 100 can also be one of the many hardware modules of the computing device or control unit, or it can be embedded in the field programmable gate array circuit to accelerate the tensor operation in the process of multimodal feature extraction, cross-modal attention correlation calculation and fire source localization with hardware parallel logic, or it can be a special application integrated circuit for tunnel fire source detection and localization processing.

[0062] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A real-time fire source detection method for extra-long highway tunnels based on multimodal fusion, characterized in that, include: Step 1: Based on the collected wind speed data inside the tunnel, determine the dynamic propagation delay of the smoke and temperature airflow. Step 2: Based on the dynamic propagation delay of smoke temperature airflow, phase shift compensation and spatiotemporal alignment processing are performed on visible light video stream, infrared video stream, smoke concentration time series data and ambient temperature time series data to obtain aligned visual tensor and aligned sensor matrix. Step 3: Extract spatial semantic features of flame contour and thermal gradient from the aligned visual tensor to obtain a visual feature map, and extract temporal evolution features of fluctuating smoke temperature gradient from the aligned sensor matrix to obtain an environmental feature vector. Step four involves cross-modal feature interaction and deep fusion of the visual feature map and the environmental feature vector to obtain a fused feature tensor. Step 5: Use the object detection regression network to predict the fire source bounding box and calculate the category probability of the fused feature tensor to obtain the fire confidence level. Combine the pre-set calibration matrix to perform perspective inverse transformation on the centroid coordinates of the bounding box with the highest confidence level to obtain the fire source coordinates.

2. The method for real-time fire source detection in extra-long highway tunnels based on multimodal fusion according to claim 1, characterized in that, Step one includes: based on wind speed data and pre-calibrated sensor spatial distance, the convection-diffusion law is used to dynamically solve the propagation time of the smoke-temperature airflow from the fire source to the sensor to obtain the propagation delay.

3. The method for real-time fire source detection in extra-long highway tunnels based on multimodal fusion according to claim 1, characterized in that, Step two includes: Using the propagation delay as a negative time bias, inverse phase shift compensation is performed on the time series data of smoke concentration and ambient temperature to obtain compensated smoke data and compensated temperature data. Affine registration and stacking of visible light and infrared video streams are performed to obtain an aligned visual tensor. The compensated smoke data and compensated temperature data are resampled and normalized according to frame timestamps and then stitched together to obtain an aligned sensor matrix.

4. The method for real-time fire source detection in extra-long highway tunnels based on multimodal fusion according to claim 1, characterized in that, Step three includes: The flame contour and thermal gradient spatial features at different receptive field scales were scanned on the aligned visual tensor to obtain visual feature maps. Temporal gradient abrupt change features are extracted from the aligned sensor matrix to obtain the environmental feature vector.

5. The method for real-time fire source detection in extra-long highway tunnels based on multimodal fusion according to claim 1, characterized in that, Step four includes: After spatially flattening the visual feature map, it is projected onto the same-dimensional hidden space along with the environment feature vector to obtain the mapped visual vector and the mapped environment vector. A query matrix is ​​generated by mapping visual vectors, and a key matrix and a value matrix are generated by mapping environment vectors. Cross-modal association calculations and weighted aggregations are performed on the query matrix, key matrix, and value matrix to obtain an enhanced multimodal matrix. The enhanced multimodal matrix is ​​subjected to high-dimensional nonlinear feature projection and activation filtering, followed by dimensionality reduction and redundancy elimination of the target dimension to obtain the fused feature tensor.

6. The method for real-time fire source detection in extra-long highway tunnels based on multimodal fusion according to claim 1, characterized in that, Step four includes: By utilizing the classification and regression branches of the target detection head network, fire probability inference and coordinate offset regression prediction are performed on the fused feature tensor to obtain fire confidence and initial bounding box. Using fire confidence as the scoring and ranking criterion, non-maximum suppression is used to filter the initial bounding boxes to remove redundant overlapping boxes, and geometric centroids are calculated for the best selected boxes to obtain the pixel centroids. Based on the camera intrinsic and extrinsic parameter matrices extracted from the calibration matrix, an inverse homography transformation matrix is ​​constructed. Combined with the road height constraint, the pixel centroid is subjected to inverse perspective transformation and three-dimensional spatial projection to obtain the fire source coordinates.

7. The method for real-time fire source detection in extra-long highway tunnels based on multimodal fusion according to claim 1, characterized in that, The process also includes step six: inputting the fire source coordinates and fire confidence along with twin standard data into the joint loss function to comprehensively evaluate spatial bias and classification error, and then using the backpropagation gradient descent algorithm to adaptively update and optimize the current network parameters to obtain optimized network parameters.

8. The method for real-time fire source detection in extra-long highway tunnels based on multimodal fusion according to claim 7, characterized in that, Step six includes: The fire source coordinates and fire confidence scores are respectively compared with the corresponding real coordinate labels and real category labels in the twin standard data. The regression loss and classification loss are calculated and then weighted and summed to obtain the comprehensive loss value. Based on the chain rule of differentiation, the partial derivatives of the comprehensive loss value are solved in reverse layer by layer along the computational graph path of the network forward propagation. The partial derivatives of the loss with respect to each trainable node in the current network parameters are calculated and aggregated to obtain the gradient matrix. The gradient descent optimizer is used to scale the gradient matrix by step size according to a preset learning rate, and the scaled gradient is applied to the current network parameters for subtraction update to obtain the optimized network parameters.

9. The method for real-time fire source detection in extra-long highway tunnels based on multimodal fusion according to claim 3, characterized in that, Using the propagation delay as a negative time bias, inverse phase shift compensation is performed on the time series data of smoke concentration and ambient temperature to obtain compensated smoke data and compensated temperature data, including: Differential mode delay decomposition is performed on the propagation delay to obtain the smoke mode delay and temperature mode delay; The cross-modal coupling strength is dynamically estimated based on a sliding window to obtain the cross-modal coupling coefficient for the smoke mode delay and the temperature mode delay. Based on the cross-modal coupling coefficient, smoke mode delay, and temperature mode delay, differential cross-correction phase shift compensation is performed on the coupled sensing of smoke concentration time series data and ambient temperature time series data to obtain compensated smoke data and compensated temperature data.

10. A real-time fire source detection system for extra-long highway tunnels based on multimodal fusion, characterized in that, include: The dynamic propagation delay determination module is used to determine the dynamic propagation delay of the smoke and temperature airflow based on the collected wind speed data inside the tunnel. The phase shift compensation and spatiotemporal alignment module is used to perform phase shift compensation and spatiotemporal alignment processing on visible light video streams, infrared video streams, smoke concentration time series data and ambient temperature time series data based on the dynamic propagation delay of smoke temperature airflow to obtain aligned visual tensors and aligned sensor matrices. The multimodal feature extraction module is used to extract spatial semantic features of flame contour and thermal gradient from the aligned visual tensor to obtain a visual feature map, and to extract temporal evolution features of fluctuating smoke temperature gradient from the aligned sensor matrix to obtain an environmental feature vector. The cross-modal fusion module is used to perform cross-modal feature interaction and deep fusion of visual feature maps and environmental feature vectors to obtain a fused feature tensor. The fire source detection and localization module is used to predict the fire source bounding box and calculate the category probability on the fused feature tensor using the target detection regression network to obtain the fire confidence level. It also uses a pre-set calibration matrix to perform perspective inverse transformation on the centroid coordinates of the bounding box with the highest confidence level to obtain the fire source coordinates.

Images