An emergency scene video analysis method and system based on a convolutional neural network

Abstract

This invention relates to the field of emergency scene analysis technology, and more particularly to an emergency scene video analysis method and system based on convolutional neural networks. The method acquires visible light video data and corresponding thermal imaging data from the emergency scene, aligns them according to a unified timestamp, and obtains multi-source temporal input data, reducing the instability in identification caused by single-modal anomalies. Then, based on convolutional neural networks, features are extracted from the multi-source temporal input data, and quality assessments are performed on clarity, occlusion degree, signal-to-noise ratio, frame drop status, and thermal signal integrity to obtain modal feature vectors and modal reliability parameters. Furthermore, gated weighted fusion is performed based on the modal reliability parameters, and placeholder substitution and weight suppression are applied when missing modalities exist to obtain a fused feature sequence. Combined with temporal convolutional networks to extract stage evolution features, the method outputs event staging results and risk scoring sequences, enabling continuous analysis of emergency event stages and risk change trends.

Description

Technical Field

[0001] This invention relates to the field of emergency scene analysis technology, and more particularly to an emergency scene video analysis method and system based on convolutional neural networks. Background Technology

[0002] In emergency scenarios such as earthquakes, fires, explosions, hazardous chemical leaks, traffic accidents, public place emergencies, and industrial production safety accidents, the on-site conditions change rapidly and there are many interfering factors. Managers usually need to rely on on-site video to identify and assess abnormal situations. In existing technologies, most emergency video analysis methods are mainly based on visible light video to detect flames, smoke, personnel behavior, or equipment anomalies. Some solutions also introduce thermal imaging data to assist in identifying high-temperature areas or hidden thermal anomalies.

[0003] However, existing technologies have the following shortcomings: On the one hand, single visible light videos are easily affected by smoke obstruction, insufficient lighting, image blurring, and camera shake, while single thermal imaging videos are also easily affected by thermal noise, thermal signal attenuation, and resolution limitations, resulting in poor stability of anomaly identification. On the other hand, most existing multi-source video analysis methods adopt fixed splicing or fixed weight fusion methods, lacking the evaluation of modal quality factors such as clarity, degree of occlusion, signal-to-noise ratio, frame drop status, and thermal signal integrity. They cannot dynamically adjust the fusion process according to the effectiveness of different modalities at the current moment, and are prone to introducing erroneous features when some modalities are distorted or missing, reducing the accuracy of analysis. Therefore, they can only identify abnormal targets or events at a single moment, lacking modeling of the continuous development process of emergency events, making it difficult to further determine the stage of the event and the trend of risk changes over time. This results in the output results remaining at the static alarm level, failing to achieve risk evolution analysis and stage judgment in emergency command. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides an emergency scene video analysis method and system based on convolutional neural networks.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: An emergency scene video analysis method based on convolutional neural networks includes the following steps: S1. Acquire visible light video data and corresponding thermal imaging data from the emergency site, align them according to a unified timestamp, and obtain multi-source time-series input data; S2. Based on the convolutional neural network, feature extraction is performed on the multi-source temporal input data respectively, and the quality mode set is evaluated to obtain the corresponding mode feature vector and mode confidence parameter. The quality mode set includes sharpness, occlusion degree, signal-to-noise ratio, frame drop status and thermal signal integrity. S3. Based on the modality confidence parameter, perform gated weighted fusion processing on each modality feature vector, and when there is a missing modality, replace the missing modality feature with a preset placeholder feature, while suppressing the fusion weight corresponding to the missing modality, to obtain a fusion feature sequence. S4. Input the fused feature sequence into a temporal convolutional network to extract the stage evolution features of the emergency event within a continuous time window, output the corresponding event staging results, and perform constrained cumulative calculation on the risk increment based on the event staging results to obtain a risk score sequence that changes continuously over time. S5. Based on the risk scoring sequence and event staging results, generate video analysis results of the emergency scene, including the type of abnormal event, risk level, and stage of the event.

[0006] Further, S1 includes the following steps: Acquire the visible light video stream output by the visible light acquisition device at the emergency site, and acquire the thermal imaging video stream output by the thermal imaging acquisition device corresponding to the visible light video stream; A collection timestamp is added to the visible light video stream and the thermal imaging video stream respectively, and video frame extraction processing is performed on the visible light video stream and the thermal imaging video stream according to a preset time window to obtain the visible light video frame sequence and the thermal imaging video frame sequence. Based on the acquisition timestamp of each visible light video frame in the visible light video frame sequence, the thermal imaging video frame sequence is subjected to same timestamp matching processing to obtain candidate thermal imaging video frames corresponding to each visible light video frame. For candidate thermal imaging video frames that are inconsistent with the acquisition timestamp of the visible light video frames, perform neighboring frame selection processing or inter-frame interpolation processing to obtain the target thermal imaging video frames corresponding to each visible light video frame. Each visible light video frame is combined with the corresponding target thermal imaging video frame in chronological order to obtain multi-source time-series input data.

[0007] Furthermore, the convolutional neural network is trained through the following steps: Historical multi-source time-series samples were constructed by using visible light video streams and thermal imaging video streams from historical emergency sites. Each historical multi-source time-series sample was labeled with the type of abnormal event, the stage of the event, and the corresponding labels for clarity, degree of occlusion, signal-to-noise ratio, frame drop status, and thermal signal integrity to obtain a training sample set. Visible light video frames and thermal imaging video frames from the training sample set are input into a convolutional neural network for convolution, activation, and downsampling operations to obtain visible light intermediate feature maps and thermal imaging intermediate feature maps. Global pooling and vector mapping are then performed on the visible light intermediate feature maps and thermal imaging intermediate feature maps to obtain the predicted modality feature vector. Based on the visible light intermediate feature map and the thermal imaging intermediate feature map, the quality assessment branch output is extracted respectively, and the quality assessment branch output is compared with the corresponding sharpness label, occlusion label, signal-to-noise ratio label, frame drop status label and thermal signal integrity label to calculate the error and obtain the quality assessment loss. Based on the predicted modality feature vector, event type prediction and event stage prediction are performed, and the prediction results are compared with the corresponding abnormal event type label and event stage label to calculate the error and obtain the feature extraction loss. A joint loss function is constructed based on the quality assessment loss and feature extraction loss. The network parameters of the visible light convolutional neural network and the thermal imaging convolutional neural network are then updated in reverse based on the joint loss function until the joint loss function converges, resulting in a trained convolutional neural network.

[0008] Furthermore, the joint loss function is as follows: ; in, This is the loss value; This represents the number of training samples; This represents the confidence value of the target modality corresponding to the i-th training sample. This represents the confidence value of the predicted mode output by the convolutional neural network for the i-th training sample. This represents the integrity value of the target thermal signal corresponding to the i-th training sample. This represents the completeness value of the predicted thermal signal output by the convolutional neural network for the i-th training sample. Let be the target label value of the i-th training sample on the k-th type of abnormal event; Let be the probability that the i-th training sample belongs to the k-th type of abnormal event predicted by the convolutional neural network. This represents the total number of categories of abnormal events. Let be the target label value of the i-th training sample at the m-th event stage; Let be the probability that the convolutional neural network predicts the i-th training sample as belonging to the m-th event stage; This represents the total number of categories in the event phase. , , and These are the weighting coefficients.

[0009] Further, S3 includes the following steps: Based on the visible light modal confidence parameters and thermal imaging modal confidence parameters, vector concatenation is performed with the corresponding visible light modal feature vector and thermal imaging modal feature vector, respectively, and then input into the gating mapping layer to perform matrix multiplication and activation calculation to obtain the initial gating value for visible light and the initial gating value for thermal imaging. Normalization is performed on the initial visible light gate value and the initial thermal imaging gate value to obtain the visible light gate weight and the thermal imaging gate weight. Based on the comparison results between the visible light modal confidence parameter and the thermal imaging modal confidence parameter and the preset missing value judgment threshold, the corresponding missing value marker is generated. Based on the missing marker, the historical mean feature vector corresponding to the missing mode is called to replace the missing mode feature vector, and the gating weight corresponding to the missing mode is corrected to the preset suppression weight, so as to obtain the corrected mode feature vector and the corrected gating weight. Based on the modified modal feature vectors and the modified gating weights, element-wise product calculation and cross-modal summation are performed respectively to obtain the fusion feature vectors corresponding to each time step. The fusion feature vectors corresponding to each time step are then combined in chronological order to obtain the fusion feature sequence.

[0010] Further, S4 includes the following steps: Based on the fused feature sequence, time segmentation is performed according to a preset sliding time window to obtain multiple continuous time segments; Each of the continuous time segments is input into a temporal convolutional network, and one-dimensional convolution, dilated convolution, and temporal pooling operations are performed to obtain the stage evolution features corresponding to each continuous time segment. The event stage classification calculation is performed based on the evolution characteristics of each stage to obtain the event staging results corresponding to each continuous time segment. The stage transition constraint correction is performed based on the event staging results of adjacent continuous time segments to obtain the target event staging results arranged in chronological order. Based on the phased results of the target event and the corresponding stage evolution characteristics, the risk increment value corresponding to each continuous time segment is calculated, and the risk increment value is cumulatively calculated under constraints according to the chronological order of the event stages to obtain the risk score sequence.

[0011] Furthermore, the temporal convolutional network is trained through the following steps: Obtain the fusion feature sequence corresponding to the historical emergency scene, and determine the start time, stage transition time and handling time of the abnormal event based on the video timeline, event handling record, alarm log and manual annotation results corresponding to each fusion feature sequence; Based on the start time, stage transition time, and disposal time, each fused feature sequence is segmented and sliced ​​according to a preset sliding time window to obtain several historical continuous time segments. Based on the position of each historical continuous time segment in the original event timeline, the corresponding event stage label and risk score label are marked. Based on the event stage labels and risk score labels, historical continuous time segments belonging to the same event stage are classified, and training samples are extracted from the historical continuous time segments corresponding to each event stage according to a preset ratio to construct a time series training set with balanced stage distribution. Each historical continuous time segment in the time-series training set is input into the temporal convolutional network for training. The error is calculated based on the output results and the corresponding event stage labels and risk score labels. The network parameters of the temporal convolutional network are updated in reverse based on the error calculation results until the training loss converges, and the trained temporal convolutional network is obtained.

[0012] Furthermore, generating video analysis results of the emergency scene based on the risk scoring sequence and event staging results includes the following steps: Based on the risk score value corresponding to each time step in the risk scoring sequence, interval mapping is performed according to the preset risk level division threshold to obtain the risk level result corresponding to each time step. The event staging results and the risk level results corresponding to each time step are time-aligned and correlated to obtain time-series analysis results containing event stage fields and risk level fields; Based on the time period in the time series analysis results, locate the target video frame that triggers the risk change; Based on the time-series analysis results and the target video frame, output the video analysis results.

[0013] Furthermore, based on the time period in the time series analysis results where the risk level increases, the target video frames that trigger the risk change include: By comparing the risk level results corresponding to adjacent time steps in the time series analysis results, the time segment in which the risk level changes from low to high is determined, and the risk change segment is obtained. Based on the temporal position of the risk change segment in the fused feature sequence, the target time window is obtained by reverse indexing the continuous time segments corresponding to the risk change segment. Based on the start and end times corresponding to the target time window, video frames corresponding to the target time window are extracted from the visible light video frame sequence and the thermal imaging video frame sequence to obtain a candidate target video frame set. The modal credibility parameters, event stage results, and risk score change magnitudes corresponding to the candidate target video frame set are comprehensively sorted, and the video frame with the highest score is selected as the target video frame that triggers the risk change.

[0014] An emergency scene video analysis system based on convolutional neural networks, applied to any of the aforementioned emergency scene video analysis methods based on convolutional neural networks, includes: The data acquisition module is used to acquire visible light video data and corresponding thermal imaging data from the emergency site, and align them according to a unified timestamp to obtain multi-source time-series input data. The quality assessment module is used to extract features from the multi-source temporal input data based on a convolutional neural network, and to assess the clarity, occlusion, signal-to-noise ratio, frame drop status, or thermal signal integrity of each source data, so as to obtain the corresponding modal feature vector and modal confidence parameters. The gated fusion module is used to perform gated weighted fusion processing on each modality feature vector according to the modality confidence parameter, and when there is a missing modality, replace the missing modality feature with a preset placeholder feature, while suppressing the fusion weight corresponding to the missing modality to obtain a fusion feature sequence. The temporal evolution analysis module is used to input the fused feature sequence into a temporal convolutional network, extract the stage evolution features of the emergency event within a continuous time window, output the corresponding event staging results, and perform constrained cumulative calculation of the risk increment based on the event staging results to obtain a risk score sequence that changes continuously over time. The analysis result generation module is used to generate video analysis results of the emergency scene based on the risk scoring sequence and event staging results. The video analysis results include the abnormal event type, risk level, and event stage.

[0015] This application also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described in any of the foregoing claims.

[0016] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the preceding claims.

[0017] The beneficial effects of this invention are as follows: By acquiring visible light video data and corresponding thermal imaging data from emergency scenes and aligning them according to a unified timestamp, this invention obtains multi-source time-series input data, enabling visible light information and thermal imaging information to establish a correspondence under the same time reference. This overcomes the limitations of existing technologies that mainly rely on single visible light video or single thermal imaging video for analysis, and reduces the impact of factors such as smoke obstruction, insufficient lighting, image blurring, thermal noise, and thermal signal attenuation on the stability of anomaly identification. Furthermore, based on a convolutional neural network, feature extraction is performed on the multi-source time-series input data, and the clarity, obstruction degree, signal-to-noise ratio, frame drop status, or thermal signal integrity of each source data are evaluated to obtain modal feature vectors and modal credibility parameters. This allows the system to extract scene anomaly features while simultaneously quantifying the effectiveness of each modality at the current moment, thereby solving the problem that existing multi-source video analysis methods only use fixed splicing or fixed weight fusion and cannot reflect the quality differences between different modalities. Next, based on the modality credibility parameter, gated weighted fusion processing is performed on the feature vectors of each modality. When a missing modality exists, a preset placeholder feature replaces the missing modality feature, while suppressing the fusion weights corresponding to the missing modality, resulting in a fused feature sequence. This allows the fusion process to be dynamically adjusted according to modality quality, maintaining the continuity and stability of feature expression even when some modalities are distorted or missing. This avoids low-quality or missing modalities from bringing erroneous features into the subsequent analysis process, improving the accuracy and robustness of video analysis in complex emergency environments. Finally, the fused feature sequence is input into a temporal convolutional network to extract the stage evolution features of the emergency event within a continuous time window, outputting the corresponding event staging results. Based on the event staging results, the risk increment is constrained and accumulated to obtain a risk score sequence that changes continuously over time. This expands the analysis process from single-moment anomaly identification in existing technologies to modeling the continuous development process of emergency events. It can not only identify anomalies but also determine the stage of the event and the trend of risk level changes over time. Attached Figure Description

[0018] Figure 1 This is a flowchart of the steps of an emergency scene video analysis method based on convolutional neural networks in this invention.

[0019] Figure 2 This is a flowchart of the training steps of the temporal convolutional network in this invention. Detailed Implementation

[0020] Please see Figures 1-2 As shown, this invention relates to an emergency scene video analysis method based on a convolutional neural network, comprising the following steps: S1. Acquire visible light video data and corresponding thermal imaging data from the emergency site, align them according to a unified timestamp, and obtain multi-source time-series input data; S2. Based on the convolutional neural network, feature extraction is performed on the multi-source temporal input data respectively, and the quality mode set is evaluated to obtain the corresponding mode feature vector and mode confidence parameter. The quality mode set includes sharpness, occlusion degree, signal-to-noise ratio, frame drop status and thermal signal integrity. S3. Based on the modality confidence parameter, perform gated weighted fusion processing on each modality feature vector, and when there is a missing modality, replace the missing modality feature with a preset placeholder feature, while suppressing the fusion weight corresponding to the missing modality, to obtain a fusion feature sequence. S4. Input the fused feature sequence into a temporal convolutional network to extract the stage evolution features of the emergency event within a continuous time window, output the corresponding event staging results, and perform constrained cumulative calculation on the risk increment based on the event staging results to obtain a risk score sequence that changes continuously over time. S5. Based on the risk scoring sequence and event staging results, generate video analysis results of the emergency scene, including the type of abnormal event, risk level, and stage of the event.

[0021] In some embodiments, a visible light video stream is first output by a visible light acquisition device on-site, and a thermal imaging video stream corresponding to the same monitoring area is output by a thermal imaging acquisition device. To avoid time misalignment between the two types of videos due to differences in sampling frequency, startup delay, and transmission buffer between different devices, this embodiment does not directly match the two video streams frame by frame. Instead, it first adds acquisition timestamps to the visible light video stream and the thermal imaging video stream respectively, and then performs video frame extraction processing within a preset time window. For example, in a chemical storage tank area leak monitoring scenario, a visible light camera outputs video at 25 frames per second, and a thermal imaging device outputs thermal video at 9 frames per second. If the frames are directly aligned according to the frame number, the 100th frame of the visible light image and the 100th frame of the thermal imaging image do not correspond to the same physical moment, which can easily cause a mismatch between the leak plume boundary in the visible light image and the high-temperature disturbance area in the thermal imaging. To this end, this embodiment performs timestamp matching in the thermal imaging video frame sequence based on the acquisition timestamp of each visible light video frame to obtain candidate thermal imaging video frames. If the time difference between a candidate thermal imaging video frame and its corresponding visible light video frame exceeds a preset allowable threshold, then neighboring frame selection processing or inter-frame interpolation processing is further performed to obtain the target thermal imaging video frame that is temporally closest to each visible light video frame. The multi-source temporal input data obtained in this way is not a simple dual-channel image, but rather establishes a synchronous correspondence between the target morphology, texture boundaries, and smoke diffusion contours in the visible light image and the thermal anomaly regions, thermal gradient changes, and thermal boundary expansions in the thermal imaging image under a unified time reference, providing a foundation for subsequent cross-modal quality assessment and fusion.

[0022] In step S2, this embodiment extracts features from the multi-source temporal input data based on a convolutional neural network and simultaneously quantifies the quality status of each source data. It should be noted that while existing technologies often use convolutional neural networks to extract image features, most schemes assume the input modality is reliable, focusing only on what is in the image and not on whether the current image value is reliable. This scheme directly incorporates modality quality assessment into the feature generation stage, enabling the network to output modality feature vectors along with additional modality reliability parameters, thus providing a decision-making basis for subsequent fusion. Specifically, a visible light video frame sequence is input into a visible light convolutional neural network. After convolution, nonlinear activation, and downsampling operations, a visible light feature map representing target contours, edge textures, smoke diffusion patterns, personnel postures, and equipment appearance changes is obtained. A thermal imaging video frame sequence is input into a thermal imaging convolutional neural network, resulting in a thermal imaging feature map representing temperature distribution, local high-temperature zone expansion, thermal boundary clarity, and the degree of spatial aggregation of thermal anomalies. Furthermore, global pooling and vector mapping are performed on the two types of feature maps to obtain visible light modal feature vectors and thermal imaging modal feature vectors, respectively. The imaging quality of the current video frame is quantitatively characterized through a quality assessment branch. For example, for visible light video frames, the image gradient statistics can be calculated to obtain the sharpness value; the occlusion degree can be calculated through the integrity of the foreground region and the coverage ratio of occluders; the signal-to-noise ratio can be calculated based on the luminance-noise variance and the signal intensity of the target region; and the frame drop status can be determined by combining the sampling timestamp interval. For thermal imaging video frames, the effective thermal area coverage ratio can be calculated as the thermal signal integrity value; and thermal imaging quality parameters can be constructed by combining thermal response continuity and background thermal noise intensity. Each quality index is normalized and mapped to form modal reliability parameters. Taking a fire scene in an underground parking garage as an example, in the early stages of the fire, the visible light image may still be able to identify the vehicle outline and local smoke drift, but the thermal imaging image is more sensitive to high-temperature points; as the smoke rapidly increases, the sharpness of the visible light image decreases significantly and the occlusion degree increases, at which point the visible light modal reliability parameter will decrease synchronously.

[0023] In step S3, the visible light modality confidence parameter is concatenated with the visible light modality feature vector, and the thermal imaging modality confidence parameter is concatenated with the thermal imaging modality feature vector. These are then input into the gating mapping layer. Through matrix multiplication and nonlinear activation calculations, initial gating values ​​for visible light and thermal imaging are obtained. After normalization, gating weights for the two modalities are formed. When the system detects that the confidence parameter of a certain modality is lower than the preset missing threshold, it does not simply discard the modality. Instead, it calls the mean feature vector of that modality obtained from historical normal samples as a placeholder feature to replace the missing modality feature. Simultaneously, the gating weight corresponding to that modality is corrected to the preset suppression weight. Taking a hazardous chemical leak scenario as an example, if the thermal imaging camera is briefly subjected to high-temperature steam interference, resulting in blurred thermal boundaries and abnormally increased thermal noise intensity, the thermal imaging confidence parameter decreases. The system reduces the weight of the thermal imaging modality in the current fusion process on the one hand, and uses historical mean thermal features as placeholders on the other hand, ensuring that the fusion vector maintains structural continuity over time, preventing sequence breakage due to the complete loss of thermal imaging features at a certain moment. Finally, through element-wise product calculation and cross-modal summation, the fused feature vectors corresponding to each time step are obtained and combined into a fused feature sequence in chronological order. A dual mechanism of credibility-driven dynamic gating and structural compensation under missing states is constructed, enabling multimodal fusion to move beyond static stitching and acquire adaptive capabilities for unstable imaging conditions in emergency scenarios.

[0024] In step S4, taking a warehouse fire as an example, in the initial stage, the fused feature sequence shows the appearance of local thermal anomalies and a slight enhancement of visible smoke texture; in the continuous development stage, the thermal anomaly area expands rapidly, smoke concentration increases, and personnel activity patterns change; in the rapid deterioration stage, the high-temperature area expands rapidly, the smoke plume rises faster, and the obstruction area increases sharply. Temporal convolutional networks can simultaneously capture abrupt changes within short timescales and continuous growth trends over longer timescales through convolutional kernels with different receptive fields, thereby forming stage evolution features and outputting corresponding event staging results. Furthermore, this embodiment does not directly output the risk score for each time window independently, but rather performs constrained cumulative calculation of the risk increment based on the event staging results. For example, when the event is in the initial warning stage, the cumulative slope of the risk increment is low; when the event enters the rapid deterioration stage, the risk increment is allowed to grow at a higher slope; when the event enters the mitigation stage, a suppressive constraint is applied to the newly added risk value. The resulting risk score sequence is not merely a simple change in classification probability, but a continuous risk representation with stage constraint significance.

[0025] In step S5, based on the risk scoring sequence and event staging results, video analysis results of the emergency scene are generated. These results include the type of abnormal event, risk level, and stage of the event. The output here is no longer a single alarm trigger signal, nor a simple bounding box or category label, but a comprehensive analysis result integrating event category, risk level, and stage status. For example, in a tunnel traffic accident scenario, the output might be "Abnormal event type: vehicle fire; current risk level: high risk; event stage: rapidly deteriorating stage." In a chemical plant leak scenario, the output might be "Abnormal event type: hazardous chemical leak with localized thermal anomaly; current risk level: medium-high risk; event stage: ongoing development stage." This output format allows emergency commanders to directly determine, based on the system results, whether to take action such as continuous monitoring, localized containment, personnel evacuation, or emergency reinforcement. More importantly, this result comes from the progressive processing of multimodal quality perception, gating fusion and time-series evolution modeling, rather than triggering threshold alarms based solely on single-frame target detection results. Therefore, its engineering value lies not only in the improvement of recognition accuracy, but also in the significantly enhanced decision-making usability of the output information.

[0026] Further, S1 includes the following steps: Acquire the visible light video stream output by the visible light acquisition device at the emergency site, and acquire the thermal imaging video stream output by the thermal imaging acquisition device corresponding to the visible light video stream; A collection timestamp is added to the visible light video stream and the thermal imaging video stream respectively, and video frame extraction processing is performed on the visible light video stream and the thermal imaging video stream according to a preset time window to obtain the visible light video frame sequence and the thermal imaging video frame sequence. Based on the acquisition timestamp of each visible light video frame in the visible light video frame sequence, the thermal imaging video frame sequence is subjected to same timestamp matching processing to obtain candidate thermal imaging video frames corresponding to each visible light video frame. For candidate thermal imaging video frames that are inconsistent with the acquisition timestamp of the visible light video frames, perform neighboring frame selection processing or inter-frame interpolation processing to obtain the target thermal imaging video frames corresponding to each visible light video frame. Each visible light video frame is combined with the corresponding target thermal imaging video frame in chronological order to obtain multi-source time-series input data.

[0027] Specifically, the visible light acquisition equipment deployed at the emergency site continuously outputs a visible light video stream, while the thermal imaging acquisition equipment simultaneously outputs a thermal imaging video stream. This "correspondence" does not simply mean that the two devices are installed in the same area, but rather that they share a common coverage area for the monitored target. For example, in a hazardous chemical storage tank leak monitoring scenario, the visible light camera focuses on the tank valve area and the ground diffusion area, while the thermal imaging equipment focuses on the same area to collect changes in heat distribution. Although both observe the same object, in actual operation, the visible light equipment may sample at 25 frames per second, while the thermal imaging equipment may only sample at 8 or 9 frames per second. Therefore, the two video streams are naturally at different time scales. If they are directly correlated according to the video frame number, the 100th frame of the visible light image and the 100th frame of the thermal imaging image would actually come from different absolute times, thus undermining the premise of subsequent multimodal joint analysis. Therefore, after data access, a timestamp is first added to each frame of the visible light image and each frame of the thermal imaging image. The timestamp preferably uses an absolute time identifier calibrated between the device's local clock and the system's unified clock, but microsecond or millisecond-level incremental time sequence numbers can also be used. In this way, the original video stream is no longer a simple collection of image frames, but is transformed into a time-series object stream containing image content and acquisition time. Subsequently, video frame extraction processing is performed on the visible light video stream and the thermal imaging video stream according to preset time windows, respectively, to obtain visible light video frame sequences and thermal imaging video frame sequences. This time window is not a simple equally spaced sampling parameter, but rather the basic discrete granularity for subsequent time-series modeling. For example, in the early monitoring scenario of a fire on a subway platform, to balance real-time performance and subsequent network computational burden, the time window can be set to 200 milliseconds, that is, a representative visible light video frame is selected every 200 milliseconds, and the thermal imaging frame closest to the center moment within this window is simultaneously extracted from the thermal imaging video stream. Furthermore, to enhance the stability of frame extraction, this embodiment can calculate indicators such as image blur, average brightness, and motion amplitude within each preset time window, and prioritize the selection of frames with higher image clarity and more complete target information as representative frames for that window. After obtaining the visible light video frame sequence and the thermal imaging video frame sequence, this embodiment performs timestamp matching processing on the thermal imaging video frame sequence based on the acquisition timestamp of each visible light video frame in the visible light video frame sequence to obtain candidate thermal imaging video frames corresponding to each visible light video frame. Visible light frames are used as the primary time reference because in most emergency situations, visible light video has higher temporal resolution and finer motion capture capabilities, making it suitable as the primary sequence for anchoring. Algorithmically, the acquisition timestamp of the i-th visible light video frame can be set as t_i. One or more thermal imaging video frames with acquisition times closest to t_i are retrieved from the thermal imaging video frame sequence and identified as candidate thermal imaging video frames for that visible light frame.If multiple frames in the thermal imaging video frame sequence have similar timestamps to t_i, the frame with the smallest absolute time difference can be further selected. If there are still frames with the same time difference, the frame with higher thermal signal integrity or clearer thermal boundary can be selected as the candidate thermal imaging video frame. This step completes the transformation from retrieval and matching on a unified time axis to coarse synchronization. However, in practical applications, the low thermal imaging sampling frequency and unstable device buffer refresh cycle often result in the acquisition timestamps of candidate thermal imaging video frames not being completely consistent with the acquisition timestamps of the corresponding visible light video frames. Therefore, this embodiment further performs neighboring frame selection processing or inter-frame interpolation processing on candidate thermal imaging video frames with inconsistent timestamps to obtain the target thermal imaging video frame corresponding to each visible light video frame. The so-called neighboring frame selection processing refers to directly selecting the thermal imaging frame with the closest time as the target thermal imaging video frame when the time difference between the thermal imaging frame and the visible light frame is less than a preset tolerance threshold. Inter-frame interpolation, in this context, refers to generating a compensation frame by performing temporal interpolation on adjacent thermal imaging frames when the time difference exceeds a tolerance threshold and two adjacent thermal imaging frames are located on opposite sides of the visible light frame's time point. For example, if the target visible light frame timestamp is 10.400 seconds, but the thermal imaging sequence only contains frames at 10.333 seconds and 10.500 seconds, then linear interpolation can be performed on the corresponding pixels or thermal feature values ​​in the two thermal imaging images according to the time distance weight, generating a target thermal imaging video frame approximately at 10.400 seconds. In a more preferred implementation, instead of directly interpolating the original pixels, a low-level thermal feature map can be extracted from the thermal imaging frame first, and then temporal interpolation can be performed on the feature map to reduce the impact of thermal noise on the interpolation result. This processing has a higher temporal approximation capability compared to the simple nearest neighbor frame selection in existing technologies, and is particularly suitable for rapidly changing stages of emergency events. For example, in a scenario where the high-temperature point rises rapidly before an explosion, the boundary of the thermal anomaly region will expand significantly in a short period of time. If only the nearest neighbor frame is selected, the state before the change may be incorrectly mapped to the state after the change. However, by using inter-frame interpolation, the thermal field distribution at the target time can be recovered more accurately. After obtaining the target thermal imaging video frame corresponding to each visible light video frame, this embodiment combines each visible light video frame with the corresponding target thermal imaging video frame in chronological order to obtain multi-source time-series input data. Here, "combination" is not simply stacking two images into a dual-channel matrix, but rather constructing a multimodal data structure with time steps as index units. In each time step, the data unit includes at least: the visible light video frame at the current time, the target thermal imaging video frame after time matching or interpolation compensation, and a unified time identifier corresponding to that time step. In some embodiments, this time step may also include basic metadata required for subsequent quality assessment, such as the original time difference, interpolation markers, or frame selection method markers.The multi-source time-series input data constructed in this way is essentially a time-ordered dual-modal sample sequence, with each sample representing the combined state of visible light observation and thermal observation at the same physical moment.

[0028] Furthermore, the convolutional neural network is trained through the following steps: Historical multi-source time-series samples were constructed by using visible light video streams and thermal imaging video streams from historical emergency sites. Each historical multi-source time-series sample was labeled with the type of abnormal event, the stage of the event, and the corresponding labels for clarity, degree of occlusion, signal-to-noise ratio, frame drop status, and thermal signal integrity to obtain a training sample set. Visible light video frames and thermal imaging video frames from the training sample set are input into a convolutional neural network for convolution, activation, and downsampling operations to obtain visible light intermediate feature maps and thermal imaging intermediate feature maps. Global pooling and vector mapping are then performed on the visible light intermediate feature maps and thermal imaging intermediate feature maps to obtain the predicted modality feature vector. Based on the visible light intermediate feature map and the thermal imaging intermediate feature map, the quality assessment branch output is extracted respectively, and the quality assessment branch output is compared with the corresponding sharpness label, occlusion label, signal-to-noise ratio label, frame drop status label and thermal signal integrity label to calculate the error and obtain the quality assessment loss. Based on the predicted modality feature vector, event type prediction and event stage prediction are performed, and the prediction results are compared with the corresponding abnormal event type label and event stage label to calculate the error and obtain the feature extraction loss. A joint loss function is constructed based on the quality assessment loss and feature extraction loss. The network parameters of the visible light convolutional neural network and the thermal imaging convolutional neural network are then updated in reverse based on the joint loss function until the joint loss function converges, resulting in a trained convolutional neural network.

[0029] It should be noted that, firstly, historical multi-source time-series samples are constructed using visible light video streams and thermal imaging video streams from historical emergency scenes. These "historical emergency scenes" can originate from fire drill videos, chemical leak accident replay videos, tunnel fire monitoring videos, warehouse fire early warning videos, or videos collected from artificially constructed simulated scenarios. To make the training samples more consistent with real emergency environments, this embodiment preferably collects video data containing various complex disturbance factors, such as smoke obstruction, low illumination, lens shake, and motion blur in the visible light mode, and thermal reflection, thermal saturation, thermal noise enhancement, and local thermal signal interruption in the thermal imaging mode. Then, following the aforementioned unified timestamp alignment mechanism, time alignment, windowed frame extraction, and matching compensation processing are performed on the historical visible light video streams and historical thermal imaging video streams to obtain historical multi-source time-series samples. Each historical multi-source time-series sample is not a single visible light image or thermal imaging image, but rather a dual-modal sample unit with a clear correspondence established at the same time step, containing the visible light video frame, thermal imaging video frame, and corresponding time stamp information for that moment. After constructing the sample set, each historical multi-source time-series sample is labeled with the type of abnormal event, the event stage, and the corresponding labels for clarity, occlusion degree, signal-to-noise ratio, frame drop status, and thermal signal integrity, thus obtaining the training sample set. Visible light video frames and thermal imaging video frames from the training sample set are then input into a convolutional neural network for convolution, activation, and downsampling operations, respectively, to obtain intermediate feature maps for visible light and thermal imaging. In terms of implementation, the visible light and thermal imaging modal can employ a dual-branch convolutional neural network with identical structure but independent parameters, or a network structure with independent front layers and partially shared back layers. Preferably, the visible light branch focuses on extracting visual features such as texture gradients, edge contours, smoke diffusion patterns, flame flickering areas, and changes in human posture, while the thermal imaging branch focuses on extracting thermal field features such as temperature hotspots, thermal boundary expansion, local heat cluster aggregation, and abrupt changes in thermal gradients. The convolution operation extracts spatial local patterns from the input image through the local receptive field, the activation operation maps the linear convolution response to a discriminative nonlinear representation, and the downsampling operation improves the stability of high-level semantic representation by reducing spatial resolution. For example, in the training samples of warehouse fires, the shallow convolutional kernels in the visible light branch preferentially respond to flame edges and smoke textures, the middle convolutional kernels gradually integrate the flame clusters and rising smoke plumes, and the deep convolutional kernels form an abstract representation of the "spatial layout of the fire." In the thermal imaging branch, the shallow convolutional kernels can respond to local high-temperature spots and hot edges, the middle convolutional kernels further characterize the expansion of high-temperature areas and the aggregation of thermal gradients, and the deep convolutional kernels form a representation of the "spatial evolution morphology of thermal anomalies." After obtaining the intermediate feature maps of visible light and thermal imaging, global pooling and vector mapping are performed based on these intermediate feature maps to obtain the predicted modal feature vectors.The global pooling operation here aims to compress high-dimensional features in spatial dimensions into a fixed-length global representation, enabling the network output to be directly used as a modal feature vector by subsequent fusion modules. Vector mapping processing can further compress the pooled features to a unified dimension through fully connected layers or linear projection layers, ensuring that the two modal features can be operated on element-wise in the subsequent gated fusion stage. For example, after global average pooling, the visible light intermediate feature map yields a set of global response values ​​representing the scene's appearance state, which are then compressed into 128-dimensional or 256-dimensional visible light modal feature vectors through a mapping layer; the thermal imaging intermediate feature map is also mapped to the same dimension in the same way to form a thermal imaging modal feature vector. The innovation of this step lies in extending the training objective of the convolutional network from the final classification accuracy in traditional classification networks to generating intermediate modal representations that can directly participate in quality-aware fusion, so that the trained features are not only effective for classification tasks, but also have structured representational value for subsequent cross-modal decision-making. Based on the visible light intermediate feature map and the thermal imaging intermediate feature map, the output of the quality assessment branch is extracted respectively. The quality assessment branch output is then compared with the corresponding sharpness label, occlusion degree label, signal-to-noise ratio label, frame drop state label, and thermal signal integrity label to calculate the error and obtain the quality assessment loss. Here, the quality assessment branch can be understood as an auxiliary prediction head branching from the intermediate feature map. Its goal is not to determine the anomaly type, but to learn to regress or classify the current input imaging quality state from deep features. Taking the visible light modality as an example, the visible light intermediate feature map not only contains event semantics but also a certain degree of imaging information, such as whether the edge sharpness is obvious, whether the local texture is continuous, and whether the occlusion area has expanded. The quality assessment branch utilizes these intermediate features to predict sharpness, occlusion degree, signal-to-noise ratio, and frame drop state respectively. Further, event type prediction and event stage prediction are performed based on the predicted modality feature vector, and the prediction results are compared with the corresponding abnormal event type label and event stage label to calculate the error and obtain the feature extraction loss. The event type prediction here is used to learn the anomaly category corresponding to the sample, such as fire, leak, deflagration, personnel entrapment, abnormal equipment temperature rise, etc.; the event stage prediction is used to learn the stage of the emergency event, such as the initial warning stage, the continuous development stage, the rapid deterioration stage, or the handling and mitigation stage. It is important to emphasize that the event stage prediction is not a static classification and labeling common in existing technologies, but a pre-training task that corresponds to the stage evolution modeling in the subsequent temporal convolutional network. That is to say, this embodiment aims to enable the convolutional neural network to perceive the stage semantic tendency corresponding to the time step when extracting modal features at a single time step. For example, in the same fire scene, the visible light features of the initial stage may be characterized by the initial appearance of local smoke and a small flame area, while the features of the rapid deterioration stage are characterized by a large-area flame expansion and obvious thermal field expansion. Through the supervision of event stage labels, the network can encode the discriminative information of the development interval of the event in the modal feature vector.

[0030] Furthermore, the joint loss function is as follows: ; in, This is the loss value; This represents the number of training samples; This represents the confidence value of the target modality corresponding to the i-th training sample. This represents the confidence value of the predicted mode output by the convolutional neural network for the i-th training sample. This represents the integrity value of the target thermal signal corresponding to the i-th training sample. This represents the completeness value of the predicted thermal signal output by the convolutional neural network for the i-th training sample. Let be the target label value of the i-th training sample on the k-th type of abnormal event; Let be the probability that the i-th training sample belongs to the k-th type of abnormal event predicted by the convolutional neural network. This represents the total number of categories of abnormal events. Let be the target label value of the i-th training sample at the m-th event stage; Let be the probability that the convolutional neural network predicts the i-th training sample as belonging to the m-th event stage; This represents the total number of categories in the event phase. , , and These are the weighting coefficients.

[0031] Specifically, the loss value is first averaged across all training samples to eliminate the influence of sample size on the loss value scale, thereby ensuring training stability across different batches. At the level of a single training sample, the joint loss is composed of four types of errors. The first type of error characterizes the deviation between the modal confidence level output by the network and the target modal confidence level. This target modal confidence level is not a fixed label but is generated based on the clarity, occlusion level, signal-to-noise ratio, and frame drop status of the visible light image, combined with the overall availability of thermal imaging, reflecting the reliability of the current input modality at that moment. When the network still gives a high confidence prediction for low-quality images, this error term increases significantly, prompting the network to pay more attention to image quality factors during feature extraction. The second type of error characterizes the deviation between the thermal signal integrity level output by the network and the target thermal signal integrity level. Its target value reflects the continuity and integrity of the effective thermal area in thermal imaging, such as whether the thermal boundary is clear, whether the thermal distribution is continuous, and whether there is missing thermal signal or noise interference. Through the constraint of this error term, the network can learn the ability to discriminate "effective thermal information" in thermal imaging data during training, rather than relying solely on thermal images for event recognition. The third type of error characterizes the difference between the network's prediction of abnormal event types and the target label. This part compares the predicted probabilities of each category with the true category labels, enabling the network to gradually build the ability to distinguish between different abnormal events (such as fires, leaks, abnormal temperature rises, etc.). If the network's prediction probability of the true category is too low, this error term increases, thereby driving the network to strengthen the extraction of key discriminative features. The fourth type of error characterizes the difference between the network's prediction of event stages and the target stage label, guiding the network to possess stage semantic expression capabilities at the single-frame feature level. For example, different stages have significant differences in image representation; through this error term constraint, the network can learn to distinguish the development state of an event from spatial features. These four types of errors are weighted and combined in the joint loss function using weight coefficients. Each weight coefficient is used to adjust the contribution ratio of different tasks in the overall optimization process, to avoid one type of error dominating the training process.

[0032] Further, S3 includes the following steps: Based on the visible light modal confidence parameters and thermal imaging modal confidence parameters, vector concatenation is performed with the corresponding visible light modal feature vector and thermal imaging modal feature vector, respectively, and then input into the gating mapping layer to perform matrix multiplication and activation calculation to obtain the initial gating value for visible light and the initial gating value for thermal imaging. Normalization is performed on the initial visible light gate value and the initial thermal imaging gate value to obtain the visible light gate weight and the thermal imaging gate weight. Based on the comparison results between the visible light modal confidence parameter and the thermal imaging modal confidence parameter and the preset missing value judgment threshold, the corresponding missing value marker is generated. Based on the missing marker, the historical mean feature vector corresponding to the missing mode is called to replace the missing mode feature vector, and the gating weight corresponding to the missing mode is corrected to the preset suppression weight, so as to obtain the corrected mode feature vector and the corrected gating weight. Based on the modified modal feature vectors and the modified gating weights, element-wise product calculation and cross-modal summation are performed respectively to obtain the fusion feature vectors corresponding to each time step. The fusion feature vectors corresponding to each time step are then combined in chronological order to obtain the fusion feature sequence.

[0033] Specifically, at each time step, the visible light modal confidence parameters and thermal imaging modal confidence parameters output from the previous step are first acquired, and then concatenated with the corresponding visible light modal feature vectors and thermal imaging modal feature vectors, respectively. This concatenation operation is not merely a formal data concatenation; rather, it organizes modal content information and modal reliability information into a unified input. This allows the subsequent gating mapping layer to consider both the semantic distribution of the feature itself and whether the feature originates from high-quality imaging conditions when calculating weights. For example, in the early monitoring scenario of a subway station fire, the visible light modal feature vector might primarily encode smoke texture, personnel outlines, and equipment edge information, while the corresponding visible light modal confidence parameters reflect the clarity, occlusion level, and signal-to-noise ratio of the current image. The thermal imaging modal feature vector encodes high-temperature patches, thermal boundary expansion, and temperature distribution patterns, while the corresponding thermal imaging modal confidence parameters reflect the integrity of the thermal signal and the stability of the thermal field. After concatenating these two types of information, the resulting gating input vector no longer simply represents what was seen, but rather what was seen and to what extent this content is reliable. This differs from existing technologies that generate attention weights solely based on feature values, because this scheme uses modal quality as an explicit input for weight calculation, rather than an implicit assumption. Subsequently, the concatenated visible light gated input vector and thermal imaging gated input vector are input into the gated mapping layer, respectively, to perform matrix multiplication and activation calculations, yielding the initial gate values ​​for visible light and thermal imaging. The gated mapping layer can be understood as a lightweight parameterized mapping unit that learns the correspondence between "feature content" and "quality state" in the input vector, outputting the initial response intensity to be assigned to the current modality. Matrix multiplication linearly combines the dimensions of the input vector, establishing a trainable coupling between different feature dimensions and the confidence dimension; activation calculation maps the linear response to a nonlinear space, enabling the network to express higher-order combinatorial relationships. For example, in a vehicle fire scenario within a tunnel, when both obvious plume textures and high occlusion levels exist in the visible light modal features, the gating mapping layer can learn the mapping relationship that "although abnormal textures exist, the reliability of the current visible light features decreases due to severe occlusion," thus outputting a relatively low initial visible light gating value. Conversely, if the high-temperature region boundaries in the thermal imaging modal features are clear and the thermal signal integrity is high, the thermal imaging branch can output a higher initial thermal imaging gating value. After obtaining the initial visible light gating value and the initial thermal imaging gating value, normalization is performed on both to obtain the visible light gating weight and the thermal imaging gating weight. The purpose of normalization is to map the initial response values ​​of each modality to a unified, comparable scale, allowing direct comparison of the relative contributions of different modalities at the same time step. Preferably, normalization constraints can be used to keep the sum of the gating weights of the two modalities within a preset range, thereby preventing unconstrained numerical amplification of a certain modality.For example, in the development stage of a warehouse fire, if the initial gating value of the thermal imaging modality is significantly higher than that of the visible light modality, the normalized thermal imaging gating weight will dominate, while the visible light gating weight will be compressed. Conversely, in scenarios without thermal anomalies but with abnormal human behavior, the visible light gating weight can be higher than the thermal imaging gating weight. Furthermore, this embodiment does not assign weights solely based on the gating mapping results, but simultaneously generates corresponding missing markers based on the comparison results between the visible light modality confidence parameters and the thermal imaging modality confidence parameters and the preset missing threshold. After generating the missing markers, the historical mean feature vector corresponding to the missing modality is called to replace the missing modality feature vector, and the gating weight corresponding to the missing modality is corrected to a preset suppression weight, resulting in the corrected modality feature vector and the corrected gating weight. Here, the historical mean feature vector refers to the reference feature obtained by statistically averaging the feature vectors of a specific modality in the corresponding scene category or stage during the training phase or in historical stable samples, used to maintain the structural integrity of the feature space when a modality is missing. It is important to emphasize that this embodiment does not simply set the missing modalities to zero. Directly clearing the features of missing modalities would cause abrupt changes in feature distribution, leading subsequent temporal networks to misinterpret "modal absence" as "event state change." Furthermore, it would result in the fusion vector dimension remaining unchanged, but the numerical distribution deviating from the common region observed during training. This solution uses historical mean feature vectors as placeholders, ensuring that the missing modalities retain the "average background state of this type of scenario" in the numerical space, thereby mitigating the distributional break caused by sudden modal failure. Simultaneously, the gating weights corresponding to the missing modalities are modified to preset suppression weights, significantly reducing the actual contribution of the modalities to the current fusion result, even though they participate in the computation as placeholder features. For example, in a chemical plant leak scenario, if the thermal imaging equipment outputs abnormally high-noise images for a period due to strong heat reflection, the thermal imaging modality is marked as missing. In this case, the system calls the historical mean thermal feature vector to replace the current thermal modal features to maintain temporal continuity and compresses the thermal imaging gating weights to extremely low values, making the fusion result primarily dominated by the visible light modality. Once the thermal imaging modality returns to normal, its reliability parameter rebounds, and the corresponding gating weights increase again, thereby automatically restoring the dual-modal collaborative analysis capability.

[0034] Further, S4 includes the following steps: Based on the fused feature sequence, time segmentation is performed according to a preset sliding time window to obtain multiple continuous time segments; Each of the continuous time segments is input into a temporal convolutional network, and one-dimensional convolution, dilated convolution, and temporal pooling operations are performed to obtain the stage evolution features corresponding to each continuous time segment. The event stage classification calculation is performed based on the evolution characteristics of each stage to obtain the event staging results corresponding to each continuous time segment. The stage transition constraint correction is performed based on the event staging results of adjacent continuous time segments to obtain the target event staging results arranged in chronological order. Based on the phased results of the target event and the corresponding stage evolution characteristics, the risk increment value corresponding to each continuous time segment is calculated, and the risk increment value is cumulatively calculated under constraints according to the chronological order of the event stages to obtain the risk score sequence.

[0035] In some embodiments, the fused feature sequence is first temporally segmented according to a preset sliding time window. This sliding time window is not merely for data slicing, but rather for constructing local sequence units with temporal context, ensuring that each consecutive time segment simultaneously contains the current state and historical trends. Furthermore, the sliding window moves in an overlapping manner, allowing adjacent time segments to share some time steps, thus guaranteeing a smooth transition of subsequent stage determination results on the time axis without abrupt changes due to window segmentation. After obtaining multiple consecutive time segments, each time segment is input into a temporal convolutional network to perform one-dimensional convolution operations, dilated convolution operations, and temporal pooling operations to obtain the corresponding stage evolution features. The one-dimensional convolution operation here is mainly used to extract local change patterns in the time dimension, such as a rapid rise in heat signal, a sudden increase in population density, or a concentrated occurrence of speed fluctuations in a short period of time. Dilated convolution operation expands the receptive field by introducing an interval sampling mechanism, enabling the network to capture long-distance dependencies across time segments without significantly increasing computational load, such as the cross-stage change process of a fire from initial spread to full-scale spread. Temporal pooling operation is used to compress and filter temporal features, making the stage evolution features more stable and representative. After obtaining the stage evolution features corresponding to each consecutive time segment, event stage classification calculation is performed based on these features to obtain the initial event staging results. This classification process does not rely solely on the static features of a single time segment, but is based on the evolution patterns extracted by the temporal convolutional network, thus distinguishing situations such as "short-term abnormal fluctuations" from "continuous deterioration trends," which are difficult to differentiate in a single frame. However, relying solely on the classification results may still lead to unreasonable stage jumps, for example, a time segment may be misclassified as a "mitigation stage," while the preceding and following time segments are still in the "development stage." Therefore, this embodiment further performs stage transition constraint correction based on the event staging results of adjacent consecutive time segments. This constraint is not a simple smoothing process, but rather a correction of jumps that do not conform to the actual evolutionary logic based on preset stage transition rules. For example, event stages usually follow a unidirectional or weakly retreating path of "start-development-deterioration-management". If there is a situation where there is a jump from the "start stage" directly to the "mitigation stage", the constraint rules correct this abnormal jump to a more reasonable intermediate stage. Through this correction mechanism, the target event staging results are obtained in chronological order and satisfy the evolutionary logic. After obtaining the target event staging results, the risk increment value of each consecutive time segment is further calculated in combination with the corresponding stage evolution characteristics. The risk increment value here is used to characterize the change in risk level of the current time segment relative to the previous time segment. Its calculation is not based on a fixed threshold, but on the dynamic change information reflected in the stage evolution characteristics. For example, in a warehouse fire scenario, if the area of ​​the thermal anomaly region expands rapidly and the temperature gradient increases significantly in a certain time segment, the corresponding risk increment value is large; if there are only slight fluctuations, the risk increment value is small.Furthermore, this embodiment does not directly use the risk increment value as the final output. Instead, it performs constrained cumulative calculations on each risk increment value according to the chronological order of the event stages to obtain a risk score sequence. This accumulation process takes into account the evolutionary relationship between stages. For example, in the development and deterioration stages, the risk increment value will be continuously superimposed, thereby forming a rapidly rising risk score curve; while in the handling or mitigation stage, the cumulative growth is slowed down or suppressed through constraint mechanisms, so that the risk score gradually tends to stabilize or decline.

[0036] Furthermore, the temporal convolutional network is trained through the following steps: Obtain the fusion feature sequence corresponding to the historical emergency scene, and determine the start time, stage transition time and handling time of the abnormal event based on the video timeline, event handling record, alarm log and manual annotation results corresponding to each fusion feature sequence; Based on the start time, stage transition time, and disposal time, each fused feature sequence is segmented and sliced ​​according to a preset sliding time window to obtain several historical continuous time segments. Based on the position of each historical continuous time segment in the original event timeline, the corresponding event stage label and risk score label are marked. Based on the event stage labels and risk score labels, historical continuous time segments belonging to the same event stage are classified, and training samples are extracted from the historical continuous time segments corresponding to each event stage according to a preset ratio to construct a time series training set with balanced stage distribution. Each historical continuous time segment in the time-series training set is input into the temporal convolutional network for training. The error is calculated based on the output results and the corresponding event stage labels and risk score labels. The network parameters of the temporal convolutional network are updated in reverse based on the error calculation results until the training loss converges, and the trained temporal convolutional network is obtained.

[0037] Specifically, the process begins by acquiring the fused feature sequences corresponding to historical emergency scenes. These sequences are then combined with video timelines, event handling records, alarm logs, and manually annotated results to align and analyze key time nodes within each historical sequence. The key here is not simply recording the event's occurrence time, but precisely locating the start time, transition points, and intervention times. The start time corresponds to the moment the anomaly first appears; the transition point corresponds to the critical point where the event changes from one development state to another, such as moving from an initial anomaly to a sustained development stage or from a development stage to a rapid deterioration stage; and the intervention time corresponds to the moment when external intervention begins to impact the event. Through the structured extraction of these key time points, the original continuous time sequence is endowed with clear stage boundary information. Based on this, the fused feature sequence is segmented according to a preset sliding time window, based on the start time, transition points, and intervention times, resulting in several historical continuous time segments. It is important to emphasize that this slicing process is not a simple equal-length division, but rather a collaborative process with time-annotation information, ensuring that each time segment possesses clear stage semantics within the original event's evolution. For example, in a chemical leak scenario, a time segment that falls entirely within the sustained diffusion phase is labeled as that phase; if a time segment crosses a phase boundary, the dominant phase label can be determined based on its center time or proportion rule. Simultaneously, based on the time segment's position in the event's evolution, a risk score label is assigned. This risk score is not a discrete category but a continuous or graded value reflecting risk intensity, used to characterize the degree of danger of the event within that time segment. In this way, each historical continuous time segment simultaneously possesses a phase label and a risk label, thus transforming the original time series into structured samples suitable for multi-task learning. Subsequently, based on the event phase label and risk score label, historical continuous time segments belonging to the same event phase are categorized, and training data is extracted from the samples of each phase according to a preset ratio to construct a time series training set with a balanced phase distribution. The key to this step is addressing the common problem of uneven phase distribution in emergency data. In real-world scenarios, certain phases (such as the stabilization or mitigation phases) are long and involve large amounts of data, while critical phases (such as the rapid deterioration phase) are short but crucial for risk assessment. If the original data is used directly for training, the network may become biased towards the majority of phases, resulting in insufficient ability to identify critical phases. This embodiment groups data by phase and performs proportional sampling to maintain a relatively balanced distribution of samples across phases in the training set, ensuring that the temporal convolutional network can fully learn the evolutionary characteristics of different phases during training. For example, in a tunnel fire dataset, sampling ratios can be set for the initial, development, deterioration, and response phases, preventing the model from ignoring key patterns in the deterioration phase due to an excessive number of samples in the initial phase. After constructing the training set, each historical continuous time segment is input into the temporal convolutional network for training.The described temporal convolutional network models feature changes over time using one-dimensional convolutional and dilated convolutional structures. Its input is a fused feature sequence corresponding to consecutive time segments, and its output includes event stage prediction results and risk score prediction results. During training, error calculation is performed based on the network output results and the corresponding event stage labels and risk score labels. The stage labels constrain the network's ability to distinguish stage categories, while the risk score labels constrain the network's ability to regress risk intensity. Finally, the network parameters of the temporal convolutional network are updated in reverse based on the error calculation results, allowing the network to gradually adjust its convolutional kernel weights and parameters in the dilated structure to enhance its responsiveness to key temporal patterns.

[0038] Furthermore, generating video analysis results of the emergency scene based on the risk scoring sequence and event staging results includes the following steps: Based on the risk score value corresponding to each time step in the risk scoring sequence, interval mapping is performed according to the preset risk level division threshold to obtain the risk level result corresponding to each time step. The event staging results and the risk level results corresponding to each time step are time-aligned and correlated to obtain time-series analysis results containing event stage fields and risk level fields; Based on the time period in the time series analysis results, locate the target video frame that triggers the risk change; Based on the time-series analysis results and the target video frame, output the video analysis results.

[0039] In some embodiments, the risk score value corresponding to each time step in the risk scoring sequence is first mapped to a predetermined risk level threshold to obtain the risk level result for each time step. The risk score value here is a continuous risk metric output by the aforementioned temporal convolutional network based on continuous-time modeling. It reflects the degree of danger of an abnormal event at the current time step, but for command personnel, directly using continuous values ​​for judgment is not intuitive enough. Therefore, this embodiment divides the continuous risk score into several risk level intervals by setting a predetermined risk level threshold. For example, time steps with risk score values ​​in the lower interval can be mapped to a low-risk level, the middle interval to a medium-risk level, and the higher interval to a high-risk level. For different application scenarios, the threshold can be set based on historical statistical results or expert experience. Taking an underground parking garage fire scenario as an example, in the initial stage of a fire, although there is local smoke and a slight temperature rise, the risk score value is still in a low range, so the system outputs a low-risk level; when flames form and the temperature rises rapidly, the risk score value crosses the next higher threshold, and the system outputs a medium-to-high-risk level. Through this interval mapping process, the continuous risk score is transformed into a risk level result with clear semantic boundaries. After obtaining the risk level results corresponding to each time step, the event staging results and the risk level results corresponding to each time step are time-aligned and associated, resulting in a time-series analysis result containing an event stage field and a risk level field. Time alignment is necessary because although the risk scoring sequence and event staging results both originate from the same fused feature sequence, their temporal resolution may differ due to variations in sliding window length, step size, or classification output granularity. This embodiment uses a unified time index to map the risk level result at each time step to the event stage result at the same time location, thereby establishing a joint description of "current risk level" and "current stage" at the same moment. The association and combination process is not simply a mechanical concatenation of two fields, but rather forms a time-series analysis unit for subsequent retrieval and output. Each unit includes at least a time identifier, an event stage field, and a risk level field. In some implementations, the original risk score value can also be included to support more refined sorting and positioning. Subsequently, the target video frame triggering the risk change is located based on the time segment where the risk level increases in the time-series analysis result. The purpose of this step is to identify the key frames from a continuous time series that best represent the causes of escalating risk and turning points in events, in order to assist emergency personnel in quickly tracing back the source of abnormal changes. Existing technologies typically only transmit the current frame or frames at fixed time intervals after an alarm is detected, making it difficult to guarantee that the returned images actually correspond to the key nodes of risk changes.This embodiment does not simply extract an image from a single point in time. Instead, it first identifies time segments in the time-series analysis results where the risk level changes from low to high. Then, it maps these time segments back to the original video timeline to locate the target video frame that best represents the escalating risk. Finally, based on the time-series analysis results and the target video frame, it outputs video analysis results. These results include at least an event stage field, a risk level field, and the corresponding target video frame. In practical applications, an abnormal event type field can also be included, thus forming a complete analysis result that includes "when, what happened, what stage, what the risk level is, and what the key footage is." For example, in a large shopping mall fire scenario, the output could be: the current abnormal event type is fire, the event stage is a continuing development stage, the risk level is high risk, and the corresponding target video frame is the footage of open flame spreading and people being evacuated at a specific point in time. In this way, on-duty personnel no longer need to replay long video segments one by one; instead, they can directly make rapid judgments and decisions based on the key frames and structured fields output by the system.

[0040] Furthermore, based on the time period in the time series analysis results where the risk level increases, the target video frames that trigger the risk change include: By comparing the risk level results corresponding to adjacent time steps in the time series analysis results, the time segment in which the risk level changes from low to high is determined, and the risk change segment is obtained. Based on the temporal position of the risk change segment in the fused feature sequence, the target time window is obtained by reverse indexing the continuous time segments corresponding to the risk change segment. Based on the start and end times corresponding to the target time window, video frames corresponding to the target time window are extracted from the visible light video frame sequence and the thermal imaging video frame sequence to obtain a candidate target video frame set. The modal credibility parameters, event stage results, and risk score change magnitudes corresponding to the candidate target video frame set are comprehensively sorted, and the video frame with the highest score is selected as the target video frame that triggers the risk change.

[0041] In some embodiments, the risk level results corresponding to adjacent time steps in the time series analysis are first compared to determine the time segment in which the risk level changes from low to high, thus obtaining the risk change segment. This comparison operation does not simply determine whether the risk level at a certain time step is higher than a threshold, but rather examines the relationship between the level changes between adjacent time steps sequentially along the time axis. For example, when the risk level at a certain time step is low, and the next time step increases it to medium risk, or further increases it to high risk, the system marks the corresponding time span as the risk change segment. If the risk level gradually increases over multiple consecutive time steps, this continuous increase process can be considered as an extended risk change segment. The significance of this approach is that risk improvement is usually not caused by a single frame abrupt change, but rather by the combined effect of feature evolution over multiple consecutive time steps. Taking an underground parking garage fire scenario as an example, the initial time step may only show localized thermal anomalies, corresponding to low risk; subsequently, the smoke concentration increases, the thermal anomaly area expands, and the risk level increases to medium risk; further on, the flame boundary clearly appears, and the risk level enters high risk. At this point, the system does not only record the single moment when the risk first increases, but identifies the time interval from low risk to medium risk, and then to high risk as the risk change segment. After obtaining the risk change segment, based on the time position of the risk change segment in the fused feature sequence, the system uses a reverse index to find the corresponding continuous time segment, thus obtaining the target time window. Here, the reverse index is the original time segment used to generate the result, traced back from the time series analysis result. Since the risk scoring sequence and event staging result in S4 are both obtained by segmenting the fused feature sequence under a sliding time window, each risk level change actually corresponds to one or a group of continuous time segments. In this embodiment, the system determines the corresponding continuous time segment in the fused feature sequence based on the start and end positions of the risk change segment on the time axis, and accordingly recovers the target time window when the risk transition occurs. This time window is not necessarily equal to the length of a single time step, but preferably covers a local interval before and after the risk change, thereby preserving the background state before the change, the key state during the change, and the stable state after the change. For example, in a chemical leak scenario, if a certain risk change segment corresponds to a rapid temperature increase and an expansion of the plume in visible light, the target time window preferably covers several frames before and after this temperature increase, so that the most representative images can be found from the complete local context when extracting subsequent candidate frames. After determining the target time window, based on the start and end times corresponding to the target time window, video frames corresponding to the target time window are extracted from the visible light video frame sequence and the thermal imaging video frame sequence to obtain a set of candidate target video frames.The extraction process here does not involve extracting only a single video frame, but rather simultaneously searching for frames at the corresponding time points in both the visible light video frame sequence and the thermal imaging video frame sequence to form a candidate frame set consistent with the target time window. Since the correspondence between visible light frames and thermal imaging frames under a unified timestamp has already been established in S1, the dual-modal frames can be extracted synchronously in this step according to the start and end times. The candidate target video frame set preferably includes dual-modal video frames at multiple time points, rather than a single frame, thus providing options for subsequent comprehensive sorting. For example, in a warehouse fire scenario, within a target time window lasting approximately 4 seconds, the candidate target video frame set might include thermal imaging frames where the flame has not yet clearly appeared but the high-temperature zone is rapidly expanding, visible light frames where smoke is significantly denser, and synchronous dual-modal frames showing the first appearance of open flame. By retaining these candidate frames, the system can select the image that best represents the triggering cause of the risk change in the subsequent sorting stage, rather than mechanically selecting the center frame or the starting frame of the window. This enhances the stability and accuracy of key frame selection. After obtaining the candidate target video frame set, the modal reliability parameters, event stage results, and risk score change magnitude corresponding to the candidate target video frame set are comprehensively sorted, and the video frame with the highest score is selected as the target video frame that triggers the risk change. This comprehensive sorting process is the key algorithmic step that distinguishes this embodiment from the prior art. In existing solutions, key frame selection is often based on a single indicator, such as the highest anomaly probability, the most obvious brightness change, or the highest temperature. However, in emergency situations, truly representative key frames usually need to meet multiple conditions simultaneously: First, the modal input corresponding to the frame should be reliable; otherwise, even if the risk score is high, the image may lack reference value due to blurring, occlusion, or thermal noise. Second, the frame should be near the event stage transition or stage escalation; otherwise, although there is an anomaly, it may not be the trigger node for risk escalation. Third, the frame should correspond to a large risk score change magnitude, that is, the risk change in the time steps before and after it is most obvious at that position. Based on this, this embodiment constructs a comprehensive score for each frame in the candidate target video frame set. Specifically, the visible light modal reliability parameter and the thermal imaging modal reliability parameter corresponding to the current frame can be weighted and combined to characterize the overall imaging reliability of the frame. The event stage result corresponding to the frame is mapped to a stage weight, for example, frames near the transition from a continuous development stage to a rapid deterioration stage are given higher weights. Then, the difference or slope of the risk score before and after the position of the frame is calculated to characterize the magnitude of the risk score change. When a frame simultaneously meets the conditions of high modal reliability, being near a critical stage transition, and having a large increase in risk score, its comprehensive ranking score is the highest, and it is therefore selected as the target video frame.

[0042] An emergency scene video analysis system based on convolutional neural networks includes: The data acquisition module is used to acquire visible light video data and corresponding thermal imaging data from the emergency site, and align them according to a unified timestamp to obtain multi-source time-series input data. The quality assessment module is used to extract features from the multi-source temporal input data based on a convolutional neural network, and to assess the clarity, occlusion, signal-to-noise ratio, frame drop status, or thermal signal integrity of each source data, so as to obtain the corresponding modal feature vector and modal confidence parameters. The gated fusion module is used to perform gated weighted fusion processing on each modality feature vector according to the modality confidence parameter, and when there is a missing modality, replace the missing modality feature with a preset placeholder feature, while suppressing the fusion weight corresponding to the missing modality to obtain a fusion feature sequence. The temporal evolution analysis module is used to input the fused feature sequence into a temporal convolutional network, extract the stage evolution features of the emergency event within a continuous time window, output the corresponding event staging results, and perform constrained cumulative calculation of the risk increment based on the event staging results to obtain a risk score sequence that changes continuously over time. The analysis result generation module is used to generate video analysis results of the emergency scene based on the risk scoring sequence and event staging results. The video analysis results include the abnormal event type, risk level, and event stage.

[0043] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0044] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0045] The above embodiments are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. An emergency scene video analysis method based on convolutional neural networks, characterized in that, Includes the following steps: S1. Acquire visible light video data and corresponding thermal imaging data from the emergency site, align them according to a unified timestamp, and obtain multi-source time-series input data; S2. Based on the convolutional neural network, feature extraction is performed on the multi-source temporal input data respectively, and the quality mode set is evaluated to obtain the corresponding mode feature vector and mode confidence parameter. The quality mode set includes sharpness, occlusion degree, signal-to-noise ratio, frame drop status and thermal signal integrity. S3. Based on the modality confidence parameter, perform gated weighted fusion processing on each modality feature vector, and when there is a missing modality, replace the missing modality feature with a preset placeholder feature, while suppressing the fusion weight corresponding to the missing modality, to obtain a fusion feature sequence. S4. Input the fused feature sequence into a temporal convolutional network to extract the stage evolution features of the emergency event within a continuous time window, output the corresponding event staging results, and perform constrained cumulative calculation on the risk increment based on the event staging results to obtain a risk score sequence that changes continuously over time. S5. Based on the risk scoring sequence and event staging results, generate video analysis results of the emergency scene, including the type of abnormal event, risk level, and stage of the event.

2. The emergency scene video analysis method based on convolutional neural networks according to claim 1, characterized in that, S1 includes the following steps: Acquire the visible light video stream output by the visible light acquisition device at the emergency site, and acquire the thermal imaging video stream output by the thermal imaging acquisition device corresponding to the visible light video stream; A collection timestamp is added to the visible light video stream and the thermal imaging video stream respectively, and video frame extraction processing is performed on the visible light video stream and the thermal imaging video stream according to a preset time window to obtain the visible light video frame sequence and the thermal imaging video frame sequence. Based on the acquisition timestamp of each visible light video frame in the visible light video frame sequence, the thermal imaging video frame sequence is subjected to same timestamp matching processing to obtain candidate thermal imaging video frames corresponding to each visible light video frame. For candidate thermal imaging video frames that are inconsistent with the acquisition timestamp of the visible light video frames, perform neighboring frame selection processing or inter-frame interpolation processing to obtain the target thermal imaging video frames corresponding to each visible light video frame. Each visible light video frame is combined with the corresponding target thermal imaging video frame in chronological order to obtain multi-source time-series input data.

3. The emergency scene video analysis method based on convolutional neural networks according to claim 2, characterized in that, The convolutional neural network is trained through the following steps: Historical multi-source time-series samples were constructed by using visible light video streams and thermal imaging video streams from historical emergency sites. Each historical multi-source time-series sample was labeled with the type of abnormal event, the stage of the event, and the corresponding labels for clarity, degree of occlusion, signal-to-noise ratio, frame drop status, and thermal signal integrity to obtain a training sample set. Visible light video frames and thermal imaging video frames from the training sample set are input into a convolutional neural network for convolution, activation, and downsampling operations to obtain visible light intermediate feature maps and thermal imaging intermediate feature maps. Global pooling and vector mapping are then performed on the visible light intermediate feature maps and thermal imaging intermediate feature maps to obtain the predicted modality feature vector. Based on the visible light intermediate feature map and the thermal imaging intermediate feature map, the quality assessment branch output is extracted respectively, and the quality assessment branch output is compared with the corresponding sharpness label, occlusion label, signal-to-noise ratio label, frame drop status label and thermal signal integrity label to calculate the error and obtain the quality assessment loss. Based on the predicted modality feature vector, event type prediction and event stage prediction are performed, and the prediction results are compared with the corresponding abnormal event type label and event stage label to calculate the error and obtain the feature extraction loss. A joint loss function is constructed based on the quality assessment loss and feature extraction loss. The network parameters of the visible light convolutional neural network and the thermal imaging convolutional neural network are then updated in reverse based on the joint loss function until the joint loss function converges, resulting in a trained convolutional neural network.

4. The emergency scene video analysis method based on convolutional neural networks according to claim 1, characterized in that, S3 includes the following steps: Based on the visible light modal confidence parameters and thermal imaging modal confidence parameters, vector concatenation is performed with the corresponding visible light modal feature vector and thermal imaging modal feature vector, respectively, and then input into the gating mapping layer to perform matrix multiplication and activation calculation to obtain the initial gating value for visible light and the initial gating value for thermal imaging. Normalization is performed on the initial visible light gate value and the initial thermal imaging gate value to obtain the visible light gate weight and the thermal imaging gate weight. Based on the comparison results between the visible light modal confidence parameter and the thermal imaging modal confidence parameter and the preset missing value judgment threshold, the corresponding missing value marker is generated. Based on the missing marker, the historical mean feature vector corresponding to the missing mode is called to replace the missing mode feature vector, and the gating weight corresponding to the missing mode is corrected to the preset suppression weight, so as to obtain the corrected mode feature vector and the corrected gating weight. Based on the modified modal feature vectors and the modified gating weights, element-wise product calculation and cross-modal summation are performed respectively to obtain the fusion feature vectors corresponding to each time step. The fusion feature vectors corresponding to each time step are then combined in chronological order to obtain the fusion feature sequence.

5. The emergency scene video analysis method based on convolutional neural networks according to claim 1, characterized in that, S4 includes the following steps: Based on the fused feature sequence, time segmentation is performed according to a preset sliding time window to obtain multiple continuous time segments; Each of the continuous time segments is input into a temporal convolutional network, and one-dimensional convolution, dilated convolution, and temporal pooling operations are performed to obtain the stage evolution features corresponding to each continuous time segment. The event stage classification calculation is performed based on the evolution characteristics of each stage to obtain the event staging results corresponding to each continuous time segment. The stage transition constraint correction is performed based on the event staging results of adjacent continuous time segments to obtain the target event staging results arranged in chronological order. Based on the phased results of the target event and the corresponding stage evolution characteristics, the risk increment value corresponding to each continuous time segment is calculated, and the risk increment value is cumulatively calculated under constraints according to the chronological order of the event stages to obtain the risk score sequence.

6. The emergency scene video analysis method based on convolutional neural networks according to claim 5, characterized in that, The temporal convolutional network is trained through the following steps: Obtain the fusion feature sequence corresponding to the historical emergency scene, and determine the start time, stage transition time and handling time of the abnormal event based on the video timeline, event handling record, alarm log and manual annotation results corresponding to each fusion feature sequence; Based on the start time, stage transition time, and disposal time, each fused feature sequence is segmented and sliced ​​according to a preset sliding time window to obtain several historical continuous time segments. Based on the position of each historical continuous time segment in the original event timeline, the corresponding event stage label and risk score label are marked. Based on the event stage labels and risk score labels, historical continuous time segments belonging to the same event stage are classified, and training samples are extracted from the historical continuous time segments corresponding to each event stage according to a preset ratio to construct a time series training set with balanced stage distribution. Each historical continuous time segment in the time-series training set is input into the temporal convolutional network for training. The error is calculated based on the output results and the corresponding event stage labels and risk score labels. The network parameters of the temporal convolutional network are updated in reverse based on the error calculation results until the training loss converges, and the trained temporal convolutional network is obtained.

7. The emergency scene video analysis method based on convolutional neural networks according to claim 1, characterized in that, The process of generating video analysis results of the emergency scene based on the risk scoring sequence and event staging results includes the following steps: Based on the risk score value corresponding to each time step in the risk scoring sequence, interval mapping is performed according to the preset risk level division threshold to obtain the risk level result corresponding to each time step. The event staging results and the risk level results corresponding to each time step are time-aligned and correlated to obtain time-series analysis results containing event stage fields and risk level fields; Based on the time period in the time series analysis results, locate the target video frame that triggers the risk change; Based on the time-series analysis results and the target video frame, output the video analysis results.

8. An emergency scene video analysis system based on convolutional neural networks, applied to the emergency scene video analysis method based on convolutional neural networks as described in any one of claims 1-7, characterized in that, include: The data acquisition module is used to acquire visible light video data and corresponding thermal imaging data from the emergency site, and align them according to a unified timestamp to obtain multi-source time-series input data. The quality assessment module is used to extract features from the multi-source temporal input data based on a convolutional neural network, and to assess the clarity, occlusion, signal-to-noise ratio, frame drop status, or thermal signal integrity of each source data, so as to obtain the corresponding modal feature vector and modal confidence parameters. The gated fusion module is used to perform gated weighted fusion processing on each modality feature vector according to the modality confidence parameter, and when there is a missing modality, replace the missing modality feature with a preset placeholder feature, while suppressing the fusion weight corresponding to the missing modality to obtain a fusion feature sequence. The temporal evolution analysis module is used to input the fused feature sequence into a temporal convolutional network, extract the stage evolution features of the emergency event within a continuous time window, output the corresponding event staging results, and perform constrained cumulative calculation of the risk increment based on the event staging results to obtain a risk score sequence that changes continuously over time. The analysis result generation module is used to generate video analysis results of the emergency scene based on the risk scoring sequence and event staging results. The video analysis results include the abnormal event type, risk level, and event stage.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images