Fireworks safety dynamic early warning method and system based on multi-modal fusion

Abstract

The application provides a firework safety dynamic early warning method and system based on multi-modal fusion, relates to the field of firework early warning technology, and breaks through the limitations of static weight and logical fragmentation in traditional multi-modal fusion methods, and innovatively introduces a double-path dynamic weight generation and selection mechanism based on safety physical rules and time sequence consistency.The mechanism can intelligently identify the reliability of different modal data according to the real-time working conditions of the launching site, and realize adaptive balance between physical rationality and time sequence stability, so as to realize high-precision, low-delay and strong-robust dynamic early warning of composite safety risks in a complex and highly-interfered outdoor firework scene, and significantly improve the safety guarantee level of large-scale activities.

Description

Technical Field

[0001] This invention relates to the field of fireworks display early warning technology, and in particular to a dynamic early warning method and system for fireworks display safety based on multimodal fusion. Background Technology

[0002] In outdoor fireworks displays such as large-scale cultural performances and celebrations, safety risks are characterized by their suddenness, rapid spread, and wide impact. Accidents can easily result in casualties and property damage. Early safety measures relied primarily on manual inspections and experience-based judgment, which suffered from slow response times, numerous blind spots, and strong subjectivity. With the development of sensing and automation technologies, the industry has gradually introduced single-modal monitoring methods such as video surveillance, infrared thermal imaging, and sound pressure sensing to attempt to achieve automatic early warning during the fireworks display process.

[0003] However, single-modal methods exhibit significant limitations in complex outdoor environments: visible light video-based trajectory recognition suffers a sharp drop in image signal-to-noise ratio at night, under strong light reflection, in smoky conditions, or in rain or snow, making it difficult to stably extract pyrotechnic trajectories; while infrared thermal imaging can detect high-temperature areas, it is prone to temperature saturation or false alarms during high-density continuous pyrotechnic displays or in the presence of background heat sources (such as stage lighting or vehicle engines), failing to effectively distinguish between normal residual heat and abnormal ignition hotspots; although sound pressure sensors have some sensitivity to pyrotechnic events, ambient noise (such as crowd noise, sound systems, and wind noise) often masks or falsely triggers non-standard pyrotechnic signals, reducing the reliability of the judgment. Furthermore, traditional systems generally do not incorporate meteorological parameters (such as wind speed, wind direction, and humidity) and real-time personnel location information into the risk assessment system, resulting in a severe disconnect between the early warning logic and on-site dynamics;

[0004] To overcome the limitations of single-modality data, some recent studies have attempted to simply overlay or fuse data from multiple sensors, such as combining video detection results with infrared alarm signals using an AND logic, or integrating acoustic and thermal imaging outputs through threshold weighting. However, these methods typically employ fixed weights or static decision rules, failing to consider the reliability variations of different modalities under specific operating conditions. For instance, in windy weather, the risk of falling sparks increases significantly, and personnel location and meteorological data should be given higher weights; conversely, in low-visibility conditions, the reliability of visible light video decreases, and its contribution should be dynamically suppressed. Existing fusion strategies lack dynamic evaluation mechanisms for modal quality and have not established explicit correlations with the physical laws of fireworks displays (such as minimum safe distance, ground ignition temperature threshold, and upper limit of sound pressure energy), leading to deviations between the fusion results and the actual safety situation. Furthermore, most systems neglect the consistency constraints of multimodal events over time, failing to effectively identify isolated false positive signals caused by transient sensor interference or local anomalies, further exacerbating false alarms or missed alarms.

[0005] Therefore, there is an urgent need for technical solutions for dynamic early warning methods and systems for fireworks display safety based on multimodal fusion. Summary of the Invention

[0006] To address the aforementioned technical problems, embodiments of the present invention provide a dynamic early warning method for fireworks display safety based on multimodal fusion, specifically including the following steps: S1. Synchronously acquire visible light video streams, infrared thermal image sequences, sound pressure timing signals, real-time meteorological parameters, and personnel location coordinates of the fireworks display area; S2. Perform inter-frame difference processing on the visible light video stream to extract the trajectory point set of the fireworks; perform temperature gradient clustering on the infrared thermal image sequence to generate a high-temperature anomaly region mask; perform short-time Fourier transform on the sound pressure time sequence signal to identify the energy peak of the non-standard combustion and explosion frequency band. S3. Based on the modal data, which includes: fireworks trajectory point set, high temperature anomaly area mask, non-standard combustion and explosion frequency band energy peak, real-time meteorological parameters and personnel location coordinates, the modal data are weighted and fused according to the physical constraints of fireworks display safety and the consistency relationship between the time sequence of multimodal events to generate a fused feature vector of the current fireworks display area. S31. Based on the firework trajectory point set, high temperature anomaly area mask, non-standard combustion and explosion frequency band energy peak, real-time meteorological parameters and personnel location coordinates, three safety risk quantitative indicators are calculated. The three quantitative indicators of safety risks include: personnel intrusion risk coefficient, spark ignition risk coefficient, and abnormal combustion and explosion confidence coefficient. S32. Perform the first verification on the three safety risk quantification indicators based on the physical constraints of fireworks display safety, and determine the first fusion weight based on the first verification results; S321. Obtain the safety physical constraint boundary values ​​associated with each safety risk quantification index, including the minimum safe distance boundary value associated with the personnel intrusion risk coefficient, the maximum permissible surface temperature boundary value associated with the spark fall ignition risk coefficient, and the maximum sound pressure energy boundary value associated with the abnormal combustion and explosion confidence coefficient. S322. Perform physical constraint verification: Determine whether the personnel intrusion risk coefficient is greater than the minimum safe distance boundary value. If so, mark the personnel intrusion risk coefficient as a physical boundary violation state; otherwise, mark it as a physical compliance state. Determine whether the spark fall ignition risk coefficient is greater than the maximum allowable surface temperature boundary value. If so, mark the spark fall ignition risk coefficient as a physical boundary violation state; otherwise, mark it as a physical compliance state. Determine whether the abnormal combustion and explosion confidence coefficient is greater than the maximum sound pressure energy boundary value. If so, mark the abnormal combustion and explosion confidence coefficient as a physical boundary violation state; otherwise, mark it as a physical compliance state. S323. Based on the marking results of physical boundary violation and physical compliance status, calculate the boundary violation degree for each security risk quantification indicator. calculate: If any security risk quantification indicator is marked as physically out of bounds, the ratio of the current security risk quantification indicator to the associated security physical constraint boundary value is calculated as the degree of out of bounds. If any security risk quantification indicator is marked as a physical compliance status, then the ratio of the current security risk quantification indicator to the associated security physical constraint boundary value is calculated, and the current ratio is restricted to the interval [0, 1] by a truncation function as the out-of-bounds degree; S324. Apply a monotonically increasing nonlinear mapping function to the out-of-bounds degree of each security risk quantification indicator to generate the first physical confidence factor, the second physical confidence factor, and the third physical confidence factor. S325. Normalize the first physical confidence factor, the second physical confidence factor, and the third physical confidence factor to obtain each normalized physical confidence factor. Combine each normalized physical confidence factor and use the combination result as the first fusion weight. S33. Perform a second verification on the three security risk quantification indicators based on the consistency relationship of multimodal event time series, and determine the second fusion weight based on the second verification results; S331. Construct a time sliding window of length M, where M is a positive integer greater than or equal to 3, and extract the personnel intrusion risk coefficient sequence, the Martian fall ignition risk coefficient sequence, and the abnormal combustion and explosion confidence coefficient sequence recorded in the most recent M sampling periods from the historical data cache. S332, Perform timing consistency verification: Based on the historical personnel intrusion risk coefficient sequence, the maximum value of the absolute difference between adjacent sampling periods is calculated as the upper limit threshold for personnel movement mutation; the personnel intrusion risk coefficient of the current sampling period is subtracted from the personnel intrusion risk coefficient of the previous sampling period and the absolute value is taken to obtain the current personnel risk change; if the current personnel risk change is greater than the upper limit threshold for personnel movement mutation, the personnel intrusion risk coefficient of the current sampling period is marked as a time-series abnormal state, otherwise it is marked as a time-series stable state. Based on the historical Mars fall ignition risk coefficient sequence, the standard deviation of the current sequence is calculated as the thermal radiation fluctuation threshold; the difference between the Mars fall ignition risk coefficient of the current sampling period and the mean of the current sequence is taken as the absolute value to obtain the current Mars risk deviation; if the current Mars risk deviation is greater than the thermal radiation fluctuation threshold, the Mars fall ignition risk coefficient of the current sampling period is marked as a time-series abnormal state, otherwise it is marked as a time-series stable state. Based on the historical abnormal combustion and explosion confidence coefficient sequence, the difference between the maximum value and the mean value of the current sequence is calculated as the upper limit threshold for combustion and explosion confidence anomalies; the difference between the abnormal combustion and explosion confidence coefficient of the current sampling period and the mean value of the current sequence is used to obtain the current combustion and explosion confidence deviation; if the current combustion and explosion confidence deviation is greater than the upper limit threshold for combustion and explosion confidence anomalies, the abnormal combustion and explosion confidence coefficient of the current sampling period is marked as a time-series abnormal state; otherwise, it is marked as a time-series stable state. S333. Based on the labeling results of time-series abnormal states and time-series stable states, perform time-series inconsistency measurement calculations for each security risk quantification indicator: If any security risk quantification indicator is marked as a time-series anomalous state, the absolute deviation between the current security risk quantification indicator and the mean of the historical security risk quantification indicator series is calculated as a time-series inconsistency measure. If any security risk quantification indicator is marked as a time-stable state, then the time-series inconsistency measure of the current security risk quantification indicator is set to zero. S334. Take the negative value of the time series inconsistency measure for each security risk quantification indicator and input it into the exponential function to generate the first time series confidence factor, the second time series confidence factor, and the third time series confidence factor. S335. Normalize the first time series confidence factor, the second time series confidence factor, and the third time series confidence factor to obtain each normalized time series confidence factor. Combine each normalized time series confidence factor and use the combination result as the second fusion weight. S34. Based on three quantitative indicators of safety risks, a comprehensive indicator is calculated; S341. Determine whether at least one of the three quantitative safety risk indicators is marked as a physical out-of-bounds state or a temporal anomaly state. S342. If at least one of the three security risk quantification indicators is marked as a physical out-of-bounds state or a timing abnormal state, the maximum value among the three security risk quantification indicators shall be determined as the comprehensive indicator. S343. If none of the three safety risk quantitative indicators are marked as physical out-of-bounds state and none of them are marked as temporal abnormal state, then the arithmetic mean of the three safety risk quantitative indicators shall be determined as the comprehensive indicator. S35. Based on the comprehensive index, select the target fusion weight from the first fusion weight and the second fusion weight, and perform weighted fusion of the fireworks trajectory point set, high temperature anomaly area mask, non-standard combustion and explosion frequency band energy peak, real-time meteorological parameters and personnel position coordinates based on the target fusion weight to generate the fusion feature vector of the current fireworks display area. S351. Set the preset threshold for the comprehensive index; S352. Compare the comprehensive index with the preset threshold: if the comprehensive index is greater than the preset threshold, select the first fusion weight as the target fusion weight; if the comprehensive index is less than or equal to the preset threshold, select the second fusion weight as the target fusion weight. S353. Convert the firework trajectory point set, high temperature anomaly area mask, non-standard combustion and explosion frequency band energy peak, real-time meteorological parameters and personnel position coordinates into feature vectors of a unified dimension to obtain the first mode feature vector, the second mode feature vector, the third mode feature vector, the fourth mode feature vector and the fifth mode feature vector; S354. Based on the target fusion weights, perform element-wise weighting on the first modality feature vector, the second modality feature vector, the third modality feature vector, the fourth modality feature vector, and the fifth modality feature vector, respectively; S355. Sum the weighted modal feature vectors element by element to generate the fusion feature vector of the current fireworks display area. S4. Input the fused feature vector of the current fireworks display area into the pre-trained risk level discrimination model to generate a safety risk level identifier for the current fireworks display area, and trigger an early warning command based on the safety risk level identifier of the current fireworks display area.

[0007] This embodiment also discloses a dynamic early warning system for fireworks display safety based on multimodal fusion, including the following modules: Data acquisition module: used to synchronously acquire visible light video streams, infrared thermal image sequences, sound pressure timing signals, real-time meteorological parameters, and personnel location coordinates of the fireworks display area; Data processing module: Connected to the data acquisition module, it is used to perform inter-frame difference processing on visible light video streams to extract the trajectory point set of fireworks; perform temperature gradient clustering on infrared thermal image sequences to generate high temperature anomaly region masks; and perform short-time Fourier transform on sound pressure time-series signals to identify energy peaks in non-standard combustion and explosion frequency bands. Feature vector acquisition module: connected to the data processing module and the data acquisition module, used to perform weighted fusion of each modal data based on the following: fireworks trajectory point set, high temperature anomaly area mask, non-standard combustion and explosion frequency band energy peak, real-time meteorological parameters and personnel location coordinates, according to the physical constraints of fireworks display safety and the consistency relationship of multimodal event time sequence, to generate a fused feature vector of the current fireworks display area; Warning command triggering module: connected to the feature vector acquisition module, used to input the fused feature vector of the current fireworks display area into the pre-trained risk level discrimination model, generate the safety risk level identifier of the current fireworks display area, and trigger warning commands based on the safety risk level identifier of the current fireworks display area.

[0008] The embodiments of the present invention have the following technical effects: This invention significantly improves the accuracy of risk warning and environmental adaptability in fireworks display scenarios by constructing a multimodal dynamic fusion mechanism guided by both safety physical rules and temporal consistency. Unlike existing technologies that employ fixed weights or simple threshold stacking fusion strategies, this invention first extracts risk features with clear safety semantics from multi-source data such as visible light, infrared, sound pressure, meteorology, and personnel positioning. Based on these features, three core risk indicators are calculated: personnel intrusion risk coefficient, spark fall ignition risk coefficient, and abnormal combustion and explosion confidence coefficient. Then, two sets of fusion weights are generated in parallel: the first set of weights is driven by safety physical constraints (such as minimum safe distance, surface ignition temperature threshold, and maximum permissible sound pressure energy), quantifying the credibility of the current modality at the level of physical rationality by judging whether each risk indicator exceeds its limits and the degree of deviation; the second set of weights is driven by temporal consistency logic, using a sliding time window to analyze the historical trends of each modality's risk indicators, identifying abrupt changes, fluctuations, or isolated anomalies, thereby assessing their stability in the time dimension. Finally, based on the comprehensive risk level, a better set of fusion weights is dynamically selected to weight and fuse the multimodal features, achieving adaptive decision switching between physical credibility priority and temporal stability priority, effectively avoiding false alarms or missed alarms caused by the failure of a single criterion. In summary, this invention overcomes the limitations of static weights and logical fragmentation in traditional multimodal fusion methods. It innovatively introduces a dual-path dynamic weight generation and selection mechanism based on safety physical rules and temporal consistency. This mechanism can intelligently determine the reliability of different modal data according to the real-time operating conditions at the fireworks display site and achieve an adaptive balance between physical rationality and temporal stability. Thus, in complex and highly interference-prone outdoor fireworks scenarios, it can achieve high-precision, low-latency, and robust dynamic early warning of complex safety risks, significantly improving the safety assurance level of large-scale events. Attached Figure Description

[0009] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0010] Figure 1 This is a flowchart of the dynamic early warning method for fireworks display safety based on multimodal fusion provided in this embodiment of the invention; Figure 2 This is a framework diagram of a dynamic early warning system for fireworks display safety based on multimodal fusion provided in an embodiment of the present invention. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0012] Example 1: As Figure 1 As shown, the present invention provides a dynamic early warning method for fireworks display safety based on multimodal fusion, comprising the following steps: S1. Synchronously acquire visible light video streams, infrared thermal image sequences, sound pressure timing signals, real-time meteorological parameters, and personnel location coordinates of the fireworks display area; It is worth noting that in this embodiment, firstly, high-frame-rate visible light cameras and infrared thermal imagers are deployed around the firing area. The fields of view of the two cameras overlap and their spatial calibration is consistent, ensuring that the same physical area can be matched in both modes. At the same time, multiple high-sensitivity sound pressure sensor arrays are uniformly deployed within the safety boundary to capture the transient acoustic features generated by the explosion event. All visual and acoustic devices are synchronized at the nanosecond level via PTP (Precise Time Protocol) to ensure that the visible light video stream, infrared thermal image sequence, and sound pressure timing signal are strictly aligned on the time axis. Secondly, an integrated weather station is installed near the firing point to collect parameters such as wind speed, wind direction, ambient temperature, and relative humidity in real time. Finally, for on-site staff and spectators, a high-precision positioning system based on UWB (Ultra-Wideband) or RTK-GNSS is used to obtain their two-dimensional or three-dimensional position coordinates in real time, and these coordinates are aligned with the aforementioned sensor data through timestamps.

[0013] S2. Perform inter-frame difference processing on the visible light video stream to extract the trajectory point set of the fireworks; perform temperature gradient clustering on the infrared thermal image sequence to generate a high-temperature anomaly region mask; perform short-time Fourier transform on the sound pressure time sequence signal to identify the energy peak of the non-standard combustion and explosion frequency band. It is worth noting that in this embodiment, firstly, for the synchronously acquired visible light video stream, a three-frame difference method is used, that is, the current frame is differiating from the previous frame and the next frame at the pixel level to suppress static background interference. Then, by setting a joint threshold for brightness and motion amplitude, foreground pixels with significant dynamic characteristics are selected. Subsequently, morphological closing operations and connected component analysis are performed on the difference results to remove noise spots, and the center points of spatially adjacent foreground regions in consecutive frames are connected in chronological order to form a set of firework trajectory points. Secondly, for each frame in the infrared thermal image sequence, non-uniformity correction and ambient temperature adjustment are first performed. Temperature compensation is performed, and then the temperature gradient magnitude of each pixel and its local neighborhood is calculated. Based on the gradient distribution histogram, an adaptive K-means clustering algorithm is used to divide the image into high-gradient boundary regions and low-gradient homogeneous regions. Connected regions with temperatures significantly higher than the environmental threshold (e.g., ≥80℃) and located in low-gradient homogeneous regions are marked as potential high-temperature sources. Finally, a high-temperature anomaly region mask is generated by binarization output. Finally, a short-time Fourier transform (STFT) is performed on the sound pressure time-series signal with a window length of 512 points and an overlap rate of 50% to obtain the time spectrum. Based on this, the focus is on the frequency range (e.g., 0.5–2 kHz or 5–10 kHz) outside the preset standard combustion and explosion frequency band (e.g., 2–5 kHz), and the energy integral of each frequency band within the time window is calculated. The energy peak of the non-standard combustion and explosion frequency band with energy significantly higher than the background noise baseline is identified by the sliding window peak detection algorithm as the acoustic criterion for abnormal combustion and explosion events.

[0014] S3. Based on the modal data, which includes: fireworks trajectory point set, high temperature anomaly area mask, non-standard combustion and explosion frequency band energy peak, real-time meteorological parameters and personnel location coordinates, the modal data are weighted and fused according to the physical constraints of fireworks display safety and the consistency relationship between the time sequence of multimodal events to generate a fused feature vector of the current fireworks display area. S31. Based on the firework trajectory point set, high temperature anomaly area mask, non-standard combustion and explosion frequency band energy peak, real-time meteorological parameters and personnel location coordinates, three safety risk quantitative indicators are calculated. The three quantitative indicators of safety risks include: personnel intrusion risk coefficient, spark ignition risk coefficient, and abnormal combustion and explosion confidence coefficient. It is worth noting that in this embodiment, firstly, when calculating the personnel intrusion risk coefficient, the spatial relationship between the coordinates of all personnel positions and the preset safety warning area (dynamically expanded by the maximum height of the fireworks and the wind deflection distance) is determined. If a personnel position falls into this area, its Euclidean distance to the nearest warning boundary is normalized to the [0,1] interval. The closer the distance, the higher the risk value. Finally, the maximum value among all personnel is taken as the personnel intrusion risk coefficient for the current frame. Secondly, when calculating the spark ignition risk coefficient, the slope of the final falling segment of the fireworks trajectory point set and the wind speed and wind direction in real-time meteorological parameters are combined to estimate the ground projection area where the sparks may fall. Then, the spatial intersection operation is performed between this projection area and the high temperature anomaly area mask. The proportion of the intersection area to the total projection area is calculated and multiplied by the relative difference between the ambient temperature and the ignition threshold of surface combustibles (such as 200℃) (when the ambient temperature is lower than the threshold, it is linearly decayed). The result is the spark ignition risk coefficient. Finally, when calculating the confidence coefficient of abnormal combustion and explosion, the energy peak with the largest amplitude in the non-standard combustion and explosion frequency band is extracted, and its ratio is calculated with the average energy baseline of historical standard combustion and explosion events in the same frequency band. If the ratio is greater than 1, it is logarithmically compressed and mapped to the [0,1] interval; otherwise, it is set to 0. This value is the confidence coefficient of abnormal combustion and explosion. It is worth further explaining that the personnel intrusion risk coefficient, by comparing personnel locations with dynamic warning zones in real time, can accurately identify unauthorized entry into high-risk areas, avoiding the lag and misjudgment caused by relying on manual inspections or rough zone divisions, thus achieving immediate perception of threats to personal safety during the ignition process; secondly, the spark ignition risk coefficient integrates ballistic trajectory prediction, meteorological influences, and surface thermal anomaly information, quantifying the possibility of secondary fires caused by sparks in a physically interpretable way, overcoming the one-sidedness of relying solely on temperature thresholds or wind speed as a single parameter, and demonstrating... This significantly enhances the ability to predict the risk of ignition of ground combustibles. Finally, the confidence coefficient for abnormal combustion and explosion is quantified based on the degree of energy deviation in the acoustic frequency domain. It does not rely on complex models and can effectively capture abnormal combustion events such as misfires, premature explosions, and off-center explosions simply by comparing with the standard acoustic baseline for combustion and explosion. It has the advantages of fast response and strong resistance to visual obstruction interference. The three indicators start from the three core safety dimensions of people, fire, and explosion, respectively, and form a complementary and semantically clear risk characterization system. This provides a structured, interpretable, and traceable input basis for subsequent multimodal fusion, effectively supporting highly reliable dynamic early warning decisions.

[0015] S32. Perform the first verification on the three safety risk quantification indicators based on the physical constraints of fireworks display safety, and determine the first fusion weight based on the first verification results; S321. Obtain the safety physical constraint boundary values ​​associated with each safety risk quantification index, including the minimum safe distance boundary value associated with the personnel intrusion risk coefficient, the maximum permissible surface temperature boundary value associated with the spark fall ignition risk coefficient, and the maximum sound pressure energy boundary value associated with the abnormal combustion and explosion confidence coefficient. It is worth noting that, in this embodiment, the minimum safe distance boundary value associated with the personnel intrusion risk coefficient is determined by geometric extrapolation based on the maximum launch height of the fireworks product and historical wind deviation statistics, combined with the location of the ignition point, to determine the farthest landing point of the ground projection. A fixed buffer distance (e.g., 50 meters) is added to this value as the minimum safe distance boundary value. The maximum permissible surface temperature boundary value associated with the spark ignition risk coefficient is determined by referring to the measured ignition temperature threshold of common combustibles (e.g., hay, paper), and after correction for environmental humidity, a conservative value (e.g., 200°C) is taken as the upper limit. The maximum sound pressure energy boundary value associated with the abnormal explosion confidence coefficient is determined by performing a short-time Fourier transform on the sound pressure time-series signals collected from multiple standard ignition events, statistically analyzing the historical maximum value of the energy peak in the non-standard frequency band, and setting a certain margin (e.g., increasing by 10%) as the maximum sound pressure energy boundary value.

[0016] S322. Perform physical constraint verification: Determine whether the personnel intrusion risk coefficient is greater than the minimum safe distance boundary value. If so, mark the personnel intrusion risk coefficient as a physical boundary violation state; otherwise, mark it as a physical compliance state. Determine whether the spark fall ignition risk coefficient is greater than the maximum allowable surface temperature boundary value. If so, mark the spark fall ignition risk coefficient as a physical boundary violation state; otherwise, mark it as a physical compliance state. Determine whether the abnormal combustion and explosion confidence coefficient is greater than the maximum sound pressure energy boundary value. If so, mark the abnormal combustion and explosion confidence coefficient as a physical boundary violation state; otherwise, mark it as a physical compliance state. It is worth noting that in this embodiment, the personnel intrusion risk coefficient essentially reflects the spatial proximity of personnel to the dangerous area; the larger the value, the closer the distance and the higher the risk. The minimum safe distance boundary value represents the insurmountable safety baseline determined by physical deduction. Therefore, when this coefficient exceeds the boundary value, it means that the actual distance is less than the safety threshold, constituting a physical breach. Similarly, the spark fall ignition risk coefficient combines surface temperature and flammability conditions; if it exceeds the maximum permissible surface temperature boundary value, it indicates that the current thermal environment has reached or exceeded the ignition critical state. If the abnormal combustion and explosion confidence coefficient exceeds the maximum sound pressure energy boundary value, it indicates that the acoustic energy has significantly deviated from the normal combustion and explosion range, and there is an unexpected explosion behavior. All three are judged based on "whether the physical safety threshold has been breached," which conforms to the basic principles of engineering safety control. Secondly, this judgment method adopts a binary judgment approach of "out of bounds / compliance," avoiding decision delays or misjudgments caused by fuzzy intervals, and enabling rapid and deterministic state identification. At the same time, this method does not rely on complex reasoning or learning mechanisms, requires only one numerical comparison, and has extremely low computational overhead, making it suitable for fireworks display monitoring scenarios with high real-time requirements. In addition, each indicator and its corresponding physical boundary have clear dimensions and physical meanings, which facilitates understanding, calibration, and traceability by operation and maintenance personnel, improving the interpretability and maintainability of the system. Finally, the reason this judgment is the only viable option is that the core objective of safety monitoring is to intercept danger before it occurs. Physical constraint boundary values ​​are rigid limitations determined by objective physical laws such as ignition characteristics, material ignition points, and acoustic baselines, leaving no room for subjective adjustment. If we abandon the direct comparison logic of "whether it has crossed the boundary" and instead use relative change rates, trend predictions, or other indirect criteria, we risk losing the ability to effectively prevent sudden dangers due to response delays or excessively high tolerance for misjudgments. Therefore, only by directly comparing real-time quantitative indicators with preset physical boundaries can we ensure the rigor, timeliness, and reliability of safety judgments. This is an inevitable choice determined by the high-risk, rapid-response, and strongly physically coupled characteristics of fireworks displays.

[0017] S323. Based on the marking results of physical boundary violation and physical compliance status, calculate the boundary violation degree for each security risk quantification indicator. calculate: If any security risk quantification indicator is marked as physically out of bounds, the ratio of the current security risk quantification indicator to the associated security physical constraint boundary value is calculated as the degree of out of bounds. If any security risk quantification indicator is marked as a physical compliance status, then the ratio of the current security risk quantification indicator to the associated security physical constraint boundary value is calculated, and the current ratio is restricted to the interval [0, 1] by a truncation function as the out-of-bounds degree; It is worth noting that in this embodiment, for cases marked as physically compliant, the current security risk quantification index needs to be compared with the corresponding security physical constraint boundary value, and the ratio needs to be truncated: if the ratio is less than or equal to one, the ratio is directly retained as the boundary; if the ratio is greater than one, it should not theoretically occur, but may be caused by instantaneous sensor disturbances or numerical errors, so it is forcibly set to one. The reason for using the above calculation method is as follows: The core function of the boundary deviation is to uniformly characterize the degree of deviation of various risk indicators from their physical safety boundaries. When an indicator is in compliance, its value will not exceed the boundary value. At this time, the ratio will naturally fall between zero and one, which can directly reflect the degree of "approaching the boundary". If the ratio slightly exceeds one due to non-physical factors such as measurement noise, it will be truncated to ensure that it is not misjudged as boundary deviation, while maintaining the consistency of the output range. This processing not only preserves the continuous sensitivity within the compliance range, but also avoids the interference of abnormal fluctuations on the system's judgment. This method achieves numerical normalization of the out-of-bounds degree under all operating conditions. Regardless of whether the indicator is in a compliant or out-of-bounds state, its out-of-bounds degree has a clear physical reference and consistent dimensions, which facilitates subsequent unified threshold determination or horizontal comparison of multiple indicators. Secondly, the truncation operation is simple, deterministic, and parameterless, without introducing additional complexity, thus ensuring real-time calculation efficiency. Thirdly, in the compliant state, the out-of-bounds degree is limited to the range [0, 1], making "out-of-bounds degree equal to one" the critical marker of physical out-of-bounds. The logic is clear and the boundaries are distinct, which is conducive to building a reliable state machine or alarm mechanism. Finally, because security monitoring has extremely high requirements for numerical stability and logical rigor, if the ratio under compliant conditions is not truncated, even a slight overshoot (such as 1.02) may be misread as a high-risk signal in subsequent processing, causing a false alarm, even if it is not marked as an out-of-bounds error. Conversely, if smooth compression or other nonlinear mappings are used, the true degree of "approaching the boundary" will be blurred, weakening the risk identification capability. Therefore, only by adopting a hard truncation to [0, 1] can we ensure numerical robustness while strictly aligning the physical state with the mathematical expression, ensuring that the system behavior is completely consistent with the security logic.

[0018] S324. Apply a monotonically increasing nonlinear mapping function to the out-of-bounds degree of each security risk quantification indicator to generate the first physical confidence factor, the second physical confidence factor, and the third physical confidence factor. It is worth noting that in this embodiment, firstly, a preset nonlinear mapping relationship is configured for each boundary violation degree. This relationship increases slowly when the input value is close to zero, and increases rapidly after the input value is close to or exceeds one. In actual calculation, the boundary violation degree is used as input, and the corresponding output value is obtained by looking up a table or by piecewise linear interpolation. This output is the corresponding physical confidence factor. The mapping relationship can be pre-calibrated using engineering experience or historical data. Furthermore, on the one hand, the nonlinear mapping can amplify the differences in high-risk areas (i.e., boundary violations close to or greater than one), so that small boundary violations are reflected as a significant increase in the confidence factor, thereby enhancing the system's sensitivity to critical dangerous states. On the other hand, in low-risk areas (boundary violations much less than one), a smooth response is maintained to avoid unnecessary alarms caused by normal fluctuations, thus improving system stability. At the same time, the monotonically increasing characteristic ensures the consistency logic between the risk level and the confidence factor, that is, the higher the risk, the larger the confidence factor, which conforms to the principle of intuitive safety judgment and facilitates subsequent fusion decision-making.

[0019] S325. Normalize the first physical confidence factor, the second physical confidence factor, and the third physical confidence factor to obtain each normalized physical confidence factor. Combine each normalized physical confidence factor and use the combination result as the first fusion weight. It is worth noting that in this embodiment, the first physical confidence factor, the second physical confidence factor, and the third physical confidence factor are normalized respectively, specifically using a linear minimum-maximum normalization method: subtract the minimum value among all current physical confidence factors from each physical confidence factor, and then divide by the difference between the maximum value and the minimum value, thereby mapping it to the interval between zero and one, to obtain each normalized physical confidence factor; subsequently, the three normalized physical confidence factors are combined in an arithmetic mean manner, that is, the three are added together and then divided by three, and the result is used as the first fusion weight.

[0020] S33. Perform a second verification on the three security risk quantification indicators based on the consistency relationship of multimodal event time series, and determine the second fusion weight based on the second verification results; S331. Construct a time sliding window of length M, where M is a positive integer greater than or equal to 3, and extract the personnel intrusion risk coefficient sequence, the Martian fall ignition risk coefficient sequence, and the abnormal combustion and explosion confidence coefficient sequence recorded in the most recent M sampling periods from the historical data cache. It is worth noting that in this embodiment, a time sliding window of length M is first constructed, where M is a positive integer not less than three, to cover a historical period that sufficiently reflects dynamic trends. Subsequently, three types of safety risk quantification index sequences recorded in the most recent M sampling periods are synchronously extracted from the historical data cache. These are the personnel intrusion risk coefficient sequence, the spark fall ignition risk coefficient sequence, and the abnormal combustion and explosion confidence coefficient sequence. This window slides forward one position with each new sampling period to ensure that the data used is always up-to-date and has temporal continuity, providing a reliable foundation for subsequent temporal consistency verification.

[0021] S332, Perform timing consistency verification: Based on the historical personnel intrusion risk coefficient sequence, the maximum value of the absolute difference between adjacent sampling periods is calculated as the upper limit threshold for personnel movement mutation; the personnel intrusion risk coefficient of the current sampling period is subtracted from the personnel intrusion risk coefficient of the previous sampling period and the absolute value is taken to obtain the current personnel risk change; if the current personnel risk change is greater than the upper limit threshold for personnel movement mutation, the personnel intrusion risk coefficient of the current sampling period is marked as a time-series abnormal state, otherwise it is marked as a time-series stable state. Based on the historical Mars fall ignition risk coefficient sequence, the standard deviation of the current sequence is calculated as the thermal radiation fluctuation threshold; the difference between the Mars fall ignition risk coefficient of the current sampling period and the mean of the current sequence is taken as the absolute value to obtain the current Mars risk deviation; if the current Mars risk deviation is greater than the thermal radiation fluctuation threshold, the Mars fall ignition risk coefficient of the current sampling period is marked as a time-series abnormal state, otherwise it is marked as a time-series stable state. Based on the historical abnormal combustion and explosion confidence coefficient sequence, the difference between the maximum value and the mean value of the current sequence is calculated as the upper limit threshold for combustion and explosion confidence anomalies; the difference between the abnormal combustion and explosion confidence coefficient of the current sampling period and the mean value of the current sequence is used to obtain the current combustion and explosion confidence deviation; if the current combustion and explosion confidence deviation is greater than the upper limit threshold for combustion and explosion confidence anomalies, the abnormal combustion and explosion confidence coefficient of the current sampling period is marked as a time-series abnormal state; otherwise, it is marked as a time-series stable state. It is worth noting that in this embodiment, firstly, personnel movement has inertial characteristics, and the risk coefficient should not change drastically in a short period of time. Therefore, the maximum value of the absolute difference between adjacent periods is used as the upper limit threshold of the sudden change, which can effectively capture non-physical jumps (such as sensor failure or false detection). The risk of sparks falling and igniting is affected by environmental thermal disturbances, and its natural fluctuations should be randomly distributed around the mean. Therefore, the standard deviation is used to measure the normal fluctuation range, and deviations exceeding this range are considered abnormal. The confidence level of abnormal combustion and explosion is usually stable during normal combustion and only rises briefly during actual combustion and explosion events. Therefore, the difference between the maximum value and the mean is used as the upper limit of the abnormality, which can identify sudden high confidence signals that significantly deviate from the normal pattern. Secondly, all three criteria are customized based on the inherent dynamic characteristics of their respective indicators, eliminating the need for a unified threshold and improving the accuracy of the discrimination. At the same time, all calculations rely solely on historical data within the sliding window, requiring no external prior models, making them highly adaptable and easy to deploy. Furthermore, clearly classifying the state into "temporal anomalies" or "temporal stability" facilitates subsequent differentiated processing and enhances the robustness of the system. Finally, it is worth noting that if the mean or variance criterion is used for personnel risk, it will be impossible to effectively identify acute events such as sudden intrusion; if the difference method is used for Mars risk, normal slow warming will be misjudged as abnormal; if a fixed threshold is used for the confidence level of combustion and explosion, it will be difficult to adapt to the baseline changes in different stages of the launch. Therefore, it is necessary to use the most suitable temporal consistency criterion according to the physical behavior characteristics of each type of indicator. This is the only reasonable path determined by the essential differences of the three types of risks in the time dimension.

[0022] S333. Based on the labeling results of time-series abnormal states and time-series stable states, perform time-series inconsistency measurement calculations for each security risk quantification indicator: If any security risk quantification indicator is marked as a time-series anomalous state, the absolute deviation between the current security risk quantification indicator and the mean of the historical security risk quantification indicator series is calculated as a time-series inconsistency measure. If any security risk quantification indicator is marked as a time-stable state, then the time-series inconsistency measure of the current security risk quantification indicator is set to zero. It is worth noting that in this embodiment, when the current sequence is determined to be abnormal, it means that the current value deviates significantly from the historical behavior pattern. At this time, the absolute deviation between the current value and the mean of the historical sequence can quantitatively characterize its "degree of abnormality". When the current sequence is determined to be stable, it means that the current value is within the normal fluctuation range and there is no need to assign inconsistency, so it is directly set to zero. This conditional assignment not only preserves the quantitative expression of abnormal information, but also avoids excessive punishment of the normal state. This calculation method achieves two advantages: firstly, it calculates deviations only under abnormal conditions, reducing invalid calculations and improving efficiency; secondly, it forces the inconsistencies in stable states to zero, enabling a clear distinction between no anomalies and low anomalies when generating confidence factors, avoiding fuzzy transitions and improving the accuracy of fusion decisions. Secondly, if the deviation is calculated for all states, the steady state will also produce non-zero inconsistency, making it impossible to distinguish between normal fluctuations and real anomalies. If relative deviation or other complex measures are used, scale sensitivity or computational delay may be introduced. Only by enabling the deviation measure in the event of an anomaly and setting it to zero in the event of stability can the binary judgment logic of the time series verification be strictly aligned while maintaining simplicity. This is a necessary design to achieve efficient and reliable time series credibility assessment.

[0023] S334. Take the negative value of the time series inconsistency measure for each security risk quantification indicator and input it into the exponential function to generate the first time series confidence factor, the second time series confidence factor, and the third time series confidence factor. It is worth noting that in this embodiment, for each security risk quantification indicator, the time-series inconsistency measure is first negative, and then the negative value is used as input to the exponential function for mapping, thereby generating the first time-series confidence factor, the second time-series confidence factor, and the third time-series confidence factor respectively. Specifically, when the inconsistency measure is zero (i.e., the state is stable), the output of the exponential function is one, indicating complete confidence; when the inconsistency measure increases, the negative value becomes smaller, the output of the exponential function decays rapidly, and the confidence factor decreases accordingly. Among them, the exponential function has a natural monotonically decreasing property, which can intuitively transform the more inconsistent and unreliable the semantics into a decrease in numerical value. At the same time, its smoothness near the zero point and rapid decay under large deviations can both tolerate minor anomalies and strongly suppress serious anomalies, effectively enhancing the risk sensitivity of the confidence factor and providing high-quality input for subsequent weight fusion.

[0024] S335. Normalize the first time series confidence factor, the second time series confidence factor, and the third time series confidence factor to obtain each normalized time series confidence factor. Combine each normalized time series confidence factor and use the combination result as the second fusion weight. It is worth noting that in this embodiment, the three time-series confidence factors are normalized and combined in a manner completely consistent with that described in S325, that is, the arithmetic mean is taken after minimum-maximum linear normalization, and the result is used as the second fusion weight.

[0025] S34. Based on three quantitative indicators of safety risks, a comprehensive indicator is calculated; S341. Determine whether at least one of the three quantitative safety risk indicators is marked as a physical out-of-bounds state or a temporal anomaly state. S342. If at least one of the three security risk quantification indicators is marked as a physical out-of-bounds state or a timing abnormal state, the maximum value among the three security risk quantification indicators shall be determined as the comprehensive indicator. S343. If none of the three safety risk quantitative indicators are marked as physical out-of-bounds state and none of them are marked as temporal abnormal state, then the arithmetic mean of the three safety risk quantitative indicators shall be determined as the comprehensive indicator. It is worth noting that in this embodiment, the status labels of the three safety risk quantification indicators are jointly checked to determine whether at least one is marked as physically out of bounds or in a time-series abnormal state. This judgment is based on the dual status information output from the previous steps and is implemented using a logical "OR" operation. As long as any indicator shows a physical out of bounds or a time-series abnormality, it is considered that the whole is in an unsteady state or a high-risk situation. This design can effectively capture single-point sudden risks and avoid the local danger being masked by the normality of other dimensions. This is in line with the principle in safety engineering that the weakest link determines the system's safety. If a more stringent judgment of simultaneous anomalies of multiple indicators is adopted, it will be difficult to respond to single-source sudden risks commonly seen in fireworks display scenarios. If time-series abnormalities are ignored, it will be impossible to identify potential threats that have not yet crossed the boundaries but have deviated from the normal pattern. Therefore, this judgment logic is a necessary choice that balances sensitivity and reliability. Based on the above judgment results, the calculation method of the comprehensive index is dynamically selected in steps S342 and S343: when there is at least one abnormal state, the maximum value of the three indicators is taken as the comprehensive index; when all indicators are physically compliant and temporally stable, the arithmetic mean of the three indicators is used as the comprehensive index. The maximum value strategy ensures that the most unfavorable risk is fully highlighted under abnormal conditions, preventing high-risk signals from being diluted by the averaging effect; while the arithmetic mean strategy effectively integrates multi-dimensional information under normal conditions, suppresses random fluctuations, and improves the stability of the assessment. If the average value is still used under abnormal conditions, the risk will be underestimated; if the maximum value is used uniformly throughout the time, false alarms may be triggered by normal fluctuations. Only by dynamically switching the aggregation method according to the state can accurate safety perception be achieved in high-risk, highly dynamic fireworks display scenarios, ensuring that no abnormality is overlooked and no normal situation is mistakenly disturbed.

[0026] S35. Based on the comprehensive index, select the target fusion weight from the first fusion weight and the second fusion weight, and perform weighted fusion of the fireworks trajectory point set, high temperature anomaly area mask, non-standard combustion and explosion frequency band energy peak, real-time meteorological parameters and personnel position coordinates based on the target fusion weight to generate the fusion feature vector of the current fireworks display area. S351. Set the preset threshold for the comprehensive index; It is worth noting that in this embodiment, firstly, based on a large amount of historical data on fireworks displays, the distribution range of comprehensive indicators under safe and controllable conditions is statistically analyzed, and combined with expert experience and safety standards, the percentile of this distribution is selected as the initial threshold. Subsequently, a small-scale trial run is conducted in the early stage of actual deployment, and the threshold is fine-tuned based on the balance between false alarm rate and false alarm rate. Finally, a fixed threshold is determined that can effectively identify high-risk conditions without frequently triggering alarms.

[0027] S352. Compare the comprehensive index with the preset threshold: if the comprehensive index is greater than the preset threshold, select the first fusion weight as the target fusion weight; if the comprehensive index is less than or equal to the preset threshold, select the second fusion weight as the target fusion weight. It is worth noting that in this embodiment, if the comprehensive index exceeds the threshold, it indicates that the overall risk level is high. In this case, the first fusion weight (i.e., the weight generated based on the physical confidence factor) is selected as the target fusion weight. Conversely, if the comprehensive index does not exceed the threshold, the system is considered to be in a low-risk steady state, and the second fusion weight (i.e., the weight generated based on the time-series confidence factor) is selected. This is because in high-risk scenarios, indicators with stronger physical boundary compliance are given priority to ensure a safety baseline. In low-risk scenarios, more attention is paid to the temporal continuity and stability of the indicators to improve the accuracy of the assessment. This dynamic switching mechanism ensures that the fusion strategy always matches the current risk situation, avoiding the conservative or overly aggressive problems caused by a "one-size-fits-all" approach.

[0028] S353. Convert the firework trajectory point set, high temperature anomaly area mask, non-standard combustion and explosion frequency band energy peak, real-time meteorological parameters and personnel position coordinates into feature vectors of a unified dimension to obtain the first mode feature vector, the second mode feature vector, the third mode feature vector, the fourth mode feature vector and the fifth mode feature vector; It is worth noting that in this embodiment, the raw data from different sensing sources are uniformly converted into numerical feature vectors. Specifically, the firework trajectory point set is mapped into a fixed-length position sequence vector after coordinate normalization and interpolation alignment; the high-temperature anomaly area mask is processed morphologically to extract connected component features and encode them into a region attribute vector; the energy peak of the non-standard combustion and explosion frequency band is converted into an energy distribution vector through spectrum segment integration; real-time meteorological parameters (such as wind speed, humidity, and temperature) directly form an environmental parameter vector; and the personnel position coordinates are spatially gridded or distance transformed to form a position relationship vector. All vectors are filled or truncated to the same dimension to ensure consistency in subsequent calculations.

[0029] S354. Based on the target fusion weights, perform element-wise weighting on the first modality feature vector, the second modality feature vector, the third modality feature vector, the fourth modality feature vector, and the fifth modality feature vector, respectively; It is worth noting that in this embodiment, the target fusion weight is selected according to S352, and the weight is expanded in dimension to make it consistent with the length of each modal feature vector. Then, element-wise multiplication is performed on the five modal feature vectors respectively, that is, each element in the target fusion weight is multiplied with the element at the same position in the corresponding modal feature vector, so as to obtain five weighted feature vectors. This process realizes the differential adjustment of the importance of different feature dimensions.

[0030] S355. Sum the weighted modal feature vectors element by element to generate the fusion feature vector of the current fireworks display area. It is worth noting that in this embodiment, the five weighted feature vectors are added element by element at the same position, that is, the first element of the first vector is added to the first element of the other four vectors to obtain the first element of the fused feature vector. This process is repeated until all dimensions are summed, and finally a fused feature vector with the same dimension as the original modality vector is generated. This vector integrates multi-source information and highlights the most credible feature components in the current risk situation through dynamic weights, providing a highly discriminative input representation for subsequent risk decisions.

[0031] S4. Input the fused feature vector of the current fireworks display area into the pre-trained risk level discrimination model to generate a safety risk level identifier for the current fireworks display area, and trigger an early warning command based on the safety risk level identifier of the current fireworks display area. It is worth noting that in this embodiment, the fused feature vector of the current fireworks display area is input into a pre-trained risk level discrimination model, which outputs a corresponding safety risk level identifier. This identifier is usually divided into multiple discrete levels (e.g., low risk, medium risk, high risk, and extremely high risk). Then, based on the output level, the corresponding level of warning instruction is triggered, such as voice broadcast, light warning, automatic suspension of the fireworks display device, or activation of emergency isolation measures. The risk level discrimination model is constructed as follows: First, a large amount of multimodal raw data from historical fireworks displays is collected, including fireworks trajectory records, infrared thermal imaging videos, acoustic spectrum signals, meteorological monitoring logs, and personnel location information. The actual safety consequences of each fireworks display event are simultaneously labeled (such as no abnormality, minor danger, serious danger, explosion accident, etc.). These consequence labels are mapped to unified risk level labels after expert review, forming the label system required for supervised learning. Based on this, a corresponding fusion feature vector is generated for each historical sample according to the process described in S353 to S355 above, thereby constructing a complete training set. Each sample consists of a fixed-dimensional fusion feature vector and its corresponding risk level label. The samples in the training set cover a variety of firework scales, environmental conditions, crowd densities and anomaly types to ensure that the model has good generalization ability. The model structure adopts a lightweight multilayer perceptron (MLP) or support vector machine (SVM). Its input layer dimension is consistent with the fused feature vector, and the number of output layer nodes is equal to the number of risk level categories. The activation function is Softmax to output the probability distribution of each category. During training, the cross-entropy loss function is used for optimization. The parameters are updated iteratively through the Adam optimizer, and an early stopping mechanism is introduced to prevent overfitting. At the same time, a stratified sampling strategy is adopted to divide the training set, validation set and test set to ensure that the proportion of each risk level is balanced in each subset. After the model is trained, its accuracy, recall, and F1 score are evaluated on an independent test set. If all indicators meet the preset safety standards (e.g., overall accuracy is not less than 92%, and recall for high-risk categories is not less than 95%), the risk level discrimination model is deployed. At this point, the pre-trained risk level discrimination model is completed and used to receive fused feature vectors in real time and output reliable safety risk level identifiers to support subsequent graded early warning decisions.

[0032] Example 2: Figure 2 As shown, this invention also discloses a dynamic early warning system for fireworks display safety based on multimodal fusion, comprising the following modules: Data acquisition module: used to synchronously acquire visible light video streams, infrared thermal image sequences, sound pressure timing signals, real-time meteorological parameters, and personnel location coordinates of the fireworks display area; Data processing module: Connected to the data acquisition module, it is used to perform inter-frame difference processing on visible light video streams to extract the trajectory point set of fireworks; perform temperature gradient clustering on infrared thermal image sequences to generate high temperature anomaly region masks; and perform short-time Fourier transform on sound pressure time-series signals to identify energy peaks in non-standard combustion and explosion frequency bands. Feature vector acquisition module: connected to the data processing module and the data acquisition module, used to perform weighted fusion of each modal data based on the following: fireworks trajectory point set, high temperature anomaly area mask, non-standard combustion and explosion frequency band energy peak, real-time meteorological parameters and personnel location coordinates, according to the physical constraints of fireworks display safety and the consistency relationship of multimodal event time sequence, to generate a fused feature vector of the current fireworks display area; Warning command triggering module: connected to the feature vector acquisition module, used to input the fused feature vector of the current fireworks display area into the pre-trained risk level discrimination model, generate the safety risk level identifier of the current fireworks display area, and trigger warning commands based on the safety risk level identifier of the current fireworks display area.

[0033] To implement the above embodiments, this application also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided in the foregoing embodiments.

[0034] To implement the above embodiments, this application also proposes a computer program product, including a computer program that, when executed by a processor, implements the methods provided in the foregoing embodiments.

[0035] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A dynamic early warning method for fireworks display safety based on multimodal fusion, characterized in that, Includes the following steps: S1. Synchronously acquire visible light video streams, infrared thermal image sequences, sound pressure timing signals, real-time meteorological parameters, and personnel location coordinates of the fireworks display area; S2. Perform inter-frame difference processing on the visible light video stream to extract the trajectory point set of the fireworks; perform temperature gradient clustering on the infrared thermal image sequence to generate a high-temperature anomaly region mask. Short-time Fourier transform is performed on the sound pressure timing signal to identify the energy peak value of the non-standard combustion and explosion frequency band; S3. Based on the modal data, which includes: fireworks trajectory point set, high temperature anomaly area mask, non-standard combustion and explosion frequency band energy peak, real-time meteorological parameters and personnel location coordinates, the modal data are weighted and fused according to the physical constraints of fireworks display safety and the consistency relationship between the time sequence of multimodal events to generate a fused feature vector of the current fireworks display area. S4. Input the fused feature vector of the current fireworks display area into the pre-trained risk level discrimination model to generate a safety risk level identifier for the current fireworks display area, and trigger an early warning command based on the safety risk level identifier of the current fireworks display area.

2. The method for dynamic early warning of fireworks display safety based on multimodal fusion according to claim 1, characterized in that, The process involves weighted fusion of various modal data, including: fireworks trajectory point sets, high-temperature anomaly area masks, energy peaks in non-standard combustion and explosion frequency bands, real-time meteorological parameters, and personnel location coordinates. This fusion is based on the physical constraints of fireworks display safety and the temporal consistency relationship of multimodal events, generating a fused feature vector for the current fireworks display area. This vector includes: S31. Based on the firework trajectory point set, high temperature anomaly area mask, non-standard combustion and explosion frequency band energy peak, real-time meteorological parameters and personnel location coordinates, three safety risk quantitative indicators are calculated. S32. Perform the first verification on the three safety risk quantification indicators based on the physical constraints of fireworks display safety, and determine the first fusion weight based on the first verification results; S33. Perform a second verification on the three security risk quantification indicators based on the consistency relationship of multimodal event time series, and determine the second fusion weight based on the second verification results; S34. Based on three quantitative indicators of safety risks, a comprehensive indicator is calculated; S35. Based on the comprehensive index, select the target fusion weight from the first fusion weight and the second fusion weight, and perform weighted fusion of the fireworks trajectory point set, high temperature anomaly area mask, non-standard combustion and explosion frequency band energy peak, real-time meteorological parameters and personnel position coordinates based on the target fusion weight to generate the fusion feature vector of the current fireworks display area.

3. The method for dynamic early warning of fireworks display safety based on multimodal fusion according to claim 2, characterized in that, The three quantitative indicators of safety risks include: personnel intrusion risk coefficient, spark ignition risk coefficient, and abnormal combustion and explosion confidence coefficient.

4. The method for dynamic early warning of fireworks display safety based on multimodal fusion according to claim 3, characterized in that, The first verification of the three safety risk quantification indicators based on the physical constraints of fireworks display safety is performed, and the first fusion weight is determined based on the first verification result, including: S321. Obtain the safety physical constraint boundary values ​​associated with each safety risk quantification index, including the minimum safe distance boundary value associated with the personnel intrusion risk coefficient, the maximum permissible surface temperature boundary value associated with the spark fall ignition risk coefficient, and the maximum sound pressure energy boundary value associated with the abnormal combustion and explosion confidence coefficient. S322. Perform physical constraint verification: Determine whether the personnel intrusion risk coefficient is greater than the minimum safe distance boundary value. If so, mark the personnel intrusion risk coefficient as a physical boundary violation state; otherwise, mark it as a physical compliance state. Determine whether the spark fall ignition risk coefficient is greater than the maximum allowable surface temperature boundary value. If so, mark the spark fall ignition risk coefficient as a physical boundary violation state; otherwise, mark it as a physical compliance state. Determine whether the abnormal combustion and explosion confidence coefficient is greater than the maximum sound pressure energy boundary value. If so, mark the abnormal combustion and explosion confidence coefficient as a physical boundary violation state; otherwise, mark it as a physical compliance state. S323. Based on the marking results of physical boundary violation and physical compliance status, calculate the boundary violation degree for each security risk quantification indicator. calculate: If any security risk quantification indicator is marked as physically out of bounds, the ratio of the current security risk quantification indicator to the associated security physical constraint boundary value is calculated as the degree of out of bounds. If any security risk quantification indicator is marked as a physical compliance status, then the ratio of the current security risk quantification indicator to the associated security physical constraint boundary value is calculated, and the current ratio is restricted to the interval [0, 1] by a truncation function as the out-of-bounds degree; S324. Apply a monotonically increasing nonlinear mapping function to the out-of-bounds degree of each security risk quantification indicator to generate the first physical confidence factor, the second physical confidence factor, and the third physical confidence factor. S325. Normalize the first physical confidence factor, the second physical confidence factor, and the third physical confidence factor to obtain each normalized physical confidence factor. Combine each normalized physical confidence factor and use the combination result as the first fusion weight.

5. The method for dynamic early warning of fireworks display safety based on multimodal fusion according to claim 4, characterized in that, The second verification based on the temporal consistency relationship of multimodal events is performed on the three security risk quantification indicators, and the second fusion weight is determined according to the second verification result, including: S331. Construct a time sliding window of length M, where M is a positive integer greater than or equal to 3, and extract the personnel intrusion risk coefficient sequence, the Martian fall ignition risk coefficient sequence, and the abnormal combustion and explosion confidence coefficient sequence recorded in the most recent M sampling periods from the historical data cache. S332, Perform timing consistency verification: Based on the historical personnel intrusion risk coefficient sequence, the maximum value of the absolute difference between adjacent sampling periods is calculated as the upper limit threshold for personnel movement mutation; the personnel intrusion risk coefficient of the current sampling period is subtracted from the personnel intrusion risk coefficient of the previous sampling period and the absolute value is taken to obtain the current personnel risk change; if the current personnel risk change is greater than the upper limit threshold for personnel movement mutation, the personnel intrusion risk coefficient of the current sampling period is marked as a time-series abnormal state, otherwise it is marked as a time-series stable state. Based on the historical Mars fall ignition risk coefficient sequence, the standard deviation of the current sequence is calculated as the thermal radiation fluctuation threshold; the difference between the Mars fall ignition risk coefficient of the current sampling period and the mean of the current sequence is taken as the absolute value to obtain the current Mars risk deviation; if the current Mars risk deviation is greater than the thermal radiation fluctuation threshold, the Mars fall ignition risk coefficient of the current sampling period is marked as a time-series abnormal state, otherwise it is marked as a time-series stable state. Based on the historical abnormal combustion and explosion confidence coefficient sequence, the difference between the maximum value and the mean value of the current sequence is calculated as the upper limit threshold for combustion and explosion confidence anomalies; the difference between the abnormal combustion and explosion confidence coefficient of the current sampling period and the mean value of the current sequence is used to obtain the current combustion and explosion confidence deviation; if the current combustion and explosion confidence deviation is greater than the upper limit threshold for combustion and explosion confidence anomalies, the abnormal combustion and explosion confidence coefficient of the current sampling period is marked as a time-series abnormal state; otherwise, it is marked as a time-series stable state. S333. Based on the labeling results of time-series abnormal states and time-series stable states, perform time-series inconsistency measurement calculations for each security risk quantification indicator: If any security risk quantification indicator is marked as a time-series anomalous state, the absolute deviation between the current security risk quantification indicator and the mean of the historical security risk quantification indicator series is calculated as a time-series inconsistency measure. If any security risk quantification indicator is marked as a time-stable state, then the time-series inconsistency measure of the current security risk quantification indicator is set to zero. S334. Take the negative value of the time series inconsistency measure for each security risk quantification indicator and input it into the exponential function to generate the first time series confidence factor, the second time series confidence factor, and the third time series confidence factor. S335. Normalize the first time-series confidence factor, the second time-series confidence factor, and the third time-series confidence factor to obtain each normalized time-series confidence factor. Combine each normalized time-series confidence factor and use the combination result as the second fusion weight.

6. The method for dynamic early warning of fireworks display safety based on multimodal fusion according to claim 5, characterized in that, The comprehensive index, calculated based on three quantitative security risk indicators, includes: S341. Determine whether at least one of the three quantitative safety risk indicators is marked as a physical out-of-bounds state or a temporal anomaly state. S342. If at least one of the three security risk quantification indicators is marked as a physical out-of-bounds state or a timing abnormal state, the maximum value among the three security risk quantification indicators shall be determined as the comprehensive indicator. S343. If none of the three safety risk quantitative indicators are marked as physical out-of-bounds states and none of them are marked as temporal abnormal states, then the arithmetic mean of the three safety risk quantitative indicators shall be determined as the comprehensive indicator.

7. The method for dynamic early warning of fireworks display safety based on multimodal fusion according to claim 6, characterized in that, The process involves selecting a target fusion weight from the first and second fusion weights based on comprehensive indicators, and then weighting and fusing the fireworks trajectory point set, high-temperature anomaly area mask, non-standard combustion and explosion frequency band energy peak, real-time meteorological parameters, and personnel location coordinates based on the target fusion weight to generate a fusion feature vector of the current fireworks display area, including: S351. Set the preset threshold for the comprehensive index; S352. Compare the comprehensive index with the preset threshold: if the comprehensive index is greater than the preset threshold, select the first fusion weight as the target fusion weight; if the comprehensive index is less than or equal to the preset threshold, select the second fusion weight as the target fusion weight. S353. Convert the firework trajectory point set, high temperature anomaly area mask, non-standard combustion and explosion frequency band energy peak, real-time meteorological parameters and personnel position coordinates into feature vectors of a unified dimension to obtain the first mode feature vector, the second mode feature vector, the third mode feature vector, the fourth mode feature vector and the fifth mode feature vector; S354. Based on the target fusion weights, perform element-wise weighting on the first modality feature vector, the second modality feature vector, the third modality feature vector, the fourth modality feature vector, and the fifth modality feature vector, respectively; S355. Sum the weighted modal feature vectors element by element to generate the fused feature vector of the current fireworks display area.

8. A dynamic early warning system for fireworks display safety based on multimodal fusion, used to execute the dynamic early warning method for fireworks display safety based on multimodal fusion as described in any one of claims 1-7, characterized in that, Includes the following modules: Data acquisition module: used to synchronously acquire visible light video streams, infrared thermal image sequences, sound pressure timing signals, real-time meteorological parameters, and personnel location coordinates of the fireworks display area; Data processing module: Connected to the data acquisition module, it is used to perform inter-frame difference processing on visible light video streams to extract the trajectory point set of fireworks; perform temperature gradient clustering on infrared thermal image sequences to generate high temperature anomaly region masks; and perform short-time Fourier transform on sound pressure time-series signals to identify energy peaks in non-standard combustion and explosion frequency bands. Feature vector acquisition module: connected to the data processing module and the data acquisition module, used to perform weighted fusion of each modal data based on the following: fireworks trajectory point set, high temperature anomaly area mask, non-standard combustion and explosion frequency band energy peak, real-time meteorological parameters and personnel location coordinates, according to the physical constraints of fireworks display safety and the consistency relationship of multimodal event time sequence, to generate a fused feature vector of the current fireworks display area; Warning command triggering module: connected to the feature vector acquisition module, used to input the fused feature vector of the current fireworks display area into the pre-trained risk level discrimination model, generate the safety risk level identifier of the current fireworks display area, and trigger warning commands based on the safety risk level identifier of the current fireworks display area.

Images