An abnormal target early warning method, system and device based on multi-modal video stream and a medium

Abstract

The application relates to the technical field of land management and protection, and discloses an abnormal target early warning method, system and device based on a multi-modal video stream and a medium; the method comprises the following steps: acquiring multi-modal data; inputting the multi-modal data into a Transform multi-branch network to extract spatial features, time features and cross-modal correlation features; dynamically allocating feature weights based on an attention mechanism to realize multi-modal feature deep fusion; identifying natural fires, geological disaster displacements and illegal intrusions as abnormal targets through an abnormal target recognition algorithm library, combining confidence judgment to output early warning information and a risk level; calling historical data to perform abnormal preposition analysis, mine key preposition behaviors and environmental data; tracking and path predicting the moving track of the abnormal target to construct a dynamic supervision area; and realizing abnormal behavior whole-process tracing.

Description

Technical Field

[0001] This invention relates to the field of intelligent monitoring and early warning technology, specifically to an abnormal target early warning method, system, device, and medium based on multimodal video streams. Background Technology

[0002] The protected areas for reserve land span a large area, and the land use status changes dynamically. Some plots are adjacent to areas that are already in operation, and the activity of moving targets is frequent. The coverage of manual patrol monitoring is limited, which can easily create monitoring blind spots and make it impossible to identify and deal with various abnormal events in a timely manner.

[0003] In existing technologies, anomaly monitoring mostly relies on cameras to collect static images or independently analyze thermal imaging and visible light single-modal data for alarms. This approach suffers from problems such as limited information dimensions, poor adaptability to complex scenarios, and high false alarm rates. Furthermore, the lack of multi-device collaborative mechanisms during monitoring makes it difficult to accurately verify suspected anomalies, resulting in insufficient early warning accuracy. In addition, traditional methods only focus on anomaly identification results, lacking the ability to trace the source of abnormal behavior and failing to identify the underlying causes and precursory signs of anomalies. This makes it difficult to prevent similar anomalies from recurring at the source, and the lack of prediction and focused monitoring of potential abnormal behaviors often leads to a reactive response only after an anomaly occurs, resulting in insufficient proactive early warning and failing to meet the needs of efficient, accurate, and proactive monitoring and early warning in land management scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide an abnormal target early warning method, system, device and medium based on multimodal video stream, so as to realize accurate identification, rapid early warning, full-process traceability and potential risk prediction of abnormal targets, and improve the intelligence level and handling efficiency of abnormal land management monitoring.

[0005] The basic solution provided by this invention is an abnormal target early warning method based on multimodal video streams, comprising the following steps: Step S1: Multimodal data acquisition, acquiring visible light video stream, thermal imaging video stream, monitoring target location data, and environmental data of the protected area, including temperature, wind speed, and air pressure data.

[0006] Step S2: Data preprocessing, timestamp synchronization calibration of the collected multimodal data, removal of blurry and noisy frames from visible light video stream and thermal imaging video stream, gridding of the management area space, assignment of spatial coordinates, terrain attributes and scene feature labels to each grid, and completion of data format standardization.

[0007] Step S3: Input the multimodal data into the abnormal target identification algorithm library. The abnormal target identification algorithm library stores identification algorithm models for natural fires, illegally intruding vehicles, illegally intruding personnel, and geological disaster displacement. Each identification algorithm model identifies the confidence level of the abnormal target in parallel based on the input multimodal data. When the confidence level of the identified target in any identification algorithm model is greater than 80%, the abnormal target, confidence level, location of occurrence, and risk level are output, and an early warning is issued.

[0008] An abnormal target recognition algorithm library is built into the remote monitoring platform and is trained and constructed based on a large number of labeled samples in the land management scenario.

[0009] The algorithm library has a built-in pre-denoising unit that removes redundant noise data generated during the fusion process using Gaussian filtering and inter-frame differencing. Multimodal data is input in parallel to four independent recognition models. Each model analyzes the corresponding feature dimensions according to its own business direction without interfering with each other. The models output the identified target, confidence level, and pixel coordinates (converted to actual geographical location) in real time. Based on preset rules, risk levels are classified. When the confidence level of any model is greater than 80%, the linkage warning module completes the audible and visual warning and data reporting.

[0010] Among them, the natural fire identification model is developed based on the improved YOLOv8 network, with the addition of a temperature feature detection branch. It prioritizes the extraction of thermal imaging temperature features and visual contour features from visible light video streams and thermal imaging video streams, while filtering out invalid background noise.

[0011] Unauthorized vehicle identification model: developed based on SSD target detection and behavior analysis network, focusing on analyzing target contours and motion temporal features in video streams while shielding static environmental noise.

[0012] Unauthorized intruder identification model: It integrates YOLOv8 object detection and OpenPose human pose estimation model to extract temporal features of human skeleton and remove irrelevant interference pixels.

[0013] Geological disaster displacement identification model: Combining optical flow method and LSTM temporal network, the model analyzes the displacement changes of surface pixels in video stream, weakening the influence of instantaneous environmental noise.

[0014] Step S4: After an abnormal target is identified, a preliminary analysis is performed on the abnormal target. If the abnormal target is an illegally intruding vehicle or an illegally intruding person, video data from a preset time window before the abnormal target is identified is retrieved to obtain the abnormal target's actions and behaviors. The actions and behaviors are then filtered for strong correlations to identify key actions. In subsequent monitoring, if a target with key actions appears, an early warning is issued for the abnormal target if no abnormality occurs.

[0015] If the abnormal target is a natural fire or geological disaster displacement, environmental data within a preset time window before abnormal target identification is obtained. By analyzing the strong correlation between the environmental data and the abnormal target, strong correlation data of the abnormal target is obtained. In the subsequent monitoring process, if strong correlation data of the abnormal target appears, an early warning of the abnormal target is issued even if no abnormality occurs.

[0016] The advantages of this solution are: 1. This solution combines proactive early warning with tiered response to enhance the initiative and relevance of early warning: By mining key behaviors and environmental data before anomalies occur, the early warning node is moved forward. At the same time, differentiated response strategies are formulated based on the degree of danger of anomaly symptoms to optimize resource allocation and ensure rapid handling of high-risk anomalies.

[0017] 2. Trajectory tracking and dynamic monitoring closed loop to strengthen anomaly control: Track the trajectory and predict the path of abnormal targets, construct a dynamic monitoring area, and form a closed loop mechanism of identification, tracking, prediction and upgrading to effectively prevent abnormal targets from escaping or causing secondary risks.

[0018] 3. Source tracing analysis: By retrieving historical data, we can mine strong correlation factors of anomalies, clarify the underlying causes of anomalies, provide data support for the prevention of similar anomalies at their source, and improve the long-term effectiveness of land management.

[0019] Furthermore, in step S4, when an illegally intruding vehicle is identified, video data of the illegally intruding vehicle 30 minutes ago is extracted, and the vehicle's failure to slow down and frequent lane changes during that period are identified as key actions. When an illegally intruding person is identified, video data of the illegally intruding person 30 minutes ago is extracted, and the person's lingering and looking around, loitering, covering their face, and carrying suspicious tools during that period are identified as key actions.

[0020] Beneficial effects: This solution accurately identifies the core pre-emptive behaviors of human anomalies, providing clear data for quickly identifying illegal intrusion targets, avoiding identification delays caused by ambiguous behavior judgments, and improving the timeliness and accuracy of pre-emptive warnings.

[0021] Furthermore, in step S4, a human pose estimation model is used to extract the motion trajectory of key skeletal points of the unauthorized intruder, accurately reconstructing the temporal characteristics of the intruder's behavior; for the unauthorized vehicle, a vehicle behavior analysis model is used to identify the vehicle's driving status, including acceleration characteristics, turning frequency, and dwell time.

[0022] Furthermore, for individuals carrying suspicious tools, vehicles speeding in, or deliberately obscuring their faces, drones will be dispatched to the scene to record the entire incident and issue warnings via loudspeaker. For individuals loitering and observing, vehicles making brief stops, or frequently changing lanes, the data sampling frequency will be increased to twice the normal frequency to enhance feature extraction density. For vehicles approaching at low speeds or individuals occasionally glancing around, key feature monitoring will be maintained, and only key behavioral feature parameters will be extracted without issuing advance warnings.

[0023] Furthermore, in step S4, when natural fires and geological disaster displacements are considered anomalous targets, the confirmation steps for strongly correlated data of the anomalous targets include: Step S41: Divide the environmental data from the 72 hours prior to disaster identification into a background baseline layer, anomaly initiation layer, and anomaly enhancement layer; Step S42: Calculate the intensity of temperature change, wind speed anomaly deviation rate, and air pressure fluctuation frequency ratio of the anomalous target for the background reference layer, the anomalous germination layer, and the anomalous reinforcement layer, respectively. Step S43: Using environmental data characteristics such as the intensity of temperature change, the rate of wind speed anomaly deviation, and the ratio of air pressure fluctuation frequency as the dependent variable sequence, and the occurrence of disaster as the result variable sequence, construct a dynamic time series regression model, introduce a time lag term, and capture the temporal correlation between environmental data anomalies and the occurrence of disasters. Step S44: Perform a causal test on the abnormal germination layer and the abnormal reinforcement layer; For the anomaly incubation layer: verify the causal relationship between slight environmental data anomalies and the increased probability of disaster occurrence, calculate the lag correlation coefficient between the anomaly characteristics of the incubation layer and the occurrence of disaster. If the lag correlation coefficient is ≥0.3, it is considered that there is a potential causal relationship. For the anomaly reinforcement layer: verify the causal relationship between significant environmental data anomalies and a sharp increase in the probability of disaster occurrence, calculate the instantaneous correlation coefficient between the anomaly characteristics of the reinforcement layer and the occurrence of disaster. If the instantaneous correlation coefficient is ≥0.6, it is considered that there is a strong causal relationship. Cross-level verification: If the same environmental data shows a potential causal relationship at the budding layer and a strong causal relationship at the reinforcement layer, and the intensity of temperature change increases synchronously with the probability of disaster occurrence, then it is initially determined to be a true causal relationship; if the relationship trend does not conform to the preset disaster evolution model, it is regarded as abnormal interference data and the interference data is removed. Step S45: Elimination of cross-interference: To address the mutual influence between environmental data, a feature separation matrix is ​​introduced to orthogonally decompose the environmental data of the background baseline layer, anomaly budding layer, and anomaly enhancement layer, eliminate cross-interference terms, and recalculate the causal correlation coefficient between each environmental data and the occurrence of the disaster. Step S46, Relevance Calculation: The relevance is calculated using the following formula: R0 = (0.3k + 0.7p) s, R0 is the correlation degree, k is the correlation coefficient of the budding layer, p is the correlation coefficient of the strengthening layer, and s is the correction coefficient, the value of which ranges from 0.8 to 1.2; Step S47: Key data screening: Set the correlation threshold to 70% and screen out environmental data with a correlation R0 greater than 70% as strongly correlated data of abnormal targets.

[0024] Beneficial effects: It clarifies the strong correlation between target anomalies and environmental factors, predicts the risk of natural anomalies by analyzing the temporal changes in environmental data, solves the problem of delayed early warning of natural anomalies, and buys time for early prevention of natural disasters.

[0025] Furthermore, after step S3, there is also a tracking step after the abnormal target is identified. Specifically, the movement trajectory of illegal dumping vehicles, construction sabotage personnel, or arsonists is tracked, the movement path is stored in the form of a sequence of movement trajectory coordinate points, the subsequent movement path is predicted and an overlapping regulatory area is constructed, the starting point of the predicted path is used as the core diffusion source, and the outline of the regulatory range is calculated and delineated in real time based on the movement speed, turning probability of the abnormal target, and the accessibility and concealment factors of the surrounding environment. If the actual movement path deviates from the predicted path by more than a threshold and exceeds the overlapping regulatory area, the risk level of the abnormal target is increased.

[0026] Beneficial effects: Enables full-process control of abnormal targets from identification to tracking. By predicting paths and constructing dynamic monitoring areas, it can lock in the possible spread range in advance. At the same time, it sets up a path deviation early warning mechanism to respond to the sudden behavior of abnormal targets in a timely manner and avoid loss of control.

[0027] Furthermore, the prediction of the subsequent movement path adopts a hybrid algorithm combining LSTM (Long Short-Term Memory) network and Kalman filtering. The algorithm takes into account the historical trajectory of the abnormal target, real-time positioning, motion parameters and environmental constraints. LSTM learns the temporal pattern of motion, Kalman filtering corrects positioning errors, and invalid paths are eliminated by combining terrain and obstacles. The algorithm updates the predicted path every 5 seconds with a sampling interval of 1 second, predicts the movement trajectory for the next 30 seconds, and provides data support for the calculation of the monitored area.

[0028] To implement the above-mentioned abnormal target early warning method, this application also provides an abnormal target early warning system based on multimodal video streams, characterized in that it includes: Multimodal data acquisition module: including visible light camera, thermal imaging sensor and GNSS positioning module mounted on the monitoring equipment, used to collect visible light video stream, thermal imaging video stream, real-time positioning data stream, environmental data and GIS resource data of the protected area, to ensure the comprehensiveness and synchronization of data acquisition; The data preprocessing module is used to realize time synchronization, noise filtering, positioning correction and data standardization of multimodal data streams. At the same time, it completes the gridding of the protected area and assigns spatial coordinates, terrain attributes and scene feature labels to provide standardized data input for subsequent recognition algorithms. Anomaly Target Recognition Algorithm Library: Stored on the remote monitoring platform, this library contains recognition algorithm models for natural fires, illegally intruding vehicles, illegally intruding personnel, and geological disaster displacement; used to identify various anomaly targets. Abnormal Behavior Source Tracing and Analysis Module: Used to retrieve historical multimodal data, and achieve strong correlation factor mining, historical action parsing, and natural data tracing through association rule mining and time series feature analysis.

[0029] Beneficial effects: The system's modules have clear division of labor and work together in a coordinated manner, forming a complete technical chain from data collection, preprocessing, feature extraction, fusion recognition to source tracing analysis. This ensures efficient processing and in-depth application of multimodal data, enabling intelligent, precise, and proactive anomaly early warning, and comprehensively improving the technical level and efficiency of land management.

[0030] This application also discloses an abnormal target early warning device based on multimodal video streams, including a memory, a processor, a communication interface, and an encryption module.

[0031] The memory is used to store computer programs and multimodal acquisition data, preprocessed data, feature data, and early warning data.

[0032] The communication interface is used for data interaction between the monitoring device and the remote monitoring platform, including data reception and command transmission. The encryption module is used for encrypted storage of abnormal video clips, early warning data, trajectory reports, and source tracing analysis data.

[0033] The processor, when executing the computer program, provides an abnormal target warning method for a multimodal video stream.

[0034] This application also discloses a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements an abnormal target early warning method based on multimodal video streams. Attached Figure Description

[0035] Figure 1 This is a flowchart of the abnormal target early warning method based on multimodal video stream in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the abnormal target early warning system based on multimodal video stream in Embodiment 2 of the present invention. Detailed Implementation

[0036] The following detailed description illustrates the specific implementation method: Example 1: As attached Figure 1 The method for early warning of abnormal targets based on multimodal video streams, as shown, is characterized by the following steps: Step S1: Multimodal data acquisition, acquiring visible light video stream, thermal imaging video stream, real-time positioning data stream and environmental data of the protected area, including temperature, wind speed and air pressure data.

[0037] Step S2: Data preprocessing, timestamp synchronization calibration of the collected multimodal data, removal of blurry and noisy frames from visible light video stream and thermal imaging video stream, gridding of the management area space, assignment of spatial coordinates, terrain attributes and scene feature labels to each grid, and completion of data standardization processing.

[0038] Step S3: Input the multimodal data into the abnormal target identification algorithm library. The abnormal target identification algorithm library stores identification algorithm models for natural fires, illegally intruding vehicles, illegally intruding personnel, and geological disaster displacement. Each identification algorithm model identifies the confidence level of the abnormal target in parallel based on the input multimodal data. When the confidence level of the identified target in any identification algorithm model is greater than 80%, the abnormal target, confidence level, location of occurrence, and risk level are output, and an early warning is issued.

[0039] After identifying abnormal targets, the system also tracks them, including the movement trajectories of vehicles illegally dumping soil, construction saboteurs, or arsonists. The movement paths are stored as a sequence of coordinate points, and subsequent movement paths are predicted. Overlapping regulatory areas are constructed, with the starting point of the predicted path as the core diffusion source. Based on the abnormal target's movement speed, turning probability, and the accessibility and concealment of the surrounding environment, the regulatory scope is calculated and outlined in real time. If the actual movement path deviates from the predicted path by more than a threshold and exceeds the overlapping regulatory area, the risk level of the abnormal target is increased.

[0040] The temporal trajectory modeling algorithm used for movement path prediction is a hybrid algorithm combining LSTM long short-term memory network and Kalman filtering.

[0041] Algorithm input: the coordinate sequence of the historical movement trajectory of the abnormal target, real-time positioning data, movement speed, turning angle, and environmental constraints such as surrounding terrain, obstacles, and passage areas.

[0042] Operational logic: The LSTM network learns the historical movement time sequence of the target, Kalman filtering corrects positioning data errors, and environmental constraints are combined to eliminate impassable paths.

[0043] Dynamic update rule: With a data sampling interval of 1 second, the latest real-time data is imported every 5 seconds, the predicted path is recalculated and updated, and the prediction step size is the movement trajectory for the next 30 seconds; Linkage rules: The predicted path is synchronously output to the regulatory scope calculation module, and a dynamic regulatory area is constructed in conjunction with the adaptive boundary generation algorithm.

[0044] Step S4: Analyze the key actions of the abnormal target before it is identified. Based on the abnormal identification results, retrieve the historical video data, environmental data and positioning trajectory data of the preset time window before the abnormal target occurs. Use the key data before the abnormal target is identified as the strong correlation data of the abnormal target. In the subsequent abnormal target identification, when the strong correlation data of the abnormal target appears, the abnormal target identification model can identify the abnormal target in advance based on the strong correlation data of the abnormal target.

[0045] For unusual targets such as illegally intruding vehicles and individuals, key data to be extracted from illegally intruding vehicles includes actions such as vehicles not slowing down when entering and frequently changing lanes. Key data to be extracted from illegally intruding individuals includes actions such as lingering and observing, loitering, covering their faces, and carrying suspicious tools.

[0046] A human pose estimation model using a YOLOv8+OpenPose fusion architecture was used to extract the motion trajectory of 17 key skeletal points of an unauthorized intruder, accurately reconstructing the temporal characteristics of the intruder's behavior. A vehicle behavior analysis model based on the SVM algorithm was used to identify the acceleration characteristics, turning frequency, and dwell time of unauthorized intruding vehicles.

[0047] For vehicles carrying suspicious tools, speeding in, or people deliberately obscuring their faces, trigger an emergency drone to fly to the scene, record the entire process, and issue a warning via loudspeaker; for people loitering and looking around, vehicles making brief stops, or frequently changing lanes, increase the data sampling frequency to twice the normal amount and strengthen the feature extraction density; for vehicles approaching at low speeds or people occasionally looking around, maintain monitoring of key features and extract only key behavioral feature parameters.

[0048] The specific implementation process for identifying the correlation data of abnormal targets caused by natural fires and geological disaster displacement is as follows: Step S41: Divide the environmental data from the 72 hours prior to disaster identification into three layers based on time: Background Baseline Layer: Environmental data from 72 to 48 hours prior to disaster identification, representing normal environmental indicators for the region; Anomaly Initiation Layer: Environmental data from 48 to 24 hours prior to disaster identification, representing the stage where slight environmental anomalies appear; Anomaly Intensification Layer: Environmental data from 24 hours prior to disaster identification to the moment of disaster identification, representing the stage where environmental anomalies continue to intensify. The completeness of data at each layer is verified, and missing data is filled in through interpolation.

[0049] Step S42: Calculate the intensity of temperature change, wind speed anomaly deviation rate, and air pressure fluctuation frequency ratio of anomalous targets in the background reference layer, anomalous germination layer, and anomalous enhancement layer, respectively. Wherein, temperature change intensity = |current layer average temperature Average temperature of background reference layer | ÷ Average temperature of background reference layer × 100%; Wind speed anomaly deviation rate = | Average wind speed of current layer Average wind speed at the background reference layer | ÷ Average wind speed at the background reference layer × 100%; Pressure fluctuation frequency ratio = Current layer pressure fluctuation frequency ÷ Background reference layer pressure fluctuation frequency.

[0050] Step S43: Using the intensity of temperature change, the rate of wind speed anomaly deviation, and the ratio of air pressure fluctuation frequency as the dependent variable sequence, and whether a disaster has occurred as the result variable sequence (1 for occurrence, 0 for non-occurrence), construct a dynamic time series regression model based on LSTM, introduce 1 to 3 time lag terms, capture the temporal correlation between environmental data anomalies and disaster occurrence, and ensure that the model training goodness of fit R² ≥ 0.85.

[0051] Step S44: Use the Granger causality test to perform causality tests at different levels and by disaster type. Anomaly incubation layer: Calculate the lag correlation coefficient between the anomaly characteristics of the incubation layer and the occurrence of the disaster, with a value range of 0 to 1. The higher the value, the stronger the potential risk. A threshold ≥ 0.3 indicates the existence of a potential causal relationship. Anomaly reinforcement layer: Calculate the synchronous correlation coefficient between the environmental anomaly characteristics of the reinforcement layer and the occurrence of the disaster, with a value range of 0 to 1. The higher the value, the greater the probability of the disaster outbreak. A threshold ≥ 0.6 indicates the existence of a strong causal relationship.

[0052] Cross-level verification: If the same environmental data shows a potential causal relationship at the nascent layer and a strong causal relationship at the intensified layer, and the intensity of temperature abrupt changes increases over time, synchronously increasing with the probability of disaster occurrence, then it is preliminarily determined to be a true causal relationship; if the relationship trend does not conform to the preset disaster evolution process, it is considered abnormal interference data and is removed. The disaster evolution process refers to the existing technologies for landslide evolution and natural fire evolution.

[0053] Step S45: Cross-interference elimination: To address the mutual influence between temperature, wind speed, and air pressure data, a 3×3 dimension feature separation matrix is ​​introduced. Principal component analysis is used to orthogonally decompose the features of the three environmental data, eliminating cross-interference terms between features. After decomposition, the variance contribution rate of each feature is ≥85%. Then, based on the decomposed pure features, the causal correlation coefficient between each environmental data and the disaster is recalculated.

[0054] Step S46: Calculate the correlation degree using the formula R0 = (0.3k + 0.7p). s calculates the correlation degree, where R0 is the correlation degree, k is the correlation coefficient of the budding layer, p is the correlation coefficient of the strengthening layer, and s is the correction coefficient.

[0055] The lag correlation coefficient k is the temporal correlation coefficient between the environmental anomalies at the budding stage and the occurrence of disasters. The value ranges from 0 to 1, with higher values ​​indicating stronger potential risks. A threshold of ≥0.3 indicates the existence of a potential causal relationship.

[0056] Real-time correlation coefficient p: The synchronous correlation coefficient between the abnormal characteristics of the enhanced layer environment and the occurrence of disasters. The value ranges from 0 to 1. The higher the value, the greater the probability of disaster outbreak. A threshold of ≥0.6 is used to determine that there is a strong causal relationship.

[0057] Correction coefficient s: A correction parameter used to compensate for sensor acquisition errors and accidental environmental disturbances. Its value ranges from 0.8 to 1.2 and is adaptively selected according to the accuracy of the field equipment.

[0058] Step S47: Key Data Screening: Set the correlation threshold to 70%, round the calculated correlation R0 to one decimal place, and screen environmental data with a correlation R0 greater than 70% as strong correlation data for abnormal targets. For example, strong correlation data for natural fires are temperature change intensity ≥ 40% and wind speed anomaly deviation rate ≥ 30%, and strong correlation data for geological disaster displacement are air pressure fluctuation frequency ratio ≥ 1.5 and wind speed anomaly deviation rate ≥ 40%.

[0059] Example 2: like Figure 2 The system shown is an abnormal target early warning system based on multimodal video streams, including a multimodal data acquisition module. The multimodal data acquisition module includes a visible light camera, a thermal imaging sensor, and a GNSS positioning module mounted on the monitoring equipment. It is used to collect visible light video streams, thermal imaging video streams, real-time positioning data streams, environmental data, and GIS resource data of the protected area, ensuring the comprehensiveness and synchronization of data acquisition.

[0060] Fixed monitoring equipment is installed at the boundaries and key nodes of the management area, such as mountainous areas with a slope greater than 25°, concentrated forest areas, and entrances and exits of idle plots. Each set of monitoring equipment includes a high-definition visible light camera, an infrared thermal imaging sensor, a GNSS positioning module, and an environmental sensor. At the same time, regional GIS resource data is accessed, including information such as topographic elevation, road distribution, vegetation cover, and building locations.

[0061] Three ground control stations are set up within the protected area, equipped with high-definition video recording equipment and remote broadcasting devices; four multi-rotor drones are deployed, with a flight time of 60 minutes, a maximum flight speed of 15 m / s, and equipped with dual visible light and thermal imaging cameras for on-site verification and tracking in emergency scenarios.

[0062] The management center deploys a remote monitoring platform, equipped with a data preprocessing module, a feature extraction module, a dynamic weighted fusion module, an abnormal behavior tracing and analysis module, an abnormal target identification algorithm library, and data storage devices. Data interaction with front-end monitoring equipment is achieved via a 5G network, with transmission latency controlled within 50ms. An encryption module is also configured, employing the AES-256 encryption algorithm to encrypt and store sensitive data during transmission.

[0063] The data preprocessing module is used to realize time synchronization, noise filtering, positioning correction and data standardization of multimodal data streams. At the same time, it completes the gridding of the management area and assigns spatial coordinates, terrain attributes and scene feature labels, providing a standardized data foundation for subsequent feature extraction and fusion.

[0064] The abnormal target recognition algorithm library stores recognition algorithm models for natural fires, illegally intruding vehicles, illegally intruding personnel, and geological disaster displacement; these are used to identify various abnormal targets.

[0065] The natural fire identification model, based on an improvement of YOLOv8, extracts thermal imaging temperature features and visual contour features from visible light video streams and thermal imaging video streams, filtering out invalid background noise. It takes temperature distribution features and visual contour features from visible light video streams and thermal imaging video streams as input, performs preliminary screening of suspected fire points through heat map threshold segmentation, and performs secondary verification by combining flame contour features, such as the irregular edges of flames, the dynamic change rate of flames, and environmental data, and outputs the fire identification confidence score.

[0066] The geological hazard displacement identification model combines optical flow and LSTM temporal network. It takes visible light video stream and monitoring target location data as input, analyzes the displacement changes of surface pixels, weakens the impact of instantaneous environmental noise, and combines GIS topographic data to calculate the displacement of surface pixels. When the average displacement of 5 consecutive frames is ≥5cm and the affected area exceeds 3 grids, a landslide warning is triggered. At the same time, wind speed and air pressure environmental data are incorporated to optimize the identification accuracy.

[0067] An illegal intrusion vehicle identification model is developed based on an SSD target detection and behavior analysis network. It takes visible light video stream and thermal imaging video stream data as input, analyzes the target contour and motion temporal features, extracts vehicle contour, size and driving trajectory features, and combines the vehicle behavior analysis model to identify key actions such as entering without deceleration and frequent lane changes, and outputs the vehicle intrusion confidence score.

[0068] The unauthorized intruder identification model integrates YOLOv8 target detection and OpenPose human pose estimation models. It takes the input visible light video stream and thermal imaging video stream data of the monitoring area as input. After detecting a human body, it extracts the motion trajectory of 17 key skeletal points of the human body, removes irrelevant interference pixels, and identifies actions such as lingering and looking around, loitering, obscuring the face, and carrying suspicious tools through temporal feature analysis, and outputs the confidence level of the intruder.

[0069] The algorithm library has a built-in pre-denoising unit that removes redundant noise data generated during the fusion process through Gaussian filtering and inter-frame differencing. Multimodal data is input in parallel to four independent recognition models; the models output the identified target, confidence level, and pixel coordinates in real time, which can be converted into actual geographical locations. Based on preset rules, risk levels are classified. When the confidence level of any model is greater than 80%, the linkage warning module completes the audible and visual warning and data reporting.

[0070] Abnormal Behavior Source Tracing and Analysis Module: Used to retrieve historical multimodal data, and achieve strong correlation factor mining, historical action parsing, and natural data tracing through association rule mining and time series feature analysis.

[0071] The temporal trajectory modeling algorithm uses a Long Short-Term Memory (LSTM) network combined with Kalman filtering. It takes the historical trajectory coordinates of the abnormal target as input, sampling interval of 1 second, and data on movement speed and turning angle to construct a trajectory prediction model. The real-time positioning data is updated every 5 seconds to dynamically correct the predicted path, and the prediction step size is set to the movement path in the next 30 seconds.

[0072] The adaptive boundary generation algorithm, with the predicted path start point as the core, calculates the accessibility coefficient of the surrounding area based on GIS terrain data. The coefficients are as follows: slope less than 15° is 1.0, 15° to 30° is 0.7, and greater than 30° is 0.3; no obstacles are 1.0, and obstacles are 0.5. The concealment coefficient is 0.9 if vegetation coverage is greater than or equal to 80%, 0.6 if 50% to 80%, and 0.3 if less than 50%; building obstruction is 0.8, and no obstruction is 0.3. The monitoring range outline is calculated using the following formula: R = v × t × (k1 × P + k2 × Q), where v is the abnormal target movement speed, t is the prediction time, k1 is the accessibility weight coefficient, k2 is the concealment weight coefficient (satisfying k1 + k2 = 1), P is the accessibility coefficient, and Q is the concealment coefficient. In this embodiment, k1 = 0.6 and k2 = 0.4. Based on the calculation results, a polygon fitting algorithm is used to outline the monitoring range, and the accuracy and range are adjusted every 10 seconds.

[0073] Causal correlation analysis algorithm: The Granger causality test algorithm is used to analyze the causal relationship between environmental data such as the magnitude of temperature change, duration of continuous abnormal wind speed, and frequency of air pressure fluctuation and natural fire and geological disaster displacement. The causal correlation threshold is set at 70%. When the probability value corresponding to the Granger causality test statistic of environmental data and abnormal event is less than 0.05 and the correlation strength coefficient is ≥0.7, it is judged as strongly correlated data.

[0074] The human pose estimation model adopts a YOLOv8+OpenPose fusion architecture, accurately extracting 17 key skeleton points with a skeleton point localization error of ≤2px and a temporal feature restoration accuracy of 92%. The vehicle behavior analysis model, based on the SVM algorithm, takes acceleration characteristics, steering frequency, and dwell time as input parameters, achieving a behavior recognition accuracy of 89%.

[0075] Example 3: An abnormal target early warning device based on multimodal video streams includes a memory, a processor, a communication interface, and an encryption module. The memory stores computer programs, multimodal acquisition data, preprocessed data, feature data, and early warning data. The communication interface facilitates data interaction between the monitoring device and a remote monitoring platform, including data reception and command transmission. The encryption module encrypts and stores abnormal video segments, early warning data, trajectory reports, and source analysis data. The processor, when executing the computer program, implements the abnormal target early warning method based on multimodal video streams described in Embodiment 1.

[0076] Example 4: A computer-readable storage medium is provided for storing a computer program, which, when executed by a processor, is used to implement the abnormal target warning method based on multimodal video streams in Embodiment 1.

[0077] Computer-readable storage media include USB flash drives, portable hard drives, solid-state drives (SSDs), optical discs, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. The storage capacity of the storage media is ≥16G, and it adopts the FAT32 / NTFS / exFAT file system.

[0078] The computer-readable storage medium stores a computer program, which is an executable program written in Python / C++. The program includes code for functional modules such as data acquisition, preprocessing, feature extraction, feature fusion, anomaly recognition, trajectory tracking, and source analysis. The computer program has ≥100,000 lines of code and is compiled to generate an executable file that supports running on multiple operating systems such as Windows / Linux / Android embedded systems.

[0079] When the computer program is executed by the processor, it implements all the steps of the abnormal target early warning method based on multimodal video stream as described in claims 1-7 and embodiment 1.

[0080] In addition, the computer program also includes an installation package, configuration files, model weight files, user manuals and other auxiliary files. The model weight files are the weights of the trained and converged abnormal target recognition algorithm model. The configuration files allow users to adjust parameters such as data collection frequency, feature extraction parameters, confidence threshold, and correlation threshold according to the actual management and maintenance scenario, thereby improving the adaptability of the program.

[0081] The above are merely embodiments of the present invention. Commonly known structures and characteristics are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. An abnormal target early warning method based on multimodal video stream, characterized in that, Includes the following steps: Step S1: Multimodal data acquisition, acquiring visible light video stream, thermal imaging video stream, monitoring target location data, and environmental data of the protected area, including temperature, wind speed, and air pressure data; Step S2: Data preprocessing, timestamp synchronization calibration of the collected multimodal data, removal of blurry and noisy frames from visible light video stream and thermal imaging video stream, gridding of the management area space, assignment of spatial coordinates, terrain attributes and scene feature labels to each grid, and completion of data format standardization. Step S3: Input the multimodal data into the abnormal target identification algorithm library. The abnormal target identification algorithm library stores identification algorithm models for natural fires, illegally intruding vehicles, illegally intruding personnel, and geological disaster displacement. Each identification algorithm model identifies the confidence level of the abnormal target in parallel based on the input multimodal data. When the confidence level of the identified target in any identification algorithm model is greater than 80%, the abnormal target, confidence level, location, and risk level are output, and an early warning is issued. Step S4: After an abnormal target is identified, a preliminary analysis is performed on the abnormal target. If the abnormal target is an illegally intruding vehicle or an illegally intruding person, video data from a preset time window before the abnormal target is identified is retrieved to obtain the abnormal target's actions and behaviors. The actions and behaviors are then filtered for strong correlations of abnormal occurrences to obtain key actions. In the subsequent monitoring process, if a target with key actions appears, an early warning for the abnormal target is issued even if no abnormality occurs. If the abnormal target is a natural fire or geological disaster displacement, environmental data within a preset time window before abnormal target identification is obtained. By analyzing the strong correlation between the environmental data and the abnormal target, strong correlation data of the abnormal target is obtained. In the subsequent monitoring process, if strong correlation data of the abnormal target appears, an early warning of the abnormal target is issued even if no abnormality occurs.

2. The abnormal target early warning method according to claim 1, characterized in that, In step S4, when an illegally intruding vehicle is identified, video data of the illegally intruding vehicle from 30 minutes ago is extracted, and the vehicle's failure to slow down and frequent lane changes during that period are identified as key actions. When an illegally intruding person is identified, video data of the illegally intruding person from 30 minutes ago is extracted, and the person's lingering and looking around, loitering, covering their face, and carrying suspicious tools during that period are identified as key actions.

3. The abnormal target early warning method according to claim 2, characterized in that, In step S4, a human pose estimation model is used to extract the motion trajectory of key skeletal points of the unauthorized intruder, accurately reconstructing the temporal characteristics of the intruder's behavior; for the unauthorized vehicle, a vehicle behavior analysis model is used to identify the vehicle's driving status, including acceleration characteristics, turning frequency, and dwell time.

4. The abnormal target early warning method according to claim 3, characterized in that, For vehicles carrying suspicious tools, speeding in, or people deliberately obscuring their faces, drones will be dispatched to the scene to record the entire incident and issue warnings via loudspeaker. For people loitering and observing, vehicles making brief stops, or frequently changing lanes, the data sampling frequency will be increased to twice the normal frequency to enhance the density of feature extraction. For vehicles approaching at low speeds or people occasionally glancing around, key feature monitoring will be maintained, and only key behavioral feature parameters will be extracted without issuing advance warnings.

5. The abnormal target early warning method according to claim 1, characterized in that, In step S4, when natural fire and geological disaster displacement are considered anomalous targets, the steps for determining the strongly correlated data of the anomalous targets include: Step S41: Divide the environmental data for the 72 hours prior to disaster identification into a background baseline layer, anomaly initiation layer, and anomaly reinforcement layer; Background baseline layer: Environmental data from 72 hours to 48 hours prior to disaster identification, representing the normal environmental indicators of the region; Anomaly initiation layer: Environmental data from 48 hours to 24 hours prior to disaster identification, representing the stage where the environment shows slight anomalies; Anomaly reinforcement layer: Environmental data from 24 hours prior to disaster identification to the time of disaster identification, representing the stage where environmental anomalies continue to intensify; Step S42: Calculate the intensity of temperature change, wind speed anomaly deviation rate, and air pressure fluctuation frequency ratio of anomalous targets in the background reference layer, anomalous germination layer, and anomalous enhancement layer, respectively. Wherein, temperature change intensity = |current layer average temperature Average temperature of background reference layer | ÷ Average temperature of background reference layer × 100%; Wind speed anomaly deviation rate = | Average wind speed of current layer Average wind speed at the background reference layer | ÷ Average wind speed at the background reference layer × 100%; Pressure fluctuation frequency ratio = Current layer pressure fluctuation frequency ÷ Background reference layer pressure fluctuation frequency; Step S43: Using the intensity of temperature change, the rate of wind speed anomaly deviation, and the ratio of air pressure fluctuation frequency as the dependent variable sequence, and whether a disaster has occurred as the result variable sequence (1 for occurrence, 0 for non-occurrence), construct a dynamic time series regression model based on LSTM, introduce 1 to 3 time lag terms, capture the time correlation between environmental data anomalies and disaster occurrence, and the model training goodness of fit R² ≥ 0.

85. Step S44: Perform a causal test on the abnormal germination layer and the abnormal reinforcement layer; For the anomaly incubation layer: verify the causal relationship between slight environmental data anomalies and the increased probability of disaster occurrence, calculate the lag correlation coefficient between the anomaly characteristics of the incubation layer and the occurrence of disaster. If the lag correlation coefficient is ≥0.3, it is considered that there is a potential causal relationship. For the anomaly reinforcement layer: verify the causal relationship between significant environmental data anomalies and a sharp increase in the probability of disaster occurrence, calculate the instantaneous correlation coefficient between the anomaly characteristics of the reinforcement layer and the occurrence of disaster. If the instantaneous correlation coefficient is ≥0.6, it is considered that there is a strong causal relationship. Cross-level verification: If the same environmental data shows a potential causal relationship at the budding layer and a strong causal relationship at the reinforcement layer, and the intensity of temperature change increases synchronously with the probability of disaster occurrence, then it is initially determined to be a true causal relationship; if the relationship trend does not conform to the preset disaster evolution model, it is regarded as abnormal interference data and the interference data is removed. Step S45: Elimination of cross-interference: To address the mutual influence between environmental data, a feature separation matrix is ​​introduced to orthogonally decompose the environmental data of the background baseline layer, anomaly budding layer, and anomaly enhancement layer, eliminate cross-interference terms, and recalculate the causal correlation coefficient between each environmental data and the occurrence of the disaster. Step S46, Relevance Calculation: The relevance is calculated using the following formula: R0 = (0.3k + 0.7p) s, R0 is the correlation degree, k is the correlation coefficient of the budding layer, p is the correlation coefficient of the strengthening layer, and s is the correction coefficient, the value of which ranges from 0.8 to 1.2; Step S47: Key data screening: Set the correlation threshold to 70% and screen out environmental data with a correlation R0 greater than 70% as strongly correlated data of abnormal targets.

6. The abnormal target early warning method according to claim 1, characterized in that, After step S3, there is also a tracking step after the abnormal target is identified. Specifically, the movement trajectory of illegal dumping vehicles, construction sabotage personnel or arsonists is tracked, the movement path is stored in the form of a sequence of movement trajectory coordinate points, the subsequent movement path is predicted and an overlapping monitoring area is constructed, the starting point of the predicted path is used as the core diffusion source, and the monitoring range outline is calculated and delineated in real time based on the movement speed, turning probability of the abnormal target and the accessibility and concealment of the surrounding environment. If the actual movement path deviates from the predicted path by more than a threshold and exceeds the overlapping regulatory area, the risk level of the abnormal target will be increased.

7. The abnormal target early warning method according to claim 6, characterized in that, The algorithm for predicting the subsequent movement path uses a hybrid algorithm combining LSTM (Long Short-Term Memory) network and Kalman filtering. The algorithm takes into account the historical trajectory of the abnormal target, real-time positioning, motion parameters, and environmental constraints. LSTM learns the temporal pattern of motion, Kalman filtering corrects positioning errors, and invalid paths are eliminated by combining terrain and obstacle information. The algorithm updates the predicted path every 5 seconds with a sampling interval of 1 second, predicts the movement trajectory for the next 30 seconds, and provides data support for the calculation of the monitored area.

8. An abnormal target early warning system based on multimodal video streams, characterized in that, Implementing the abnormal target early warning method based on multimodal video stream as described in any one of claims 1 to 7, comprising: Multimodal data acquisition module: including visible light camera, thermal imaging sensor and GNSS positioning module mounted on the monitoring equipment, used to collect visible light video stream, thermal imaging video stream, real-time positioning data stream, environmental data and GIS resource data of the protected area, to ensure the comprehensiveness and synchronization of data acquisition; The data preprocessing module is used to realize time synchronization, noise filtering, positioning correction and data standardization of multimodal data streams. At the same time, it completes the gridding of the protected area and assigns spatial coordinates, terrain attributes and scene feature labels to provide standardized data input for subsequent recognition algorithms. Anomaly Target Recognition Algorithm Library: Stored on the remote monitoring platform, this library contains recognition algorithm models for natural fires, illegally intruding vehicles, illegally intruding personnel, and geological disaster displacement; used to identify various anomaly targets. Abnormal Behavior Source Tracing and Analysis Module: Used to retrieve historical multimodal data, and achieve strong correlation factor mining, historical action parsing, and natural data tracing through association rule mining and time series feature analysis.

9. An abnormal target early warning device based on multimodal video stream, characterized in that, include: The memory is used to store computer programs, multimodal acquisition data, preprocessed data, feature data, and early warning data. The communication interface is used for data interaction between the monitoring equipment and the remote monitoring platform, including data reception and command transmission; The encryption module is used to encrypt and store abnormal video clips, early warning data, trajectory reports, and source tracing analysis data. The processor, when executing the computer program, implements the abnormal target early warning method based on multimodal video stream as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The device contains a computer program that, when executed by a processor, implements the abnormal target warning method based on multimodal video streams as described in any one of claims 1-7.

Images