Cloud multi-modal consistency constraint-based key abnormal event identification method and system

Abstract

The application discloses a cloud multimodal consistency constraint key abnormal event identification method and system. For the multimodal time series data generated by cameras, environmental sensors and other devices in smart city, industrial internet of things and other scenes, a completely unsupervised event-level abnormal understanding framework is proposed: through a multimodal consistency constraint mechanism, a cooperative abnormal period is accurately positioned; the abnormal period is clustered online to form an event cluster; based on Z-score significance analysis, the most abnormal feature dimension in each cluster is extracted and mapped into a human-readable semantic description; combined with the three indexes of the number of affected devices, duration and state deviation, the keyness of the event is quantified, and a ranked list is output. The application does not need to predefine event types or label data, can discover, name and sort known and unknown abnormal events at the same time, significantly improves the discovery and response efficiency of key abnormal events, and is suitable for lightweight cloud deployment of large-scale sensing networks.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of artificial intelligence and intelligent monitoring technology, and particularly relates to an unsupervised key abnormal event identification method and system based on cloud multi-modal data collaborative analysis. The present application is deployed on a cloud server, receives multi-modal time series data streams from heterogeneous devices such as video cameras, environmental sensors, Internet of Things terminals and the like in a city perception network, automatically discovers collaborative abnormal patterns without any manually labeled abnormal event labels, generates human understandable event semantic descriptions, quantifies the influence weight of each event on the system state, and outputs a sorted key abnormal event list to provide decision support for intelligent operation and maintenance. BACKGROUND

[0002] With the development of smart city and Internet of Things technology, a large number of heterogeneous sensing devices are deployed in the city environment, continuously generating multi-modal time series data. However, the existing abnormal analysis technology has obvious deficiencies:

[0003] Firstly, supervised event identification methods (such as video behavior recognition based on deep learning) rely on a large amount of labeled data, but abnormal events are sparse and diverse in types, with extremely high labeling costs, and cannot identify new event types that have not been seen in training.

[0004] Secondly, traditional unsupervised anomaly detection methods (such as Isolation Forest, LSTM autoencoder) usually only output "whether abnormal" or "abnormal device list", lacking understanding of the semantic level of abnormality. For example, the system may report "multiple device abnormalities", but cannot explain whether this is "regional fog interference" or "local power failure".

[0005] Thirdly, there is a lack of priority mechanism in multi-event concurrent scenarios. When device offline and environmental interference occur simultaneously, existing systems have difficulty determining which event has the greatest impact on the current business, leading to inefficient allocation of operation and maintenance resources.

[0006] Finally, existing solutions focus on single-device anomalies and ignore the collaborative relationship between devices. For example, fog may cause the image quality of all cameras in the region to decrease, and if judged in isolation, it will be misjudged as multiple independent failures, but in fact it belongs to the same environmental event.

[0007] Therefore, there is an urgent need for an intelligent analysis solution that can automatically discover, name and sort key abnormal events from multi-modal data without labels.

[0008] It is particularly pointed out that although existing research has attempted to use clustering for abnormal grouping, it has failed to solve the following three core problems:

[0009] (1) How to ensure that the clustering object is a real system-level event rather than noise? → Existing methods directly cluster the original data, which is easily disturbed by single-device jitter;

[0010] (2) How to give the clustering results an interpretable semantic? → Most solutions only output "Cluster #3", which cannot be used for human-computer interaction;

[0011] (3) How to evaluate the criticality of events? → Lack of unified quantitative indicators to support operation and maintenance decisions. SUMMARY

[0012] The purpose of the application is to provide a cloud multi-modal consistency constraint key abnormal event recognition method and system, which realizes an unsupervised, interpretable and sortable key abnormal event recognition closed loop by screening high-quality abnormal periods through multi-modal consistency constraint, automatic semantic generation based on statistical significance and three-dimensional key evaluation model. The application aims to solve the following four interrelated technical problems:

[0013] Problem 1: How to accurately identify system-level coordinated anomalies from massive multi-modal data, rather than single-device noise?

[0014] Problem 2: How to automatically assign interpretable semantics to abnormal patterns under unlabeled conditions?

[0015] Problem 3: How to quantify the criticality of different abnormal events to support operation and maintenance priority decisions?

[0016] Problem 4: How to realize a lightweight, efficient and deployable cloud analysis architecture?

[0017] Technical solution: A cloud multi-modal consistency constraint key abnormal event recognition method, comprising the following steps:

[0018] Step 1, receiving multi-modal time series data streams from multiple heterogeneous sensing devices, the data streams including video metadata, environmental sensor data and device status codes.

[0019] Step 2, for the multi-modal time series data stream, locate the coordinated abnormal period through the multi-modal consistency constraint mechanism that fuses autoencoder reconstruction error and cross-modal prediction residual.

[0020] Step 3, unsupervised clustering of the multi-modal feature vectors of the coordinated abnormal period to form multiple abnormal event clusters.

[0021] Step 4, for each abnormal event cluster, calculate its standardized deviation from the normal state in each feature dimension, and select the features of the top K dimensions with the largest deviation.

[0022] Step 5, converting the features of the top K dimensions to natural language descriptions through a pre-set feature identifier-semantic phrase mapping table as semantic labels of abnormal events.

[0023] Step 6, calculate the criticality score based on the number of devices affected, the duration, and the state deviation degree of the event.

[0024] Step 7, output the list of critical abnormal events in descending order of criticality score.

[0025] In step 2, for modalities , the autoencoder reconstruction error is:

[0026] (1)

[0027] wherein, is the data stream of modality t at time , and is the reconstruction result of the data stream of modality t at time .

[0028] The cross-modality prediction residual is calculated as follows:

[0029] The multi-modality feature vector is obtained by processing the multi-modality time series data stream through the autoencoder. For any two different modalities k and j , a lightweight predictor is trained to map modality k to the feature space of modality j; the L2 norm distance between the predicted value and the true value is calculated as the residual term.

[0030] The comprehensive abnormal score obtained by fusing the autoencoder reconstruction error and the cross-modality prediction residual is used as the multi-modality consistency constraint mechanism for fusing the autoencoder reconstruction error and the cross-modality prediction residual. The calculation formula of the comprehensive abnormal score is:

[0031] (2)

[0032] wherein, is the number of modalities, is a cross-modality predictor composed of a single-layer multi-layer perception (MLP), specifically a fully connected neural network containing an input layer, a single hidden layer, and an output layer, used to map from the feature space of modality to the feature space of modality , and α and β are weights, with typical values α = 0.6 and β = 0.4.

[0033] The cross-modality predictor composed of a single-layer multi-layer perception (MLP) has a lightweight and efficient architecture, usually containing only hundreds to thousands of parameters, and is suitable for real-time inference on the cloud.

[0034] Only when multiple modalities simultaneously exhibit high reconstruction error and cross-modality prediction failure, the comprehensive abnormal score only slightly elevated, ensuring that detected are coordinated anomalies.

[0035] In step 4, the standardized deviation is calculated using Z-score:

[0036] (3)

[0037] wherein, is the Z-score value, is the mean and standard deviation of the feature dimension j in the normal state. is the value of the feature dimension j in the state, and are the mean and standard deviation of the feature dimension j in the normal state, respectively.

[0038] In step 6, for each anomaly event e, the criticality score is calculated as:

[0039] (4)

[0040] wherein, is the number of devices affected by the anomaly event e, reflecting the spatial influence range; is the total number of devices, is the event duration, embodying the time persistence; is the maximum event duration, is the average anomaly score in the event window, i.e., the mean of s(t), representing the severity of state deviation; α, β, γ are configurable weights, and α+β+γ=1.

[0041] A cloud-based multi-modal consistency constraint key abnormal event identification system, comprising the following modules:

[0042] A data access module for receiving multi-modal time series data streams from multiple heterogeneous sensing devices, the data streams including video metadata, environmental sensor data, and device status code receiving multi-modal time series data streams;

[0043] A coordinated anomaly detection module configured to process the multi-modal time series data streams, and locate coordinated anomaly periods through a multi-modal consistency constraint mechanism that fuses autoencoder reconstruction error and cross-modal prediction residual;

[0044] An event clustering module configured to perform unsupervised clustering on the multi-modal feature vectors of the coordinated anomaly periods, forming multiple anomaly event clusters;

[0045] A semantic generation module configured to calculate, for each anomaly event cluster, the standardized deviation of each feature dimension relative to the normal state, and select the top K dimensions with the largest deviation, and the features of the top KThe features of the one dimension are converted into natural language description through a preset feature identifier-semantic phrase mapping table;

[0046] a criticality evaluation module for calculating a criticality score based on the event impact device quantity, duration and state deviation degree;

[0047] an event output module for pushing the sorted critical abnormal event list.

[0048] The implementation process of the system is the same as the above method, and will not be repeated.

[0049] A computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the above computer program to realize the steps of the cloud multi-modal consistency constrained critical abnormal event identification method as described above.

[0050] A computer readable storage medium has a computer program / instruction stored thereon, and the computer program / instruction is executed by a processor to realize the steps of the cloud multi-modal consistency constrained critical abnormal event identification method as described above. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 The method flowchart of the embodiments of the present application. DETAILED DESCRIPTION

[0052] The present application will be further illustrated below in conjunction with specific embodiments, which should be understood as merely illustrating the present application and not limiting the scope of the present application. After reading the present application, those skilled in the art can make various modifications to the present application, and all such modifications fall within the scope defined by the appended claims.

[0053] A cloud multi-modal consistency constrained critical abnormal event identification method includes the following steps:

[0054] Step 1, receiving multi-modal time series data streams from multiple heterogeneous perception devices, the data streams including video metadata, environmental sensor data and device state code.

[0055] Step 2, for the multi-modal time series data stream, locate the collaborative abnormal period through a multi-modal consistency constraint mechanism that fuses autoencoder reconstruction error and cross-modal prediction residual.

[0056] Step 3, unsupervised clustering of the multi-modal feature vectors of the collaborative abnormal period to form multiple abnormal event clusters. When the semantic description of all abnormal event clusters cannot match known patterns, it is marked as an unknown abnormal event.

[0057] Step 4, for each abnormal event cluster, calculate its standardized deviation in each feature dimension relative to the normal state, and select the top K dimensional features with the largest deviation.

[0058] Step 5, convert the top K dimensional features to natural language descriptions through a pre-set feature-semantic phrase mapping table, as semantic labels of abnormal events.

[0059] Step 6, calculate the criticality score based on the number of affected devices, duration, and state deviation.

[0060] Step 7, output the list of key abnormal events in descending order of criticality score.

[0061] The method mainly includes five stages, and the core is to filter real system-level anomalies through a multi-modal consistency constraint mechanism, and then realize unsupervised event semanticization and criticality sorting. The correspondence between each stage and the problem and formula is shown in Table 1. The specific implementation process is as follows: for video metadata, PM2.5, heartbeat and other multi-modal time series data generated by cameras, environmental sensors and other devices in smart city, industrial Internet of Things and other scenarios, first, accurately locate the cooperative abnormal period through the multi-modal consistency constraint mechanism that fuses the reconstruction error of autoencoder and the cross-modal prediction residual; second, perform online clustering on the abnormal period to form event clusters; third, extract the most abnormal feature dimension in each cluster based on Z-score significance analysis, and map it to human-readable semantic description; finally, quantify the criticality of events by combining the number of affected devices, duration and state deviation, and output the sorted list.

[0062] Table 1 Five-stage technical brief description

[0063]

[0064] The "P95" in the table refers to the 95th percentile of the comprehensive abnormal score s(t) under the historical normal state. Here, the 95th percentile means that under the historical normal state, 95% of the comprehensive abnormal scores are lower than the 95th percentile, and only 5% exceed the 95th percentile due to random fluctuations. Taking the 95th percentile as the abnormality determination threshold can effectively suppress false positives.

[0065] To distinguish between real environmental events and isolated device-level failures, a multi-modal consistency constraint mechanism is introduced. Among them, the real environmental event refers to an abnormal phenomenon caused by external physical environment changes or systematic disturbances, which synchronously appears on multiple heterogeneous sensing modalities, typical scenarios include:

[0066] In the traffic flow video monitoring system: regional fog / sand causes visibility to decrease, heavy rain causes road water reflection, holiday large passenger flow causes video target dense shielding, construction fence causes lane occupation or camera view shielding, regional power supply fluctuation causes multiple video stream synchronous lag or black screen;

[0067] In environmental pollution emergency monitoring: construction site dust surge causes PM 2.5 Synchronous deterioration with visibility, VOCs (volatile organic compounds) concentration rises sharply caused by chemical leakage and accompanied by abnormal temperature and humidity, regional smoke caused by straw burning causes video quality to decrease and air quality index (AQI) to rise, heavy rain causes river water quality turbidity and pH value mutation.

[0068] Such events have spatial coordination (affect multiple adjacent devices) and modal consistency (synchronous abnormality in video, environmental sensing, device state and other data dimensions), while single device failure (such as a camera lens contamination, a sensor drift) usually only affects local modal, lacking cross-modal response.

[0069] The core idea of the multi-modal consistency constraint mechanism is that: real system-level abnormal events usually cause coordinated and synchronous state deviation in multiple heterogeneous sensing modalities, while single device failure often only affects its own modality, and is difficult to produce consistent response in other modalities. Based on this cognition, the present application does not rely on single modality abnormal index, but constructs a comprehensive abnormal score function integrating multi-source information, which is used to quantify the abnormal degree of the whole system at the current time. The score function considers the following two key dimensions:

[0070] Multi-modal autoencoder: an unsupervised deep neural network, whose structure consists of "encoder" and "decoder" two parts, the encoder outputs features, and the decoder outputs reconstruction results; the encoder compresses the input data into low-dimensional hidden representation, and the decoder tries to restore the original input from the representation. In the present embodiment method, it is used to evaluate the reconstructability of each modality data under normal operation mode. It takes the current multi-modal feature vector as input, and outputs the reconstruction result through the encoding-decoding structure; the size of the reconstruction error reflects whether each modality deviates from the normal state at the current time. Since only historical normal data is used for training, any input that significantly deviates from the normal mode (such as sensor mutation or video anomaly) will cause reconstruction difficulty, resulting in high error.

[0071] Cross-modal predictor: a cross-modal predictor composed of single-layer multi-layer perception (MLP) is a lightweight feedforward neural network, which is used to learn the internal correlation rules between different sensing modalities (for example, "high PM 2.5Concentration is often accompanied by a decrease in the confidence level of video target detection. Its working principle is similar to "inferring the unknown from the known": using some modalities (such as environmental sensor data) as input, it predicts reasonable values ​​for other modalities (such as video metadata); the magnitude of the prediction residual (i.e., the difference between the actual value and the predicted value) reflects whether the cooperative consistency between modalities has been disrupted. When the system encounters a global disturbance (such as a sandstorm or power outage), the statistical relationship between multimodalities is broken, and the residual increases significantly, thus revealing system-level anomalies.

[0072] Both the autoencoder and cross-modal prediction components are unsupervised pre-trained during the application system initialization phase using historical normal operation data (i.e., historical time periods without anomaly alarms before deployment). After training, the model parameters are frozen, and only forward computation is performed during the online inference phase, requiring no manual annotation or online fine-tuning or parameter updates. Although the feedforward network structure itself is a general technique, its application in modeling multimodal consistency and combining it with the autoencoder to construct a comprehensive anomaly score is unique. s (t), and its application to the unsupervised critical event identification process constitutes the innovation of this invention.

[0073] The autoencoder reconstruction error and the cross-modal prediction residual are weighted and fused to form a unified comprehensive anomaly score s(t). This score has the following important functions:

[0074] Suppressing noise interference: Single device jitter may increase the reconstruction error of a certain mode, but due to the lack of cross-modal consistency, its overall score is still low, thus it is effectively filtered out;

[0075] Enhanced event sensitivity: When real-world environmental events (such as regional fog) simultaneously trigger multimodal responses such as video quality degradation, reduced visibility, and abnormal sensor readings, the reconstruction error and prediction residual increase synchronously, significantly improving s(t) and ensuring that critical events are not missed.

[0076] Provides a quantifiable threshold basis: s ( t As a unified input for subsequent abnormal period extraction, event clustering, and criticality assessment, it ensures the end-to-end coherence and engineerability of the entire analysis process.

[0077] Enhanced event sensitivity: When real-world environmental events (such as regional fog) simultaneously trigger multimodal responses such as video quality degradation, reduced visibility, and abnormal sensor readings, the reconstruction error and prediction residual increase synchronously, significantly improving s(t) and ensuring that critical events are not missed.

[0078] It provides a quantifiable threshold basis: s(t) serves as a unified input for subsequent abnormal period extraction, event clustering, and criticality assessment, ensuring end-to-end consistency and engineering feasibility of the entire analysis process.

[0079] Specifically, to distinguish real environment events from device failures, cross-modal consistency constraints are introduced. The comprehensive anomaly score is defined as follows:

[0080] (1) Autoencoder reconstruction error (modal ):

[0081]

[0082] (2) Cross-modal prediction residual (modal ):

[0083]

[0084] (3) Comprehensive anomaly score:

[0085]

[0086] where is the number of modalities, is the light autoencoder of modality , and is a cross-modal predictor consisting of a single-layer multi-layer perception (MLP) (default = 0.6, β = 0.4), specifically a fully connected neural network containing an input layer, a single hidden layer, and an output layer, used to map from the feature space of modality to the feature space of modality . This structure is lightweight and efficient, usually containing only a few hundred to a few thousand parameters, suitable for real-time inference in the cloud.

[0087] Technical logic: only when multiple modalities simultaneously exhibit high reconstruction error and cross-modal prediction failure, does the comprehensive anomaly score significantly increase → ensuring that the detected is a coordinated anomaly.

[0088] In unsupervised anomaly analysis, clustering algorithms (such as K-means, DBSCAN, etc.) will automatically divide the detected abnormal data into several groups, usually numbered in the form of "Cluster #1" "Cluster #2" "Cluster #3" (corresponding ). These numbers only represent "this is the nth grouping", and do not contain any information about the nature of the event itself - for example, "Cluster #3" may be caused by video quality degradation due to fog, or signal loss due to device power failure, but the application system cannot determine its true meaning from the number itself. When facing "Cluster #3 anomaly", the operation and maintenance personnel still need to manually backtrack the original data and check one by one, and cannot directly take targeted measures. Therefore, relying solely on such cluster results without semantics makes it difficult to support efficient and automated intelligent operation and maintenance decisions.

[0089] Only obtaining the clustering results (e.g., "Cluster #3") cannot support actual operation and maintenance decisions. The core problem to be solved is to generate a human-understandable semantic description (e.g., "Video confidence drops + PM concentration surge") for each abnormal event cluster without any event label or pre-defined rule. 2.5 Traditional methods either rely on a rule library (which is difficult to cover unknown events) or directly output raw features (which lack interpretability), and neither is suitable for large-scale dynamic sensing networks.

[0090] To this end, a lightweight semantic generation mechanism based on statistical significance is proposed: the essence of an abnormal event is determined by its "most significant deviation" from the normal state in a few feature dimensions. Specifically, for each feature dimension j in the cluster , the standardized deviation (Z-score) is calculated:

[0091]

[0092] where and are the mean and standard deviation of the j-th dimension when the system is running normally. The larger the Z-score, the more diagnostic value the j-th dimension has. Then, the top K feature dimensions with the largest Z-score values are selected, where K is a pre-set integer (default K=2) to control the balance between simplicity and information quantity of the event description. Through a lightweight mapping table (e.g., "video_confidence" → "Video confidence"), the event description is converted into a natural language phrase and concatenated.

[0093] This mechanism does not require labeling and does not rely on a rule library, and can accurately describe known events and generate reasonable semantics for new abnormal events (e.g., "Unknown abnormal event #K"), significantly improving human-machine collaboration efficiency and filling the technical gap in event-level interpretability in unsupervised scenarios. The mapping table is only used for result display and does not participate in the abnormality judgment logic.

[0094] In actual operation and maintenance scenarios with multiple concurrent events, simply identifying abnormal events is not enough to guide resource scheduling - the system must answer: "Which event is the most urgent? Which one should be handled first?" To this end, a criticality evaluation mechanism is introduced to quantify the comprehensive impact of each abnormal event on the overall sensing network stability and business continuity, and generate an actionable priority ranking list accordingly.

[0095] Traditional methods usually only sort by abnormal score, ignoring the actual impact range and duration characteristics of the event. For example, a sensor drift with a high abnormal score but only affecting a single device should have a lower operation and maintenance priority than a regional power fluctuation affecting dozens of video channels with a slightly lower score.

[0096] Therefore, a ternary criticality evaluation model integrating the influence range, time dimension and state severity is proposed.

[0097] Specifically, for each abnormal event e, its criticality score is calculated:

[0098]

[0099] wherein, is the number of devices affected by event e, reflecting the spatial influence range; is the event duration, embodying the time persistence; is the average abnormal score in the event window (i.e., the mean of s(t)), representing the severity of state deviation. The weight coefficient can be configured according to business needs (default ), and satisfies .

[0100] This model has significant engineering advantages: all calculations are lightweight (only involving counting, division and mean), without complex reasoning, and can be executed in the cloud in real time; at the same time, the output result directly corresponds to the operation and maintenance decision, the higher the criticality score KeyScore, the more critical the event, and the system automatically ranks it to the top of the alarm list. The actual measurement shows that this mechanism makes the response speed of the operation and maintenance personnel to high-impact events increase by more than 5 times, effectively avoiding resource mismatch.

[0101] Technical logic: the higher the KeyScore, the greater the threat of the event to system stability → directly guiding resource scheduling.

[0102] As shown in Figure 1 , the cloud multi-modal consistency constraint key abnormal event recognition method starts from the continuous inflow of multi-modal time series data, including video metadata, environmental sensor data and device status code. With a fixed time window (such as 5 seconds) as a unit, the following five stages are executed periodically in turn:

[0103] Stage 1: Multi-modal collaborative anomaly detection: fuse each modal autoencoder reconstruction error and cross-modal prediction residual , construct multi-modal consistency constraint, generate comprehensive abnormal score .

[0104] Stage 2: Abnormal period extraction: based on dynamic threshold (taking the 95th percentile P95 of the historical normal state ) and time continuity constraint (step, threshold, continuous ≥5 time steps over threshold), filter out candidate abnormal event windows with actual operation and maintenance significance, output candidate abnormal event window set.

[0105] Stage 3: Unsupervised event clustering: extract the mean vector of multi-modal features within each abnormal window, adopt online K-means++ clustering, and merge similar clusters with cosine similarity higher than 0.85 to form a set of semantically consistent abnormal event clusters, and output abnormal event clusters .

[0106] Stage 4: Automatic semantic generation: for each event cluster , calculate its Z-score deviation in each feature dimension relative to the normal state, i.e., for event cluster , each feature dimension j , calculate ; select the top-K (default K=2) most significant dimensions, and generate human-readable event descriptions (such as "video confidence drops + PM 2.5 concentration surges") through a pre-set lightweight feature-semantic mapping table to obtain event semantic labels.

[0107] Stage 5: Criticality evaluation and sorting: combine the three indicators of event impact device number, duration, and state deviation, and calculate the criticality score according to the weighted formula,

[0108] ,

[0109] , and output a list of critical abnormal events sorted in descending order of criticality score.

[0110] The sorted list of critical abnormal events contains semantic descriptions and priority of criticality.

[0111] The method continuously receives multi-modal time series data streams in the cloud, periodically executes the above five-stage process in fixed time windows (which can be set to 5 seconds), and constantly outputs an updated event priority list, forming an end-to-end automated closed loop from perception to decision.

[0112] The method of the present application addresses the problems of the prior art, adopts the technical solutions, and can achieve the technical effects shown in Table 2.

[0113] Table 2

[0114]

[0115] The cloud multi-modal consistency constrained critical abnormal event recognition system includes six functional modules: data access module, collaborative anomaly detection module, event clustering module, automatic semantic generation module, criticality evaluation module, and event output module.

[0116] A data access module is configured to receive multi-modal time-series data streams from a plurality of heterogeneous sensing devices, the data streams comprising video metadata, environmental sensor data, and device status codes.

[0117] A collaborative anomaly detection module is configured to locate collaborative anomaly time periods by fusing multi-modal consistency constraints of autoencoder reconstruction error and cross-modal prediction residual for the multi-modal time-series data streams.

[0118] An event clustering module is configured to pair multi-modal feature vectors of the collaborative anomaly time periods for unsupervised clustering to form a plurality of anomaly event clusters.

[0119] A semantic generation module is configured to calculate, for each anomaly event cluster, a standardized offset of the cluster in each feature dimension relative to a normal state, and select a top K number of dimensions with the largest offset, and convert the K number of dimensions into natural language descriptions through a preset feature identifier-semantic phrase mapping table.

[0120] A criticality evaluation module is configured to calculate a criticality score based on the number of devices affected by an event, the duration of the event, and the state deviation of the event.

[0121] An event output module is configured to push a sorted list of critical anomaly events.

[0122] The plurality of heterogeneous sensing devices refer to various types of heterogeneous devices in a city sensing network, such as video cameras, air quality sensors, temperature and humidity probes, and edge computing terminals. The multi-modal time-series data streams include, but are not limited to, video metadata (such as target detection confidence, frame rate), environmental monitoring values (such as PM 2.5 , VOC concentration), and device status codes (such as heartbeat packets, online / offline flags). After the multi-modal data is accessed, the collaborative anomaly detection module identifies system-level anomalies and extracts continuous anomaly time periods; then the event clustering module groups the anomaly time periods into event clusters; the semantic generation module assigns readable descriptions to the event clusters; the criticality evaluation module calculates the priority of the event clusters; and finally, the event output module pushes a sorted list of critical anomaly events, forming an end-to-end unsupervised event recognition closed loop. The event output module pushes the sorted list of critical anomaly events to a cloud operation and maintenance alarm system or a visual monitoring platform through a standardized interface, supports automatic generation of work orders, large-screen display, and automatic response strategy triggering, and realizes seamless connection from anomaly recognition to operation and maintenance actions.

[0123] The following presents a deployment of the method or system of the present application, key technical processes, and actual measurement results.

[0124] (1) Method or system deployment

[0125] The application can be flexibly deployed in a public cloud or a private cloud environment, and a typical implementation takes the Ali Cloud platform as an example:

[0126] In the public cloud mode, the ecs.g7.large instance (2-core CPU, 4GB memory) deployed in the Ali Cloud North China 2 (Beijing) region has an operating system of Alibaba Cloud Linux 3;

[0127] In the private cloud mode, it can be deployed in the customer's local IDC or government cloud special zone based on the Ali Cloud Apsara Stack, meeting the data non-domain and equal protection compliance requirements.

[0128] Access multi-source sensing data streams through standard Internet of Things protocols (such as MQTT, HTTP / 2), including:

[0129] Video metadata (target detection quantity, confidence, frame rate);

[0130] Environmental sensing data (PM 2.5 , temperature and humidity, water level);

[0131] Device status code (heartbeat packet, voltage, online status).

[0132] Note: The data acquisition and transmission module is implemented by using the prior art.

[0133] (2) Key technology process

[0134] The overall data flow of the application is: multi-modal access → collaborative anomaly detection → online event clustering → semantic generation → key assessment → high-confidence alarm output.

[0135] After obtaining the comprehensive anomaly score sequence , the application extracts the effective abnormal event window through dynamic threshold and time continuity constraints. Specifically, the dynamic threshold is set to the 95th percentile (P95) of the system in the historical normal operating state . Only when the consecutive time step (τ≥5, corresponding to the duration ≥5 seconds) satisfies s, it is determined as a candidate abnormal event window, effectively suppressing transient noise interference.

[0136] The execution steps of the key technology process are shown in Table 3. Among them, “Top-2” refers to the top two feature dimensions in each abnormal event cluster Z-score value, and “P95” refers to the 95th percentile of the comprehensive anomaly score sequence in the historical normal state.

[0137] Table 3 Execution steps of key technology process

[0138]

[0139] (3) Actual effect

[0140] To verify the effectiveness of the application in different scales and scenarios, the experiment adopts a "public benchmark data + self-built micro perception network" dual-track verification strategy:

[0141] (1) The evaluation is carried out on the WADI (Water Distribution Testbed) water supply network test platform public data set. The data set is built by Nanyang Technological University of Singapore as a downstream supporting system of the SWaT system, simulates the real urban water supply and distribution process, contains 123 pressure, flow, liquid level and other sensors and actuators, covers 15 types of attack events (such as pipe burst, valve tampering), and is closer to the cooperative anomaly scene in large-scale perception network. Under the condition of complete unlabeled, the application accurately identifies the abnormal period, and sorts the events according to the influence severity through the KeyScore mechanism, and the key event detection rate reaches 94.1%, which is significantly better than the baseline methods such as LSTM-AE and Multimodal VAE.

[0142] (2) In the self-built micro city perception environment (containing 3 1080p video streams, PM 2.5 sensors and Raspberry Pi heartbeat signals, continuously collecting for 8 hours and injecting real events such as "watering car dust" and "edge node power failure", the application successfully maps the original multi-modal anomaly to an interpretable semantic description (such as "PM 2.5 concentration suddenly increased to + video screen temporary blockage"), and realizes automatic clustering and labeling of unknown events.

[0143] Comprehensive shows that the application can support accurate sorting of known anomalies and interpretable discovery of unknown events at the same time under the premise of only relying on unlabeled multi-modal flow, verifying its practicality and generalization ability in the lightweight cloud-edge collaborative architecture.

[0144] Obviously, those skilled in the art should understand that each step of the cloud multi-modal consistency constraint key abnormal event identification method of the above-mentioned embodiments of the present application or each module of the cloud multi-modal consistency constraint key abnormal event identification system can be realized by a general computing device, which can be concentrated on a single computing device or distributed on a network composed of multiple computing devices, and optionally, they can be realized by program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in different order, or they can be made into individual integrated circuit modules, or multiple modules or steps among them can be made into a single integrated circuit module to realize. Thus, the embodiments of the present application are not limited to any particular combination of hardware and software.

Claims

1. A method for cloud multi-modal consistency-constrained critical abnormal event identification, characterized in that, The method comprises the following steps: Step 1, receiving multi-modal time series data streams from multiple heterogeneous sensing devices, the data streams comprising video metadata, environmental sensor data and device status codes; Step 2, locating a collaborative anomaly period for the multi-modal time series data streams by a multi-modal consistency constraint mechanism fusing autoencoder reconstruction error and cross-modal prediction residual; Step 3, unsupervised clustering multi-modal feature vectors of the collaborative anomaly period to form multiple anomaly event clusters; Step 4, for each abnormal event cluster, calculate its normalized deviation from normal state in each feature dimension, and select the top K dimensions with the largest deviation. Step 5, the features of the one dimension are converted into natural language descriptions as semantic labels of the abnormal event through a preset feature identification-semantic phrase mapping table. K Step 5, the features of the one dimension are converted into natural language descriptions as semantic labels of the abnormal event through a preset feature identification-semantic phrase mapping table. Step 6, calculating a criticality score based on the number of affected devices, duration and state deviation of each event; Step 7, outputting a list of critical anomaly events in descending order of criticality score; In step 2, for the modalities , the autoencoder reconstruction error is: wherein, is t a data stream of the time modality, is a reconstruction result of the data stream of the time modality, t is a reconstruction result of the data stream of the time modality. The cross-modal prediction residual is calculated by the following method: The multi-modal feature vectors are obtained by processing the multi-modal time series data stream through the auto-encoder, and the predictor is trained to map the feature space of modality k to the feature space of modality j. k and j ​​ calculating the L2 norm distance between the predicted value and the true value as a residual term; taking a comprehensive anomaly score of the fusion of the autoencoder reconstruction error and the cross-modal prediction residual as the multi-modal consistency constraint mechanism of the fusion of the autoencoder reconstruction error and the cross-modal prediction residual; the calculation formula of the comprehensive anomaly score is: wherein is the number of modalities; is a predictor, in particular a fully connected neural network comprising an input layer, a single hidden layer and an output layer, for mapping from the feature space of modality to the feature space of modality ; a, b are weights, is the autoencoder reconstruction error. 2.The cloud multi-modal consistency-constrained key abnormal event recognition method of claim 1, wherein, In the step 4, the standardized offset is calculated by Z-score: wherein, is a standard score value, is is the value of the characteristic dimension j in the state, and are the mean and standard deviation, respectively, of the characteristic dimension j in the normal state. 3.The cloud multi-modal consistency-constrained key abnormal event recognition method of claim 1, wherein, In the step 6, the calculation formula of the criticality score of each anomaly event e is: wherein, is the number of devices affected by the abnormal event e, reflecting the spatial impact range; is the total number of devices, is the duration of the event, embodying the time persistence; is the maximum duration of the event, is the average abnormal score within the event window, i.e., the mean of s(t), representing the severity of the state deviation; a, b, g are configurable weights, and a + b + g = 1.

4. A system for implementing the method of key abnormal event identification with cloud multi-modal consistency constraints as claimed in claim 1, characterized in that, The method comprises the following modules: A data access module for receiving multi-modal time series data streams from multiple heterogeneous sensing devices, the data streams comprising video metadata, environmental sensor data and device status codes; A collaborative anomaly detection module configured to locate a collaborative anomaly period for the multi-modal time series data streams by a multi-modal consistency constraint mechanism fusing autoencoder reconstruction error and cross-modal prediction residual; An event clustering module configured to unsupervisedly cluster multi-modal feature vectors of the collaborative anomaly period to form multiple anomaly event clusters; The semantic generation module is configured to calculate, for each abnormal event cluster, a standardized offset of the cluster in each feature dimension relative to the normal state, and select the top K dimensional features with the largest offset K The top K dimensional features are converted into natural language descriptions through a preset feature identifier-semantic phrase mapping table. A criticality evaluation module for calculating a criticality score based on the number of affected devices, duration and state deviation of each event; An event output module for pushing a sorted list of critical anomaly events.

5. A computer device, characterized by: The computer device comprises a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to implement the steps of the cloud multi-modal consistency constraint critical anomaly event identification method according to any one of claims 1-3.

6. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that: The computer program / instructions are executed by the processor to implement the steps of the cloud multi-modal consistency constraint critical anomaly event identification method according to any one of claims 1-3.

Images