A multi-modal security event real-time detection and response system based on an end-cloud collaborative architecture

Abstract

The application belongs to the field of artificial intelligence and intelligent security technology, and in particular relates to a multi-modal security event real-time detection and response system based on an end-cloud collaborative architecture, which comprises an edge perception node, a regional edge server and a cloud collaborative processing platform, and through joint encapsulation of a structured semantic vector and an integrity label, the verifiability and context integrity of edge uploaded data are ensured; the local semantic graph construction mechanism of the regional edge server is utilized to complete efficient preliminary judgment under the premise of preserving original multi-modal correlation, dynamic alignment between cloud global semantic modeling and edge local reasoning is realized, and cognitive bias caused by feature truncation is avoided.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and intelligent security technology, specifically a multimodal security event real-time detection and response system based on an edge-cloud collaborative architecture. Background Technology

[0002] The security needs of smart cities, smart parks, and critical infrastructure continue to upgrade. Intelligent security systems based on the integration of artificial intelligence and the Internet of Things have become the core support for ensuring public safety and operational order. The core requirement of the new generation of security systems is multimodal perception and real-time response capabilities. It is necessary to integrate heterogeneous data sources such as video, audio, and infrared to realize the transformation from post-event tracing to pre-event warning and in-event handling. The edge-cloud collaborative architecture has become the mainstream technical path because it can balance latency, bandwidth, and accuracy constraints: edge devices undertake lightweight inference tasks, while the cloud relies on its computing power advantage to complete complex model training and global strategy optimization. Current mainstream edge-cloud collaborative security systems adopt a two-level processing paradigm of edge initial screening and cloud verification: front-end edge nodes (smart cameras, edge gateways) are equipped with lightweight models to perform preliminary feature extraction and coarse anomaly judgment on local multimodal data; after triggering a preset threshold, the relevant data is uploaded to the cloud, where a high-precision large model performs secondary verification and links alarm, scheduling and other response mechanisms. This architecture effectively alleviates the network congestion problem of uploading full data to the cloud, meets the basic requirement of sub-second response, and its task layering realizes the design logic of optimal resource allocation, which has good engineering practicality and deployment economy in specific scenarios.

[0003] Existing technologies suffer from deep-seated core contradictions: The lack of a dynamic and verifiable semantic consistency guarantee mechanism between edge-side local decision-making and cloud-side global state; existing solutions treat edge and cloud as functionally isolated serial units; edge computing power limitations prevent the resolution of deep spatiotemporal correlations in multimodal signals, leading to initial screening prone to semantic ambiguity or contextually missing false alarms; while the cloud possesses stronger inference capabilities, it receives compressed / truncated intermediate features, making it difficult to reconstruct the original context for correction. When facing new and unknown threats, updated model parameters from the cloud cannot be fed back to the edge in real time to form a closed-loop learning process, resulting in an inefficient cycle of edge misjudgment and cloud correction leading to repeated misjudgments; in high-concurrency, low-bandwidth environments, the uploading of numerous invalid features also consumes communication resources and misses response windows. This contradiction stems from the structural neglect of the semantic continuity of the perception, decision-making, and feedback closed-loop in the edge-cloud interaction paradigm, a problem that conventional optimization methods cannot fundamentally solve.

[0004] Therefore, this invention provides a multimodal security event real-time detection and response system based on an edge-cloud collaborative architecture. Summary of the Invention

[0005] In order to overcome the shortcomings of the prior art, at least one technical problem raised in the background art is solved.

[0006] The technical solution adopted by this invention to solve its technical problem is as follows: A multimodal security event real-time detection and response system based on an edge-cloud collaborative architecture includes multiple edge sensing nodes set up in the physical field, one or more regional edge servers, and a cloud collaborative processing platform located in a remote data center. The components form a hierarchical distributed processing topology through a secure communication link. The edge sensing nodes are responsible for the acquisition of raw multimodal data and preliminary feature extraction, the regional edge servers undertake the tasks of local context fusion and initial event judgment, and the cloud collaborative processing platform performs global semantic modeling, cross-domain correlation analysis and strategy optimization. It also sends the updated semantic rules and lightweight model parameters to the edge side through a feedback channel to realize the dynamic evolution and cognitive synchronization of the entire system.

[0007] Preferably, the edge sensing node includes a video acquisition unit, an audio pickup unit, an infrared thermal imaging unit, a vibration sensing unit, and an environmental monitoring unit. Each unit achieves microsecond-level sampling alignment through a hardware timestamp synchronization module and is connected to an embedded secure processing unit via a local bus interface. The secure processing unit has a built-in trusted execution environment for running a formally verified lightweight multimodal feature encoder, whose output is a structured semantic vector containing event type identifiers, confidence intervals, spatiotemporal coordinate indices, and a hash digest of the original data. Before being uploaded, the structured semantic vector must be digitally signed through an identity authentication protocol based on asymmetric encryption to ensure that the data source is traceable and has not been tampered with.

[0008] Preferably, the regional edge server is configured with a dual-port shared memory buffer, one end of which connects to the uplink data stream from multiple edge sensing nodes, and the other end is accessed by the local inference engine. The inference engine integrates a multimodal attention fusion module and a context state machine, which can dynamically weight the contribution of different modal signals according to the current scene, and generate a local semantic graph by combining the event sequence within the historical sliding window. When there is a subgraph structure in the local semantic graph that meets the preset risk pattern matching conditions, the regional edge server triggers a first-level response action, including activating the local alarm device, locking the trajectory of the suspicious target, and generating an incremental semantic packet containing complete contextual dependencies. The incremental semantic packet is compressed using differential encoding and attached with an integrity verification tag issued by the security element chip, and then pushed to the cloud collaborative processing platform through a priority queue scheduling mechanism.

[0009] Preferably, the cloud-based collaborative processing platform includes a global semantic modeling unit, a cross-domain correlation analysis unit, a strategy optimization unit, and a model distillation unit. The global semantic modeling unit receives incremental semantic packets from multiple regional edge servers, reconstructs a global event evolution map using graph neural networks, and identifies potential cross-regional collaborative threat patterns based on this. The cross-domain correlation analysis unit interfaces with external information systems, introducing auxiliary contexts such as geographic information, personnel identity databases, and facility operation status to enhance the completeness of event interpretation. The strategy optimization unit dynamically adjusts the response strategy set based on the latest threat situation and encapsulates the strategy change instructions into security policy objects with version numbers and time constraints. The model distillation unit generates a lightweight sub-model adapted to a specific hardware platform based on a large-scale multimodal joint representation model trained in the cloud, combined with the computing power constraints and energy consumption characteristics of edge devices, and then binds it with the corresponding semantic rule set before sending it to the target regional edge server.

[0010] Preferably, this system introduces a two-way semantic consistency verification mechanism at the edge-cloud interaction level: On the one hand, after receiving the incremental semantic packet, the cloud collaborative processing platform first verifies the validity of its integrity label and digital signature, and then maps it to the global semantic space for conflict detection. If a logical contradiction is found with the existing knowledge system, a reverse interrogation process is initiated, requiring the source region edge server to send back the original multimodal fragment for reproducing the reasoning process; On the other hand, when the region edge server receives a new model or policy object from the cloud, it also needs to perform local consistency verification, including checking whether the model input and output dimensions match the current deployment of the perception node configuration and whether the policy action exceeds the capability boundary of the local executor. Only after confirming that there is no conflict can the new version be activated.

[0011] Preferably, to support the system's continuous learning capability in the face of unknown threats, this invention designs a closed-loop feedback mechanism based on active learning: when the cloud-based collaborative processing platform identifies a new event pattern that cannot be covered by existing semantic rules, it automatically marks the relevant samples as high-value exploration samples and sends a targeted sampling instruction to the corresponding regional edge server; this instruction specifies the modal combination, time window, and spatial range to be supplemented for collection, and the regional edge server temporarily increases the data collection granularity of the relevant sensing nodes accordingly, and uploads the enhanced original data fragments in encryption; the cloud uses such high-quality labeled samples for incremental training and injects the extracted new semantic rules into the global knowledge base, while triggering a new round of model distillation and policy update process.

[0012] Preferably, this system adopts a hierarchical bandwidth guarantee mechanism to address communication bottlenecks in high-concurrency, low-bandwidth environments: all uplink data streams are divided into three categories according to semantic urgency: critical event streams, regular state streams, and background noise streams; critical event streams carry the highest transmission priority, employ power line carrier and wireless dual-channel redundant transmission, and enable forward error correction coding; regular state streams are aggregated to the regional edge server via RS485 bus and then uploaded in batches after compression; background noise streams are transmitted with the lowest priority only during periods of bandwidth idle time, or are directly archived locally at the edge without being uploaded; downlink control commands are also divided into immediate-effective and delayed-effective types based on their timeliness, with the former forcibly preempting transmission resources through a dedicated control channel, and the latter being piggybacked into the gaps between normal data frames.

[0013] Preferably, all secure communication in the system is implemented based on an extended transport layer security protocol. This protocol embeds a device identity certificate chain verification step into the standard handshake process and integrates a device-unique entropy source generated by a physically unclonable function during the session key derivation process, thereby preventing man-in-the-middle attacks and replay attacks. In addition, a two-way heartbeat keep-alive mechanism is established between the cloud collaborative processing platform and the regional edge server. If either party fails to receive a valid heartbeat packet from the other party multiple times in a row, it will determine that the link is abnormal and activate the local emergency response plan.

[0014] Preferably, the regional edge server is equipped with an interrupt-driven event scheduler, which listens for state change interrupt signals from each edge sensing node and dynamically allocates processing threads based on the modality type and historical activity of the state change interrupt signal. For high-frequency vibration or sudden audio events, the scheduler prioritizes calling low-latency inference paths and skips some unnecessary feature normalization steps to shorten the response path. For slowly changing environmental parameters or static video images, they are handled uniformly by the background batch processing queue to avoid frequent wake-ups of the main processor and energy waste.

[0015] Preferably, the global semantic modeling unit in the cloud-based collaborative processing platform organizes event knowledge using a hierarchical graph representation method: the bottom layer consists of atomic event nodes, recording the basic facts triggered by a single modality; the middle layer consists of composite event nodes, describing the logical combination of multiple atomic events under spatiotemporal constraints; the top layer consists of intent inference nodes, representing the behavioral purpose of potential threat subjects; each layer is connected by weighted directed edges, with the weight values ​​dynamically adjusted by the online learning module based on historical verification results; when a new incremental semantic package introduces new atomic events, the system automatically attempts to aggregate upwards to generate possible composite events and evaluates their association strength with existing intent nodes. If the association strength exceeds a preset threshold, a higher-level intent update process is triggered.

[0016] Preferably, when generating lightweight sub-models, the model distillation unit not only considers the compression ratio of parameter quantity and computational complexity, but also explicitly constrains the spatial distribution characteristics of the output semantic vectors, so that they can still maintain inter-class separability and intra-class compactness similar to the original large model in the dimensionality-reduced embedding space. This constraint is achieved by introducing a contrastive learning term into the distillation loss function, which forces that the embedding distance of the same event category in the teacher-student model output is less than the minimum distance between different categories. After distillation, the lightweight sub-model is packaged and distributed together with its corresponding embedding space mapping matrix to ensure that the edge side can accurately restore the relative semantic positional relationship when performing inference.

[0017] Preferably, the system deploys a local cache proxy module between the regional edge server and the edge sensing node. This module maintains a local copy of the semantic vectors uploaded in the most recent period and their corresponding original data pointers. When the cloud initiates a reverse challenge request, the regional edge server does not need to re-collect data, but directly retrieves the matching original fragments from the cache proxy and sends them back. The cache proxy uses a circular buffer structure to manage storage space and dynamically adjusts the retention period according to the event confidence. High-confidence events are retained for a longer time to support deep auditing, while low-confidence events are quickly rotated to release resources.

[0018] Preferably, the security policy objects generated by the policy optimization unit are written in a declarative rule language. Each rule consists of three parts: preconditions, action set, and revocation condition. The preconditions are expressed in the form of logical predicates and can reference the state of any node in the global semantic graph. The action set defines specific response operations, such as audible and visual alarms, access control linkage, and camera reversal. The revocation condition specifies the triggering conditions for policy failure, such as the target leaving the monitored area or being manually confirmed as a false alarm. Before being issued, all policy objects must be checked for logical conflicts or execution deadlocks using formal verification tools to ensure the overall consistency of the policy set.

[0019] Preferably, the system supports a multi-tenant isolation deployment mode, where the data flow and policy sets of different security responsibility entities are logically completely isolated; the cloud collaborative processing platform maintains an independent semantic knowledge subgraph for each tenant, and only shares the anonymized high-level intent nodes during cross-tenant event correlation analysis; the regional edge server automatically routes data to the corresponding tenant processing pipeline based on the ownership identifier of the access device, preventing unauthorized access to sensitive information.

[0020] The beneficial effects of this invention are as follows: The present invention provides a multimodal security event real-time detection and response system based on an edge-cloud collaborative architecture, which ensures the verifiability and contextual integrity of data uploaded from the edge through the joint encapsulation of structured semantic vectors and integrity tags. It is possible to utilize the local semantic graph construction mechanism of the regional edge server to complete the efficient initial judgment while preserving the original multimodal associations; It achieves dynamic alignment between cloud-based global semantic modeling and edge-based local reasoning, avoiding cognitive bias caused by feature truncation; It ensures the continuous evolution of system capabilities through targeted sampling and incremental training when new threats are detected; and maintains reliable transmission of critical events in high-load network environments through hierarchical bandwidth guarantee and dual-channel redundant transmission mechanism. Based on formal verification of policy objects and model distillation constraints, the consistency of behavioral logic and semantic space in the process of edge-cloud collaboration is guaranteed. By combining bidirectional heartbeat keep-alive with local cache proxy, the robustness and response continuity of the system are effectively improved in scenarios with unstable links. Attached Figure Description

[0021] The invention will now be further described with reference to the accompanying drawings.

[0022] Figure 1 This is a structural block diagram of the multimodal security event real-time detection and response system based on an edge-cloud collaborative architecture, as described in this invention. Figure 2 This is a schematic diagram of the local processing and interaction process between the regional edge server and the edge sensing node in this invention. Detailed Implementation

[0023] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0024] like Figures 1-2 As shown in the figure, an embodiment of the present invention provides a multimodal security event real-time detection and response system based on an edge-cloud collaborative architecture. The system comprises multiple edge sensing nodes located in the physical field, one or more regional edge servers, and a cloud-based collaborative processing platform located in a remote data center. The above components form a hierarchical distributed processing topology through a secure communication link. The edge sensing nodes are responsible for the acquisition of raw multimodal data and preliminary feature extraction. The regional edge servers undertake the tasks of local context fusion and initial event judgment. The cloud-based collaborative processing platform performs global semantic modeling, cross-domain correlation analysis, and strategy optimization. It also sends the updated semantic rules and lightweight model parameters to the edge side through a feedback channel, realizing the dynamic evolution and cognitive synchronization of the entire system.

[0025] The edge sensing node includes a video acquisition unit, an audio pickup unit, an infrared thermal imaging unit, a vibration sensing unit, and an environmental monitoring unit. The video acquisition unit uses an image sensor; the audio pickup unit integrates a dual-microphone array; the infrared thermal imaging unit has a temperature measurement range of -20℃ to 150℃; the vibration sensing unit is a MEMS accelerometer; and the environmental monitoring unit integrates temperature, humidity, and air pressure sensors. and PM2.5 sensor; As mentioned above, the above units achieve sampling alignment through a hardware timestamp synchronization module, which is based on the Time Protocol (PTP). All raw data is connected to the embedded security processing unit via the local bus interface.

[0026] The embedded secure processing unit includes an integrated Trusted Execution Environment (TEE) that runs a formally verified lightweight multimodal feature encoder. The lightweight multimodal feature encoder consists of five parallel sub-networks, each corresponding to one of the five perceptual modalities. Each sub-network uses a depthwise separable convolutional structure with the number of parameters controlled to within 0.8M. At a main frequency of 1.5GHz, the inference latency is less than 15ms. The encoder outputs a structured semantic vector, whose data structure is defined as follows: Event type identifier (EventID): a 32-bit unsigned integer, mapped according to a predefined ontology library; Confidence Interval: Two floating-point numbers [L, U], representing the 95% confidence lower and upper bounds of the Bayesian posterior probability; SpatioTemporalIndex: Includes GPS coordinates (WGS-84 format), UTC timestamp (nanoseconds), and spatial region encoding; Original data hash digest: A 256-bit digest value calculated using the SHA3-256 algorithm on the original data segment (including 500ms before and after the time window).

[0027] The structured semantic vector must be digitally signed using an identity authentication protocol based on asymmetric encryption before it can be uploaded. As mentioned above, each edge sensing node is pre-set with a unique device certificate, and its private key is stored in a secure element protected by a Physically Unclonable Function (PUF). The signing process uses the ECDSA-P256 algorithm, and the generated digital signature is 64 bytes, which is appended to the end of the semantic vector to form a complete message packet. The message packet is encapsulated through the DTLS1.2 protocol and transmitted via the uplink interface of the regional edge server.

[0028] In some implementations, the regional edge server is configured with a dual-port shared memory buffer, one port of which is connected to uplink data streams from multiple edge sensing nodes (supporting up to 128 concurrent nodes). The other port is accessed by the local inference engine; The local inference engine accesses and integrates a multimodal attention fusion module and a context state machine; the multimodal attention fusion module adopts a gated cross-attention mechanism, with N structured semantic vectors (N≤128) as input and the fused context enhancement vector as output; The formula for calculating attention weights is as follows: in, The query vector (generated from the current scene context) For the first The key vector of each modality The key vector dimension (128 in this case) is the fused vector. ,in For the first A value vector of modalities.

[0029] The context state machine maintains a sliding time window (default window length is 30s, step size is 1s) to record the sequence of historical events. The internal state transitions of the state machine are defined by a finite state automaton (FSA), which contains 12 states, covering typical scenarios such as normal inspection, suspicious loitering, illegal intrusion, and equipment malfunction. When the fusion vector triggers the state transition condition, the system generates a local semantic graph, which is a directed graph. This indicates that the nodes Corresponding to atomic events, edges It represents the causal or temporal dependency between events, and the edge weight is determined by the time interval and semantic similarity.

[0030] When a subgraph structure that meets the preset risk pattern matching conditions exists in the local semantic graph, the region edge server triggers a level one response action. The risk pattern is defined using a graph isomorphism template. For example, if three consecutive infrared heat sources appear in the restricted area and are accompanied by abnormal audio, it corresponds to a three-node subgraph. The matching algorithm uses VF2 subgraph isomorphism detection, and the threshold is set to a matching degree ≥ 0.85. After triggering, the system performs the following operations: The system activates local audio-visual alarms, locks onto the trajectory of suspicious targets (by continuously outputting target bounding boxes via the video tracking module), and generates incremental semantic packages containing complete contextual dependencies.

[0031] The incremental semantic packet is compressed using differential coding, transmitting only the changed parts of the graph relative to the previous version (added nodes, deleted edges, weight updates), and serialized using Protocol Buffers, achieving an average compression ratio of 6.2:1. The incremental semantic packet is attached with an integrity verification tag issued by the security element chip. The tag is generated based on HMAC-SHA256, and the key is dynamically derived from PUF. The incremental semantic packet is pushed to the cloud collaborative processing platform through a priority queue scheduling mechanism. The scheduler allocates transmission priorities based on the urgency of the events: Critical events (such as unauthorized intrusion) are assigned the highest priority (DSCP value 46), routine events (such as equipment self-test) are assigned medium priority (DSCP value 18), and background noise (such as fluctuations in ambient temperature and humidity) is assigned the lowest priority (DSCP value 8).

[0032] In some implementations, the cloud-based collaborative processing platform is set up on a Kubernetes cluster, with each machine configured with dual processors, DDR4 memory, and network interfaces. The cloud-based collaborative processing platform includes a global semantic modeling unit, a cross-domain correlation analysis unit, a strategy optimization unit, and a model distillation unit.

[0033] The global semantic modeling unit receives incremental semantic packets from multiple regional edge servers and reconstructs the global event evolution graph using a graph neural network (GNN). This neural network adopts the GraphSAGE architecture, uses LSTM as the aggregation function, has 3 layers, and has 256 hidden dimensions. As mentioned earlier, the map update process is as follows: First, all incremental packets are parsed into a graph operation instruction set; In the global graph Perform these operations on; Local re-embedding is performed on the affected subgraph; The embedding vector has a dimension of 512, which is used for subsequent similarity calculation and cluster analysis.

[0034] The cross-domain association analysis unit interfaces with external information systems, including Geographic Information System (GIS) APIs, LDAP directories for personnel authentication, and MQTT topics for facility operation status. For example, when an unauthorized person is detected in a certain area, the system calls LDAP to query the person's permission level and combines it with GIS to obtain the distance between the person's current location and the nearest authorized access control. If the distance is greater than 50m and there is no access record, the event risk level is increased.

[0035] The strategy optimization unit dynamically adjusts the response strategy set based on the latest threat situation; The strategy is written in a declarative rule language, with the following syntax: ; Before being deployed, all policy objects must be checked for logical conflicts or execution deadlocks using a formal verification tool (based on the TLA+ specification). After successful verification, the policy object is encapsulated as a security policy object with a version number (e.g., v2.3.1) and time constraints (e.g., TTL=7200s).

[0036] The model distillation unit is based on a large-scale multimodal joint representation model (teacher model) trained in the cloud. This model has 120M parameters, takes raw multimodal data as input, and outputs a 1024-dimensional semantic vector. The distillation process generates lightweight sub-models (student models) for specific edge hardware platforms. The distillation loss function is defined as: in, This is the task loss (cross-entropy). For knowledge distillation loss (KL divergence). For comparison of learning items; The contrastive learning term mandates that the embedding distance of the same event category in the teacher-student model output is less than the minimum distance between different categories. Its mathematical expression is: in, and The student model and the teacher model are respectively used for the first time. Embedding of class event output, Let be the boundary constant (set to 0.2); After distillation, the lightweight sub-model (with ≤1.5M parameters) along with its corresponding embedding space mapping matrix (512×256 linear transformation) is packaged into an ONNX format model file and sent to the target region edge server through a secure channel.

[0037] In some implementations, the system introduces a two-way semantic consistency verification mechanism at the edge-cloud interaction layer; After receiving the incremental semantic packet, the cloud-based collaborative processing platform first verifies the validity of its integrity label and digital signature: it verifies the ECDSA signature using the device public key and the HMAC label using the shared key. Map it to the global semantic space for conflict detection; Conflict detection is based on a semantic logic rule base, such as the rule that the same target cannot appear in two areas more than 100m apart at the same time. If a logical contradiction is found (such as a violation of the above rules), a reverse challenge process is initiated, sending a challenge request to the source region edge server, requesting the return of the original multimodal fragment.

[0038] On the other hand, when the regional edge server receives a new model or policy object from the cloud, it also needs to perform local consistency verification. The verification includes checking whether the model's input and output dimensions match the current deployment of the perception nodes (for example, if the new model requires 6 inputs but there are only 5 sensors locally, then loading will be rejected). Does the policy action exceed the capabilities of the local executor? (For example, if the policy requires activating a water cannon but there is no such device locally, it is marked as non-executable.) After verification, the system will run the new version in shadow mode for 10 minutes, during which the old and new strategies will be executed in parallel and the output differences will be compared. If the difference rate is less than 1%, the new version will be officially activated.

[0039] Furthermore, to support the system's continuous learning capability in the face of unknown threats, when the cloud-based collaborative processing platform identifies a new event pattern that cannot be covered by existing semantic rules (for example, discovering a high-density, low-confidence cluster of isolated points through clustering algorithms), it automatically marks the relevant samples as high-value exploration samples and sends targeted sampling instructions to the corresponding edge server. The directional sampling command specifies the modal combination to be supplemented (such as adding infrared and audio synchronous sampling), time window and spatial range; Based on this, the regional edge server temporarily increases the data collection granularity of relevant sensing nodes: video frame rate is increased to 60fps, audio sampling rate is increased to 96kHz, and infrared refresh rate is increased to 10Hz. The enhanced raw data fragments were uploaded after being encrypted with AES-256-GCM. The cloud uses these high-quality labeled samples for incremental training (using online learning algorithms such as AdaGrad) and injects the extracted new semantic rules into the global knowledge base, while triggering a new round of model distillation and policy update processes.

[0040] In some implementations, the system employs a tiered bandwidth guarantee mechanism to address communication bottlenecks in high-concurrency, low-bandwidth environments; Following the above, all uplink data streams are divided into three categories according to their semantic urgency: Critical event streams (such as intrusion and fire), routine status streams (such as device heartbeat and environmental parameters), and background noise streams (such as static image change rate < 0.1%). The critical event stream carries the highest transmission priority, employs dual-channel redundant transmission of power line carrier (PLC, HomePlug AV2 standard) and wireless (Wi-Fi 65GHz band), and enables forward error correction coding (FEC, code rate 1 / 2, using LDPC code). The regular status stream is aggregated to the regional edge server via RS485 bus and then uploaded in batches after compression. Background noise streams are transmitted with the lowest priority only during periods of bandwidth idle (determined by network congestion detection results), or are archived locally at the edge (automatically overwritten after 7 days) without being uploaded. Downlink control commands are also divided into immediate-effective types (such as emergency lockout) and delayed-effective types (such as policy update) based on their timeliness. The former forcibly preempts transmission resources through a dedicated control channel (reserving 10% bandwidth), while the latter is embedded in the gaps between normal data frames for piggyback transmission.

[0041] Furthermore, all secure communication in the system is implemented based on an extended transport layer security protocol; The protocol embeds a device identity certificate chain verification step into the standard TLS 1.3 handshake process and integrates a device-unique entropy source generated by a physically unclonable function during the session key derivation process. The salt value of the key derivation function (HKDF) is formed by concatenating the 128-bit entropy source output by the PUF with a timestamp, ensuring that the key is unique and unpredictable for each session. A two-way heartbeat keep-alive mechanism is established between the cloud-based collaborative processing platform and the regional edge server. The heartbeat packet is sent every 30 seconds and includes the current load status and link quality indicators of both parties. If either party fails to receive a valid heartbeat packet from the other party for three consecutive times (90 seconds), it will determine that the link is abnormal and activate the local emergency response plan, including switching to offline inference mode, enabling local caching strategy, and attempting to rebuild the connection through a backup communication link (such as 4G LTE).

[0042] In some implementations, the regional edge server is equipped with an interrupt-driven event scheduler. The interrupt-driven event scheduler runs in the Linux kernel real-time patch (PREEMPT_RT) environment and listens for state change interrupt signals from each edge sensing node; The scheduler dynamically allocates processing threads based on the modal type and historical activity of the state change interrupt signal: For high-frequency vibration (>5Hz) or sudden audio events (energy surge >20dB), the scheduler prioritizes calling low-latency inference paths (skipping Z-score normalization and PCA dimensionality reduction steps) to keep the end-to-end latency within 25ms. For slowly changing environmental parameters (change rate < 0.1% / s) or static video images (inter-frame SSIM > 0.99), the background batch processing queue handles them uniformly, waking up the main processor every 10 seconds to avoid energy waste caused by frequent wake-ups.

[0043] Furthermore, the global semantic modeling unit in the cloud-based collaborative processing platform uses a hierarchical graph representation method to organize event knowledge; The underlying layer consists of atomic event nodes (such as infrared detection of a human body), which record the basic facts of a single modality triggering. The middle layer consists of composite event nodes (such as suspected unauthorized intrusion), which describe the logical combination of multiple atomic events under spatiotemporal constraints (e.g., infrared human body, restricted area, unauthorized card). The top layer represents the intent inference node (e.g., deliberate sabotage), characterizing the behavioral purpose of the potential threat actor. Layers are connected by weighted directed edges, with the weights dynamically adjusted by the online learning module based on historical verification results. For example, if 95 out of the past 100 infrared human body + restricted area intrusions were manually confirmed as real intrusions, then the corresponding edge weight is set to 0.95; When a new atomic event is introduced by a new incremental semantic package, the system automatically attempts to aggregate upwards to generate possible composite events (using predefined combination rule templates) and evaluates the association strength with existing intent nodes (calculated by cosine similarity). If it exceeds a preset association threshold, the higher-level intent update process is triggered.

[0044] It should be noted that when generating lightweight sub-models, the model distillation unit not only considers the compression ratio of parameter quantity and computational complexity, but also explicitly constrains the spatial distribution characteristics of the output semantic vectors. As mentioned earlier, by introducing a contrastive learning term into the distillation loss function, we can ensure that the dimensionality-reduced embedding space can still maintain inter-class separability and intra-class compactness similar to the original large model. Experiments show that, in a 256-dimensional embedding space, the average intra-class distance is 0.32, and the minimum inter-class distance is 0.87, satisfying the requirements. The constraints.

[0045] Furthermore, the system deploys a local caching proxy module between the regional edge server and the edge sensing node; The local cache proxy module runs on a separate coprocessor, maintaining a local copy of the semantic vectors uploaded within the last 72 hours and their corresponding original data pointers (stored on NVMe SSD). The cache proxy uses a circular buffer structure to manage storage space and dynamically adjusts the retention period based on event confidence: upper limit of the confidence interval. The event is retained for 72 hours. Retain for 24 hours Keep for 1 hour; When the cloud initiates a reverse challenge request, the regional edge server does not need to recollect data, but directly retrieves the matching original fragment from the cached proxy and sends it back, with a return latency of less than 200ms.

[0046] In some implementations, the security policy objects generated by the policy optimization unit are written in a declarative rule language. Each rule consists of three parts: preconditions, action set, and revocation conditions. Preconditions are expressed in the form of logical predicates and can reference the state of any node in the global semantic graph. The action set defines specific response operations, such as audible and visual alarms, access control linkage, and camera turning. The revocation conditions specify the triggering conditions for policy failure. Before all policy objects are issued, they must be checked for logical conflicts or execution deadlocks using formal verification tools to ensure the overall consistency of the policy set.

[0047] Furthermore, the system supports a multi-tenant isolation deployment mode, where the data flow and policy sets of different security responsibility entities (such as Park A and Park B) are logically completely isolated. The cloud-based collaborative processing platform maintains an independent semantic knowledge subgraph for each tenant (stored in different namespaces of the Neo4j graph database), and only shares the anonymized high-level intent nodes during cross-tenant event correlation analysis (e.g., only sharing the existence of suspicious behavior without sharing specific location or personnel information). The regional edge server automatically routes data to the corresponding tenant's processing pipeline based on the ownership identifier of the access device (such as the OU field in the device certificate) to prevent unauthorized access to sensitive information.

[0048] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal security event real-time detection and response system based on an edge-cloud collaborative architecture, characterized in that, include: Multiple edge sensing nodes set up in the physical field, one or more regional edge servers, and a cloud-based collaborative processing platform located in a remote data center; Edge sensing nodes are used to collect multimodal raw data and generate structured semantic vectors. The structured semantic vectors include event type identifiers, confidence intervals, spatiotemporal coordinate indexes, and raw data hash digests. Before uploading, they are attached with digital signatures for identity authentication based on asymmetric encryption. The regional edge server is configured with a dual-port shared memory buffer. One port connects to the uplink data stream from multiple edge sensing nodes, and the other port is accessed by the local inference engine. The local inference engine integrates a multimodal attention fusion module and a context state machine to dynamically weight the contribution of different modal signals according to the current scene, and generates a local semantic graph by combining the event sequence in the historical sliding window. When there is a subgraph structure in the local semantic graph that meets the preset risk pattern matching conditions, a first-level response action is triggered, and an incremental semantic package containing complete contextual dependencies is generated. The incremental semantic package is compressed using differential encoding and an integrity verification label is attached before being pushed to the cloud collaborative processing platform. The cloud-based collaborative processing platform includes a global semantic modeling unit, a cross-domain correlation analysis unit, a strategy optimization unit, and a model distillation unit. The global semantic modeling unit receives incremental semantic packets from multiple regional edge servers, reconstructs the global event evolution map using graph neural networks, and identifies cross-regional collaborative threat patterns. The cross-domain correlation analysis unit interfaces with external information systems to introduce auxiliary context; The strategy optimization unit dynamically adjusts the response strategy set based on the latest threat situation and encapsulates the strategy change instructions into security policy objects with version numbers and time constraints; the model distillation unit generates a lightweight sub-model adapted to a specific hardware platform based on the large-scale multimodal joint representation model trained in the cloud and combined with the computing power constraints of edge devices, and then binds it with the corresponding semantic rule set and sends it to the edge server in the target area. Among them, the system introduces a two-way semantic consistency verification mechanism at the edge-cloud interaction level: after receiving the incremental semantic packet, the cloud verifies the validity of its integrity label and digital signature, and maps it to the global semantic space for conflict detection. If a logical contradiction is found, the reverse interrogation process is initiated to request the return of the original multimodal fragment. When a regional edge server receives a new model or policy object from the cloud, it performs a local consistency check, including checking whether the model's input and output dimensions match the current perception node configuration and whether the policy actions exceed the local executor's capability boundaries. After confirming that there are no conflicts, it activates the new version.

2. A multimodal security event real-time detection and response system based on an edge-cloud collaborative architecture according to claim 1, characterized in that, The edge sensing node includes a video acquisition unit, an audio pickup unit, an infrared thermal imaging unit, a vibration sensing unit, and an environmental monitoring unit. Each unit achieves sampling alignment through a hardware timestamp synchronization module and is connected to the embedded security processing unit via a local bus interface. The embedded secure processing unit has a built-in trusted execution environment for running a formally verified lightweight multimodal feature encoder.

3. A multimodal security event real-time detection and response system based on an edge-cloud collaborative architecture according to claim 1, characterized in that, The regional edge server is equipped with an interrupt-driven event scheduler. The interrupt-driven event scheduler listens for state change interrupt signals from edge sensing nodes and dynamically allocates processing threads based on the state change interrupt signals and historical activity. For high-frequency vibrations or sudden audio events, the interrupt-driven event scheduler prioritizes low-latency inference paths and skips some unnecessary feature normalization steps to shorten the response path.

4. A multimodal security event real-time detection and response system based on an edge-cloud collaborative architecture according to claim 1, characterized in that, The global semantic modeling unit in the cloud-based collaborative processing platform organizes event knowledge using a hierarchical graph representation method, as follows: The underlying layer consists of atomic event nodes, which record the basic facts of a single modality triggering. The middle layer consists of composite event nodes, which describe the logical combination of multiple atomic events under spatiotemporal constraints; The top layer is the intent inference node, which represents the behavioral purpose of potential threat actors; The layers are connected by weighted directed edges, and the weight values ​​are dynamically adjusted by the online learning module based on historical verification results. When a new incremental semantic packet introduces a new atomic event, the system automatically attempts to aggregate upwards to generate possible composite events and evaluates their association strength with existing intent nodes. If the association strength exceeds a preset threshold, the higher-level intent update process is triggered.

5. A multimodal security event real-time detection and response system based on an edge-cloud collaborative architecture according to claim 1, characterized in that, When generating lightweight sub-models, the model distillation unit introduces a contrastive learning term into the distillation loss function, requiring that the embedding distance of the same event category in the teacher-student model output is less than the minimum distance between different categories, so as to explicitly constrain the output semantic vector to maintain inter-class separability and intra-class compactness in the dimensionality-reduced embedding space; after distillation, the lightweight sub-model and its corresponding embedding space mapping matrix are packaged and sent out together.

6. A multimodal security event real-time detection and response system based on an edge-cloud collaborative architecture according to claim 1, characterized in that, The system deploys a local cache proxy module between the regional edge server and the edge sensing node. The local cache proxy module maintains a local copy of the semantic vectors uploaded in the most recent period and their corresponding original data pointers. When the cloud initiates a reverse challenge request, the regional edge server directly retrieves the matching original fragment from the cache proxy and sends it back. The cache agent uses a ring buffer structure to manage storage space and dynamically adjusts the retention period based on event confidence.

7. A multimodal security event real-time detection and response system based on an edge-cloud collaborative architecture according to claim 1, characterized in that, The security policy objects generated by the policy optimization unit are written in a declarative rule language. Each declarative rule consists of three parts: preconditions, action set, and revocation condition. Preconditions are expressed in the form of logical predicates and can reference the state of any node in the global semantic graph. The action set defines the specific response operation. The revocation condition specifies the triggering condition for policy failure. All policy objects must be checked for logical conflicts or execution deadlocks by a formal verification tool before being issued.

8. A multimodal security event real-time detection and response system based on an edge-cloud collaborative architecture according to claim 1, characterized in that, The system employs a tiered bandwidth guarantee mechanism: All uplink data streams are categorized into three types based on their semantic urgency: critical event streams, routine status streams, and background noise streams. The critical event stream employs redundant transmission via both power line carrier and wireless channels, and forward error correction coding is enabled. Regular status streams are aggregated via the bus and then compressed and uploaded in batches. Background noise streams are transmitted with the lowest priority only during periods of bandwidth idleness or archived locally at the edge. Downlink control commands are divided into immediate-effective and delayed-effective types based on their timeliness. The immediate-effective type forcibly preempts transmission resources through a dedicated control channel, while the delayed-effective type is embedded in the gaps between normal data frames for piggyback transmission.

9. A method for real-time detection and response of multimodal security events based on an edge-cloud collaborative architecture, applicable to a real-time detection and response system for multimodal security events based on an edge-cloud collaborative architecture according to any one of claims 1-8, characterized in that, Includes the following steps: S1: Edge sensing nodes collect multimodal raw data, which is then synchronized with hardware timestamps. The embedded security processing unit generates structured semantic vectors and attaches digital signatures before uploading. S2: The regional edge server receives semantic vectors from multiple edge-aware nodes and constructs a local semantic graph through a multimodal attention fusion module and a context state machine. If a subgraph structure matching a preset risk pattern exists in the graph, a first-level response action is triggered and an incremental semantic packet with an integrity verification label is generated. S3: The cloud-based collaborative processing platform receives incremental semantic packets, verifies their validity, reconstructs the global event evolution map, identifies cross-regional threats, and optimizes response strategies in conjunction with external context. S4: The cloud generates lightweight sub-models adapted to edge hardware through model distillation, and then distributes them after binding them with the semantic rule set; S5: The regional edge server performs local consistency checks on the newly issued models or policies, and activates them after confirming that there are no conflicts. S6: When the cloud identifies a new event pattern, it sends a targeted sampling instruction to the corresponding edge server to improve the data collection granularity within the specified modality and spatiotemporal range, and performs incremental training and policy updates based on the enhanced data.

10. A method for real-time detection and response of multimodal security events based on an edge-cloud collaborative architecture according to claim 9, characterized in that, It also includes two-way heartbeat keep-alive and link anomaly emergency response steps: a two-way heartbeat mechanism is established between the cloud collaborative processing platform and the regional edge server. If either party fails to receive a valid heartbeat packet multiple times in a row, it will determine that the link is abnormal and activate the local emergency response plan, including switching to offline inference mode, enabling local caching strategy, and attempting to rebuild the connection through backup communication link.

Images