Community security and protection method and system based on cloud side-end cooperation and multi-modal analysis

Abstract

The invention discloses a community security and protection method and system based on cloud side-end cooperation and multi-modal analysis. Deploying a multi-mode sensor to collect image, audio, environment and trajectory data; the edge nodes extract and align modal features through deep learning, target recognition, behavior tracking and anomaly detection are completed after fusion, and when the anomaly exceeds a threshold value, the corresponding features are uploaded to the cloud; the cloud constructs a global knowledge base based on historical features, calculates an abnormal probability, and issues a decision trigger response to the edge and the terminal if the abnormal probability exceeds a threshold value. According to the invention, multiple sensing data are integrated, high-precision continuous sensing of people, vehicles, objects and environments and real-time detection of abnormal behaviors are realized by using deep learning and a neural network, and false alarms and missing alarms are significantly reduced under complex illumination, shielding and noise. A lightweight model is deployed at edge nodes, edge and edge cloud collaborative decision is made, real-time and deep analysis is balanced, dependence on bandwidth and the cloud is reduced, and the method is suitable for residences, parks, villages in cities and other multi-element smart community scenes.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, target tracking, and behavior recognition, and specifically to a community security method and system that combines cloud-edge-device collaboration and multimodal analysis. Background Technology

[0002] In recent years, with the deepening of smart city construction and the rapid development of artificial intelligence technology, intelligent security systems based on a "cloud-edge-device" collaborative architecture have become an important research direction for community security management. Modern community security scenarios are characterized by diversified monitoring equipment, varied data modalities, and complex security needs. Traditional single video surveillance systems are no longer sufficient to meet the intelligent requirements for personnel behavior analysis and abnormal event early warning. Especially when facing complex security tasks in community scenarios such as personnel crossing paths, abnormal gatherings, and security equipment wearing detection, how to achieve efficient collection, real-time analysis, and intelligent decision-making of multi-source heterogeneous data has become a key technical challenge that urgently needs to be solved in the current community security field.

[0003] The cloud-edge-device collaborative architecture provides a novel technological path for community security by organically integrating cloud computing, edge computing, and terminal devices. Under this architecture, terminal devices are responsible for multimodal data acquisition, edge nodes perform real-time analysis, and the cloud completes deep learning and decision optimization, forming a hierarchical intelligent processing system. However, existing systems still face several key technical bottlenecks in practical applications: in multimodal data fusion, there is a lack of unified feature representation and effective correlation analysis methods between data from different sources and with different characteristics; in target tracking, the accuracy of target correlation across cameras and across time periods still needs improvement; in behavior understanding, the accuracy and generalization ability for identifying abnormal human behavior in complex scenarios are insufficient; more importantly, existing systems generally lack the ability to systematically model community security knowledge, making it difficult to form an organic closed loop between low-level data perception and high-level decision-making and early warning.

[0004] The introduction of multimodal analysis technology offers a new approach to solving the aforementioned problems. By integrating various AI technologies such as computer vision, speech recognition, and sensor data processing, it is possible to achieve comprehensive perception of human behavior, equipment status, and environmental factors in a community environment. Especially at the edge computing level, the application of lightweight multimodal fusion algorithms can significantly improve the accuracy and real-time performance of abnormal behavior detection.

[0005] One existing technology is the distributed AIoT smart video surveillance technology based on edge-cloud collaboration, proposed in the paper "Distributed real-time object detection based on edge-cloud collaboration for smart video surveillance applications." This technology proposes a novel edge-cloud collaborative architecture to achieve real-time object detection in smart cities. Addressing the bandwidth limitations and insufficient real-time performance of centralized video surveillance systems, a distributed edge computing framework is constructed. This framework employs an edge-cloud collaborative mechanism, where edge nodes act as distributed computing units, and the cloud is responsible for global knowledge integration and model deployment. At the technical implementation level, media data is preprocessed at the edge through lightweight encoding, and AI models utilize a dynamic deployment mechanism to achieve cloud-edge collaboration. The paper also explores edge device management methods based on distributed message middleware and optimization strategies for video data transmission in low-bandwidth environments. The drawbacks of this technology are: firstly, while the edge-cloud collaborative architecture reduces bandwidth pressure, the limited computing power of edge nodes may struggle to handle complex AI models (such as high-precision, multi-target, small-target, or occluded scenarios), leading to decreased detection accuracy. Secondly, the heterogeneity of edge devices (such as varying computing power and sensor performance) may affect the overall system stability. Second, the dynamic model deployment mechanism relies on real-time communication between the cloud and the edge. Network fluctuations or brief outages may prevent edge nodes from updating their models in a timely manner, impacting the intelligent analysis capabilities of the monitoring system. Furthermore, issues related to resource scheduling, task allocation, and fault recovery at edge nodes can lead to additional operational costs. Third, despite the adoption of lightweight coding and low-bandwidth optimization strategies, the massive amounts of data generated by edge devices during large-scale deployments can still put pressure on the cloud's data integration and analysis capabilities, especially during emergencies (such as dense crowd monitoring), potentially causing bottlenecks in cloud computing resources.

[0006] The second existing technology is the collaborative cloud-edge video query system called SurveilEdge, proposed in the paper "Surveiledge: Real-time video query based on collaborative cloud-edge deep learning," designed to improve real-time query performance in large-scale monitoring scenarios. This technology addresses the high latency and bandwidth bottlenecks of traditional cloud processing, as well as the limited computing power of edge devices. The authors designed a cloud-edge collaborative layered processing architecture. This system uses an intelligent task allocation mechanism to perform lightweight feature extraction tasks on edge devices, while offloading complex deep learning inference tasks to the cloud, achieving a balance between latency and accuracy. In terms of technical implementation, SurveilEdge introduces a customized convolutional neural network training scheme, significantly reducing training time and improving model accuracy. The drawbacks of this technology are: firstly, while SurveilEdge effectively reduces system latency and bandwidth pressure through its cloud-edge collaborative architecture, it still relies on edge devices for feature extraction, which is limited by their computing power and model complexity. Despite employing a lightweight neural network, the system may still experience a decrease in recognition accuracy in scenarios involving occlusion, low light, and complex crowds, affecting its adaptability in real-world security environments. Secondly, while the system possesses a certain dynamic task scheduling mechanism, the lack of a unified global scheduling optimization strategy during high-concurrency operation across multiple edge nodes may lead to uneven resource utilization and fluctuations in inference performance. Furthermore, its model updates and policy distribution rely on the cloud; if the network is unstable or cloud computing is limited, edge nodes may fail to respond promptly, reducing the system's flexibility and robustness. Thirdly, SurveilEdge is more suited to handling structured query tasks and lacks sufficient support for multi-semantic, open-ended, or cross-time-segment video analysis, making it difficult to address more complex user needs. In multi-view, multi-device joint analysis scenarios, the system lacks an efficient semantic fusion mechanism. Additionally, the original design had weak considerations for privacy protection, failing to explicitly address data encryption and user identity isolation technologies, posing certain information security risks.

[0007] The third existing technology is a distributed deep learning model for intelligent video surveillance scenarios proposed in the paper "Distributed deep learning model for intelligent video surveillance systems with edge computing." This model combines an edge computing architecture to achieve efficient and low-latency video processing capabilities. The overall architecture follows the design concept of "cloud-edge collaboration and model decentralization," retaining computationally intensive model training tasks in the cloud while distributing inference tasks to edge devices, thereby effectively alleviating data transmission bandwidth pressure and improving response speed. In its implementation, the system adopts a modular deep neural network structure. Edge nodes deploy lightweight models that have undergone pruning and quantization to complete tasks such as keyframe extraction, object detection, and local behavior recognition; while the cloud handles tasks such as multi-node model fusion, global behavior analysis, and model update and optimization. Furthermore, the system designs a hierarchical communication mechanism for efficient model synchronization and feature aggregation between edge devices and the cloud. Simultaneously, to enhance task collaboration capabilities, the system introduces a collaborative inference mechanism between edge nodes, achieving data redundancy tolerance and enhanced inference accuracy. The drawbacks of this technology are as follows: First, although the system decentralizes model inference tasks to the edge to reduce latency, the heterogeneity and resource constraints of edge devices still limit model inference capabilities. This is especially true when handling high-resolution video, dense scenes, or small object recognition tasks, where lightweight models may experience accuracy degradation, impacting the system's reliability in practical applications. Second, model synchronization and updates rely on unified scheduling and management in the cloud. If cloud communication latency increases or malfunctions, edge nodes will struggle to obtain the latest model parameters in a timely manner, leading to a decrease in the system's intelligence and adaptability. Furthermore, while collaborative inference among edge nodes improves robustness, the lack of a clear resource scheduling priority strategy may cause localized performance bottlenecks under uneven load conditions. Third, when facing multi-scenario, multi-task monitoring needs, the system lacks a flexible task reconfiguration mechanism, making it difficult to dynamically adjust the model structure and processing strategies according to the scenario, thus reducing the system's generalization ability. Additionally, regarding privacy protection, although large-scale video uploads are avoided, the lack of robust data encryption, access control, and identity anonymization mechanisms still presents certain security risks at the edge. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of existing methods and propose a community security method and system based on cloud-edge-device collaboration and multimodal analysis. The main problems addressed by this invention are: first, to solve the problems of inconsistent sampling frequencies, semantic space separation, and large scale differences in multi-source heterogeneous data such as images, audio, and trajectories in community security; second, to solve the problems of low accuracy and poor robustness of traditional single-modal perception in complex environments; and third, to solve the problems of high data return latency and high bandwidth consumption caused by traditional centralized processing.

[0009] To address the aforementioned problems, this invention proposes a community security method based on cloud-edge-device collaboration and multimodal analysis, the method comprising:

[0010] Deploy various heterogeneous sensors to collect multimodal data including images, audio, environmental parameters, and trajectories;

[0011] Edge nodes are deployed, and the feature fusion module of the edge nodes uses deep learning methods to extract each modality feature from the multimodal data, maps each modality feature to a unified feature space and aligns them to form fused features;

[0012] The multimodal perception module of the edge node receives the fused features, performs multimodal target recognition and behavior tracking, and then performs abnormal behavior detection. When the abnormal score is greater than a preset threshold, the features corresponding to the abnormal score are uploaded to the cloud for further analysis.

[0013] The cloud uses all features corresponding to historical abnormal scores to build and maintain a global knowledge base, and calculates the abnormal probability of the features corresponding to the abnormal scores uploaded to the cloud in combination with the global knowledge base. When the abnormal probability is greater than a preset threshold, the decision result is fed back to the edge node and user terminal device to trigger a preset response operation.

[0014] Preferably, the deployment of multiple heterogeneous sensors to collect multimodal data, including images, audio, environmental parameters, and trajectories, specifically involves:

[0015] The deployed heterogeneous sensors include cameras, microphones, infrared sensors, position sensors, and temperature and humidity sensors. The cameras are used for image capture, the microphones are responsible for audio acquisition, the infrared sensors and position sensors are used for behavior trajectory analysis, and the temperature and humidity sensors are used for environmental monitoring. The collected multimodal data is transmitted from the sensors to the edge nodes through the low-latency, low-bandwidth CoAP protocol.

[0016] Preferably, in the deployment of edge nodes, the feature fusion module of the edge nodes uses deep learning methods to extract modal features from the multimodal data, maps the modal features to a unified feature space and aligns them to form fused features, specifically:

[0017] Timekeeping , This represents the total number of data points across all modalities.

[0018] Extracting features from image data: Processing video image data from a camera using a convolutional neural network. It automatically extracts image features, including object contours and textures, through convolution and pooling operations. :

[0019]

[0020] in It is the input image. It is a weight matrix. It is a bias term. It's a convolution operation. It is a pooling layer. The obtained image features are used; and a pre-trained model is employed. Improve the efficiency and accuracy of feature extraction. express Residual connections in a network; express The set of all learnable parameters in a network;

[0021] Feature extraction from audio data: Long Short-Term Memory (LSTM) networks are used to process audio data captured by the microphone, extracting temporal features, including sound patterns and frequency characteristics. During feature extraction, spectral analysis preprocessing is performed on the audio data to ensure the accuracy and robustness of the features.

[0022]

[0023]

[0024]

[0025] in It is a time-domain audio signal. This represents the short-time Fourier transform, which converts a time-domain audio signal into a frequency-domain representation. ; then The input is fed into an LSTM to obtain the audio feature representation. , These are the hidden states of the LSTM. This is the cell information of the LSTM;

[0026] Extracting features from time-series data: For temperature and humidity time-series data, a Long Short-Term Memory (LSTM) network is used to extract their temporal features, capturing the trends of temperature and humidity environmental parameters over time.

[0027]

[0028] in This represents time-series data of temperature and humidity. Representing temporal characteristics, These are the hidden states of the LSTM. It is the cell information of LSTM; based on the frequency characteristics of the data, the time series is processed in segments using a sliding window method to enhance the ability to capture changing trends;

[0029] Feature extraction of trajectory data: For the position coordinate sequence trajectory data of a moving target, the spatiotemporal convolutional network ST-CNN and gated recurrent unit GRU are used to jointly extract trajectory features and capture the motion pattern of the trajectory.

[0030]

[0031] in This represents the input position coordinate sequence trajectory data. This represents the obtained trajectory features; These are the weights and bias matrices of the convolutional layer; It is a non-linear activation function; These are the parameters of the GRU layer;

[0032] Multimodal feature fusion: Mapping and aligning features from different modalities to a unified feature space.

[0033]

[0034] This indicates the obtained fusion features. , , , This represents the learnable weight matrix. , , , This represents a learnable bias vector. This represents the activation function. , , , It's about weight allocation. , , , The values ​​are 0.5, 0.2, 0.2, and 0.1 respectively.

[0035] Preferably, the multimodal perception module of the edge node receives the fused features, performs multimodal target recognition and behavior tracking, and then performs abnormal behavior detection. When the abnormal score is greater than a preset threshold, the features corresponding to the abnormal score are uploaded to the cloud for further analysis. Specifically:

[0036] The fused features of the input are utilized using YOLOv4. Perform multimodal target recognition:

[0037]

[0038] in This represents the probability of object recognition and classification. Corresponding time Detected target, Indicates category; It is a learnable weight matrix; Indicates dynamic weights. An encoder that represents visual features;

[0039] Multimodal behavior tracking using the DeepSORT tracking algorithm:

[0040]

[0041] in Indicates the target at time. The state vector, Indicate category The relevant dynamic matrix, This represents the state vector from the previous step. It is a learnable cross-modal projection matrix. This represents the trajectory filtering gating function. These are the trajectory features;

[0042] Joint target recognition and behavior tracking are enhanced with the following features:

[0043]

[0044] in This represents the enhanced features obtained; This represents a two-layer fully connected feedforward neural network used for nonlinear fusion; symbol Indicates a splicing operation;

[0045] Abnormal behavior detection and cloud upload operation:

[0046]

[0047] in Indicates abnormal scores; Indicates the category of behavior; Spatiotemporal dependence of multimodal features is captured; LSTM is used to analyze trajectory dynamics features; when Push the data to the cloud;

[0048] The feature corresponding to the abnormal score pushed to the cloud is the first Features of edge nodes :

[0049]

[0050] in It is a learnable matrix. It is an activation function, and [;;] represents the concatenation operation.

[0051] Preferably, the cloud platform constructs and maintains a global knowledge base using all features corresponding to historical abnormal scores, and calculates the abnormal probability by combining the global knowledge base with the features corresponding to the abnormal scores uploaded to the cloud platform, specifically as follows:

[0052] time Historical abnormal scores Based on the characteristics corresponding to all abnormal scores at the current moment and time global knowledge base The calculation yielded:

[0053]

[0054] in It is the first The learnable weights of each feature vector, and ;

[0055] time global knowledge base via the time Historical abnormal scores and time global knowledge base Update:

[0056]

[0057] in Controlling the rate of forgetting;

[0058] Calculate the probability of anomalies :

[0059]

[0060]

[0061] Among the moments of Indicates an exception label. This indicates a normal label; Indicates the activation function; , Represents the weight matrix and bias matrix; This indicates a dedicated 2-layer sensor for anomaly detection. This represents the activation function. These are all learnable parameters.

[0062] Accordingly, the present invention also provides a community security system based on cloud-edge-device collaboration and multimodal analysis, comprising:

[0063] The data acquisition unit is used to deploy various heterogeneous sensors to collect multimodal data, including images, audio, environmental parameters, and trajectories.

[0064] The feature fusion unit is used to deploy edge nodes. The feature fusion module of the edge nodes uses deep learning methods to extract each modality feature from the multimodal data, maps each modality feature to a unified feature space and aligns them to form fused features.

[0065] The edge analysis unit is used by the multimodal perception module of the edge node to receive the fused features, perform multimodal target recognition and behavior tracking, and then perform abnormal behavior detection. When the abnormal score is greater than a preset threshold, the features corresponding to the abnormal score are uploaded to the cloud for further analysis.

[0066] The cloud analysis unit is used to construct and maintain a global knowledge base using all features corresponding to historical abnormal scores in the cloud, and to calculate the abnormal probability of the features corresponding to the abnormal scores uploaded to the cloud in combination with the global knowledge base. When the abnormal probability is greater than a preset threshold, the decision result is fed back to the edge node and user terminal device to trigger a preset response operation.

[0067] Implementing this invention has the following beneficial effects:

[0068] This invention integrates multi-sensor data and utilizes deep learning and neural networks to achieve high-precision continuous perception and real-time detection of abnormal behavior in people, vehicles, objects, and the environment, significantly reducing false alarms and missed alarms under complex lighting, occlusion, and noise conditions. Lightweight models are deployed at edge nodes for local preprocessing, fusion, and judgment, enabling millisecond-level alarms. Edge-to-edge and edge-to-cloud collaborative decision-making balances real-time and deep analysis, reducing bandwidth and cloud dependency, making it suitable for diverse smart community scenarios such as residential buildings, industrial parks, and urban villages. Attached Figure Description

[0069] Figure 1This is an overall flowchart of a community security method based on cloud-edge-device collaboration and multimodal analysis according to an embodiment of the present invention;

[0070] Figure 2 This is a structural diagram of a community security system based on cloud-edge-device collaboration and multimodal analysis, according to an embodiment of the present invention. Detailed Implementation

[0071] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] Figure 1 This is a general flowchart of a community security method based on cloud-edge-device collaboration and multimodal analysis according to an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0073] S1 deploys a variety of heterogeneous sensors to collect multimodal data including images, audio, environmental parameters, and trajectories;

[0074] S2, deploy edge nodes. The feature fusion module of the edge nodes uses deep learning methods to extract each modality feature from the multimodal data, maps each modality feature to a unified feature space and aligns them to form fused features;

[0075] S3, the multimodal perception module of the edge node receives the fused features, performs multimodal target recognition and behavior tracking, and then performs abnormal behavior detection. When the abnormal score is greater than a preset threshold, the features corresponding to the abnormal score are uploaded to the cloud for further analysis.

[0076] S4, the cloud uses all features corresponding to historical abnormal scores to build and maintain a global knowledge base, and calculates the abnormal probability of the features corresponding to the abnormal scores uploaded to the cloud in combination with the global knowledge base. When the abnormal probability is greater than a preset threshold, the decision result is fed back to the edge node and user terminal device to trigger a preset response operation.

[0077] Step S1 is as follows:

[0078] The deployed heterogeneous sensors include cameras, microphones, infrared sensors, position sensors, and temperature and humidity sensors. The cameras are used for image capture, the microphones are responsible for audio acquisition, the infrared sensors and position sensors are used for behavior trajectory analysis, and the temperature and humidity sensors are used for environmental monitoring. The collected multimodal data is transmitted from the sensors to the edge nodes through the low-latency, low-bandwidth CoAP protocol.

[0079] Step S2 is as follows:

[0080] Timekeeping , This represents the total number of data points across all modalities.

[0081] Edge nodes utilize the lightweight deep learning model TinyYOLO;

[0082] S2-1, Extracting features from image data: Processing video image data from the camera using a convolutional neural network. It automatically extracts image features, including object contours and textures, through convolution and pooling operations. :

[0083]

[0084] in It is the input image. It is a weight matrix. It is a bias term. It's a convolution operation. It is a pooling layer. The obtained image features are used; and a pre-trained model is employed. Improve the efficiency and accuracy of feature extraction. express Residual connections in a network; express The set of all learnable parameters in a network;

[0085] S2-2, Feature Extraction of Audio Data: The audio data collected by the microphone is processed using a Long Short-Term Memory (LSTM) network to extract temporal features, including sound patterns and frequency characteristics. During feature extraction, spectral analysis preprocessing is performed on the audio data to ensure the accuracy and robustness of the features.

[0086]

[0087]

[0088]

[0089] in It is a time-domain audio signal. This represents the short-time Fourier transform, which converts a time-domain audio signal into a frequency-domain representation. ; then The input is fed into an LSTM to obtain the audio feature representation. , These are the hidden states of the LSTM. This is the cell information of the LSTM;

[0090] S2-3, Extracting features from time series data: For temperature and humidity time series data, a Long Short-Term Memory (LSTM) network is used to extract their temporal features, capturing the trends of temperature and humidity environmental parameters over time.

[0091]

[0092] in This represents time-series data of temperature and humidity. Representing temporal characteristics, These are the hidden states of the LSTM. It is the cell information of LSTM; based on the frequency characteristics of the data, the time series is processed in segments using a sliding window method to enhance the ability to capture changing trends;

[0093] S2-4, Extracting Trajectory Data Features: For the position coordinate sequence trajectory data of a moving target, the spatiotemporal convolutional network ST-CNN and gated recurrent unit GRU are used to jointly extract trajectory features and capture the motion pattern of the trajectory.

[0094]

[0095] in This represents the input position coordinate sequence trajectory data. This represents the obtained trajectory features; These are the weights and bias matrices of the convolutional layer; It is a non-linear activation function; These are the parameters of the GRU layer;

[0096] S2-5, Fusing Multimodal Features: Mapping and aligning the features of each modality to a unified feature space.

[0097]

[0098] This indicates the obtained fusion features. , , , This represents the learnable weight matrix. , , , This represents a learnable bias vector. This represents the activation function. , , , It's about weight allocation. , , , The values ​​are 0.5, 0.2, 0.2, and 0.1 respectively.

[0099] Step S3 is as follows:

[0100] S3-1, Utilize YOLOv4 to process the input fused features. Perform multimodal target recognition:

[0101]

[0102] in This represents the probability of object recognition and classification. Corresponding time Detected target, Indicates the category (people, vehicles, animals, etc.); It is a learnable weight matrix; Indicates dynamic weights. An encoder that represents visual features;

[0103] S3-2, Multimodal behavior tracking using the DeepSORT tracking algorithm:

[0104]

[0105] in Indicates the target at time. The state vector, Indicate category The relevant dynamic matrix, This represents the state vector from the previous step. It is a learnable cross-modal projection matrix. This represents the trajectory filtering gating function. These are the trajectory features;

[0106] S3-3, Enhanced features are obtained by combining target recognition and behavior tracking:

[0107]

[0108] in This represents the enhanced features obtained; This represents a two-layer fully connected feedforward neural network used for nonlinear fusion; symbol Indicates a splicing operation;

[0109] S3-4, Abnormal Behavior Detection and Cloud Upload Operation:

[0110]

[0111] in Indicates abnormal scores; Indicates the category of behavior; Spatiotemporal dependence of multimodal features is captured; LSTM is used to analyze trajectory dynamics features; when Push the data to the cloud;

[0112] The feature corresponding to the abnormal score pushed to the cloud is the first Features of edge nodes :

[0113]

[0114] in It is a learnable matrix. It is an activation function, and [;;] represents the concatenation operation.

[0115] Step S4 is as follows:

[0116] S4-1, Time Historical abnormal scores Based on the characteristics corresponding to all abnormal scores at the current moment and time global knowledge base The calculation yielded:

[0117]

[0118] in It is the first The learnable weights of each feature vector, and ;

[0119] time global knowledge base via the time Historical abnormal scores and time global knowledge base Update:

[0120]

[0121] in Controlling the rate of forgetting;

[0122] Calculate the probability of anomalies :

[0123]

[0124]

[0125] Among the moments of Indicates an exception label. This indicates a normal label; Indicates the activation function; , Represents the weight matrix and bias matrix; This indicates a dedicated 2-layer sensor for anomaly detection. This represents the activation function. These are all learnable parameters.

[0126] Accordingly, the present invention also provides a community security system based on cloud-edge-device collaboration and multimodal analysis, such as... Figure 2 As shown, it includes:

[0127] Data acquisition unit 1 is used to deploy various heterogeneous sensors to collect multimodal data, including images, audio, environmental parameters, and trajectories.

[0128] Feature fusion unit 2 is used to deploy edge nodes. The feature fusion module of the edge nodes uses deep learning methods to extract each modality feature from the multimodal data, maps each modality feature to a unified feature space and aligns them to form fused features.

[0129] Edge analysis unit 3 is used by the multimodal perception module of the edge node to receive the fused features, perform multimodal target recognition and behavior tracking, and then perform abnormal behavior detection. When the abnormal score is greater than a preset threshold, the features corresponding to the abnormal score are uploaded to the cloud for further analysis.

[0130] The cloud analysis unit 4 is used to construct and maintain a global knowledge base using all features corresponding to historical abnormal scores in the cloud, and to calculate the abnormal probability of the features corresponding to the abnormal scores uploaded to the cloud in combination with the global knowledge base. When the abnormal probability is greater than a preset threshold, the decision result is fed back to the edge node and the user terminal device to trigger a preset response operation.

[0131] Therefore, this invention integrates multi-sensor data and utilizes deep learning and neural networks to achieve high-precision continuous perception and real-time detection of abnormal behavior in people, vehicles, objects, and the environment, significantly reducing false alarms and missed alarms under complex lighting, occlusion, and noise conditions. Lightweight models are deployed at edge nodes for local preprocessing, fusion, and judgment, enabling millisecond-level alarms. Edge-to-edge and edge-to-cloud collaborative decision-making balances real-time and deep analysis, reducing bandwidth and cloud dependency, making it suitable for diverse smart community scenarios such as residential areas, industrial parks, and urban villages.

[0132] The above provides a detailed description of a community security method and system based on cloud-edge-device collaboration and multimodal analysis provided by embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A community security method based on cloud-edge-device collaboration and multimodal analysis, characterized in that, The method includes: Deploy various heterogeneous sensors to collect multimodal data, including images, audio, environmental parameters, and trajectories; Edge nodes are deployed, and the feature fusion module of the edge nodes uses deep learning methods to extract each modality feature from the multimodal data, maps each modality feature to a unified feature space and aligns them to form fused features; The multimodal perception module of the edge node receives the fused features, performs multimodal target recognition and behavior tracking, and then performs abnormal behavior detection. When the abnormal score is greater than a preset threshold, the features corresponding to the abnormal score are uploaded to the cloud for further analysis. The cloud uses all features corresponding to historical abnormal scores to build and maintain a global knowledge base, and calculates the abnormal probability of the features corresponding to the abnormal scores uploaded to the cloud in combination with the global knowledge base. When the abnormal probability is greater than a preset threshold, the decision result is fed back to the edge node and user terminal device to trigger a preset response operation.

2. The community security method based on cloud-edge-device collaboration and multimodal analysis as described in claim 1, characterized in that, The deployment of multiple heterogeneous sensors to collect multimodal data, including images, audio, environmental parameters, and trajectories, specifically includes: The deployed heterogeneous sensors include cameras, microphones, infrared sensors, position sensors, and temperature and humidity sensors; cameras are used for image capture, microphones are responsible for audio acquisition, infrared sensors and position sensors are used for behavior trajectory analysis, and temperature and humidity sensors are used for environmental monitoring; the collected multimodal data is transmitted from the sensors to the edge nodes through a low-latency, low-bandwidth protocol.

3. The community security method based on cloud-edge-device collaboration and multimodal analysis as described in claim 1, characterized in that, The deployed edge nodes, whose feature fusion module uses deep learning methods to extract modal features from the multimodal data, map these features to a unified feature space and align them to form fused features, specifically: Timekeeping , This represents the total number of data points across all modalities. Extracting features from image data: Processing video image data from a camera using a convolutional neural network. It automatically extracts image features, including object contours and textures, through convolution and pooling operations. : in It is the input image. It is a weight matrix. It is a bias term. It's a convolution operation. It is a pooling layer. These are the obtained image features; And utilize pre-trained models Improve the efficiency and accuracy of feature extraction. express Residual connections in a network; express The set of all learnable parameters in a network; Extracting features from audio data: The audio data collected by the microphone is processed using a Long Short-Term Memory (LSTM) network to extract temporal features from the audio, including sound patterns and frequency features; During feature extraction, the audio data undergoes spectral analysis preprocessing to ensure the accuracy and robustness of the features: in It is a time-domain audio signal. This represents the short-time Fourier transform, which converts a time-domain audio signal into a frequency-domain representation. ; Then The input is fed into an LSTM to obtain the audio feature representation. , These are the hidden states of the LSTM. This is the cell information of the LSTM; Extracting features from time-series data: For temperature and humidity time-series data, a Long Short-Term Memory (LSTM) network is used to extract their temporal features, capturing the trends of temperature and humidity environmental parameters over time. in This represents time-series data of temperature and humidity. Representing temporal characteristics, These are the hidden states of the LSTM. It is the cell information of LSTM; based on the frequency characteristics of the data, the time series is processed in segments using a sliding window method to enhance the ability to capture changing trends; Feature extraction of trajectory data: For the position coordinate sequence trajectory data of a moving target, the spatiotemporal convolutional network ST-CNN and gated recurrent unit GRU are used to jointly extract trajectory features and capture the motion pattern of the trajectory. in This represents the input position coordinate sequence trajectory data. This represents the obtained trajectory features; These are the weights and bias matrices of the convolutional layer; It is a non-linear activation function; These are the parameters of the GRU layer; Multimodal feature fusion: Mapping and aligning features from different modalities to a unified feature space. This indicates the obtained fusion features. , , , This represents the learnable weight matrix. , , , This represents a learnable bias vector. This represents the activation function. , , , It's about weight allocation.

4. The community security method based on cloud-edge-device collaboration and multimodal analysis as described in claim 3, characterized in that, The multimodal perception module of the edge node receives the fused features, performs multimodal target recognition and behavior tracking, and then performs abnormal behavior detection. When the abnormal score is greater than a preset threshold, the features corresponding to the abnormal score are uploaded to the cloud for further analysis. Specifically: The fused features of the input are utilized using YOLOv4. Perform multimodal target recognition: in This represents the probability of object recognition and classification. Corresponding time Detected target, Indicates category; It is a learnable weight matrix; Indicates dynamic weights, An encoder that represents visual features; Multimodal behavior tracking using the DeepSORT tracking algorithm: in Indicates the target at time. The state vector, Indicates category The relevant dynamic matrix, This represents the state vector from the previous step. It is a learnable cross-modal projection matrix. This represents the trajectory filtering gating function. These are the trajectory features; Joint target recognition and behavior tracking are enhanced with the following features: in This represents the enhanced features obtained; This represents a two-layer fully connected feedforward neural network used for nonlinear fusion; symbol Indicates a splicing operation; Abnormal behavior detection and cloud upload operation: in Indicates abnormal scores; Indicates the category of behavior; Spatiotemporal dependencies are used to capture multimodal features; LSTM is used to analyze trajectory dynamics features. when Push the data to the cloud; The feature corresponding to the abnormal score pushed to the cloud is the first Features of edge nodes : in It is a learnable matrix. It is an activation function, and [;;] represents the concatenation operation.

5. A community security method based on cloud-edge-device collaboration and multimodal analysis as described in claim 1, characterized in that, The cloud platform constructs and maintains a global knowledge base using all features corresponding to historical abnormal scores, and calculates the abnormal probability based on the features corresponding to the abnormal scores uploaded to the cloud, specifically: time Historical abnormal scores Based on the characteristics corresponding to all abnormal scores at the current moment and time global knowledge base The calculation yielded: in It is the first The learnable weights of each feature vector, and ; time global knowledge base via the time Historical abnormal scores and time global knowledge base Update: in Controlling the rate of forgetting; Calculate the probability of anomalies : Among the moments of Indicates an exception label. This indicates a normal label; Indicates the activation function; , Represents the weight matrix and bias matrix; This indicates a dedicated 2-layer sensor for anomaly detection. This represents the activation function. These are all learnable parameters.

6. A community security system featuring cloud-edge-device collaboration and multimodal analysis, characterized in that, The system includes: The data acquisition unit is used to deploy various heterogeneous sensors to collect multimodal data, including images, audio, environmental parameters, and trajectories. The feature fusion unit is used to deploy edge nodes. The feature fusion module of the edge nodes uses deep learning methods to extract each modality feature from the multimodal data, maps each modality feature to a unified feature space and aligns them to form fused features. The edge analysis unit is used by the multimodal perception module of the edge node to receive the fused features, perform multimodal target recognition and behavior tracking, and then perform abnormal behavior detection. When the abnormal score is greater than a preset threshold, the features corresponding to the abnormal score are uploaded to the cloud for further analysis. The cloud analysis unit is used to construct and maintain a global knowledge base using all features corresponding to historical abnormal scores in the cloud, and to calculate the abnormal probability of the features corresponding to the abnormal scores uploaded to the cloud in combination with the global knowledge base. When the abnormal probability is greater than a preset threshold, the decision result is fed back to the edge node and user terminal device to trigger a preset response operation.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images