Community abnormal behavior recognition method based on multi-source data

Abstract

The present application belongs to the technical field of community safety monitoring, and particularly relates to a community abnormal behavior recognition method based on multi-source data, comprising the following steps: collecting video data, acoustic data and Internet of Things equipment data, carrying out denoising processing and feature extraction to obtain video features, acoustic features and Internet of Things features; constructing an initial model, training the initial model through the video features, acoustic features and Internet of Things features to obtain updated features; obtaining a plurality of normal behavior features from a community normal behavior library, carrying out clustering processing to obtain a plurality of normal behavior cluster centers; calculating the updated features and the normal behavior cluster centers to obtain an abnormal score; judging whether the abnormal score is greater than a preset value, if yes, it is an abnormal behavior; if no, it is a normal behavior. The present application can solve some deficiencies existing in the prior art of community abnormal behavior recognition through monitors or audio, and has a good market application prospect.

Description

Technical Field

[0001] This invention belongs to the field of community security monitoring technology, specifically relating to a method for identifying abnormal community behavior based on multi-source data. Background Technology

[0002] Currently, many communities identify abnormal behavior through surveillance cameras or audio recognition, but both methods have shortcomings.

[0003] The shortcomings of using surveillance cameras for identification are as follows: 1. Poor environmental adaptability: The accuracy of the monitor will drop significantly at night or in backlit scenes, and in rainy, snowy or foggy weather, the monitor is likely to lose moving targets. 2. Limited behavioral understanding, unable to identify non-visual threats (such as verbal conflicts or explosions), and ineffective in detecting occluded targets (such as a significant increase in the error rate of identity recognition under mask camouflage). 3. High economic cost: Full coverage requires the deployment of a large number of 4K cameras, which significantly increases costs.

[0004] The shortcomings of using audio for recognition are as follows: 1. Severe noise interference, easily disturbed by other noises (such as the sound of keys falling, square dance music, etc.); 2. Spatial positioning is ambiguous, and traditional sound source positioning has a large error.

[0005] Therefore, this application provides a method for identifying abnormal community behavior based on multi-source data to solve the above problems.

[0006] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0007] The purpose of this invention is to provide a community abnormal behavior identification method based on multi-source data, in order to solve the problems of poor environmental adaptability, limited understanding of behavior, and high economic cost of existing identification methods using monitors, as well as the serious noise interference and ambiguous spatial positioning of identification methods using audio.

[0008] To achieve the above objectives, the present invention provides the following technical solution: A method for identifying abnormal community behavior based on multi-source data includes the following steps: S1. Collect video data through a high-definition camera, acoustic data through a directional microphone array, and data from IoT devices through access control card swiping to provide raw data for subsequent data processing; S2. Use adaptive Gaussian filtering to denoise the video data to obtain denoised video data; use Wiener filtering to denoise the acoustic data to obtain denoised acoustic data. S3. Use the SlowFast network to extract features from the denoised video data to obtain video features; The denoised acoustic data is used for speech recognition using the Mel spectrum algorithm to obtain the recognized acoustic data; the recognized acoustic data is then encoded using the Transformer model to obtain acoustic features. IoT features are obtained by extracting features from IoT device data using a sliding window algorithm; S4. Construct an initial model by training the initial model using video features, acoustic features, and IoT features to obtain updated features; S5. Obtain multiple normal behavior features from the community normal behavior database, and use the K-means clustering algorithm to cluster the multiple normal behavior features to obtain multiple normal behavior cluster centers; S6. Calculate the cluster centers of the updated features and normal behaviors; S7. Determine whether the abnormal score is greater than the preset value. If yes, it is an abnormal behavior; otherwise, it is a normal behavior.

[0009] Preferably, in S4, an initial model is constructed, and the initial model is trained to obtain updated features. This specifically includes the following steps: S41. Define the node corresponding to the video feature as visual node v1, the node corresponding to the acoustic feature as audio node v2, and the node corresponding to the IoT feature as IoT node v3. S42. If the distance between the high-definition camera and the directional microphone array is less than the preset distance, then generate a superedge e1={v1,v2}. If the video features, acoustic features, and IoT features all meet the conditions of abnormal events in the preset database, then a hyperedge e2={v1,v2,v3} will be generated. S43. Calculate the hyperedge weight using the following formula: ; in, For the weight of the superedge, For superedge, For the first Features corresponding to a visual node, audio node, or IoT node. For the first Features corresponding to a visual node, audio node, or IoT node S44, when When the weight is less than the weight threshold, the corresponding superedge is removed; S45. Aggregate the hyperedges using the following formula: ; in, As an aggregation feature, For superedge, The LeakyReLU activation function is used. For the first Features corresponding to a visual node, audio node, or IoT node. and These are learnable parameters; S46. Calculate the loss value for the aggregated features using the following formula: ; in, This is the loss value. These are preset parameters. These are similar abnormal features. As an aggregation feature, for In addition to Features For distance; S47. If the loss value is less than the preset value, then execute S48; otherwise, update the parameters of the initial model according to the loss value and return to S42. S48. Update each feature using the following formula: ; in, For the first The updated features corresponding to each visual node, audio node, or IoT node. As an aggregation feature, For superedge, For the first Features corresponding to a visual node, audio node, or IoT node. For the weight of the superedge, These are learnable parameters.

[0010] As a preferred method, in S6, the updated features and the cluster centers of normal behavior are calculated using the following formula: ; in, For abnormal scoring, For the first A cluster center of normal behavior, For the first Features corresponding to a visual node, audio node, or IoT node.

[0011] Compared with the prior art, the present invention has the following beneficial effects: (1) The community abnormal behavior identification method based on multi-source data of the present invention can connect video clips, audio features and sensor events at the same time, realize the interaction between multimodalities, enhance the correlation of multimodalities, and realize the unified representation of heterogeneous data.

[0012] (2) The community abnormal behavior identification method based on multi-source data of the present invention realizes the learning ability of small samples, reduces the cost of collecting samples, and improves the performance of abnormal identification.

[0013] (3) The community abnormal behavior identification method based on multi-source data of the present invention realizes model learning through weight update, improves anti-interference ability, reduces false alarm rate, and improves model dynamic adaptation and robustness.

[0014] (4) The community abnormal behavior identification method based on multi-source data of the present invention greatly optimizes the calculation steps, reduces the calculation complexity, and optimizes the calculation efficiency compared with the existing methods. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the method steps in an embodiment of the present invention. Detailed Implementation

[0016] The technical solution of this invention patent will be clearly and completely described below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention.

[0017] In the description of this invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention.

[0018] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to a connection within two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0019] See attached document Figure 1 The community abnormal behavior identification method based on multi-source data of the present invention includes the following steps: S1. Collect video data through a high-definition camera, acoustic data through a directional microphone array, and data from IoT devices through access control card swiping to provide raw data for subsequent data processing; S2. Use adaptive Gaussian filtering to denoise the video data to obtain denoised video data; use Wiener filtering to denoise the acoustic data to obtain denoised acoustic data. S3. Use the SlowFast network to extract features from the denoised video data to obtain video features; The denoised acoustic data is used for speech recognition using the Mel spectrum algorithm to obtain the recognized acoustic data; the recognized acoustic data is then encoded using the Transformer model to obtain acoustic features. IoT features are obtained by extracting features from IoT device data using a sliding window algorithm; S4. Construct the initial model, which includes a hyperedge generation module, a weight calculation module, and an aggregation module. Train the initial model using video features, acoustic features, and IoT features to obtain updated features. This includes the following steps: S41. Define the node corresponding to the video feature as visual node v1, the node corresponding to the acoustic feature as audio node v2, and the node corresponding to the IoT feature as IoT node v3. S42. If the distance between the high-definition camera and the directional microphone array is less than the preset distance, then generate a superedge e1={v1,v2}. If the video features, acoustic features, and IoT features all meet the conditions of abnormal events in the preset database, then a hyperedge e2={v1,v2,v3} will be generated. S43. Calculate the hyperedge weight using the following formula: ; in, For the weight of the superedge, For superedge, For the first Features corresponding to a visual node, audio node, or IoT node. For the first Features corresponding to a visual node, audio node, or IoT node S44, when When the weight is less than the weight threshold, the corresponding superedge is removed; S45. Aggregate the hyperedges using the following formula: ; in, As an aggregation feature, For superedge, The LeakyReLU activation function is used. For the first Features corresponding to a visual node, audio node, or IoT node. and These are learnable parameters; S46. Calculate the loss value for the aggregated features using the following formula: ; in, This is the loss value. These are preset parameters. These are similar abnormal features. As an aggregation feature, for In addition to Features For distance; S47. If the loss value is less than the preset value, then execute S48; otherwise, update the parameters of the initial model according to the loss value and return to S42. S48. Update each feature using the following formula: ; in, For the first The updated features corresponding to each visual node, audio node, or IoT node. As an aggregation feature, For superedge, For the first Features corresponding to a visual node, audio node, or IoT node. For the weight of the superedge, These are learnable parameters; S5. Obtain multiple normal behavior features from the community normal behavior database, and use the K-means clustering algorithm to cluster the multiple normal behavior features to obtain multiple normal behavior cluster centers; S6. Calculate the cluster centers of the updated features and normal behavior clusters using the following formula: ; in, For abnormal scoring, For the first A cluster center of normal behavior, For the first Features corresponding to a visual node, audio node, or IoT node; S7. Determine whether the abnormal score is greater than the preset value. If yes, it is an abnormal behavior; otherwise, it is a normal behavior.

[0020] In this embodiment, an anomaly score greater than 1.8 indicates abnormal behavior; a score less than or equal to 1.8 indicates normal behavior. It should be noted that normal behavior includes residents taking walks and delivery personnel making deliveries; abnormal behavior includes strangers loitering in building entrances for extended periods and vehicles illegally parking in fire lanes.

[0021] To achieve multimodal data collection across the entire community, this embodiment deployed various types of sensing devices. For video acquisition, 68 Intel RealSense D455 RGB-D cameras were deployed, averaging one per 150 square meters. These devices support 1080p resolution color video streaming and depth map acquisition at a frame rate of 30fps, simultaneously acquiring two-dimensional visual information and three-dimensional spatial location of targets. The cameras were primarily deployed at key locations such as the community perimeter, building entrances, underground parking garages, and public activity areas.

[0022] For audio acquisition, a ReSpeaker 6-Mic circular microphone array was used, with a total of 32 sets deployed, averaging one unit per 200 square meters. This equipment has six pickup channels, a sampling rate of 16kHz, and can achieve sound source localization and beamforming, effectively suppressing ambient noise. The microphone array and camera were deployed in tandem to ensure synchronized audio-visual coverage.

[0023] Regarding IoT sensors, Panasonic AMN34112 infrared sensors were deployed at the entrance of each building unit for personnel entry and exit detection; Hikvision DS-K2600 access control systems were deployed at all unit doors to record card swipe and facial recognition door opening logs; vibration fiber optic sensors, totaling 860 meters in length, were deployed along the community perimeter wall; and door and window magnetic sensors and smoke detectors were recommended for each household (this test only included public areas). In addition, environmental sensors were deployed at five locations within the community to collect temperature, humidity, and light intensity data for illumination compensation to assist in video analysis.

[0024] The software modules running on the edge nodes include: a SlowFast network that takes an input video stream and outputs a 1024-dimensional behavioral feature vector; a Mel spectrum and Transformer model that converts a 16kHz audio stream into a 256-dimensional speaker embedding vector; and a sensor data processing module that performs sliding window statistics on infrared, access control, and vibration data to generate 64-dimensional temporal features. All feature extraction is completed at the edge, with only the feature vectors uploaded instead of the raw data, significantly reducing network bandwidth and cloud storage pressure.

[0025] The cloud center deploys two NVIDIA A10 GPU servers, each equipped with 24GB of video memory, for initial model training and inference, cross-building global anomaly analysis, and online model updates. The cloud receives feature vectors from each edge node, constructs a global hypergraph, performs multimodal fusion decision-making, and pushes the warning results to property management terminals and security personnel's mobile devices.

[0026] The following are actual examples: At 19:23:15:200 on May 17, 2024, a clear sound of breaking glass was detected by the microphone array on the east side of Building 12 in the community. The audio processing module was immediately triggered, extracting the audio segment one second before and after that moment. After noise reduction and frame-by-frame windowing, a Mel spectrogram was generated and encoded into a 256-dimensional voiceprint embedding vector using a CNN.

[0027] Meanwhile, at 19:23:15:220, the RGB-D camera on the east side of Building 12 captured a rapidly falling object in the frame. The video processing module used a SlowFast network to extract features from this clip, obtaining a 1024-dimensional behavioral feature vector, and simultaneously calculated the object's trajectory and speed using a depth map.

[0028] At 19:23:15:250, the vibration sensors on the ground floor of Building 12 were triggered, generating a voltage signal exceeding 200mV. The sensor data processing module extracted the peak value, duration, and energy integral of this waveform to generate a 64-dimensional time-series feature vector.

[0029] The edge node packages the feature vectors of these three modalities, along with their timestamps and device IDs, and uploads them to the cloud. Upon receiving the data, the cloud first performs spatiotemporal alignment verification. The timestamps of the three devices are synchronized via the NTP protocol, with a maximum time difference of 50 milliseconds. The physical distance between the devices is calculated using pre-stored location coordinates. The distance between the camera and microphone is 8.5 meters, and the distance between the camera and the vibration sensor is 12.3 meters, both within the preset spatiotemporal correlation threshold range (time window 100 milliseconds, spatial radius 20 meters).

[0030] The cloud-based hypergraph neural network generates a hyperedge containing three nodes: video node v1, audio node v2, and vibration node v3. The distance between this embedding and the cluster centers in the normal behavior pattern library is calculated; the minimum distance is 2.47, while the system's preset anomaly detection threshold is 1.8. Since 2.47 significantly exceeds the threshold, the system classifies it as an abnormal event.

[0031] Subsequently, the system activated the multimodal correlation and tracing module. By analyzing the starting position of the falling object in the video frames, combined with the camera's field of view and depth information, the source of the throwing was located to window 3 on the 15th floor of building 12. Through audio wave arrival time difference calculation, the direction of the sound source was consistent with the visual positioning result. The system automatically generated a chain of evidence, including video clips of the falling trajectory, acoustic waveforms of broken glass, and waveforms from vibration sensors.

[0032] From the occurrence of the incident to the generation of the alert, the entire process took 0.3 seconds. 2.1 seconds later, the system automatically activated a drone deployed in the community to verify the situation on-site. 5.7 seconds later, the complete chain of evidence and handling recommendations were pushed to the property management terminal and the mobile phones of security personnel. Subsequent verification revealed that the incident was caused by a flowerpot accidentally falling from a resident's balcony; fortunately, no one was injured, but the timely alert from the system prevented potential risks.

[0033] The community abnormal behavior recognition method based on multi-source data of the present invention enhances the correlation of multimodal data through video features, acoustic features and IoT features, realizes the interaction between multimodal data, and achieves unified representation of heterogeneous data. It improves the anomaly detection performance through hyperedge weights, realizes the learning ability of small samples, reduces the cost of sample collection, and improves the dynamic adaptation and robustness of the model.

[0034] Compared to existing technologies, this invention integrates multi-dimensional features to identify abnormal behavior, breaking the limitations of relying solely on video surveillance. It enables multi-dimensional behavior monitoring across the entire community, taking into account factors such as personnel behavior, environment, and facilities. This allows for the timely detection of abnormal behaviors hidden beneath normal appearances and occurring in blind spots of surveillance, enhancing the comprehensiveness of abnormal behavior monitoring. This is beneficial for solving community safety issues and avoiding problems such as untimely detection and ineffective responses to abnormal behaviors within the community.

[0035] The foregoing description of specific exemplary embodiments of the invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it will be apparent that many changes and variations can be made in accordance with the foregoing teachings. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application, thereby enabling those skilled in the art to implement and utilize various different exemplary embodiments of the invention, as well as various different choices and variations. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

1. A method for identifying abnormal community behavior based on multi-source data, characterized in that, Includes the following steps: S1. Collect video data, acoustic data, and IoT device data to provide raw data for subsequent data processing; S2. Perform noise reduction processing on the video data to obtain the noise-reduced video data; The acoustic data is denoised to obtain the denoised acoustic data. S3. Perform feature extraction on the denoised video data, denoised acoustic data, and IoT device data to obtain video features, acoustic features, and IoT features; S4. Construct an initial model by training the initial model using video features, acoustic features, and IoT features to obtain updated features; S5. Obtain multiple normal behavior features from the community normal behavior database, perform clustering processing, and obtain multiple normal behavior cluster centers; S6. Calculate the anomaly score by comparing the updated features with the cluster centers of normal behavior; S7. Determine whether the abnormal score is greater than the preset value. If yes, it is an abnormal behavior; otherwise, it is a normal behavior.

2. The community abnormal behavior identification method based on multi-source data according to claim 1, characterized in that, The S1 collects video data through a high-definition camera, acoustic data through a directional microphone array, and data from IoT devices through access control card swiping.

3. The community abnormal behavior identification method based on multi-source data according to claim 1, characterized in that, In S2, adaptive Gaussian filtering is used to denoise the video data, and Wiener filtering is used to denoise the acoustic data.

4. The community abnormal behavior identification method based on multi-source data according to claim 1, characterized in that, In S3, the SlowFast network is used to extract features from the denoised video data, and the sliding window algorithm is used to extract features from the IoT device data.

5. The community abnormal behavior identification method based on multi-source data according to claim 1, characterized in that, S3 performs feature extraction on the denoised acoustic data, including the following steps: Speech recognition is performed on the denoised acoustic data using the Mel spectrum algorithm to obtain the recognized acoustic data. The acoustic features are obtained by encoding the recognized acoustic data using the Transformer model.

6. The community abnormal behavior identification method based on multi-source data according to claim 1, characterized in that, In S4, an initial model is built, trained, and updated features are obtained. This process includes the following steps: S41. Define the node corresponding to the video feature as visual node v1, the node corresponding to the acoustic feature as audio node v2, and the node corresponding to the IoT feature as IoT node v3. S42. If the distance between the high-definition camera and the directional microphone array is less than the preset distance, then generate a superedge e1={v1,v2}. If the video features, acoustic features, and IoT features all meet the conditions of abnormal events in the preset database, then a hyperedge e2={v1,v2,v3} will be generated. S43. Calculate the hyperedge weight using the following formula: ； in, For the weight of the superedge, For superedge, For the first Features corresponding to a visual node, audio node, or IoT node. For the first Features corresponding to a visual node, audio node, or IoT node S44, when When the weight is less than the weight threshold, the corresponding superedge is removed; S45. Aggregate the hyperedges using the following formula: ； in, As an aggregation feature, For superedge, The LeakyReLU activation function is used. For the first Features corresponding to a visual node, audio node, or IoT node. and These are learnable parameters; S46. Calculate the loss value for the aggregated features using the following formula: ； in, The loss value. These are preset parameters. These are similar abnormal features. As an aggregation feature, for In addition to Features For distance; S47. If the loss value is less than the preset value, then execute S48; otherwise, update the parameters of the initial model according to the loss value and return to S42. S48. Update each feature using the following formula: ； in, For the first The updated features corresponding to each visual node, audio node, or IoT node. As an aggregation feature, For superedge, For the first Features corresponding to a visual node, audio node, or IoT node. For the weight of the superedge, These are learnable parameters.

7. The community abnormal behavior identification method based on multi-source data according to claim 1, characterized in that, S5 uses the K-means clustering algorithm to cluster multiple normal behavioral features.

8. The community abnormal behavior identification method based on multi-source data according to claim 1, characterized in that, In S6, the updated features and the cluster centers of normal behavior are calculated using the following formula: ； in, For abnormal scoring, For the first A cluster center of normal behavior, For the first Features corresponding to a visual node, audio node, or IoT node.

Images