Abnormal behavior multi-mode identification method and system for smart community

Abstract

The invention relates to the technical field of abnormal behavior recognition, in particular to an abnormal behavior multi-mode recognition method and system for a smart community. Comprising the following steps: multi-modal data acquisition and preprocessing: synchronously acquiring spatio-temporal dynamic data of a target area through a millimeter wave radar, an infrared sensor, a microphone array and a panoramic camera deployed in a community, wherein the spatio-temporal dynamic data comprises a human body motion track, temperature field distribution, environment voiceprint characteristics and a visual image sequence; wherein the millimeter-wave radar adopts a frequency modulation continuous wave system to sample micro-Doppler characteristics of a human body, the infrared sensor realizes temperature field reconstruction through the pyroelectric array, the microphone array collects voiceprint signals through the array structure, and the panoramic camera collects a high-frame-rate image sequence; and environment interference coupling correction: aiming at acoustic reflection characteristics of different materials in the community, and combining an illumination abrupt change rate, an object reflectivity and an air humidity parameter, the method can improve the recognition accuracy and robustness in a complex community environment.

Description

Technical Field

[0001] This invention relates to the field of abnormal behavior recognition technology, and in particular to a method and system for multimodal recognition of abnormal behavior in smart communities. Background Technology

[0002] In the field of smart community security management, multimodal recognition technology for abnormal behavior is a core support for ensuring residents' safety. It achieves real-time monitoring of dangerous behaviors by integrating multi-source data such as vision, audio, and infrared. Existing technologies for this type of recognition often rely on the simple superposition or independent processing of single-modal features, revealing significant limitations in complex community environments. Specifically, existing technologies fail to effectively handle the coupled interference of multiple environmental factors: the difference in acoustic reflection characteristics between concrete walls and wooden benches within the community leads to audio feature distortion; sudden changes in midday sunlight combined with the high reflectivity of metal fences can cause visual feature distortion; and changes in air humidity can simultaneously exacerbate sound velocity attenuation and light scattering. These interferences are not independent but rather form a superimposed effect. Existing methods only correct for single interferences, resulting in distorted confidence levels for multimodal features. Simultaneously, the heterogeneity of multi-source devices leads to prominent temporal mismatch issues: frame rate deviations between different brands of cameras cause temporal misalignment of visual features; temperature drift caused by equipment heating in summer can deviate from temperature detection values. Existing technologies use uniform weights for temporal alignment without considering the impact of differences in device types on temporal consistency, further reducing feature reliability. Furthermore, the judgment of abnormal behavior is detached from the dynamic characteristics of the scene: the same rapid arm swing is normal behavior in the fitness area, but may be abnormal in the children's rest area. However, the existing technology sets fixed behavior thresholds for different scenes and does not adjust the judgment logic according to the scene's safety level and real-time crowd density, resulting in a high false alarm rate and difficulty in adapting to the safety management needs of diverse community scenarios.

[0003] Based on the above problems, there is an urgent need for an abnormal behavior recognition technology that can accurately handle the coupling interference of multiple environmental factors, solve the timing mismatch of heterogeneous devices, and realize the dynamic association between behavior and scene, so as to improve the recognition accuracy and robustness in complex community environments. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies and propose a multimodal method for identifying abnormal behavior in smart communities, comprising: Multimodal data acquisition and preprocessing: Spatiotemporal dynamic data of the target area are simultaneously acquired by millimeter-wave radar, infrared sensors, microphone arrays and panoramic cameras deployed in the community, including human motion trajectory, temperature field distribution, environmental acoustic features and visual image sequences. Among them, millimeter-wave radar uses frequency-modulated continuous wave mode to sample human micro-Doppler features, infrared sensors realize temperature field reconstruction through pyroelectric array, microphone array collects acoustic signals through array structure, and panoramic camera collects high frame rate image sequences. Environmental interference coupling correction: Based on the acoustic reflection characteristics of different materials in the community, and combined with the light abrupt change rate, object reflectivity and air humidity parameters, a multi-factor coupling model is constructed to correct the confidence of the original multimodal features, forming a corrected feature set; Heterogeneous device timing alignment: Based on the corrected feature set, a timing consistency model including frame rate deviation, sensor temperature drift and timestamp synchronization error is established, and spatiotemporal alignment of multi-source device data is achieved through a weighted fusion algorithm; Scene and behavior correlation reasoning: Construct a three-dimensional decision space that integrates scene security level, real-time crowd density and action speed threshold. Based on the output of the time-series consistency model, output an anomaly judgment index through a progressive weighted algorithm. When the index exceeds the preset threshold, a multi-level early warning mechanism is triggered.

[0005] Preferably, in the multimodal data acquisition and preprocessing step: Millimeter-wave radar uses time-division multiplexing technology to achieve range resolution through multiple transmission channels, and combines Kalman filtering algorithm to predict the trajectory of moving targets in order to eliminate trajectory jumps caused by non-uniform motion. The infrared sensor integrates a thermoelectric cooler and achieves baseline stability adjustment when the ambient temperature changes through closed-loop temperature control. At the same time, a median filtering algorithm is used to denoise the temperature field data to suppress the interference of sudden noise on the temperature field distribution. The microphone array uses a direction-of-arrival estimation algorithm to locate the sound source, combines blind source separation technology to separate environmental noise from human voice signals, and uses a feature dimensionality reduction algorithm to compress the extracted voiceprint feature vector. The panoramic camera uses a fisheye lens in conjunction with a distortion correction algorithm to eliminate optical distortion, and uses optical flow to perform motion compensation on adjacent frames to reduce temporal misalignment of image sequences caused by device jitter.

[0006] More preferably, the environmental interference coupling correction step includes: Establish material acoustic coefficient matrix Concrete, wood, and metal each have different acoustic attenuation coefficients, which are used to quantify the influence of materials on sound wave propagation. Real-time monitoring of light-induced mutation rate Dynamic detection of sudden changes in illumination is achieved by combining a photosensitive sensor array with a Gaussian mixture model, in order to distinguish between natural illumination changes and sudden light source interference. A humidity influencing factor is introduced. Air humidity data is collected through a humidity sensor. Based on the correlation characteristics between humidity and the speed of sound and the speed of light, the confidence of audio and video features is jointly corrected to compensate for signal attenuation in high humidity environments. When correcting illumination distortion in visual features, the differences in light reflection on different object surfaces are quantified by combining the object's reflectivity with the cosine of the incident angle of light, so as to eliminate brightness deviations caused by different object materials.

[0007] More preferably, the device heterogeneous timing alignment step includes: A frame rate deviation compensation model is established. By comparing the deviation between the actual frame rate and the standard frame rate of each device, different deviation weights are assigned according to the device type. Different weights are used for visual devices and acoustic devices to match their signal characteristics. To address the temperature drift characteristics of infrared sensors, a correction function based on temperature deviation and the device's temperature drift coefficient is designed. The temperature deviation is calculated by real-time acquisition of the sensor's operating temperature to compensate for the impact of temperature changes on the detected values. A network time protocol combined with a hardware phase-locked loop is used to achieve timestamp synchronization of multiple devices. A dynamic time warping algorithm is used to perform feature interpolation on the synchronized time series data to eliminate timing misalignment caused by differences in sampling periods of different devices. During the time alignment process, the time features of different devices are weighted and fused using the confidence level of the corrected feature set as the weight. Features with higher confidence levels have a larger proportion in the fusion, thereby improving the reliability of the time consistency model.

[0008] A multimodal recognition system for abnormal behavior in smart communities, applied to any one of the above-described methods for multimodal recognition of abnormal behavior in smart communities, includes: The multimodal sensing layer consists of a millimeter-wave radar array, an infrared sensor matrix, a microphone array, and a panoramic camera. Each sensor is connected to the data aggregation unit via a communication interface and transmits data in real time using a preset communication protocol. Millimeter-wave radar arrays are deployed at community road intersections and activity squares to collect micro-Doppler features of human movement; Infrared sensor matrices are installed at the entrance of the unit building and inside the elevator car to collect the temperature field distribution within the area; Microphone arrays are deployed in children's activity areas and elderly rest areas to collect environmental sound patterns and human vocal signals; Panoramic cameras cover public areas of the community and are used to capture high-definition visual image sequences; Edge computing layer: Deploys edge computing servers, integrates neural network acceleration engines, and enables real-time feature extraction and preliminary processing of multimodal data, including: Millimeter-wave radar signal processing module: Uses short-time Fourier transform to perform time-frequency analysis on frequency-modulated continuous wave signals and extracts the micro-Doppler features of moving targets; Infrared image processing module: Noise reduction of temperature field images is achieved through nonlocal mean filtering algorithm, and human target and environment are segmented by watershed algorithm; Audio signal processing module: Extracts voiceprint features based on Mel frequency cepstral coefficients and uses a hidden Markov model for speech activity detection; Visual analysis module: It uses target detection algorithms combined with multi-target trackers to achieve real-time detection and trajectory tracking of key points on the human body; Cloud-based decision-making layer: Building deep learning models based on cloud servers, including: Environmental interference correction network: A convolutional neural network combined with an attention mechanism is used to correct the confidence of multimodal features, and the attention weights are dynamically adjusted according to environmental parameters; Temporal alignment network: Design a long short-term memory network-conditional random field architecture to learn the impact of device heterogeneity on temporal consistency and output the temporally aligned feature sequence; Scene-behavior association network: Construct a graph neural network that integrates community map data, real-time pedestrian density data, and historical behavior data to output an anomaly detection index; Early warning execution layer: includes sound and light alarms, mobile terminal push module and community management platform interface, supports triggering early warnings in stages according to the anomaly judgment index, where yellow warnings are only pushed to the community security terminal, orange warnings are pushed to the community management platform at the same time, and red warnings are linked to the public security system to achieve data interoperability.

[0009] More preferably, the environmental interference correction network uses the following formula to correct the confidence level of multimodal features: ; in: The corrected confidence level for multimodal features; The original feature confidence score is output by the feature extraction module of the edge computing layer. For audio interference weights, For video interference weights, and ; The acoustic coefficient of the material is 0.8 for concrete, 0.3 for wood, and 0.9 for metal. The distance between the sound source and the microphone array; The main frequency of the audio signal; The light-induced mutation rate; The reflectivity of the object is 0.9 for metal and 0.2 for fabric. The angle of incidence of the light ray; Humidity sensitivity coefficient; This refers to air humidity.

[0010] More preferably, the time alignment network calculates the time consistency score using the following formula:

[0011] in: Assess time series consistency. The number of time-series feature frames; For the first The confidence level of the frame's modified features is calculated using the formula in claim 6; The frame rate sensitivity coefficient is set to 0.7 for the camera and 0.3 for the microphone. For the first The actual frame rate of the device; Standard frame rate; The weighting is affected by temperature drift; For the first The temperature deviation of a sensor is the difference between the actual operating temperature and the calibration temperature of the sensor. The temperature drift coefficient of the sensor is set to 1.2 for infrared devices and 0.5 for millimeter-wave radar. For the first The frame timestamp synchronization error correction coefficient is obtained by joint calibration of the network time protocol and the hardware phase-locked loop.

[0012] More preferably, the scene-behavior association network calculates the anomaly determination index using the following formula: ; in: This is an anomaly detection index; The temporal consistency score is calculated by the temporal alignment network using the following formula; The scenario-behavior correlation coefficient is obtained by training on historical accident data from the community. Target speed of action; The average motion speed threshold for the current scene is defined, with different thresholds for the fitness area, rest area, and children's area. Real-time pedestrian density for the scene; Based on the scene safety level, the power distribution room, children's area, and fitness area correspond to 10, 8, and 5 respectively; The highest security level for the community is represented by a value of 10.

[0013] More preferably, the signal processing modules of the edge computing layer work together in the following manner: The millimeter-wave radar signal processing module sends the extracted micro-Doppler features to the data fusion unit in the form of data frames. Each frame contains a timestamp, target ID, and micro-Doppler spectrum data. The infrared image processing module spatially registers the segmented human body temperature field data with the human body target bounding box output by the visual analysis module. The registration process is achieved through coordinate system transformation, and the transformation parameters are pre-calibrated based on the sensor installation position. The audio signal processing module performs activity detection on the extracted voiceprint features. When voice activity is detected, the visual analysis module is triggered to focus on tracking the direction of the sound source. The size of the tracking window is dynamically adjusted according to the sound source localization results. The processing results of each module are exchanged through shared memory. The interaction process uses a mutex lock mechanism to avoid data conflicts, and the processing latency is controlled within 500ms to meet real-time requirements.

[0014] More preferably, the sensor deployment of the multimodal sensing layer satisfies the following logic: The millimeter-wave radar array and the panoramic camera are deployed at the same location, with the installation height difference between the millimeter-wave radar array and the panoramic camera not exceeding 0.5m to ensure the consistency of the spatial coordinates of the motion trajectory and the visual image. The overlap between the beam coverage of the millimeter-wave radar array and the field of view of the panoramic camera is not less than 90%. The sampling period of the infrared sensor matrix is ​​kept as an integer multiple of the sampling period of the microphone array to ensure time alignment between the temperature field data and the acoustic data. The sampling period of the infrared sensor is 50ms and the sampling period of the microphone array is 25ms. The clock source of all sensors is synchronized with the community's unified time server. The synchronization period is 1 second. The synchronization process is achieved through network data packet interaction. The data packet contains the sending timestamp and the receiving timestamp for calculating the clock deviation. For areas with signal obstruction within the community, a multi-sensor cross-coverage strategy is adopted, meaning that the same target area is covered by at least two different types of sensors. Obstruction areas include building shadow areas, tree obstruction areas, and corner blind spots. The cross-coverage sensor data is redundantly verified through a data aggregation unit. When the confidence level of one sensor data is lower than 0.3, data from another sensor is used to supplement it.

[0015] Technical effects: This invention achieves significant technical effects through a three-tiered progressive technical solution: it creatively constructs a chain of environmental interference coupling correction, heterogeneous device timing alignment, and scene-behavior correlation reasoning processing, specifically addressing the core problem in the background technology—feature distortion caused by multi-factor environmental factors, material acoustic reflection, light reflection, and humidity coupling interference, correcting feature confidence through a multi-factor coupling model; addressing timing mismatch caused by device heterogeneity, frame rate deviation, and temperature drift, achieving spatiotemporal alignment through a timing consistency model; and overcoming the limitations of scene dynamics in behavior determination, achieving scene adaptation through a three-dimensional decision space. Ultimately, this improves the accuracy and robustness of anomaly identification in complex community environments, meeting diverse security management needs. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a multimodal method for identifying abnormal behavior in smart communities, as described in this application. Figure 2 This is a block diagram of a multimodal recognition system for abnormal behavior in smart communities, as described in this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0018] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0019] Traditional methods for identifying abnormal behavior in smart communities suffer from the following technical problems: existing methods often rely on single-modal data or simply fuse multimodal data, failing to consider the coupling effect of multiple factors in the complex environment of a community, such as the superimposed effects of acoustic and light reflections from different materials; the heterogeneous characteristics of multi-source devices, such as frame rate and temperature drift, lead to temporal mismatch, reducing data consistency; and the association between behavior and scene is only judged simply, without considering dynamic parameters such as scene security level and pedestrian density, resulting in insufficient accuracy and robustness in anomaly identification, making it difficult to adapt to the diverse needs of community scenarios.

[0020] Based on this, please refer to Figure 1This embodiment provides a multimodal identification method for abnormal behavior in smart communities, including the following steps: Multimodal data acquisition and preprocessing: Spatiotemporal dynamic data of the target area are simultaneously acquired by millimeter-wave radar, infrared sensors, microphone arrays and panoramic cameras deployed in the community, including human motion trajectory, temperature field distribution, environmental acoustic features and visual image sequences. Among them, millimeter-wave radar uses frequency-modulated continuous wave mode to sample human micro-Doppler features, infrared sensors realize temperature field reconstruction through pyroelectric array, microphone array collects acoustic signals through array structure, and panoramic camera collects high frame rate image sequences. Environmental interference coupling correction: Based on the acoustic reflection characteristics of different materials in the community, and combined with the light abrupt change rate, object reflectivity and air humidity parameters, a multi-factor coupling model is constructed to correct the confidence of the original multimodal features, forming a corrected feature set; Heterogeneous device timing alignment: Based on the corrected feature set, a timing consistency model including frame rate deviation, sensor temperature drift and timestamp synchronization error is established, and spatiotemporal alignment of multi-source device data is achieved through a weighted fusion algorithm; Scene-behavior association reasoning: Construct a three-dimensional decision space that integrates scene security level, real-time crowd density and action speed threshold. Based on the output of the time-series consistency model, an anomaly judgment index is output through a progressive weighted algorithm. When the index exceeds the preset threshold, a multi-level early warning mechanism is triggered.

[0021] This solution employs a three-tiered progressive processing chain. First, it corrects the coupling interference from multiple environmental factors. Then, it addresses the timing mismatch between heterogeneous devices. Finally, it combines dynamic parameters of the scenario to achieve behavioral correlation reasoning. This makes multimodal features closer to the real state, timing data alignment more accurate, and behavior anomaly judgment more in line with the characteristics of the community scenario. It effectively improves the reliability and adaptability of anomaly identification and provides more accurate decision support for smart community security management.

[0022] Traditional multimodal data acquisition processes suffer from the following technical problems: In existing acquisition methods, millimeter-wave radar is prone to abrupt changes in predicting the trajectory of non-uniformly moving targets; infrared sensors are susceptible to sudden noise interference, leading to distortion of the temperature field distribution; acoustic feature redundancy affects subsequent processing efficiency; and panoramic cameras experience temporal misalignment of image sequences due to device jitter. These problems result in insufficient raw data quality, affecting the accuracy of subsequent feature extraction.

[0023] Based on this, in the multimodal data acquisition and preprocessing steps: the millimeter-wave radar uses time-division multiplexing technology to achieve range resolution through multiple transmission channels, and combines Kalman filtering algorithm to predict the trajectory of moving targets to eliminate trajectory jumps caused by non-uniform motion; the infrared sensor integrates a thermoelectric cooler, and achieves baseline stability adjustment when the ambient temperature changes through temperature closed-loop control, while using median filtering algorithm to denoise the temperature field data to suppress the interference of sudden noise on the temperature field distribution; the microphone array achieves sound source localization through direction-of-arrival estimation algorithm, combines blind source separation technology to separate ambient noise from human voice signals, and uses feature dimensionality reduction algorithm to compress the dimension of the extracted voiceprint feature vector; the panoramic camera uses a fisheye lens with distortion correction algorithm to eliminate optical distortion, and uses optical flow method to perform motion compensation on adjacent frames to reduce temporal misalignment of image sequences caused by device jitter.

[0024] This solution addresses the specific characteristics and defects of different sensors by employing targeted processing strategies: Kalman filtering ensures trajectory continuity, temperature closed-loop control and median filtering improve the reliability of the temperature field, blind source separation and dimensionality reduction optimize the quality of acoustic features, distortion correction and optical flow compensation improve the temporal consistency of images, and enhance the integrity and accuracy of multimodal data from the data source, providing high-quality input for subsequent environmental interference correction and temporal alignment.

[0025] Traditional environmental interference processing methods have the following technical problems: Existing technologies often target single types of interference, such as only processing illumination or only processing acoustic attenuation, without considering the coupling effect of multiple factors such as acoustic reflection, illumination changes, and air humidity in the community environment, resulting in incomplete feature confidence correction; insufficient attention is paid to marginal factors that affect visual features, such as object reflectivity and light incident angle, and the correlation between humidity and audio and video features is not utilized, so that the corrected features still have biases, affecting the accuracy of subsequent recognition.

[0026] Based on this, the environmental interference coupling correction step includes: establishing the material acoustic coefficient matrix K. m The system employs different acoustic attenuation coefficients for concrete, wood, and metal to quantify the impact of materials on sound wave propagation. It also monitors the real-time illumination change rate ΔL, using a photosensitive sensor array combined with a Gaussian mixture model to dynamically detect illumination changes and distinguish between natural illumination variations and sudden light source interference. Furthermore, it introduces a humidity factor, collecting air humidity data through a humidity sensor and using the correlation between humidity and the speed of sound and light to jointly correct the confidence levels of audio and video features, compensating for signal attenuation in high-humidity environments. Finally, when correcting illumination distortion in visual features, it combines the object's reflectivity with the cosine of the incident angle of light to quantify the differences in light reflection across different object surfaces, eliminating brightness deviations caused by different object materials.

[0027] This scheme overcomes the limitations of single-interference processing by constructing a multi-factor coupled correction mechanism: the material acoustic coefficient matrix accurately quantifies the attenuation differences of different materials on sound waves, the Gaussian mixture model realizes the fine distinction of light abrupt changes, the humidity influence factor is associated with the synchronous correction of audio and video features, and the combination of object reflectivity and incident angle improves the visual features' resistance to light interference, making the corrected multimodal features closer to the real state, providing more reliable basic data for subsequent time-series alignment, and reducing feature distortion caused by environmental interference.

[0028] Traditional device timing alignment methods suffer from the following technical problems: When dealing with timing issues involving multiple devices, existing technologies do not adequately consider the impact of device heterogeneity, such as frame rate deviation and sensor temperature drift. They often use uniform weights for timing fusion without dynamically adjusting the weights based on device type and feature confidence. The time synchronization mechanism is simple and relies on software protocols without combining hardware calibration methods, resulting in large timestamp synchronization errors. Furthermore, there is a lack of effective interpolation compensation strategies for timing misalignments caused by differences in sampling periods of different devices, resulting in consistency deviations in the aligned timing data and affecting the spatiotemporal correlation of multimodal features.

[0029] Based on this, the heterogeneous timing alignment steps include: establishing a frame rate deviation compensation model, comparing the deviation between the actual frame rate and the standard frame rate of each device, and assigning different deviation weights based on the device type, wherein visual devices and acoustic devices use differentiated weights to match their signal characteristics; designing a correction function based on temperature deviation and device temperature drift coefficient for the temperature drift characteristics of infrared sensors, calculating the temperature deviation by real-time acquisition of sensor operating temperature to compensate for the impact of temperature changes on the detection value; using a network time protocol combined with a hardware phase-locked loop to achieve timestamp synchronization of multiple devices, and performing feature interpolation on the synchronized timing data through a dynamic time warping algorithm to eliminate timing misalignment caused by differences in sampling periods of different devices; during the timing alignment process, using the confidence level of the corrected feature set as the weight, weighted fusion of timing features of different devices is performed, with features having a higher confidence level accounting for a larger proportion in the fusion, in order to improve the reliability of the timing consistency model.

[0030] This solution addresses the timing issues of heterogeneous devices through a multi-level timing optimization strategy: differentiated frame rate weights adapt to device characteristics, a temperature drift correction function compensates for the impact of temperature on detection values, a hardware-software combined synchronization mechanism reduces timestamp errors, and dynamic time warping and confidence-weighted fusion improve timing consistency. This enables precise alignment of multi-source device data in the spatiotemporal dimension, ensures the temporal correlation of multimodal features, and provides spatiotemporally consistent feature inputs for scene-behavior association reasoning.

[0031] Traditional abnormal behavior recognition systems suffer from the following technical problems: existing systems mostly adopt a simple architecture of acquisition, transmission, and centralized processing, with weak edge processing capabilities, resulting in large latency and insufficient real-time performance of multimodal data transmission; poor inter-module coordination, lack of targeted deployment of sensors in the perception layer, unclear functional division between the edge computing layer and the cloud decision-making layer, and failure to fully utilize the real-time processing advantages of edge computing and the large-scale data analysis capabilities of the cloud; unclear early warning mechanism classification, single response strategy for different levels of anomalies, making it difficult to meet the diverse security management needs of communities, and the overall efficiency and adaptability of the system are insufficient.

[0032] Based on this, please refer to Figure 2 This embodiment provides a smart community abnormal behavior multimodal recognition system for implementing the method, including: The multimodal sensing layer consists of a millimeter-wave radar array, an infrared sensor matrix, a microphone array, and a panoramic camera. Each sensor is connected to the data aggregation unit via a communication interface and transmits data in real time using a preset communication protocol. Among them: millimeter-wave radar arrays are deployed at community road intersections and activity squares to collect micro-Doppler features of human movement; infrared sensor matrices are installed at the entrances of unit buildings and inside elevator cars to collect temperature field distribution within the area; microphone arrays are deployed in children's activity areas and elderly rest areas to collect environmental sound patterns and human voice signals; panoramic cameras cover the community's public areas to collect high-definition visual image sequences. Edge computing layer: Deploy edge computing servers, integrate neural network acceleration engines, and realize real-time feature extraction and preliminary processing of multimodal data; This includes: a millimeter-wave radar signal processing module that uses short-time Fourier transform to perform time-frequency analysis on frequency-modulated continuous wave signals and extracts the micro-Doppler features of moving targets; an infrared image processing module that uses a non-local mean filtering algorithm to reduce noise in temperature field images and combines it with a watershed algorithm to segment human targets from the environment; an audio signal processing module that extracts voiceprint features based on Mel frequency cepstral coefficients and uses a hidden Markov model for speech activity detection; and a visual analysis module that uses a target detection algorithm combined with a multi-target tracker to achieve real-time detection and trajectory tracking of key human points. Cloud-based decision layer: Deep learning models are built on cloud servers, including: an environmental interference correction network that uses a convolutional neural network combined with an attention mechanism to correct the confidence of multimodal features, with attention weights dynamically adjusted according to environmental parameters; a temporal alignment network that uses a long short-term memory network-conditional random field architecture to learn the impact of device heterogeneity on temporal consistency and outputs a temporally aligned feature sequence; and a scene-behavior association network that builds a graph neural network to integrate community map data, real-time pedestrian density data, and historical behavior data to output an anomaly detection index. The early warning execution layer includes audible and visual alarms, mobile terminal push modules, and interfaces to the community management platform. It supports tiered early warning triggering based on anomaly assessment indices. Yellow warnings are only pushed to community security terminals, orange warnings are simultaneously pushed to the community management platform, and red warnings are linked to the public security system for data exchange. This system achieves efficient collaboration through a layered architecture: the perception layer deploys sensors specifically tailored to regional characteristics, ensuring targeted and comprehensive data collection; the edge computing layer handles real-time feature extraction, reducing data transmission volume and improving response speed; the cloud decision-making layer utilizes deep learning models for complex reasoning, leveraging the advantages of large-scale data processing; and the tiered early warning mechanism adapts to different anomaly levels, improving the accuracy of system response. Each layer has clearly defined functions and works closely together, fully utilizing the technological advantages of the edge and cloud to achieve real-time perception, accurate identification, and tiered response to abnormal behavior, meeting the efficiency and reliability requirements of smart community security management.

[0033] Traditional environmental interference correction models have the following technical problems: Existing technologies mostly use simple linear formulas to correct the confidence of multimodal features, without quantifying the coupled effects of different environmental factors, such as material acoustic properties, illumination changes, and air humidity, resulting in a large deviation between the correction results and the actual features; the parameters in the formula are set in a fixed manner, and the weights cannot be dynamically adjusted according to the environmental characteristics of different areas of the community; the interference correction of audio and video features is independent of each other, and the synergistic effect of factors such as humidity on the two types of features is ignored, so the corrected features are still distorted, affecting the accuracy of subsequent temporal alignment.

[0034] Based on this, the environmental interference correction network uses the following formula to correct the confidence of multimodal features: ; in: The corrected confidence level for multimodal features; The original feature confidence score is output by the feature extraction module of the edge computing layer. For audio interference weights, For video interference weights, and ; The acoustic coefficient of the material is 0.8 for concrete, 0.3 for wood, and 0.9 for metal. The distance between the sound source and the microphone array; The main frequency of the audio signal; The light-induced mutation rate; The reflectivity of the object is 0.9 for metal and 0.2 for fabric. The angle of incidence of the light ray; Humidity sensitivity coefficient; This refers to air humidity.

[0035] The core function of this formula is to quantify the coupled interference of audio echo attenuation and video illumination distortion in a community environment. By using multiple factors to collaboratively correct the confidence of the original features, it solves the problem of incomplete correction of single interference in traditional techniques.

[0036] From an overall structural perspective, the formula uses the confidence level of the original features. Based on this, precise adjustments are achieved through three correction factors: The first item is for audio interference correction. :in is the acoustic coefficient of the material, used to quantify the ability of different materials to reflect and attenuate sound waves. Concrete is given a value of 0.8 due to its high density, while wood is given a value of 0.3 due to its low porosity. The distance to the sound source is the cube of the distance, which reflects the non-linear increase of sound wave attenuation with distance. When the distance doubles, the attenuation increases to 8 times. The square of the dominant frequency reflects the characteristic that high-frequency sound waves (such as screams) decay faster than low-frequency sound waves (such as conversations); The audio interference weights are dynamically adjusted based on regional characteristics; for example, audio interference is stronger in community squares due to their openness. Take 0.6; video interference is more significant in the corridor. Take 0.3.

[0037] The second item is the video interference correction item. : Light intensity abrupt change rate measures the magnitude of change in light intensity per unit time, such as the sudden change in light intensity caused by the movement of tree shade at noon. The difference in light reflection between metal (high reflectivity, 0.9) and fabric (low reflectivity, 0.2) is used to distinguish the reflectivity of an object. It reflects direct light. , Compared to oblique projection, , The resulting brightness distortion is significant. (Video interference weight) and Complementary ( This ensures that the correction weights for the two types of disturbances are balanced.

[0038] The third item is the humidity co-correction item. : Employing an exponential function rather than a nonlinear relationship, due to air humidity The interference with the speed of sound and the speed of light increases nonlinearly, and the signal attenuation rate increases sharply when the humidity is >70%. The humidity sensitivity coefficient is obtained by fitting the correlation between the community's annual humidity data and the feature distortion rate, and is set to 0.02% to ensure stronger correction in high humidity environments.

[0039] Interlocking logic of various parameters: When a metal fence appears in the community square Nearby high sound sources At the same time, the midday sunlight suddenly changed. At that time, the formula amplifies the interference of material and distance through the audio term, strengthens the influence of lighting and reflection through the video term, and the humidity term, if Further reduce the confidence level, It accurately reflects the reliability of features. This formula breaks through the limitations of traditional single-interference correction, achieving quantitative compensation for multi-factor coupled interference, and providing high-quality feature input for subsequent time-series alignment.

[0040] This scheme achieves accurate quantification of environmental interference by constructing a nonlinear correction formula involving multiple coupled factors: the audio interference term in the formula... By leveraging the synergistic effects of material, distance, and dominant frequency, sound wave attenuation under different scenarios is quantified; video interference items... By combining abrupt changes in illumination, the reflective properties of objects, and the angle of light, it precisely corrects illumination distortions affecting visual characteristics; humidity influence factor. By using an exponential function to correlate the synchronous correction of audio and video features, the synergistic effect of high humidity on the speed of sound and the speed of light is demonstrated. and The dynamic weight allocation can be adjusted according to the characteristics of the community area (such as the square being mainly affected by audio interference, and the children's area being mainly affected by video interference), making the correction more in line with the actual environment.

[0041] The revised feature confidence score more accurately reflects the reliability of multimodal data, providing high-quality input for subsequent time series alignment and reducing feature bias caused by environmental interference. It is worth noting that the values ​​of each parameter in the formula were determined based on typical community environment testing. The reflection coefficients of different materials for sound waves in the 1kHz-4kHz range were obtained through actual measurements. Reflectivity calibration for common objects in the community, such as metal fences and fabric chairs. The formula is derived by fitting the correlation between the community's annual humidity variation and the characteristic distortion rate, ensuring the applicability of the formula under different environments.

[0042] The technical effects achieved by the above scheme include: achieving fine correction of environmental interference through multi-factor coupling formula, quantitatively compensating for the interference of audio features by material and distance, accurately correcting the distortion of video features by illumination and object reflection, incorporating the synergistic effect of humidity on the two types of features into a unified model, significantly improving the matching degree between the corrected feature confidence and the actual features, providing more reliable input data for the temporal alignment network, and reducing abnormal recognition errors caused by environmental interference.

[0043] Traditional temporal consistency models suffer from the following technical problems: Existing technologies do not incorporate feature confidence into the weighting factor when calculating temporal consistency scores; they apply equal weight to frame rate deviations and temperature drift effects across different devices, neglecting device types such as the signal characteristic differences between visual and acoustic devices; timestamp synchronization relies solely on software protocols, lacking hardware calibration methods, leading to inaccurate synchronization error correction coefficients; the formula does not reflect the impact of temporal frame count on the score, and the calculation of temperature deviation simply uses a linear relationship without considering the characteristics of device temperature drift coefficients, resulting in a mismatch between the temporal consistency score and the actual temporal correlation of the data, affecting the accuracy of scene-behavior association inference.

[0044] Based on this, the time alignment network uses the following formula to calculate the time consistency score: ; in: Assess time series consistency. The number of time-series feature frames; For the first The confidence level of the frame's modified features is calculated using the formula in claim 6; The frame rate sensitivity coefficient is set to 0.7 for the camera and 0.3 for the microphone. For the first The actual frame rate of the device; Standard frame rate; The weighting is affected by temperature drift; For the first The temperature deviation of a sensor is the difference between the actual operating temperature and the calibration temperature of the sensor. The temperature drift coefficient of the sensor is set to 1.2 for infrared devices and 0.5 for millimeter-wave radar. For the first The frame timestamp synchronization error correction coefficient is obtained by joint calibration of the network time protocol and the hardware phase-locked loop.

[0045] The core function of this formula is to quantify the timing mismatch caused by frame rate deviation, temperature drift and timestamp error of multi-source devices, and to achieve spatiotemporal alignment of heterogeneous data through weighted fusion, thereby solving the problem of uniform weight processing of device differences in traditional technology.

[0046] From an overall structural perspective, the formula calculates temporal consistency by weighted averaging of features from N frames, and the functions of each component are as follows: Sum of the average terms To eliminate the impact of differences in the number of time-series frames, such as N=200 during the morning rush hour when there is a large flow of people in the community and N=50 at night, so that time-series sequences of different durations are comparable.

[0047] Feature confidence weights The revised confidence level introduced by the above formula allows high-confidence frames, such as clear visual frames, to account for a larger proportion of the score, while reducing the impact of low-confidence frames, such as blurry audio frames, thus improving the reliability of the score.

[0048] Frame rate deviation correction item : and The relative deviation reflects the degree of timing misalignment; Differentiated values ​​are used because visual frames are more closely correlated and frame rate deviations have a greater impact, while acoustic signals are less sensitive to frame rate.

[0049] Temperature drift correction item : and The effect of product quantization temperature on infrared equipment Due to its strong temperature dependence; radar It is less affected by temperature; The temperature drift weight was determined experimentally and set to 0.6 to ensure that the temperature correction strength is adapted to the characteristics of the equipment.

[0050] Timestamp synchronization correction It is derived from the NTP protocol and hardware phase-locked loop, which dynamically compensates for the clock deviation of different devices, ranging from 0.9 to 1.1, to ensure consistent time base.

[0051] Interlocking logic of various parameters: When the infrared sensor inside the elevator... Due to rising summer temperatures Deviation, and camera frame rate At that time, the formula reduces the weight of temperature deviation frames through the temperature drift term, and weakens the impact of frame rate misalignment through the frame rate term. Dynamically balancing the two, ultimately It accurately reflects temporal consistency, providing temporally reliable features for scene-behavior association.

[0052] This scheme achieves precise alignment of data from heterogeneous devices through a multi-parameter weighted time-series consistency formula: Formula introduction As a weighting factor, frames with high confidence levels account for a larger proportion of the score, thereby improving the reliability of the score. Based on the differences in device type, values ​​are assigned to match the sensitivity differences between visual and acoustic devices to frame rate deviations; and The product quantification of the effect of temperature change on different sensors avoids errors caused by uniform temperature drift processing; By combining the calibration results of the software protocol and the hardware phase-locked loop, the accuracy of the timestamp synchronization error correction is ensured; The averaging process eliminates the interference of time frame differences on the scoring, making time series of different lengths comparable.

[0053] It is worth mentioning that the collaborative logic of each parameter in the formula is clear: when a certain frame At lower levels, even with small frame rate or temperature deviations, its effect on The contribution will also decrease, reflecting the principle of "prioritizing high-confidence features"; and The differentiated values ​​were derived from experiments on device signal characteristics. Cameras are more sensitive to frame rate deviations due to their close inter-frame correlation. Infrared devices have a higher temperature drift coefficient due to their strong temperature dependence. ).

[0054] The technical effects achieved by the above solution include: through a multi-parameter collaborative temporal consistency formula, the temporal data of different devices are accurately aligned in the spatiotemporal dimension; the impact of frame rate deviation and temperature drift is quantitatively compensated; timestamp synchronization error is effectively corrected; the weighted effect of feature confidence improves the reliability of the score; and it provides temporally consistent feature input for scene-behavior association reasoning, reducing the problem of misjudgment of behavior caused by temporal mismatch.

[0055] Traditional scene-behavior association models have the following technical problems: When judging abnormal behavior, existing technologies mostly rely solely on the speed or trajectory of the behavior itself, without considering the dynamic characteristics of the scene, such as security level and crowd density. They also set fixed thresholds for normal behavior in different scenes, ignoring the rule that the higher the scene security level, the lower the tolerance for abnormal behavior. The association between scene and behavior in the formula uses a simple multiplicative relationship, which does not reflect the impact of crowd density on the sensitivity of behavior speed. This results in consistent abnormal judgment results for the same behavior in different scenes, which is inconsistent with the actual management needs of the community and leads to insufficient accuracy in anomaly identification.

[0056] Based on this, the scenario-behavior association network uses the following formula to calculate the anomaly determination index:

[0057] in: This is an anomaly detection index; The timing consistency score is calculated using the formula in the above embodiments; The scenario-behavior correlation coefficient is obtained by training on historical accident data from the community. Target speed of action; The average motion speed threshold for the current scene is defined, with different thresholds for the fitness area, rest area, and children's area. Real-time pedestrian density for the scene; Based on the scene safety level, the power distribution room, children's area, and fitness area correspond to 10, 8, and 5 respectively; The highest security level for the community is represented by a value of 10.

[0058] The core function of this formula is to integrate the dynamic characteristics of the scene with the behavioral features, output a quantitative anomaly index, solve the problem of traditional technology judging behavioral anomalies out of context, and realize differentiated security management in the community.

[0059] From an overall structural perspective, the formula uses time-series consistency scoring. Based on this, the anomaly index is adjusted through scenario-behavior correlation items. The functions of each item are as follows: Basic items : Ensure that anomaly detection is based on time-reliable features. If the timeline is disordered, even if the behavior is abnormal, Also because This reduces the risk of false alarms.

[0060] Behavioral speed item : The anomalous contribution of nonlinear amplification overspeed behavior increases to 4 times when the speed doubles. The average speed threshold for the scene is set to adapt to the scene characteristics: 1.8 m / s for the fitness area and 0.3 m / s for the rest area. Based on pedestrian density, densely populated areas are more sensitive to speed, such as squares. At the same time The anomalous contribution is 5 times higher than that of the sparse region.

[0061] Scene security level items : The higher the safety level of the scene, the smaller this value; for power distribution rooms... When this value is 0, the tolerance for abnormal behavior is lower. The highest level of community security ensures that security levels are comparable across different scenarios.

[0062] Correlation coefficient The formula is derived by training on historical accident data from the community, such as the accident rate in children's areas where children run at high speeds, to adapt the formula to the community's risk characteristics, including children's areas. They are more sensitive to abnormal behavior.

[0063] Interlocking logic of various parameters: Children's area , ,Appear Speeding by 300% and in dense crowds At that time, the formula amplifies the effect of overspeed through the velocity square term, enhances sensitivity through the density term, and the safety level term... Further increase the weight, ultimately A significant increase triggers an orange alert, aligning with the management logic of prioritizing safety in children's areas. This formula dynamically correlates abnormal behavior with scene characteristics, making the judgment results more closely reflect the actual needs of the community.

[0064] This solution achieves precise correlation between behavior and scene by integrating anomaly detection formulas based on dynamic scene parameters: the formula uses... Based on this, ensure that anomaly detection is built upon features that are consistent with time sequence; The item reflects the logic that the higher the action speed is compared to the average speed of the scene and the higher the population density, the greater the contribution of anomalies, which is in line with the actual needs of community management where rapid movement in dense areas is more dangerous. The impact of the safety level on the quantitative scenario is that the higher the safety level, such as in a power distribution room. The closer this value is to 0, the weaker its amplification effect on abnormal behavior; that is, even minor anomalies will be judged as high risk in high-security scenarios. By training with historical data, the formula is adapted to the actual accident patterns in the community, improving the accuracy of judgments. It's worth noting that the interrelationships between the parameters in the formula are clearly defined: when When it increases, Tiny changes can lead to Significant changes demonstrate the high sensitivity of densely populated areas to speed; The values ​​cover typical community scenarios. As a high-security area, the power distribution room will be affected even if the action speed slightly exceeds the threshold. Approaching 0 and making The rapid increase aligns with the zero-tolerance management logic of high-security areas.

[0065] The technical effects achieved by the above solution include: by using a dynamic correlation formula between scenarios and behaviors, the anomaly judgment results are closely matched with the characteristics of the community scenario; the same behavior under different scenarios can be judged differently; the linkage between pedestrian density and security level improves the accuracy of anomaly identification; the basic role of time-series consistency scoring ensures the reliability of the judgment; and the false alarms and false negatives caused by insufficient scenario adaptation are effectively reduced, thus meeting the diverse security management needs of the community.

[0066] Traditional edge computing layer module collaboration suffers from the following technical problems: In existing technologies, the signal processing modules of the edge computing layer often operate independently, and data interaction uses simple data bus transmission without establishing a target ID-based association mechanism, resulting in the inability to achieve spatial registration of multimodal features of the same target; the lack of mutual exclusion lock mechanism between modules makes it easy for conflicts to occur during data interaction, resulting in large fluctuations in processing latency and the inability to guarantee real-time performance; the poor linkage between the audio module and the vision module means that when voice activity is detected, the vision module cannot adjust its tracking strategy accordingly, resulting in insufficient accuracy of visual feature extraction in the sound source region and affecting the fusion effect of multimodal features.

[0067] Based on this, the signal processing modules of the edge computing layer work collaboratively in the following ways: the millimeter-wave radar signal processing module sends the extracted micro-Doppler features to the data fusion unit in the form of data frames, each frame containing a timestamp, target ID, and micro-Doppler spectrum data; the infrared image processing module performs spatial registration between the segmented human body temperature field data and the human target bounding box output by the visual analysis module, the registration process is achieved through coordinate system transformation, and the transformation parameters are pre-calibrated based on the sensor installation position; the audio signal processing module performs activity detection on the extracted voiceprint features, and when voice activity is detected, the visual analysis module is triggered to focus on tracking the direction of the sound source, and the size of the tracking window is dynamically adjusted according to the sound source localization result; the processing results of each module are exchanged through shared memory, and the interaction process adopts a mutex lock mechanism to avoid data conflicts, and the processing latency is controlled within 500ms to meet the real-time requirements.

[0068] This solution improves the processing efficiency and accuracy of the edge computing layer by constructing a collaborative interaction mechanism between modules: the introduction of target ID and timestamp enables the spatiotemporal correlation of radar micro-Doppler features, infrared temperature data, and visual target boxes for the same target, laying the foundation for multimodal feature fusion; spatial registration is achieved through pre-calibrated coordinate system transformation parameters, ensuring that infrared temperature field data and visual target boxes correspond accurately in physical space, avoiding feature misalignment caused by differences in sensor installation positions; the audio-visual linkage mechanism enables the visual module to specifically track the sound source area, and the tracking window is dynamically adjusted according to the sound source localization result, such as enlarging the window accordingly when the sound source range expands, improving the extraction accuracy of visual features in that area; the combination of shared memory and mutex locks reduces the latency and conflict of data interaction, ensuring the stability of each module under high concurrency processing, and the 500ms latency control meets the real-time requirements of community abnormal behavior recognition.

[0069] It is worth mentioning that the logical hierarchy of module collaboration is clear: the bottom layer enables fast data interaction through shared memory, the middle layer ensures data integrity through mutex locks, the upper layer achieves association of multimodal features through target ID and spatial registration, and the top layer achieves precise processing of key areas through audio-visual linkage, forming a collaborative link for data interaction, association and fusion, and key tracking.

[0070] The technical effects achieved by the above solution include: the collaborative work of various modules in the edge computing layer enables accurate spatiotemporal correlation of multimodal features; spatial registration eliminates feature misalignment caused by differences in sensor installation positions; audio-visual linkage improves the visual feature quality of the sound source area; the mutual exclusion lock mechanism ensures the stability of data interaction; real-time control meets the community's need for rapid response to abnormal behavior; and it provides high-quality multimodal fusion features for the cloud decision-making layer, thereby improving the overall system's recognition efficiency and accuracy.

[0071] Traditional multimodal sensing layer deployment has the following technical problems: Existing technologies mostly adopt a uniform distribution strategy for sensor deployment, without selecting sensor types specifically according to the functional characteristics of different areas in the community, such as road intersections and elevator cars, resulting in incomplete feature collection in key areas; sensor clock synchronization relies solely on software protocols without hardware calibration, resulting in long synchronization cycles and large errors; blind spots exist in sensor coverage in the same area, such as shaded areas of buildings; there is a lack of cross-coverage and redundancy verification mechanisms, and when data from a certain sensor is distorted, it cannot be supplemented in time, leading to insufficient reliability of data collection and affecting the accuracy of subsequent processing.

[0072] Based on this, the sensor deployment of the multimodal perception layer satisfies the following logic: the millimeter-wave radar array and the panoramic camera are co-located, with a height difference of no more than 0.5m between them to ensure the consistency of the spatial coordinates of the motion trajectory and the visual image. The overlap between the radar beam coverage and the camera's field of view is no less than 90%. The sampling period of the infrared sensor matrix and the sampling period of the microphone array are kept as integer multiples to ensure the time alignment of temperature field data and acoustic data. The sampling period of the infrared sensor is 50ms, and the sampling period of the microphone array is 25ms. The clock source of all sensors is synchronized with the community's unified time server, with a synchronization period of 1s. The synchronization process is achieved through network data packet interaction, and the data packet contains a sending timestamp and a receiving timestamp for calculating the clock deviation. For signal obstruction areas within the community, a multi-sensor cross-coverage strategy is adopted, that is, the same target area is covered by at least two different types of sensors. Obstruction areas include building shadow areas, tree obstruction areas, and corner blind spots. The cross-covered sensor data is redundantly verified through a data aggregation unit. When the confidence of a certain sensor data is lower than 0.3, the data from another sensor is used to supplement it.

[0073] This solution improves the reliability and integrity of multimodal data acquisition through targeted sensor deployment strategies: co-location and high overlap ensure spatial consistency between radar trajectories and visual images, avoiding feature misalignment due to differences in installation location; integer multiples of sampling periods ensure precise temporal alignment of temperature field data and acoustic signature data, facilitating temporal correlation of multimodal features; a hardware-software combined clock synchronization mechanism, with a 1-second cycle plus data packet timestamp verification, reduces time synchronization errors and ensures consistent time references for data from all sensors; cross-coverage and redundancy verification strategies eliminate monitoring blind spots in obstructed areas, allowing data from another sensor to be promptly supplemented when the confidence level of one sensor data is low, ensuring data continuity.

[0074] It is worth mentioning that the deployment logic is highly targeted: millimeter-wave radar arrays are deployed at road intersections and activity squares to capture motion characteristics due to high pedestrian traffic; infrared sensor matrices are installed at unit building entrances and elevator cars to monitor temperature fields due to their high degree of enclosure; microphone arrays are deployed in children's activity areas and elderly rest areas because voiceprint monitoring is required, reflecting the principle that the type of sensor is determined by the function of the area; the cross-coverage of obscured areas is designed according to the actual environmental characteristics of the community, with radar and camera cross-coverage used in the shaded areas of buildings, where radar is not affected by light, and infrared and microphone cross-coverage used in the tree-obscured areas, where infrared has strong penetration, ensuring that all kinds of blind spots have reliable monitoring methods.

[0075] The technical effects achieved by the above solution include: targeted deployment of sensors enables more comprehensive feature collection in key areas; spatial and temporal alignment mechanisms ensure the spatiotemporal correlation of multimodal data; improved clock synchronization accuracy reduces timing errors; cross-coverage and redundancy verification strategies eliminate monitoring blind spots and improve data reliability; high-quality and highly complete raw data are provided for the edge computing layer; identification errors caused by data defects are reduced from the source of collection; and the stable operation of the entire system is supported.

[0076] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A multimodal method for identifying abnormal behavior in smart communities, characterized in that, include: Multimodal data acquisition and preprocessing: Spatiotemporal dynamic data of the target area are simultaneously acquired by millimeter-wave radar, infrared sensors, microphone arrays and panoramic cameras deployed in the community, including human motion trajectory, temperature field distribution, environmental acoustic features and visual image sequences. Among them, millimeter-wave radar uses frequency-modulated continuous wave mode to sample human micro-Doppler features, infrared sensors realize temperature field reconstruction through pyroelectric array, microphone array collects acoustic signals through array structure, and panoramic camera collects high frame rate image sequences. Environmental interference coupling correction: Based on the acoustic reflection characteristics of different materials in the community, and combined with the light abrupt change rate, object reflectivity and air humidity parameters, a multi-factor coupling model is constructed to correct the confidence of the original multimodal features, forming a corrected feature set; Heterogeneous device timing alignment: Based on the corrected feature set, a timing consistency model including frame rate deviation, sensor temperature drift and timestamp synchronization error is established, and spatiotemporal alignment of multi-source device data is achieved through a weighted fusion algorithm; Scene and behavior correlation reasoning: Construct a three-dimensional decision space that integrates scene security level, real-time crowd density and action speed threshold. Based on the output of the time-series consistency model, output an anomaly judgment index through a progressive weighted algorithm. When the index exceeds the preset threshold, a multi-level early warning mechanism is triggered.

2. The method for multimodal identification of abnormal behavior in smart communities according to claim 1, characterized in that, In the multimodal data acquisition and preprocessing steps: Millimeter-wave radar uses time-division multiplexing technology to achieve range resolution through multiple transmission channels, and combines Kalman filtering algorithm to predict the trajectory of moving targets in order to eliminate trajectory jumps caused by non-uniform motion. The infrared sensor integrates a thermoelectric cooler and achieves baseline stability adjustment when the ambient temperature changes through closed-loop temperature control. At the same time, a median filtering algorithm is used to denoise the temperature field data to suppress the interference of sudden noise on the temperature field distribution. The microphone array uses a direction-of-arrival estimation algorithm to locate the sound source, combines blind source separation technology to separate environmental noise from human voice signals, and uses a feature dimensionality reduction algorithm to compress the extracted voiceprint feature vector. The panoramic camera uses a fisheye lens in conjunction with a distortion correction algorithm to eliminate optical distortion, and uses optical flow to perform motion compensation on adjacent frames to reduce temporal misalignment of image sequences caused by device jitter.

3. The method for multimodal identification of abnormal behavior in smart communities according to claim 1, characterized in that, The environmental interference coupling correction step includes: Establish material acoustic coefficient matrix Concrete, wood, and metal each have different acoustic attenuation coefficients, which are used to quantify the influence of materials on sound wave propagation. Real-time monitoring of light-induced mutation rate Dynamic detection of sudden changes in illumination is achieved by combining a photosensitive sensor array with a Gaussian mixture model, in order to distinguish between natural illumination changes and sudden light source interference. A humidity influencing factor is introduced. Air humidity data is collected through a humidity sensor. Based on the correlation characteristics between humidity and the speed of sound and the speed of light, the confidence of audio and video features is jointly corrected to compensate for signal attenuation in high humidity environments. When correcting illumination distortion in visual features, the differences in light reflection on different object surfaces are quantified by combining the object's reflectivity with the cosine of the incident angle of light, so as to eliminate brightness deviations caused by different object materials.

4. The method for multimodal identification of abnormal behavior in smart communities according to claim 1, characterized in that, The heterogeneous timing alignment step for the devices includes: A frame rate deviation compensation model is established. By comparing the deviation between the actual frame rate and the standard frame rate of each device, different deviation weights are assigned according to the device type. Different weights are used for visual devices and acoustic devices to match their signal characteristics. To address the temperature drift characteristics of infrared sensors, a correction function based on temperature deviation and the device's temperature drift coefficient is designed. The temperature deviation is calculated by real-time acquisition of the sensor's operating temperature to compensate for the impact of temperature changes on the detected values. A network time protocol combined with a hardware phase-locked loop is used to achieve timestamp synchronization of multiple devices. A dynamic time warping algorithm is used to perform feature interpolation on the synchronized time series data to eliminate timing misalignment caused by differences in sampling periods of different devices. During the time alignment process, the time features of different devices are weighted and fused using the confidence level of the corrected feature set as the weight. Features with higher confidence levels have a larger proportion in the fusion, thereby improving the reliability of the time consistency model.

5. A multimodal recognition system for abnormal behavior in smart communities, applied to the multimodal recognition method for abnormal behavior in smart communities as described in any one of claims 1-4, characterized in that, include: The multimodal sensing layer consists of a millimeter-wave radar array, an infrared sensor matrix, a microphone array, and a panoramic camera. Each sensor is connected to the data aggregation unit via a communication interface and transmits data in real time using a preset communication protocol. Millimeter-wave radar arrays are deployed at community road intersections and activity squares to collect micro-Doppler features of human movement; Infrared sensor matrices are installed at the entrance of the unit building and inside the elevator car to collect the temperature field distribution within the area; Microphone arrays are deployed in children's activity areas and elderly rest areas to collect environmental sound patterns and human vocal signals; Panoramic cameras cover public areas of the community and are used to capture high-definition visual image sequences; Edge computing layer: Deploys edge computing servers, integrates neural network acceleration engines, and enables real-time feature extraction and preliminary processing of multimodal data, including: Millimeter-wave radar signal processing module: Uses short-time Fourier transform to perform time-frequency analysis on frequency-modulated continuous wave signals and extracts the micro-Doppler features of moving targets; Infrared image processing module: Noise reduction of temperature field images is achieved through nonlocal mean filtering algorithm, and human target and environment are segmented by watershed algorithm; Audio signal processing module: Extracts voiceprint features based on Mel frequency cepstral coefficients and uses a hidden Markov model for speech activity detection; Visual analysis module: It uses target detection algorithms combined with multi-target trackers to achieve real-time detection and trajectory tracking of key points on the human body; Cloud-based decision-making layer: Building deep learning models based on cloud servers, including: Environmental interference correction network: A convolutional neural network combined with an attention mechanism is used to correct the confidence of multimodal features, and the attention weights are dynamically adjusted according to environmental parameters; Temporal alignment network: Design a long short-term memory network-conditional random field architecture to learn the impact of device heterogeneity on temporal consistency and output the temporally aligned feature sequence; Scene-behavior association network: Construct a graph neural network that integrates community map data, real-time pedestrian density data, and historical behavior data to output an anomaly detection index; Early warning execution layer: includes sound and light alarms, mobile terminal push module and community management platform interface, supports triggering early warnings in stages according to the anomaly judgment index, where yellow warnings are only pushed to the community security terminal, orange warnings are pushed to the community management platform at the same time, and red warnings are linked to the public security system to achieve data interoperability.

6. A multimodal recognition system for abnormal behavior in smart communities according to claim 5, characterized in that, The environmental interference correction network uses the following formula to correct the confidence level of multimodal features: ； in: The corrected confidence level for multimodal features; The original feature confidence score is output by the feature extraction module of the edge computing layer. For audio interference weights, For video interference weights, and ; The acoustic coefficient of the material is 0.8 for concrete, 0.3 for wood, and 0.9 for metal. The distance between the sound source and the microphone array; The main frequency of the audio signal; The light-induced mutation rate; The reflectivity of the object is 0.9 for metal and 0.2 for fabric. The angle of incidence of the light ray; Humidity sensitivity coefficient; This refers to air humidity.

7. A multimodal recognition system for abnormal behavior in smart communities according to claim 5, characterized in that, The time-alignment network calculates the time-consistency score using the following formula: ； in: Assess time series consistency. The number of time-series feature frames; For the first The confidence level of the frame's modified features is calculated using the formula in claim 6; The frame rate sensitivity coefficient is set to 0.7 for the camera and 0.3 for the microphone. For the first The actual frame rate of the device; Standard frame rate; The weighting is affected by temperature drift; For the first The temperature deviation of a sensor is the difference between the actual operating temperature and the calibration temperature of the sensor. The temperature drift coefficient of the sensor is set to 1.2 for infrared devices and 0.5 for millimeter-wave radar. For the first The frame timestamp synchronization error correction coefficient is obtained by joint calibration of the network time protocol and the hardware phase-locked loop.

8. A multimodal recognition system for abnormal behavior in smart communities according to claim 5, characterized in that, The scenario-behavior association network calculates the anomaly detection index using the following formula: ； in: This is an anomaly detection index; The temporal consistency score is calculated by the temporal alignment network using the following formula; The scenario-behavior correlation coefficient is obtained by training on historical accident data from the community. Target speed of action; The average motion speed threshold for the current scene is defined, with different thresholds for the fitness area, rest area, and children's area. Real-time pedestrian density for the scene; Based on the scene safety level, the power distribution room, children's area, and fitness area correspond to 10, 8, and 5 respectively; The highest security level for the community is represented by a value of 10.

9. A multimodal recognition system for abnormal behavior in smart communities according to claim 5, characterized in that, The signal processing modules of the edge computing layer work together in the following ways: The millimeter-wave radar signal processing module sends the extracted micro-Doppler features to the data fusion unit in the form of data frames. Each frame contains a timestamp, target ID, and micro-Doppler spectrum data. The infrared image processing module spatially registers the segmented human body temperature field data with the human body target bounding box output by the visual analysis module. The registration process is achieved through coordinate system transformation, and the transformation parameters are pre-calibrated based on the sensor installation position. The audio signal processing module performs activity detection on the extracted voiceprint features. When voice activity is detected, the visual analysis module is triggered to focus on tracking the direction of the sound source. The size of the tracking window is dynamically adjusted according to the sound source localization results. The processing results of each module are exchanged through shared memory. The interaction process uses a mutex lock mechanism to avoid data conflicts, and the processing latency is controlled within 500ms to meet real-time requirements.

10. A multimodal recognition system for abnormal behavior in smart communities according to claim 5, characterized in that, The sensor deployment of the multimodal sensing layer satisfies the following logic: The millimeter-wave radar array and the panoramic camera are deployed at the same location, with the installation height difference between the millimeter-wave radar array and the panoramic camera not exceeding 0.5m to ensure the consistency of the spatial coordinates of the motion trajectory and the visual image. The overlap between the beam coverage of the millimeter-wave radar array and the field of view of the panoramic camera is not less than 90%. The sampling period of the infrared sensor matrix is ​​kept as an integer multiple of the sampling period of the microphone array to ensure time alignment between the temperature field data and the acoustic data. The sampling period of the infrared sensor is 50ms and the sampling period of the microphone array is 25ms. The clock source of all sensors is synchronized with the community's unified time server. The synchronization period is 1 second. The synchronization process is achieved through network data packet interaction. The data packet contains the sending timestamp and the receiving timestamp for calculating the clock deviation. For areas with signal obstruction within the community, a multi-sensor cross-coverage strategy is adopted, meaning that the same target area is covered by at least two different types of sensors. Obstruction areas include building shadow areas, tree obstruction areas, and corner blind spots. The cross-coverage sensor data is redundantly verified through a data aggregation unit. When the confidence level of one sensor data is lower than 0.3, data from another sensor is used to supplement it.

Images