A home abnormal event detection method based on multi-modal data and edge intelligent device

Abstract

The application provides a kind of based on multi-modal data's home abnormal event detection method machine system, it is related to intelligent monitoring technical field.Personnel video data, sound data and environmental sensing signal in home environment are collected; personnel video data is sequentially subjected to human body detection tracking, human body posture estimation and kinematic analysis, and preliminary visual event is obtained; keyword recognition is carried out on sound data, and preliminary acoustic event is obtained; environmental sensing signal is continuously polled, each environmental sensing signal is compared with the size of corresponding environmental event judgment threshold, and preliminary environmental event is obtained; time correlation analysis and cross validation are carried out on preliminary visual event, preliminary acoustic event and preliminary environmental event, and the type and emergency degree of final event are determined; when and only when it is determined that an emergency occurs in the environment, the processing subsystem generates an alarm data package containing the type of event, time, place and confidence, and sends the alarm data package to the pre-bound user remote terminal.

Description

Technical Field

[0001] This invention relates to the field of intelligent monitoring technology, and in particular to a method for detecting abnormal events in homes based on multimodal data and an edge intelligent device. Background Technology

[0002] With the increasing demand for home safety and health monitoring, especially for the care of elderly people living alone, children, or people with mobility difficulties, intelligent monitoring technology has been widely used. Its core objective is to monitor and promptly alert authorities in real time to abnormal events such as falls, sudden illness calls for help, or fires without disturbing the user's normal life.

[0003] One existing solution requires users to wear devices that integrate inertial sensors such as accelerometers and gyroscopes, such as smart bracelets, watches, or pendants. By continuously monitoring the readings of these sensors, the system analyzes the user's movement patterns. When the sensor data shows drastic changes that match the characteristics of a preset fall model, a fall event is determined to have occurred, and an alarm is sent to a preset contact person or service center through the built-in communication module.

[0004] This solution has the following problems:

[0005] 1. User burden and inconvenience: Requiring users (especially the elderly) to wear extra devices for extended periods can cause discomfort and psychological burden. The weight and size of the devices may affect normal activities and may even increase the risk of falls in some cases.

[0006] 2. Compliance issues: Users may remove the device for reasons such as showering, sleeping, charging, or forgetting to wear it, causing the device to malfunction in a real emergency, creating a monitoring blind spot and greatly reducing the reliability of the solution.

[0007] 3. Functional limitations: This solution can usually only detect events related to strenuous movement (such as falls), and is powerless to detect silent coma or environmental hazards such as fires.

[0008] Another existing solution involves installing a network camera at home to capture raw, high-definition video streams. These streams are then uploaded in real-time to a cloud server via the internet. The cloud server uses sophisticated artificial intelligence algorithms (such as deep learning models) to analyze the video content and identify changes in human posture, behavior, or environment. If an abnormal event (such as a fall) is detected, the cloud server generates an alarm message, which is then transmitted back to the user's mobile app or a designated recipient via the network.

[0009] This solution has the following problems:

[0010] 1. Privacy breach: Uploading and storing raw video footage from highly private spaces such as the home in the cloud poses a fundamental threat to user privacy. Once this data is leaked, misused, or subjected to unauthorized access, the consequences are unimaginable. The constant feeling of being "monitored" will also bring enormous psychological pressure to family members.

[0011] 2. High latency: Data is uploaded from the camera to the cloud, queued and analyzed by the server, and then the alarm is sent back. The network latency of the entire link can be fatal in an emergency where every second counts.

[0012] 3. High bandwidth and high cost: Continuous transmission of high-definition video streams will consume a lot of home network bandwidth and bring high cloud storage and cloud computing service subscription fees, increasing the long-term economic burden on users.

[0013] 4. High false alarm rate: Despite the powerful computing capabilities of the cloud, relying solely on a single visual modality makes it difficult to distinguish between a real fall and similar everyday actions such as lying down or bending over, easily leading to false alarms. Furthermore, it cannot detect visually inconspicuous dangers, such as cries for help. Summary of the Invention

[0014] To address the problems existing in the prior art, this invention provides a method and edge intelligent device for detecting abnormal events in the home based on multimodal data. It enables rapid, accurate, and low-cost detection and alarm for falls, voice calls for help, and environmental hazards without requiring users to wear any devices or transmitting any raw audio or video data locally.

[0015] In a first aspect, the present invention provides a method for detecting abnormal events in a home based on multimodal data. This method is implemented using an edge intelligent device, which includes a processing subsystem and a sensing subsystem. The processing subsystem includes a neural network processing unit, a multi-core central processing unit, and a graphics processing unit. The sensing subsystem includes a visual sensing module, an acoustic sensing module, and an environmental sensing module. The method includes:

[0016] (1) The visual perception module collects video data of people in the home environment, the acoustic perception module collects sound data of the home environment, and the environmental perception module monitors environmental sensing signals of the home environment.

[0017] (2) The processing subsystem sequentially performs human detection and tracking, human posture estimation and kinematic analysis on the personnel video data to obtain preliminary visual events;

[0018] (3) The processing subsystem performs keyword recognition on the sound data to obtain preliminary acoustic events;

[0019] (4) The processing subsystem continuously polls the environmental sensor signals, compares each environmental sensor signal with the corresponding environmental event judgment threshold, and obtains the preliminary environmental event.

[0020] (5) The processing subsystem performs time correlation analysis and cross-validation on preliminary visual events, preliminary acoustic events and preliminary environmental events to determine the type and urgency of the final event. The types of the final event include personnel fall events, voice distress calls and environmental hazard events. The urgency of the final event includes emergency situations and daily activities.

[0021] (6) The processing subsystem generates an alarm data packet containing the event type, time, location and confidence level and sends the alarm data packet to the pre-bound user remote terminal only when it is determined that an emergency has occurred in the environment.

[0022] Optionally, in step (2), the specific steps for performing human body detection and tracking, human posture estimation, and kinematic analysis on the personnel video data are as follows:

[0023] Human detection and tracking: A lightweight target detection model that accelerates the operation of human video data by inputting it into a neural network processing unit. The target detection model locates all human targets in the video frames and uses a tracking algorithm to assign a unique, frame-stable tracking ID to each human target.

[0024] Human pose estimation: For each tracked human target, a lightweight human pose estimation model running on a neural network processing unit is used for inference, and the video frames corresponding to the human target are converted into a set of skeleton key point coordinates in real time.

[0025] Kinematic analysis and judgment: Kinematic analysis is performed based on the temporal data stream of the skeletal key point coordinates of each human target, and the kinematic analysis results are input into a pre-trained lightweight support vector machine classifier to judge human stability, and the output is a preliminary visual event with timestamp and confidence.

[0026] Optionally, the lightweight human pose estimation model adopts a single-branch architecture, LitePose.

[0027] Optionally, the kinematic analysis and judgment employs a fall detection algorithm based on a spatiotemporal stability matrix, specifically including:

[0028] For each tracked human target, the human spatiotemporal stability matrix STSM is constructed and updated within a preset sliding time window T, as follows:

[0029] The formulas for calculating the vertical velocity of the human target's center of mass, the area of ​​the supporting polygon, the deviation of the center of mass projection, and the variance of the attitude energy are as follows:

[0030]

[0031] A SoP (t)=Area(ConvexHull(P contact (t)))

[0032] D CoM (t)=||P CoM_proj (t)-C SoP (t)||

[0033]

[0034] Among them, V CoM (t) represents the vertical velocity of the center of mass, P CoM.y (t) represents the vertical position of the human body's center of mass at time t, and Δt represents the time interval between adjacent frames. ConvexHull(P) contact (t) represents the convex hull formed by the key points of the human body in contact with the ground at time t, i.e., the area of ​​the supporting polygon, A. SoP (t) represents the area of ​​the supporting polygon, D CoM (t) represents the centroid projection deviation, P CoM,y_proj (t) represents the projection of the human body's center of mass onto the ground at time t, C SoP (t) represents the center point of the supporting polygon at time t, E Var (t) represents the attitude energy variance, Pi(t') represents the position of the i-th skeleton keypoint at time t', and P CoM (t') represents the position of the human body's center of mass at time t', and N is the total number of key points in the skeleton;

[0035] The human spatiotemporal stability matrix STSM is constructed using the calculated centroid vertical velocity, support polygon area, centroid projection deviation, and attitude energy variance.

[0036] The human spatiotemporal stability matrix (STSM) is input into a pre-trained lightweight support vector machine classifier. When the classifier outputs an unstable state and the duration of the subsequent static state is less than a preset static threshold, a high-confidence preliminary visual event is output.

[0037] Optionally, in step (3), the keyword recognition of the sound data employs a hierarchical acoustic threat confirmation algorithm, specifically including:

[0038] First-layer low-power keyword detection: Employs an ultra-lightweight keyword recognition model to detect a small number of core and urgent keywords;

[0039] The second layer of prosodic feature analysis: extract the audio segments before and after the keywords, and analyze the prosodic features of the audio segments, including the maximum time gradient of the pitch fundamental frequency, the short-time energy peak, and the average harmonic noise ratio. When the prosodic features match the real speech distress call model, a preliminary acoustic event with high confidence is output.

[0040] Optionally, in step (5), temporal correlation analysis and cross-validation are performed sequentially on the preliminary visual events, preliminary acoustic events, and preliminary environmental events, specifically including:

[0041] Based on the timestamps of the preliminary visual events, preliminary acoustic events, and preliminary environmental events, the preliminary visual events, preliminary acoustic events, and preliminary environmental events that occur within the same preset time window are grouped as a set of time-related events;

[0042] The logical verification rule set is used to cross-validate three types of events in the same time-related event group to determine the type and urgency of the final event. The logical verification rule set is pre-defined based on the semantics and confidence of the event and includes association confirmation rules, ambiguity elimination rules, and environmental hazard confirmation rules.

[0043] In a second aspect, the present invention provides an edge intelligent device, including a processing subsystem and a sensing subsystem. The processing subsystem includes a neural network processing unit, a multi-core central processing unit and a graphics processing unit. The sensing subsystem includes a visual sensing module, an acoustic sensing module and an environmental sensing module.

[0044] The visual perception module is a high-definition, wide-angle imaging module with excellent low-light performance, used to collect video data of people in the home environment;

[0045] The acoustic sensing module is an array of microphones with a high signal-to-noise ratio, used to collect sound data of the home environment;

[0046] The environmental sensing module integrates several physical sensors to monitor environmental sensing signals in the home environment;

[0047] Neural network processing units are used to provide high-throughput, low-power hardware acceleration for deep learning models;

[0048] Multi-core central processing units are used to run embedded operating systems, control upper-layer application logic, schedule tasks, process communication protocol stacks, and execute logical rules in multimodal fusion decision engines;

[0049] The graphics processing unit (GPU) is used to perform general-purpose parallel computing tasks such as graphics rendering and video stream preprocessing, thus offloading the load of the multi-core central processing unit (CPU).

[0050] The processing subsystem configures the data processing framework, deep learning model, and multimodal fusion decision engine through software programming to achieve the following functions:

[0051] The video data of personnel is sequentially subjected to human detection and tracking, human posture estimation and kinematic analysis to obtain preliminary visual events;

[0052] Keyword recognition is performed on the sound data to obtain preliminary acoustic events;

[0053] The system continuously polls environmental sensor signals and compares each environmental sensor signal with the corresponding environmental event judgment threshold to obtain preliminary environmental events.

[0054] Temporal correlation analysis and cross-validation were performed on preliminary visual events, preliminary acoustic events, and preliminary environmental events to determine the type and urgency of the final events. The types of final events included personnel fall events, voice distress call events, and environmental hazard events, and the urgency of the final events included emergency situations and daily activities.

[0055] The processing subsystem generates an alarm data packet containing the event type, time, location, and confidence level, and sends the alarm data packet to the pre-bound user remote terminal only when an emergency is determined to have occurred in the environment.

[0056] By adopting the above technical solution, the present invention has at least the following beneficial effects:

[0057] 1. This invention processes and transforms all sensitive raw data (video, audio) containing personal biometrics in real time within the edge device (video → skeleton, audio → keywords), and the analysis process is completed entirely locally in a closed loop, ensuring that no sensitive raw data is transmitted or stored externally throughout the entire process from data collection to analysis and decision-making, thus achieving an extremely high level of privacy protection.

[0058] 2. This invention uses a multimodal fusion decision engine to perform time correlation analysis and cross-validation on the analysis results of visual data, auditory data, and environmental sensor data, which effectively reduces false alarms caused by the complexity of the scene due to single-modal data and provides highly reliable alarms.

[0059] 3. This invention is a contactless detection method, which does not require users to wear any devices, thus solving the compliance problem of wearable devices. At the same time, users only need a one-time hardware investment and there are no ongoing cloud service subscription fees. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a schematic diagram of the structure of an edge intelligent device provided in an embodiment of the present disclosure;

[0062] Figure 2 This disclosure provides a flowchart of a home anomaly event detection method based on multimodal data. Detailed Implementation

[0063] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] like Figure 1 As shown, this disclosure provides an edge intelligent device, including a processing subsystem and a sensing subsystem;

[0065] (1) Perception Subsystem

[0066] The sensing subsystem is the interface between the device and the physical world. It is responsible for acquiring raw monitoring data in multiple modes, including the visual sensing module, the acoustic sensing module, and the environmental sensing module.

[0067] The visual perception module is a high-definition, wide-angle imaging module with excellent low-light performance, used to collect video data of people in the home environment, providing high-quality input for subsequent human posture analysis;

[0068] The acoustic perception module is an array of several high signal-to-noise ratio microphones used to collect sound data from the home environment, providing high-quality input for subsequent keyword recognition;

[0069] The environmental perception module integrates several physical sensors to monitor environmental sensing signals in the home environment, providing key contextual information that is neither visual nor acoustic.

[0070] (2) Processing Subsystem

[0071] The core of the processing subsystem is a high-performance, low-power heterogeneous computing unit, preferably an advanced system-on-a-chip (SoC). The SoC integrates multiple processing cores that work together to achieve optimal performance and energy efficiency, including a neural network processing unit, a multi-core CPU, and a graphics processing unit.

[0072] The Neural Processing Unit (NPU) serves as a dedicated AI acceleration engine, providing high-throughput, low-power hardware acceleration for deep learning models such as human detection, pose estimation, and keyword recognition models.

[0073] Multi-core central processing units (CPUs) are used to run embedded operating systems, control upper-layer application logic, schedule tasks, process communication protocol stacks, and execute logical rules in multimodal fusion decision engines.

[0074] Graphics processing units (GPUs) are used to perform general-purpose parallel computing tasks such as graphics rendering and video stream preprocessing (e.g., image scaling, color space conversion), thus offloading the load from multi-core central processing units.

[0075] The processing subsystem configures the data processing framework, deep learning model, and multimodal fusion decision engine through software programming to achieve the following functions:

[0076] The video data of personnel is sequentially subjected to human detection and tracking, human posture estimation and kinematic analysis to obtain preliminary visual events;

[0077] Keyword recognition is performed on the sound data to obtain preliminary acoustic events;

[0078] The system continuously polls environmental sensor signals and compares each environmental sensor signal with the corresponding environmental event judgment threshold to obtain preliminary environmental events.

[0079] Temporal correlation analysis and cross-validation were performed on preliminary visual events, preliminary acoustic events, and preliminary environmental events to determine the type and urgency of the final events. The types of final events included personnel fall events, voice distress call events, and environmental hazard events, and the urgency of the final events included emergency situations and daily activities.

[0080] The processing subsystem generates an alarm data packet containing the event type, time, location, and confidence level, and sends the alarm data packet to the pre-bound user remote terminal only when an emergency is determined to have occurred in the environment.

[0081] like Figure 2 As shown, based on the aforementioned edge intelligent device, this disclosure provides a method for detecting abnormal home events based on multimodal data. This method is abstracted into a hierarchical, multi-pipeline data processing framework, which operates entirely in a closed loop locally on the edge intelligent device, ensuring data privacy and real-time processing.

[0082] 1. The first layer is the data acquisition and preprocessing layer, which is the starting point of the data processing framework. Through the perception subsystem, it concurrently captures raw, unstructured multi-source heterogeneous data streams in real time from the visual perception module, acoustic perception module, and environmental perception module, including personnel video data, sound data, and environmental sensor signals. These data streams are then sent to the next layer for parallel and specialized processing.

[0083] 2. The second layer is the single-modal feature extraction and event preliminary judgment layer. This layer is the core computing layer of the data processing framework. It consists of three parallel and independent analysis pipelines. Each pipeline is responsible for transforming the raw data of one modality into structured and meaningful preliminary events.

[0084] (1) Visual Analysis Pipeline: The goal of this pipeline is to transform raw personnel video data into an abstract, privacy-preserving skeleton key point time-series data stream in real time, and detect preliminary visual events from it.

[0085] Human detection and tracking: A lightweight target detection model that accelerates the operation of human video data by inputting it into a neural network processing unit. The target detection model locates all human targets in the video frames and uses tracking algorithms (such as Kalman filtering) to assign a unique, frame-stable tracking ID to each human target.

[0086] Privacy-preserving pose estimation: For each tracked human target, a lightweight human pose estimation model optimized for edge devices is used for inference on the NPU, converting the video frames corresponding to the human target into a set of skeleton keypoint coordinates in real time. This step is the core of privacy protection in this invention. After the original video data is "distilled" into abstract skeleton data, it is immediately discarded from memory without any form of storage or external transmission.

[0087] Specifically, an optimized, lightweight pose estimation model based on the LitePose single-branch architecture is adopted. This model is designed for edge computing and can accurately output a set of coordinates of N skeleton key points while ensuring low computational cost.

[0088] Kinematic analysis and preliminary judgment: Kinematic analysis is performed based on the time series of the skeletal key point coordinates of each human target, and the kinematic analysis results are input into a pre-trained lightweight support vector machine classifier to judge human stability, and the preliminary visual events with timestamps and confidence scores are output.

[0089] Specifically, the kinematic analysis and judgment employs a fall detection algorithm based on the spatiotemporal stability matrix, which includes: for each tracked human target, constructing and updating the human spatiotemporal stability matrix STSM within a preset sliding time window T, as follows:

[0090] The formulas for calculating the vertical velocity of the human target's center of mass, the area of ​​the supporting polygon, the deviation of the center of mass projection, and the variance of the attitude energy are as follows:

[0091]

[0092] A SoP (t)=Area(ConvexHull(P contact(t)))

[0093] D CoM (t)=||P CoM_proj (t)-C SoP (t)||

[0094]

[0095] Among them, V CoM (t) represents the vertical velocity of the center of mass, P CoM.y (t) represents the vertical position of the human body's center of mass at time t, and Δt represents the time interval between adjacent frames. ConvexHull(P) contact (t) represents the convex hull formed by the key points of the human body in contact with the ground at time t, i.e., the area of ​​the supporting polygon, A. SoP (t) represents the area of ​​the supporting polygon, D CoM (t) represents the centroid projection deviation, P CoM,y_proj (t) represents the projection of the human body's center of mass onto the ground at time t, C SoP (t) represents the center point of the supporting polygon at time t, E Var (t) represents the attitude energy variance, Pi(t') represents the position of the i-th skeleton keypoint at time t', and P CoM (t') represents the position of the human body's center of mass at time t', and N is the total number of key points in the skeleton;

[0096] The human spatiotemporal stability matrix STSM is constructed using the calculated centroid vertical velocity, support polygon area, centroid projection deviation, and attitude energy variance.

[0097] The human spatiotemporal stability matrix (STSM) is input into a pre-trained lightweight support vector machine classifier. When the classifier outputs an unstable state and the duration of the subsequent static state is less than a preset static threshold, a preliminary visual event with high confidence is output, such as ({event:"Fall",confidence:0.89,timestamp:T1}).

[0098] (2) Acoustic analysis pipeline: Since traditional Keyword Recognition (KWS) is easily triggered by environmental sound sources such as television and radio, this pipeline adopts a hierarchical acoustic threat confirmation (HATC) model to achieve high-precision voice distress event recognition with extremely low power consumption.

[0099] The first layer of low-power keyword detection: After preprocessing (such as Voice Activity Detection (VAD) and noise reduction), the audio stream is fed into a lightweight keyword recognition model that runs on an NPU or CPU and is designed for low power consumption and continuous monitoring. It only recognizes a few core emergency keywords (such as "help", "fire", "I fell down"). The power consumption at this stage is extremely low, but a higher false alarm rate is allowed to ensure no missed detections (high recall rate).

[0100] The second layer of prosodic feature analysis: Extract audio segments (approximately 2 seconds) before and after the keywords, and analyze the prosodic features of the audio segments to determine whether they represent a genuine human emergency call.

[0101] Maximum temporal gradient of pitch fundamental frequency: When making an emergency call, the pitch fundamental frequency of the distress call usually rises sharply or fluctuates violently. Its maximum temporal gradient is extracted as a prosodic feature.

[0102] Short-time energy peak: When making an emergency call, the distress sound has the characteristic of a sudden energy burst. Its short-time peak energy is extracted as a prosodic feature.

[0103] Average harmonic noise ratio: When making an emergency call, the harmonic components of the distress sound usually decrease and the noise increases, resulting in a relatively low harmonic noise ratio. The average value of this ratio is extracted as a prosodic feature.

[0104] When the above prosodic features match the real speech distress call model, output a preliminary acoustic event with high confidence, for example, ({event:"Keyword_Help",confidence:0.95,timestamp:T2}).

[0105] (3) Environmental analysis pipeline

[0106] The pipeline continuously polls the readings of environmental sensors (such as smoke sensors). When the reading exceeds a preset environmental event judgment threshold, the pipeline outputs a preliminary environmental event ({event:"Smoke_Detected",timestamp:T3}).

[0107] 3. The third layer is the multimodal fusion decision layer. This layer is the key to improving the intelligence and reliability of the system. All preliminary events from the second layer, along with their timestamps and confidence levels, are aggregated into a multimodal fusion decision engine running on the CPU. This engine does not simply respond to any single event, but performs time correlation analysis and cross-validation on preliminary visual events, preliminary acoustic events, and preliminary environmental events to determine the type and urgency of the final event.

[0108] Temporal correlation analysis: Based on the timestamps of preliminary visual events, preliminary acoustic events, and preliminary environmental events, preliminary visual events, preliminary acoustic events, and preliminary environmental events that occur within the same preset time window are grouped as a set of temporally correlated events;

[0109] Cross-validation: A set of logical validation rules is used to cross-validate three types of time-related events within the same group to determine the final event type (personal fall event, voice distress call event, and environmental hazard event) and urgency level (emergency situation and daily activity). The logical validation rule set is pre-defined based on the semantics and confidence level of the events and includes association confirmation rules, ambiguity resolution rules, and environmental hazard confirmation rules.

[0110] Association confirmation rule: If the visual analysis pipeline outputs a high-confidence personnel fall event (e.g., a visually clear fall action), and the acoustic analysis pipeline outputs a voice distress call event that is semantically related to the personnel fall event (e.g., a cry for help was heard), then the system will associate the personnel fall event and the voice distress call event and determine that a high-priority personnel fall event has occurred.

[0111] Ambiguity resolution rule: If the visual analysis pipeline outputs a low-confidence person fall event (e.g., an action that looks like a fall but may actually be lying down), but the acoustic analysis pipeline does not output a voice call for help that is semantically related to the person fall event, and the person recovers to normal after a short time, then it is determined that no person fall event has occurred, and the event is a daily activity.

[0112] Environmental hazard confirmation rule: If the environmental analysis pipeline outputs an environmental hazard event, and the voice distress call event output by the acoustic analysis pipeline is semantically related to the environmental hazard event, then a high-priority environmental hazard event is determined to have occurred.

[0113] This fusion decision-making process effectively distinguishes between genuine emergency situations and routine activities (such as lying down to rest), resulting in an extremely low false alarm rate.

[0114] 4. The fourth layer is the event response and interaction layer. This layer is activated only when the multimodal fusion decision layer determines that an emergency has occurred in the environment. It immediately generates an alarm data packet containing the event type, time, location (device ID), and confidence level, and sends the alarm data packet to the pre-bound user remote terminal (such as a smartphone).

[0115] To verify the effectiveness of the proposed method and edge intelligent device, an experimental environment was built and compared with current mainstream technical solutions, as detailed below:

[0116] 1. Experimental Dataset

[0117] The experiment used a comprehensive, specially constructed test dataset containing 500 videos to simulate complex situations in a real home environment.

[0118] Positive samples (fall events): a total of 250 sequences, of which 150 are from public datasets (such as URFD, FDD) and the other 100 are self-recorded, covering various falling postures such as forward, backward, and side slips, tripping, and slipping off chairs.

[0119] Negative samples (non-fall events): a total of 250 sequences, all of which are large-amplitude movements in daily life that are easily confused with falls, such as sitting down quickly, lying on the bed or sofa, bending over to pick up items, doing yoga, etc.

[0120] Acoustic interference: In 100 negative samples, TV news and movie clips containing keywords such as "help" and "fire" were added as interference through background playback.

[0121] II. Experimental Comparison Scheme

[0122] Option 1: Wearable sensor: Commercial anti-fall wristband based on accelerometer.

[0123] Option 2: Cloud-based single-modal vision: Upload the video stream to the cloud and analyze it using standard server-level pose estimation algorithms.

[0124] Option 3: Local Single-Modal Vision: Judgments are made using only the STSM vision analysis pipeline proposed in this invention on edge devices.

[0125] Option 4: Local Single-Mode Acoustics: On edge devices, only the HATC acoustic analysis pipeline proposed in this invention is used for judgment.

[0126] The present invention utilizes a complete STSM vision, HATC acoustic pipeline, and TBFN fusion decision network on an edge device.

[0127] III. Experimental Results:

[0128] Table 1 shows the comparison results of each scheme in terms of key performance indicators.

[0129] Table 1 Comparison Results of Key Performance Indicators

[0130]

[0131] As can be seen from the data in Table 1, the proposed solution significantly outperforms all other comparative solutions in both accuracy (99.2%) and F1 score (99.2%), demonstrating its overall superior performance. The specificity of the proposed solution reaches 99.8%, which is directly attributed to the innovative multimodal fusion mechanism of the proposed solution. For example, when faced with only visually similar lying-down movements, it will not be identified as an emergency due to the lack of acoustic evidence of a distress call. When faced with keyword interference in the background sound, the proposed solution successfully filters it out through prosodic feature analysis, avoiding false alarms. The sensitivity of the proposed solution is as high as 98.5%, demonstrating an extremely high detection rate when real emergency events occur. This is due to the STSM visual algorithm's accurate capture of human instability and the HATC acoustic algorithm's high recall design for real distress calls.

[0132] In summary, the experimental data quantitatively demonstrate that this invention, through innovative algorithm design and multimodal fusion, reduces the false alarm rate to almost zero while maintaining an extremely high detection rate, thus solving the key pain points of existing technologies.

[0133] Although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A method for detecting abnormal home events based on multimodal data, characterized in that, This method is implemented based on an edge intelligent device, which includes a processing subsystem and a perception subsystem. The processing subsystem includes a neural network processing unit, a multi-core central processing unit, and a graphics processing unit. The perception subsystem includes a visual perception module, an acoustic perception module, and an environmental perception module. The method includes: (1) The visual perception module collects video data of people in the home environment, the acoustic perception module collects sound data of the home environment, and the environmental perception module monitors environmental sensing signals of the home environment. (2) The processing subsystem sequentially performs human body detection and tracking, human posture estimation and kinematic analysis on the personnel video data to obtain preliminary visual events; (3) The processing subsystem performs keyword recognition on the sound data to obtain preliminary acoustic events; (4) The processing subsystem continuously polls the environmental sensor signals, compares each environmental sensor signal with the corresponding environmental event judgment threshold, and obtains the preliminary environmental event; (5) The processing subsystem performs time correlation analysis and cross-validation on preliminary visual events, preliminary acoustic events and preliminary environmental events to determine the type and urgency of the final event. The types of the final event include personnel fall events, voice distress calls and environmental hazard events. The urgency of the final event includes emergency situations and daily activities. (6) When an emergency is determined to occur in the environment, the processing subsystem generates an alarm data packet containing the event type, time, location and confidence level and sends the alarm data packet to the pre-bound user remote terminal; In step (2), the specific steps for performing human body detection and tracking, human posture estimation, and kinematic analysis on the personnel video data are as follows: Human detection and tracking: A lightweight target detection model that accelerates the operation of human video data by inputting it into a neural network processing unit. The target detection model locates all human targets in the video frames and uses a tracking algorithm to assign a unique, frame-stable tracking ID to each human target. Human pose estimation: For each tracked human target, a lightweight human pose estimation model running on a neural network processing unit is used for inference, and the video frames corresponding to the human target are converted into a set of skeleton key point coordinates in real time. Kinematic analysis and judgment: Kinematic analysis is performed based on the temporal data stream of the skeletal key point coordinates of each human target, and the kinematic analysis results are input into a pre-trained lightweight support vector machine classifier to judge human stability, and output preliminary visual events with timestamps and confidence scores. The kinematic analysis and judgment employs a fall detection algorithm based on the spatiotemporal stability matrix, specifically including: For each tracked human target, the human spatiotemporal stability matrix STSM is constructed and updated within a preset sliding time window T, as follows: The formulas for calculating the vertical velocity of the human target's center of mass, the area of ​​the supporting polygon, the deviation of the center of mass projection, and the variance of the attitude energy are as follows: Among them, V CoM (t) represents the vertical velocity of the center of mass, P CoM.y (t) represents the vertical position of the human body's center of mass at time t, and Δt represents the time interval between adjacent frames. ConvexHull(P) contact (t) represents the convex hull formed by the key points of the human body in contact with the ground at time t, i.e., the area of ​​the supporting polygon, A. SoP (t) represents the area of ​​the supporting polygon, D CoM (t) represents the centroid projection deviation, P CoM,y_proj (t) represents the projection of the human body's center of mass onto the ground at time t, C SoP (t) represents the center point of the supporting polygon at time t, E Var (t) represents the attitude energy variance, Pi(t') represents the position of the i-th skeleton keypoint at time t', and P CoM (t') represents the position of the human body's center of mass at time t', and N is the total number of key points in the skeleton; The human spatiotemporal stability matrix STSM is constructed using the calculated centroid vertical velocity, support polygon area, centroid projection deviation, and attitude energy variance. The human spatiotemporal stability matrix (STSM) is input into a pre-trained lightweight support vector machine classifier. When the classifier outputs an unstable state and the duration of the subsequent static state is less than a preset static threshold, a high-confidence preliminary visual event is output. In step (5), temporal correlation analysis and cross-validation are performed sequentially on the preliminary visual events, preliminary acoustic events, and preliminary environmental events, specifically including: Based on the timestamps of the preliminary visual events, preliminary acoustic events, and preliminary environmental events, the preliminary visual events, preliminary acoustic events, and preliminary environmental events that occur within the same preset time window are grouped as a set of time-related events; The logical verification rule set is used to cross-validate three types of events within the same group of time-related events to determine the final event type and urgency. The logical verification rule set is pre-defined based on the semantics and confidence level of the events and includes association confirmation rules, ambiguity resolution rules, and environmental hazard confirmation rules. Association confirmation rule: If the visual analysis pipeline outputs a high-confidence personnel fall event, and the acoustic analysis pipeline outputs a voice distress call event that is semantically related to the personnel fall event, the system will associate the personnel fall event and the voice distress call event and determine that a high-priority personnel fall event has occurred. Ambiguity elimination rule: If the visual analysis pipeline outputs a low-confidence person fall event, but the acoustic analysis pipeline does not output a voice call for help event that is semantically related to the person fall event, and the person recovers to normal after a short time, then it is determined that no person fall event has occurred, and the event is a daily activity. Environmental hazard confirmation rule: If the environmental analysis pipeline outputs an environmental hazard event, and the voice distress call event output by the acoustic analysis pipeline is semantically related to the environmental hazard event, then a high-priority environmental hazard event is determined to have occurred.

2. The method for detecting abnormal home events based on multimodal data according to claim 1, characterized in that, The lightweight human pose estimation model adopts a single-branch architecture, LitePose.

3. The method for detecting abnormal home events based on multimodal data according to claim 1, characterized in that, In step (3), keyword recognition of the sound data employs a hierarchical acoustic threat confirmation algorithm, specifically including: First-layer low-power keyword detection: Employs an ultra-lightweight keyword recognition model to detect a small number of core and urgent keywords; The second layer of prosodic feature analysis: extract the audio segments before and after the keywords, and analyze the prosodic features of the audio segments, including the maximum time gradient of the pitch fundamental frequency, the short-time energy peak, and the average harmonic noise ratio. When the prosodic features match the real speech distress call model, a preliminary acoustic event with high confidence is output.

Images