Old people abnormal behavior identification method and system based on multi-modal fusion

Abstract

The invention discloses an old people abnormal behavior identification method and system based on multi-modal fusion, and the method comprises the steps: an edge end synchronously collects visual, inertial and environmental heterogeneous data, and extracts each modal feature after time alignment and preprocessing; vision and inertia feature weights are dynamically corrected through a cross-modal attention adapter, and modal unbalance is relieved; constructing a multi-modal fusion graph, executing a graph attention mechanism (HLGAtt) in a hyperbolic Lorentz space, and depicting a behavior hierarchical relationship; and detecting anomaly based on the graph attention network, positioning a root cause node through random walk, and outputting an anomaly category and root cause interpretation. Compared with the prior art, the method has the advantages that the anomaly identification F1 score is increased by more than 6%, the false alarm rate is lower than 0.7 times per day, and the method has the characteristics of high reliability and interpretability, can be widely applied to nursing homes, communities and family scenes, and provides technical support for intelligent nursing for the aged.

Description

Technical Field

[0001] This invention relates to the field of smart elderly care and health monitoring technology, and in particular to a method and system for identifying abnormal behaviors of the elderly based on multimodal fusion. Background Technology

[0002] As the global population ages, abnormal behaviors in the daily activities of the elderly (such as falls, missed medications, and disordered sleep patterns) have become key triggers for major health and safety incidents. Therefore, achieving accurate and efficient monitoring of abnormal behaviors in the elderly is of great significance for ensuring their health and safety and promoting the development of the smart elderly care industry.

[0003] Traditional solutions for monitoring abnormal behavior in the elderly often employ single-modal sensing technology, relying on a single sensor device such as a visual camera or a wearable accelerometer. However, such single-modal monitoring solutions have significant limitations. In real-world elderly care scenarios, such as sudden changes in lighting conditions, obstruction of the monitored target, blind spots in the monitoring perspective, or improper use of wearable devices, the accuracy of abnormal behavior detection drops sharply, easily leading to false alarms and missed alarms. This makes it difficult to meet the 24 / 7, highly reliable monitoring requirements of elderly care settings.

[0004] In recent years, multimodal fusion technology has been gradually introduced into the field of abnormal behavior recognition in the elderly. This technology, by combining heterogeneous information from multiple sources such as visual sensing data, inertial sensing data, and environmental sensing data, can theoretically effectively improve the robustness of abnormal behavior monitoring. However, existing multimodal fusion methods have obvious technical defects. Most of them adopt a simple fusion mode of early feature splicing or late decision fusion, failing to address core technical challenges such as differences in sampling frequency between heterogeneous modal data, imbalance in information weight allocation, and asynchronous temporal features. Ultimately, this results in low fusion efficiency of multimodal data, insufficient discriminative power of the fused features, and an inability to fully leverage the complementary advantages of multi-source data.

[0005] Furthermore, current mainstream abnormal behavior recognition models are generally based on building deep learning networks using Euclidean space to complete feature calculations and modeling. However, the behavioral evolution process of the elderly, from "normal to sub-abnormal to abnormal," has a naturally hierarchical structure, which is difficult for linear modeling methods using Euclidean space to accurately characterize. Simultaneously, existing models can only output category labels for abnormal behaviors, failing to trace the specific sensor data sources or environmental influencing factors that triggered the abnormal behavior, resulting in severely insufficient interpretability. These technical shortcomings significantly limit the large-scale deployment and widespread application of related abnormal behavior monitoring technologies in real-world elderly care scenarios.

[0006] In summary, there is an urgent need in this field to develop an end-to-end technology solution for monitoring abnormal behaviors of the elderly that can adaptively balance the contribution weights of multimodal data, complete behavioral feature modeling in a space with hierarchical expressive capabilities, and achieve precise localization of abnormal root causes, so as to break through the application bottlenecks of existing technologies. Summary of the Invention

[0007] This invention simultaneously acquires three types of heterogeneous data—visual, inertial, and environmental—at the edge. After time alignment and preprocessing, features of each modality are extracted. The weights of visual and inertial features are dynamically corrected using a cross-modal attention adapter (CFA) to alleviate the imbalance of modal information. The enhanced features are constructed into a multimodal fusion graph, and a graph attention mechanism (HLGAtt) is executed in hyperbolic Lorentz space to characterize the hierarchical relationship between normal and abnormal behaviors. Anomaly detection is performed based on the graph attention network, and a random walk algorithm is used to locate the root cause node causing the anomaly on the fusion graph. Finally, the anomaly category and root cause explanation are output.

[0008] This invention provides a method for identifying abnormal behaviors in the elderly based on multimodal fusion, comprising: Multimodal data is acquired through a multimodal data sensor and preprocessed. Multimodal features are generated based on multimodal data fusion; Anomaly detection and anomaly localization based on multimodal features; and Verify and analyze the results.

[0009] In one embodiment of the present invention, the acquisition and preprocessing of multimodal data includes: For inertial sensor data, the influence of dimensions is eliminated using the following formula: ; Where X represents the original data, and μ and σ are the mean and standard deviation, respectively; For missing values ​​in environmental sensor data, KNN interpolation is used to fill in the missing values ​​to ensure data continuity. Normalize and extract features from visual sensor data.

[0010] In one embodiment of the present invention, the normalization and feature extraction of visual sensor data includes: Normalization is performed using the following formula: ; in The value is the normalized value. The original values ​​to be normalized. The maximum value in the original dataset. It is the minimum value in the original dataset; Map pixel values ​​to the [0,1] range; Feature extraction is performed using the following formula: ; in To modify the activation function of the linear unit, These are the weight parameters of the convolutional layer. The input features are normalized. These are the bias parameters for the convolutional layer.

[0011] In one embodiment of the present invention, the sequence is aligned using the following formula to handle frequency differences in multimodal time series data: ; in The dynamic time-warped distance between sequence A and sequence B. To find the minimum value of π among all valid regular paths, where π is a valid regular path connecting elements of sequence A and sequence B, Let i be the i-th element of sequence A. Let j be the j-th element of sequence B. It represents the absolute difference between the two elements.

[0012] In one embodiment of the present invention, the generation of multimodal features based on multimodal data fusion includes: Given visual features Audio features Where T is the time step, D V and D A For feature dimensions; The key matrix K and value matrix V of the audio features are concatenated with the learnable parameters Pk and Pv: ; Computing cross-modal attention: ; in For concatenating the key matrix Dimensions This is a normalized exponential function used to generate attention weights; Cross-modal attention features are then passed through a bottleneck adapter: ; in For attention feature output, This is a downsampling function used for attention features. Dimensional reduction For the Gaussian error linear unit activation function, a nonlinear transformation is introduced. This is an upsampling function used to restore the dimensionality-reduced features to the target dimension; The importance of audio features can be dynamically adjusted using learnable weights: ; in Here, σ is the modality-weighted learnable parameter matrix, and σ is the sigmoid activation function. Original audio features; The final multimodal fusion features are: ; in This is a fully connected layer function used to perform dimensionality transformations or non-linear mappings on the fused features. Original visual features Modal weighting factors generated for audio features.

[0013] In one embodiment of the present invention, the method further includes constructing a multimodal fusion graph. ; Among the nodes Represents the modal feature vector, edge Indicates the correlation strength between modes; The edge weights are calculated as follows: ; in Let be the edge weight between node i and node j. Let be the set of neighboring nodes of node i. For leaky modified linear unit activation functions, This is the attention vector, used to calculate the association weights between features. This is a learnable weight matrix used to perform linear transformations on node features. Let i, j, and k be the feature vectors of nodes i, j, and k, respectively. It is an exponential function used to map activation values ​​to positive weights.

[0014] In one embodiment of the present invention, the method further includes capturing the hierarchical relationship between normal and abnormal behaviors using a hyperbolic Lorentz diagram attention mechanism: Mapping the fused features to hyperbolic space: ; The formula for calculating the Lorentz inner product is: ; in For an n-dimensional Lorentz hyperboloid, Let x be an (n+1)-dimensional real vector, with components denoted as x0, x1, ..., xn. For the Lorentz inner product, To constrain the upper half of the hyperboloid; The hyperbolic Lorentz graph attention mechanism is as follows: ; in The hyperbolic space feature vector of node i after aggregating neighborhood features. This is a hyperbolic exponential mapping centered at the origin 0 of hyperbolic space with curvature parameter c, used to map tangent space vectors back to hyperbolic space. Let be a hyperbolic logarithmic mapping centered at the origin 0 of hyperbolic space with curvature parameter c. Its function is to map hyperbolic space vectors to the tangent space of the origin. Let be the attention weight of node i to its neighboring node j. Let be the set of neighboring nodes of node i. Let be the original hyperbolic space eigenvector of the neighboring node j; HLGAtt learns different patterns through parallel branching, and its node update formula is as follows: ; ; ; in, For network hierarchical indexing, Let be the Lorentz space eigenvector of node i in the (l-1)th layer. To adapt to the linear transformation function of Lorentz geometry, The intermediate feature of node i in the l-th layer after linear transformation is denoted as . For the attention function in Lorentz space, Let be the attention weight of node i in layer l to its neighboring node j. This is a feature aggregation function in Lorentz space, which fuses neighborhood features based on attention weights. Let be the set of neighboring nodes of node i. The final Lorentz space feature after aggregating nodes i in the l-th layer.

[0015] In one embodiment of the present invention, the step of detecting and locating the cause of anomalies based on multimodal features includes: Anomaly detection based on graph attention networks; By aggregating neighborhood information through multi-layer graph convolution, node features are updated. ; in The feature vector of node i after the l-th layer calculation is the (l+1)-th layer output. For activation function, Let be the set of neighboring nodes of node i. Let be the attention weight of node i to its neighboring node j in layer l. Here is the learnable weight matrix for the l-th layer, used to perform a linear transformation on the features of neighboring nodes. Let be the input feature vector of the neighboring node j in the l-th layer; A semi-supervised learning framework is adopted to jointly optimize the supervised cross-entropy loss and the unsupervised contrastive loss: ; in The total loss function for model training. This is a balancing hyperparameter for the loss term, used to adjust the contribution ratio of different losses to the total loss. For cross-entropy loss, For regularization loss, To compare the losses; Group similar behavioral characteristics together and keep aberrant characteristics away: ; in Let P be the feature vector of the anchor sample, P be the set of positive samples corresponding to the anchor sample, N be the set of negative samples corresponding to the anchor sample (which belongs to the different class from the anchor), sim be the similarity function, τ be the temperature parameter used to adjust the smoothness of the similarity distribution (the smaller τ is, the stronger the discriminative power), and exp be the exponential function used to map the similarity value to a positive weight. After detecting the anomaly, the root cause of the anomaly is located on the multimodal fusion graph; Define the transition probability from node i to j as: ; in Let be the transition probability from node i to node j, with a value in the range [0,1]. Let be the edge weight between node i and node j, representing the strength of the association between the two nodes. Let i be the set of neighboring nodes. Let be the sum of the edge weights between node i and all its neighboring nodes; By performing multiple random walks and counting the frequency of visits to each node, a root cause weight distribution is generated: ; in Root cause score for node v, used to measure the likelihood that the node is the root cause of the failure. Let V be the number of times node v is visited, and V be the set of all nodes in the multimodal fusion graph. This is the sum of the number of visits to all nodes, used to normalize the number of visits to a single node. After sorting by weight, output the top K principal causes.

[0016] In one embodiment of the present invention, the analysis of long-term behavioral patterns is further included: The daily behavior of the elderly is modeled as a sequence of hidden states, where the hidden states represent behavioral patterns and the observed values ​​are multimodal sensor data. Given observation sequence The likelihood probability is calculated using the forward algorithm: ; in, For observation sequence In the model The likelihood probability is as follows: Let be the initial state probability vector. Let be the probability of being in state q1 at the initial moment. Here is the state transition probability matrix. To start from state q t−1 Transfer to q t The probability, To observe the emission probability matrix, For state q t Generate observation o t The probability, Let T be a sequence of hidden states. An alert is sent to the user when the behavior deviates from the normal pattern.

[0017] This invention also provides a multimodal fusion-based system for recognizing abnormal behaviors in the elderly, comprising: The data acquisition module is configured to simultaneously acquire data from visual, inertial, and environmental sensors; The preprocessing module is configured to perform time alignment, normalization, missing value imputation, and feature extraction, and output a multimodal feature vector. A cross-modal attention adapter module is configured to dynamically weight and fuse the multimodal feature vectors to generate adaptive fused features; The graph construction module is configured to construct a multimodal fusion graph using the adaptive fusion features as nodes and intermodal similarity as edge weights. The hyperbolic Lorentz graph attention mechanism module is configured to update node representations in hyperbolic space and output hyperbolic fusion features; An anomaly detection module is configured to classify the hyperbolic fusion features based on a graph attention network to obtain anomaly behavior categories; The root cause localization module is configured to calculate and output the root cause of an anomaly using a random walk algorithm when an anomaly occurs; and The early warning interface module is configured to push anomaly category and root cause information to the monitoring terminal.

[0018] The present invention has the following beneficial effects: (1) By using an adaptive weight balancing mechanism, the sampling frequency difference, information weight imbalance and time sequence asynchrony between heterogeneous modalities (visual, inertial and environmental sensing) are specifically addressed. The complementary advantages of multi-source data are fully utilized. Compared with traditional single-modal monitoring schemes and simple splicing multimodal fusion methods, the false alarm rate and false alarm rate are significantly reduced in scenarios such as changes in illumination, occlusion, blind spots, and improper device wearing, thereby improving the accuracy and environmental adaptability of abnormal behavior detection.

[0019] (2) Behavioral feature modeling is based on a space with hierarchical expression capabilities (such as hyperbolic space), which breaks through the limitations of linear modeling in Euclidean space. It can accurately capture the behavioral evolution pattern of the elderly from "normal-sub-abnormal-abnormal", enhance the ability to identify early sub-abnormal behaviors, and provide more sufficient technical support for health risk warning.

[0020] (3) Compared with existing models that only output abnormal labels, this solution can locate the specific sensor data source or environmental influencing factors that induce abnormal behavior, clarify the core cause of the abnormality, solve the pain point of traditional technology that "only knows the abnormality but not the cause", meet the actual needs of the interpretability of monitoring technology in the elderly care scenario, and facilitate the subsequent implementation of targeted intervention measures. Attached Figure Description

[0021] Figure 1 A flowchart of an embodiment of the present invention for identifying abnormal behavior of the elderly based on multimodal fusion is shown. Detailed Implementation

[0022] In the following description, the invention is described with reference to various embodiments. However, those skilled in the art will recognize that the embodiments may be practiced without one or more specific details or with other alternatives and / or additional methods, materials, or components. In other instances, well-known structures, materials, or operations are not shown or described in detail so as not to obscure the inventive points of the invention. Similarly, for illustrative purposes, specific quantities, materials, and configurations are set forth to provide a comprehensive understanding of embodiments of the invention. However, the invention is not limited to these specific details.

[0023] In this invention, the various embodiments are merely intended to illustrate the solutions of the invention and should not be construed as limiting.

[0024] In this specification, references to "an embodiment" or "this embodiment" mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the invention. The phrase "in one embodiment" appearing throughout this specification does not necessarily refer to the same embodiment in all instances.

[0025] Furthermore, the numbering of the steps in the methods of the present invention does not limit the execution order of the method steps. Unless otherwise specified, the method steps may be executed in different orders.

[0026] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0027] Figure 1 A flowchart of an embodiment of the present invention for identifying abnormal behavior of the elderly based on multimodal fusion is shown.

[0028] like Figure 1 As shown, in one embodiment of the present invention, the method for identifying abnormal behaviors of the elderly based on multimodal fusion includes: S100. Multimodal Data Acquisition and Preprocessing: The primary task of the multimodal abnormal behavior recognition system is to achieve high-quality data acquisition and effective preprocessing. This system collects daily behavioral data of the elderly in real time through various sensors deployed in the elderly care environment, ensuring comprehensive coverage of possible activity scenarios. It includes the following steps: S110, Multi-source sensor data acquisition, the system uses three types of sensors working together: Visual sensors: Deploying high-resolution industrial cameras (4096×3000 pixels, 200fps) and infrared thermal imagers (640×512 pixels, 60fps) to capture detailed appearance and temperature distribution information of the elderly during their daily activities. To protect privacy, all image data is anonymized at edge computing nodes.

[0029] Inertial sensors: Three-axis accelerometers, gyroscopes, and orientation sensors in smartphones or wearable devices collect human motion data at a frequency of 100Hz to analyze abnormal movements such as gait and falls.

[0030] Environmental sensors: Temperature, pressure, vibration and other sensors collect environmental data at a frequency of 1-500Hz via Modbus / TCP protocol, recording the elderly’s activity trajectory and environmental interaction.

[0031] To ensure time synchronization of multimodal data, the system employs a Precision Time Protocol (PTP) to ensure that the timestamp deviation of all sensors is less than 1 millisecond. The data acquisition module supports multi-channel parallel transmission, reducing latency and guaranteeing real-time performance through 5G or industrial Ethernet.

[0032] S120. Data preprocessing and feature extraction: For inertial sensor data, Z-score normalization is used to eliminate the influence of dimensions. ; Where X represents the original data, and μ and σ are the mean and standard deviation, respectively; For missing values ​​in environmental sensor data, KNN interpolation is used to fill in the missing values ​​to ensure data continuity. Normalization is performed using the following formula: ; in The value is the normalized value. The original values ​​to be normalized. The maximum value in the original dataset. It is the minimum value in the original dataset; Map pixel values ​​to the [0,1] range; Feature extraction is performed using the following formula: ; in To modify the activation function of the linear unit, These are the weight parameters of the convolutional layer. The input features are normalized. These are the bias parameters for the convolutional layer.

[0033] The following formula is used to align sequences to handle frequency differences in multimodal time series data: ; in The dynamic time-warped distance between sequence A and sequence B. To find the minimum value of π among all valid regular paths, where π is a valid regular path connecting elements of sequence A and sequence B, Let i be the i-th element of sequence A. Let j be the j-th element of sequence B. It represents the absolute difference between the two elements.

[0034] Table 1 below lists the multimodal sensor data specifications and preprocessing methods in this embodiment.

[0035] Table 1: Multimodal sensor data specifications and preprocessing methods S200. A multimodal fusion algorithm based on an attention mechanism. Multimodal fusion is the core innovation of this system, aiming to effectively integrate data from different sources to generate a more comprehensive behavioral representation. Traditional fusion methods (such as early fusion or late fusion) fail to fully consider the complex relationships between modalities, resulting in insufficient information utilization. This invention proposes a hierarchical fusion strategy that combines an attention mechanism with a graph neural network to achieve adaptive weighting of modalities.

[0036] S210, Cross-modal attention adapter: Inspired by the prefix-tuning mechanism in natural language processing, this invention designs a cross-modal fusion adapter (CFA) to dynamically adjust the contribution weights of each modality in the fusion process. The CFA consists of a prefix-tuned bottleneck attention and a modulation mechanism, specifically designed to address the problem of information imbalance between audiovisual modalities.

[0037] Given visual features Audio features Where T is the time step, D V and D A For feature dimensions; The key matrix K and value matrix V of the audio features are concatenated with the learnable parameters Pk and Pv: ; Computing cross-modal attention: ; in For concatenating the key matrix Dimensions This is a normalized exponential function used to generate attention weights; Cross-modal attention features are then passed through a bottleneck adapter: ; in For attention feature output, This is a downsampling function used for attention features. Dimensional reduction For the Gaussian error linear unit activation function, a nonlinear transformation is introduced. This is an upsampling function used to restore the dimensionality-reduced features to the target dimension; The importance of audio features can be dynamically adjusted using learnable weights: ; in Here, σ is the modality-weighted learnable parameter matrix, and σ is the sigmoid activation function. Original audio features; The final multimodal fusion features are: ; in This is a fully connected layer function used to perform dimensionality transformations or non-linear mappings on the fused features. Original visual features Modal weighting factors generated for audio features.

[0038] S220. Multimodal fusion graph construction: To capture deep-level relationships between modalities, a multimodal fusion graph is constructed. ; Among the nodes Represents the modal feature vector, edge Indicates the correlation strength between modes; The edge weights are calculated as follows: ; in Let be the edge weight between node i and node j. Let be the set of neighboring nodes of node i. For leaky modified linear unit activation functions, This is the attention vector, used to calculate the association weights between features. This is a learnable weight matrix used to perform linear transformations on node features. Let i, j, and k be the feature vectors of nodes i, j, and k, respectively. It is an exponential function used to map activation values ​​to positive weights.

[0039] S230. Hyperbolic Lorentzian Graph Attention Mechanism: To better capture the hierarchical relationship between normal and abnormal behaviors, this invention introduces the Hyperbolic Lorentzian Graph Attention (HLGAtt) mechanism. Traditional Euclidean space is difficult to represent complex hierarchical structures, while hyperbolic space can more naturally represent such relationships.

[0040] Mapping the fused features to hyperbolic space: ; The formula for calculating the Lorentz inner product is: ; in For an n-dimensional Lorentz hyperboloid, Let x be an (n+1)-dimensional real vector, with components denoted as x0, x1, ..., xn. For the Lorentz inner product, To constrain the upper half of the hyperboloid; The hyperbolic Lorentz graph attention mechanism is as follows: ; in The hyperbolic space feature vector of node i after aggregating neighborhood features. This is a hyperbolic exponential mapping centered at the origin 0 of hyperbolic space with curvature parameter c, used to map tangent space vectors back to hyperbolic space. Let be a hyperbolic logarithmic mapping centered at the origin 0 of hyperbolic space with curvature parameter c. Its function is to map hyperbolic space vectors to the tangent space of the origin. Let be the attention weight of node i to its neighboring node j. Let be the set of neighboring nodes of node i. Let be the original hyperbolic space eigenvector of the neighboring node j; HLGAtt learns different patterns through parallel branching, and its node update formula is as follows: ; ; ; in, For network hierarchical indexing, Let be the Lorentz space eigenvector of node i in the (l-1)th layer. To adapt to the linear transformation function of Lorentz geometry, The intermediate feature of node i in the l-th layer after linear transformation is denoted as . For the attention function in Lorentz space, Let be the attention weight of node i in layer l to its neighboring node j. This is a feature aggregation function in Lorentz space, which fuses neighborhood features based on attention weights. Let be the set of neighboring nodes of node i. The final Lorentz space feature after aggregating nodes i in the l-th layer.

[0041] The S300 anomaly detection and root cause localization model, based on fused multimodal features, employs a hierarchical anomaly detection strategy. It first identifies immediate abnormal behaviors (such as falls), then analyzes long-term behavioral pattern changes (such as sleep disorders), and finally locates the root cause of the anomaly, providing support for nursing decisions.

[0042] S310. Anomaly Detection Based on Graph Attention Networks: Building upon graph fusion features, we apply a Graph Attention Network (GAT) for anomaly detection. GAT aggregates neighborhood information through multi-layer graph convolutions to update node features. ; in The feature vector of node i after the l-th layer calculation is the (l+1)-th layer output. For activation function, Let be the set of neighboring nodes of node i. Let be the attention weight of node i to its neighboring node j in layer l. Here is the learnable weight matrix for the l-th layer, used to perform a linear transformation on the features of neighboring nodes. Let be the input feature vector of the neighboring node j in the l-th layer; A semi-supervised learning framework is adopted to jointly optimize the supervised cross-entropy loss and the unsupervised contrastive loss: ; in The total loss function for model training. This is a balancing hyperparameter for the loss term, used to adjust the contribution ratio of different losses to the total loss. For cross-entropy loss, For regularization loss, To compare the losses; Group similar behavioral characteristics together and keep aberrant characteristics away: ; in Let P be the feature vector of the anchor sample, P be the set of positive samples corresponding to the anchor sample, N be the set of negative samples corresponding to the anchor sample (which belongs to the different class from the anchor), sim be the similarity function, τ be the temperature parameter used to adjust the smoothness of the similarity distribution (the smaller τ is, the stronger the discriminative power), and exp be the exponential function used to map the similarity value to a positive weight.

[0043] S320. Random walk root cause localization: After an anomaly is detected, the system uses a random walk algorithm to locate the root cause of the anomaly on the multimodal fusion graph. Define the transition probability from node i to j as: ; in Let be the transition probability from node i to node j, with a value in the range [0,1]. Let be the edge weight between node i and node j, representing the strength of the association between the two nodes. Let i be the set of neighboring nodes. Let be the sum of the edge weights between node i and all its neighboring nodes; By performing multiple random walks and counting the frequency of visits to each node, a root cause weight distribution is generated: ; in Root cause score for node v, used to measure the likelihood that the node is the root cause of the failure. Let V be the number of times node v is visited, and V be the set of all nodes in the multimodal fusion graph. This is the sum of the number of visits to all nodes, used to normalize the number of visits to a single node. After sorting by weight, output the top K principal causes.

[0044] S330. Long-term behavioral pattern analysis: In addition to immediate anomaly detection, the system also integrates a Hidden Markov Model (HMM) to analyze long-term behavioral patterns. The HMM models the daily behavior of the elderly as a sequence of hidden states, where the hidden states represent behavioral patterns (such as sleep, activity, and eating), and the observations are multimodal sensor data.

[0045] Given observation sequence The likelihood probability is calculated using the forward algorithm: ; in, For observation sequence In the model The likelihood probability is as follows: Let be the initial state probability vector. Let be the probability of being in state q1 at the initial moment. Here is the state transition probability matrix. To start from state q t−1 Transfer to q t The probability, To observe the emission probability matrix, For state q t Generate observation o t The probability, Let T be a sequence of hidden states. An alert is sent to the user when the behavior deviates from the normal pattern.

[0046] Abnormal behavior corresponds to observation sequences with low likelihood probabilities, indicating a deviation from the normal pattern.

[0047] S400. System Implementation and Performance Analysis: To verify the effectiveness of the proposed algorithm, this invention implements a complete abnormal behavior recognition system and conducts performance tests on multiple datasets. The system adopts a modular design, facilitating expansion and maintenance.

[0048] S410. System Architecture and Experimental Setup: The system consists of six modules: data acquisition, preprocessing, feature extraction, multimodal fusion, anomaly detection, and root cause localization. The experiment uses XD-Violence, ShanghaiTechCampus, and the daily behavior dataset of the elderly, including both normal and abnormal behavior samples. Evaluation metrics include accuracy, precision, recall, and F1 score. ; ; ; TP (True Positive) is the number of samples that the model predicts as positive and are actually positive; FP (False Positive) is the number of samples that the model predicts as positive but are actually negative; and FN (False Negative) is the number of samples that the model predicts as negative but are actually positive.

[0049] S420. Results and Analysis: On the multimodal violence detection dataset, this embodiment achieves an average precision (AP) of 86.07%, significantly outperforming traditional methods. In the task of recognizing abnormal behavior in the elderly, the attention-based CNN-LSTM algorithm achieves a precision of 94.2%, a recall of 95.1%, and an F1 score of 94.6%.

[0050] Table 2: Comparison of Multimodal Anomaly Detection Performance (%) Experiments show that introducing components such as CFA and HLGAtt significantly improves system performance. Ablation studies show that CFA contributes approximately 3.2% to the F1 score improvement, while HLGAtt contributes approximately 2.1%. Furthermore, multimodal fusion has a clear advantage over single-modal methods in anomaly detection tasks. The system's detection performance varies across different types of anomalous behavior. It performs best in detecting obvious anomalous behaviors such as falls (F1=96.8%), while it struggles to detect progressive anomalies such as sleep disturbances (F1=89.3%). This indicates that the system is more sensitive to immediate anomalies, and its ability to detect long-term pattern changes needs improvement.

[0051] S430, computational efficiency, and practicality analysis: The system achieves an average processing latency of 87ms on an NVIDIA Tesla V100 GPU, meeting real-time monitoring requirements. Through model pruning and quantization, the model size is reduced by 63%, with an accuracy loss of only 1.2%, enabling deployment on edge devices. In field testing at a nursing home, the system successfully identified 92% of abnormal events (14 out of 15 falls and 26 out of 28 medication forgetfulness events), with a false alarm rate of only 0.7 times per day, demonstrating significant practical value.

[0052] In another embodiment of the present invention, a system for recognizing abnormal behaviors of the elderly based on multimodal fusion is provided, comprising: The data acquisition module is configured to simultaneously acquire data from visual, inertial, and environmental sensors; The preprocessing module is configured to perform time alignment, normalization, missing value imputation, and feature extraction, and output a multimodal feature vector. A cross-modal attention adapter module is configured to dynamically weight and fuse the multimodal feature vectors to generate adaptive fused features; The graph construction module is configured to construct a multimodal fusion graph using the adaptive fusion features as nodes and intermodal similarity as edge weights. The hyperbolic Lorentz graph attention mechanism module is configured to update node representations in hyperbolic space and output hyperbolic fusion features; An anomaly detection module is configured to classify the hyperbolic fusion features based on a graph attention network to obtain anomaly behavior categories; The root cause localization module is configured to calculate and output the root cause of an anomaly using a random walk algorithm when an anomaly occurs; and The early warning interface module is configured to push anomaly category and root cause information to the monitoring terminal.

[0053] The Cross-Modal Attention Adapter (CFA) module and the Hyperbolic Lorentz Graph Attention Mechanism (HLGAtt) module are deployed on the same edge computing node, with a single inference latency of no more than 100ms. The early warning interface module supports the MQTT / HTTP protocol for interfacing with nursing home management platforms or home smart terminals. The root cause analysis module outputs at least one of the following: sensor failure, sudden changes in environmental parameters, and changes in elderly behavioral patterns.

[0054] Although various embodiments of the invention have been described above, it should be understood that they are presented by way of example only and not as limitations. It will be apparent to those skilled in the art that various combinations, modifications, and alterations can be made without departing from the spirit and scope of the invention. Therefore, the breadth and scope of the invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined solely by the appended claims and their equivalents.

Claims

1. A method for identifying abnormal behaviors in the elderly based on multimodal fusion, characterized in that, include: Multimodal data is acquired through a multimodal data sensor and preprocessed. Multimodal features are generated based on multimodal data fusion; Anomaly detection and cause localization are performed based on multimodal features; as well as Verify and analyze the results.

2. The method according to claim 1, characterized in that, The process of collecting and preprocessing multimodal data includes: For inertial sensor data, the influence of dimensions is eliminated using the following formula: ； Where X represents the original data, and μ and σ are the mean and standard deviation, respectively; For missing values ​​in environmental sensor data, KNN interpolation is used to fill in the missing values ​​to ensure data continuity. Normalize and extract features from visual sensor data.

3. The method according to claim 2, characterized in that, The normalization and feature extraction of visual sensor data includes: Normalization is performed using the following formula: ； in The value is the normalized value. The original values ​​to be normalized. The maximum value in the original dataset. It is the minimum value in the original dataset; Map pixel values ​​to the [0,1] range; Feature extraction is performed using the following formula: ； in To modify the activation function of the linear unit, These are the weight parameters of the convolutional layer. The input features are normalized. These are the bias parameters for the convolutional layer.

4. The method according to claim 2, characterized in that, It also includes aligning sequences using the following formula to handle frequency differences in multimodal time series data: ； in The dynamic time-warped distance between sequence A and sequence B. To find the minimum value of π among all valid regular paths, where π is a valid regular path connecting elements of sequence A and sequence B, Let i be the i-th element of sequence A. Let j be the j-th element of sequence B. It represents the absolute difference between the two elements.

5. The method according to claim 1, characterized in that, The generation of multimodal features based on multimodal data fusion includes: Given visual features Audio features Where T is the time step, D V and D A For feature dimensions; The key matrix K and value matrix V of the audio features are concatenated with the learnable parameters Pk and Pv: ； Computing cross-modal attention: ； in For concatenating the key matrix Dimensions This is a normalized exponential function used to generate attention weights; Cross-modal attention features are then passed through a bottleneck adapter: ； in For attention feature output, This is a downsampling function used for attention features. Dimension reduction For the Gaussian error linear unit activation function, a nonlinear transformation is introduced. This is an upsampling function used to restore the dimensionality-reduced features to the target dimension; The importance of audio features can be dynamically adjusted using learnable weights: ； in Here, σ is the modality-weighted learnable parameter matrix, and σ is the sigmoid activation function. Original audio features; The final multimodal fusion features are: ； in This is a fully connected layer function used to perform dimensional transformations or non-linear mappings on the fused features. Original visual features Modal weighting factors generated for audio features.

6. The method according to claim 1, characterized in that, It also includes constructing a multimodal fusion graph. ; Among the nodes Represents the modal feature vector, edge Indicates the correlation strength between modes; The edge weights are calculated as follows: ； in Let be the edge weight between node i and node j. Let be the set of neighboring nodes of node i. For leaky modified linear unit activation functions, This is the attention vector, used to calculate the association weights between features. This is a learnable weight matrix used to perform linear transformations on node features. Let i, j, and k be the feature vectors of nodes i, j, and k, respectively. It is an exponential function used to map activation values ​​to positive weights.

7. The method according to claim 1, characterized in that, This also includes capturing the hierarchical relationship between normal and abnormal behaviors using the hyperbolic Lorenz diagram attention mechanism: Mapping the fused features to hyperbolic space: ； The formula for calculating the Lorentz inner product is: ； in For an n-dimensional Lorentz hyperboloid, Let x be an (n+1)-dimensional real vector, with components denoted as x0, x1, ..., xn. For the Lorentz inner product, To constrain the upper half of the hyperboloid; The hyperbolic Lorentz graph attention mechanism is as follows: ； in The hyperbolic space feature vector of node i after aggregating neighborhood features. This is a hyperbolic exponential mapping centered at the origin 0 of hyperbolic space with curvature parameter c, used to map tangent space vectors back to hyperbolic space. Let be a hyperbolic logarithmic mapping centered at the origin 0 of hyperbolic space with curvature parameter c. Its function is to map hyperbolic space vectors to the tangent space of the origin. Let be the attention weight of node i to its neighboring node j. Let be the set of neighboring nodes of node i. Let be the original hyperbolic space eigenvector of the neighboring node j; HLGAtt learns different patterns through parallel branching, and its node update formula is as follows: ； ； ； in, For network hierarchical indexing, Let be the Lorentz space eigenvector of node i in the (l-1)th layer. To adapt to the linear transformation function of Lorentz geometry, The intermediate feature of node i in the l-th layer after linear transformation is denoted as . For the attention function in Lorentz space, Let be the attention weight of node i in layer l to its neighboring node j. This is a feature aggregation function in Lorentz space, which fuses neighborhood features based on attention weights. Let be the set of neighboring nodes of node i. The final Lorentz space feature after aggregating nodes i in the l-th layer.

8. The method according to claim 1, characterized in that, The method of detecting and locating the cause of anomalies based on multimodal features includes: Anomaly detection based on graph attention networks; By aggregating neighborhood information through multi-layer graph convolution, node features are updated. ； in The feature vector of node i after the l-th layer calculation is the (l+1)-th layer output. For activation function, Let be the set of neighboring nodes of node i. Let be the attention weight of node i to its neighboring node j in layer l. Here is the learnable weight matrix for the l-th layer, used to perform a linear transformation on the features of neighboring nodes. Let be the input feature vector of the neighboring node j in the l-th layer; A semi-supervised learning framework is adopted to jointly optimize the supervised cross-entropy loss and the unsupervised contrastive loss: ； in The total loss function for model training. This is a balancing hyperparameter for the loss term, used to adjust the contribution ratio of different losses to the total loss. For cross-entropy loss, For regularization loss, To compare the losses; Group similar behavioral characteristics together and keep aberrant characteristics away: ； in Let P be the feature vector of the anchor sample, P be the set of positive samples corresponding to the anchor sample, N be the set of negative samples corresponding to the anchor sample (which belongs to the different class from the anchor), sim be the similarity function, τ be the temperature parameter used to adjust the smoothness of the similarity distribution (the smaller τ is, the stronger the discriminative power), and exp be the exponential function used to map the similarity value to a positive weight. After detecting the anomaly, the root cause of the anomaly is located on the multimodal fusion graph; Define the transition probability from node i to j as: ； in Let be the transition probability from node i to node j, with a value in the range [0,1]. Let be the edge weight between node i and node j, representing the strength of the association between the two nodes. Let be the set of neighboring nodes of node i. Let be the sum of the edge weights between node i and all its neighboring nodes; By performing multiple random walks and counting the frequency of visits to each node, a root cause weight distribution is generated: ； in Root cause score for node v, used to measure the likelihood that the node is the root cause of the failure. Let V be the number of times node v is visited, and V be the set of all nodes in the multimodal fusion graph. This is the sum of the number of visits to all nodes, used to normalize the number of visits to a single node. After sorting by weight, output the top K principal causes.

9. The method according to claim 1, characterized in that, It also includes analyzing long-term behavioral patterns: The daily behavior of the elderly is modeled as a sequence of hidden states, where the hidden states represent behavioral patterns and the observed values ​​are multimodal sensor data. Given observation sequence The likelihood probability is calculated using the forward algorithm: ； in, For observation sequence In the model The likelihood probability is as follows: Let be the initial state probability vector. Let be the probability of being in state q1 at the initial moment. Here is the state transition probability matrix. To start from state q t−1 Transfer to q t The probability, To observe the emission probability matrix, For state q t Generate observations o t The probability, Let T be a sequence of hidden states. An alert is sent to the user when the behavior deviates from the normal pattern.

10. A system for recognizing abnormal behaviors in the elderly based on multimodal fusion, characterized in that, include: The data acquisition module is configured to simultaneously acquire data from visual, inertial, and environmental sensors; The preprocessing module is configured to perform time alignment, normalization, missing value imputation, and feature extraction, and output a multimodal feature vector. A cross-modal attention adapter module is configured to dynamically weight and fuse the multimodal feature vectors to generate adaptive fused features; The graph construction module is configured to construct a multimodal fusion graph using the adaptive fusion features as nodes and intermodal similarity as edge weights. The hyperbolic Lorentz graph attention mechanism module is configured to update node representations in hyperbolic space and output hyperbolic fusion features; An anomaly detection module is configured to classify the hyperbolic fusion features based on a graph attention network to obtain anomaly behavior categories; The root cause localization module is configured to use a random walk algorithm to calculate and output the root cause of the anomaly when an anomaly occurs. as well as The early warning interface module is configured to push anomaly category and root cause information to the monitoring terminal.

Images