Emotional recognition and behavior pattern analysis-based early warning and intervention method for falling down of old people

Abstract

The invention discloses an elderly falling early warning and intervention method based on emotion recognition and behavior pattern analysis. The elderly falling early warning and intervention method comprises the following steps: S1, collecting a facial expression data set, a voice signal data set and a behavior data set of an elderly; s2, analyzing the collected facial expression data set by using a convolutional neural network model; s3, analyzing the voice signal data set by using a long short-term memory network model; s4, an ElderSense model is set; and S5, when the analysis result shows that the risk score is higher than a preset first threshold value, sending an early warning signal to the old people through the intelligent equipment, and when the analysis result shows that the falling risk is higher than a preset second threshold value, sending an early warning signal to the old people and caregivers thereof through the intelligent equipment. The falling risk of the old people can be comprehensively evaluated from multiple angles, and the accuracy of falling risk judgment in a complex situation is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of elderly monitoring, and in particular to an elderly fall warning and intervention method based on emotion recognition and behavior pattern analysis. Background Art

[0002] In modern society, as the aging problem becomes increasingly serious, the health and safety of the elderly have become the focus of social attention. Among them, falls among the elderly are an important public health issue, which not only causes serious physical injuries to the elderly themselves, and may even lead to death, but also brings huge economic and emotional burdens to the family and society. Therefore, it is particularly important to develop an effective fall warning and intervention system for the elderly.

[0003] In the existing technology, the elderly fall detection and warning system is mainly based on sensor technology, such as wearable sensors, ground pressure sensors and cameras in the environment. These systems can monitor the activity status of the elderly to a certain extent and issue an alarm when a fall event is detected. However, most of the existing fall detection systems rely on a single data source and simple threshold judgment logic, lacking comprehensive consideration of the emotional state and behavior patterns of the elderly, which limits the accuracy and practicality of the system's warning.

[0004] First, fall detection methods based on a single data source often ignore the multifactorial nature of fall events. Falls in the elderly are a complex biomechanical event that can be affected by a variety of factors, such as physical health, environmental obstacles, and even emotional fluctuations. Relying solely on the accelerometer in wearable devices or the camera in the environment to capture motion abnormalities often fails to fully assess the risk of falls.

[0005] Secondly, existing technologies are deficient in emotion recognition and behavior pattern analysis. The emotional state and behavior habits of the elderly are factors that cannot be ignored in fall risk assessment. Emotional fluctuations may lead to a temporary decline in physical function, and specific behavior patterns, such as frequent waking up at night, may also increase the risk of falling. However, most current fall warning systems fail to effectively integrate this information for risk assessment.

[0006] Furthermore, the existing fall warning mechanism usually lacks personalization and real-time performance. When a fall risk is detected, the system often takes a unified warning measure, without providing personalized intervention suggestions based on the specific situation and needs of the elderly. In addition, the detection and warning response of fall events are often delayed, failing to achieve real-time monitoring and immediate response.

[0007] Therefore, how to provide a fall warning and intervention method for the elderly based on emotion recognition and behavior pattern analysis is an urgent problem that technicians in this field need to solve. Summary of the invention

[0008] One purpose of the present invention is to propose a fall warning and intervention method for the elderly based on emotion recognition and behavior pattern analysis. Compared with the traditional fall detection system that relies on a single data source, the present invention can comprehensively evaluate the fall risk of the elderly from multiple angles, thereby increasing the accuracy of judging the risk of falls in complex situations.

[0009] According to an embodiment of the present invention, a method for warning and intervening in elderly people's falls based on emotion recognition and behavior pattern analysis includes the following steps:

[0010] S1, collect facial expression datasets, voice signal datasets and behavior datasets of the elderly through cameras, microphones and multi-category sensors set up indoors;

[0011] S2, using convolutional neural network model to analyze the collected facial expression dataset to identify the emotional state of the elderly;

[0012] S3, using the long short-term memory network model to analyze the speech signal dataset to identify the emotional state of the elderly and assess the intensity of emotions;

[0013] S4. Setting up the ElderSense model. The ElderSense model combines the analysis results of the convolutional neural network model and the long short-term memory network model, calculates the current fall risk score through a risk assessment layer based on a machine learning algorithm, integrates emotional state, behavior pattern and historical fall data, and gives a corresponding risk score.

[0014] S5. When the analysis result shows that the risk score is higher than the preset first threshold, a warning signal is sent to the elderly through the smart device. When the analysis result shows that the risk of falling is higher than the preset second threshold, a warning signal is sent to the elderly and their caregivers through the smart device, and the camera is used to confirm whether the elderly has fallen. If they have fallen, the police and ambulance will be automatically called.

[0015] Optionally, the facial expression dataset includes micro-expression data, eye tracking data and skin color data, the speech signal dataset includes emotional intonation data, breathing pattern data and sound change data, and the behavioral dataset includes subtle movement data, daily activity pattern data and environmental interaction data.

[0016] Optionally, the S1 specifically includes:

[0017] S11, micro-expression data E micro High-speed cameras capture subtle facial changes and eye tracking data eye Using infrared sensors to track eye movements, skin color data skin The spectral sensor analyzes the facial skin color changes and collects micro-expression data micro, Eye Tracking Data eye and skin color data C skin Integrate and construct facial expression dataset D face (T);

[0018] S12, emotional intonation data P intonation Identify emotional states by analyzing changes in the frequency and intensity of speech, breathing pattern data B breath Capture breathing rate and depth, voice change data through microphone variation The waveform analysis of the sound signal is used to detect subtle emotional fluctuations, and the collected emotional intonation data P intonation , breathing pattern data B breath and sound change data V variation Integrate and construct speech signal dataset D voice (T);

[0019] S13, subtle motion data M micro The accelerometer and gyroscope capture the body's tiny movements and daily activity pattern data. daily Identify behavioral patterns by analyzing motion sensor data, environmental interaction data interaction Environmental sensors are used to record the interaction between the elderly and their surroundings, and the collected subtle movement data is used to micro , Daily activity pattern data A daily Interaction Data with Environment I interaction Integration and construction of behavioral dataset D behavior (T).

[0020] Optionally, the micro-expression data E micro Detection involves capturing subtle changes in facial expressions over a short period of time. Using time series analysis to detect subtle changes:

[0021]

[0022] Among them, E micro (t) represents the amount of facial micro-expression change at time t, reflecting the slight changes in facial expressions of the elderly at a specific moment, which is related to emotional fluctuations. W and H represent the width and height of the analyzed image. I t (x,y),I t-1 (x, y) is the pixel intensity at position (x, y) at time t and t-1, respectively, which is used to capture the changes in facial expressions;

[0023] The small changes in facial expressions are quantified by calculating the average value of the pixel intensity difference between consecutive frames, capturing facial micro-expression movements in a short period of time;

[0024] Using eye tracking technology, we can analyze visual focus by calculating pupil position changes:

[0025]

[0026] Among them, T eye (t) represents the eye movement speed at time t, which indicates the speed of the elderly's line of sight movement, attention distribution and emotional state, The pupil position coordinates at time t reflect the specific position of the elderly's sight, which is related to their focus and interests. Represents the pupil position coordinates at time t-1, used to compare Compare and calculate eye movement velocity.

[0027] Optionally, the emotional intonation data P intonation The analysis extracts acoustic features related to emotions from speech signals and calculates the average pitch over a period of time by weighted average. The fundamental frequency is a key acoustic feature in emotion recognition, and different emotional states will cause changes in the fundamental frequency.

[0028] Breathing pattern data B breath Extracting breathing-related features from speech signals to reflect the speaker's emotional state. Changes in breathing rate reflect changes in emotional state, including tension or anxiety, which may lead to faster breathing. Analyzing breathing patterns provides additional clues to emotional state.

[0029] Sound Change Data V variation The analysis focuses on detecting small changes in speech signals that are associated with specific emotional states. The standard deviation of sound intensity over a period of time is calculated to reflect the degree of fluctuation of the sound signal. Changes in emotions, including the transition from calm to excitement, will lead to increased volatility of the sound. By analyzing the degree of sound variability, the sound characteristics associated with specific emotional states are captured.

[0030] Optionally, the subtle motion data M micro Detect potential fall risks or mood changes from subtle body movements, including tremors or unsteady stance:

[0031]

[0032] Among them, M micro (t) represents the intensity of subtle movements at time point t, reflecting the slight body movements of the elderly at that moment, including slight swaying or trembling, indicating emotional fluctuations or physical instability, and x i (t), (y i (t) and z i (t) represents the position coordinates of the ith sensor at time t on the x, y, and z axes in three-dimensional space, which are used to calculate the tiny movements of various parts of the body, and N represents the total number of sensors involved in the analysis;

[0033] Calculate the average value of the spatial changes of the position recorded by the sensor between two consecutive time points to reflect the intensity of the slight movement of the body parts. By monitoring the subtle movements, the precursors of falls or body movements caused by emotional changes can be identified in time.

[0034] Daily activity pattern data A daily Focus on identifying the behavioral habits and life rhythms of the elderly to predict risk behaviors:

[0035]

[0036] Among them, A daily (t) is the activity pattern score at time t, which comprehensively reflects the daily activity pattern of the elderly, including walking, sitting, resting, etc., as well as the frequency and regularity of activities, ω k represents the weight of a specific activity k. The parameter is set based on the contribution of the activity to the risk of falling, including that nighttime activities may have a higher weight due to insufficient light. k (t) refers to the frequency of performing a specific activity k at time t. A high frequency of certain activities, including frequent nighttime waking, indicates a high risk of falling. K represents the total number of activity types monitored;

[0037] Environmental Interaction Data I interaction Analyze how older adults interact with their living environment using data collected from environmental sensors such as light, temperature, or moving objects.

[0038] Optionally, the step S2 specifically includes:

[0039] S21, collect the facial expression data set D face (T) Perform standardization to reduce the influence of lighting, facial direction and expression intensity factors, and obtain the preprocessed facial expression dataset D′ face ;

[0040] '

[0041] S22. Apply convolutional neural network model to facial expression dataset D face Each data in (E micro (t),T eye (t),C skin (t)) to extract features and obtain the key feature vector F of facial expression face ,The process involves multiple convolutional and pooling layers to automatically learn and extract deep features of facial expressions;

[0042] S23, based on the feature vector E obtained from the feature extraction stage state (t), use the fully connected layer of the convolutional neural network model to classify the emotional state of the elderly and output the emotional state label Estate (t), emotional states include basic emotion types such as happiness, sadness, anger, and surprise;

[0043] S24, use the labeled facial expression dataset to train and optimize the convolutional neural network model, using the cross entropy loss function L cross-entropy Measure the emotion recognition performance of the model and update the model weights through the back-propagation algorithm to optimize the accuracy and robustness of the model:

[0044]

[0045] Among them, L cross-entropy is the cross entropy loss function, C is the number of emotional state categories, y o,c Is an indication of whether the oth sample belongs to category c, 0 or 1, p o,c is the probability that the model predicts that sample o is of category c.

[0046] Optionally, the step S3 specifically includes:

[0047] S31, collect the speech signal data set D voice (T) is subjected to denoising, framing and windowing. The preprocessed data set is recorded as D' voice (T);

[0048] S32, apply long short-term memory network to the preprocessed speech signal D' voice (T) Perform feature extraction, process the time series data of speech signals, and capture the dynamic characteristics and emotional changes of speech;

[0049] S33. Based on the features extracted from the long short-term memory network, use one or more fully connected layers to classify the emotional state of the elderly, evaluate the emotional intensity, and output the emotional state label E emotion (t) and emotional intensity score I interaction (t);

[0050] S34. Use the labeled speech signal dataset to train the long short-term memory network:

[0051] L total =L cross-entropy (E emotion ,Y emotion )+λL intensity (I intensity ,Y intensity );

[0052] Among them, L total is the total loss function, which combines the cross entropy loss L of the emotional state cross-entropy and loss of emotional intensity intensity , Yemotion and Y intensity are the true labels of emotional state and emotional intensity, respectively, and λ is a hyperparameter that adjusts the contribution of the two losses.

[0053] Optionally, the step S4 specifically includes:

[0054] S41, integrated convolutional neural network model and long short-term memory network model analysis results, including emotional state label E emotion (t), emotional intensity score I intensity (t), facial expression feature vector F face and speech feature vector F voice , forming a comprehensive feature set F combined (t):

[0055]

[0056] S42, the comprehensive feature set F combined (t) and historical fall data D fall Combined with the machine learning-based risk assessment layer, the current fall risk score R is calculated fall (t):

[0057] R fall (t) = α·f(F combined (t),D fall )+β·g(E state (t),I intensity (t));

[0058] Among them, f is based on the comprehensive feature set F combined (t) and historical fall data D fall is the risk assessment function, g is the risk adjustment function based on emotional state and emotional intensity, and α and β are weight parameters for adjusting the contributions of the two risks.

[0059] Optionally, S5 specifically includes:

[0060] S51, setting two preset risk thresholds, a first threshold T1 and a second threshold T2, where T2>T1, for determining the level of fall risk;

[0061] S52. When the fall risk score R calculated by the ElderSense model fall (t) exceeds the first threshold value T1 but does not exceed the second threshold value T2:

[0062]

[0063] Use smart devices to send warning signals to the elderly. The warning signals are sound, light or vibration to remind the elderly to pay attention to the risk of falling:

[0064] S53, when the fall risk score R fall (t) When exceeding the second threshold value T2:

[0065]

[0066] Not only does it send a stronger early warning signal to the elderly, but it also sends warning messages to the elderly’s caregivers through smart devices and activates the indoor camera system to detect whether the elderly actually fall;

[0067] S54. The camera system uses a motion detection algorithm to confirm whether a fall event has occurred:

[0068] F fall (t) = DetectFall(D cam (t));

[0069] If F fall (t) = True, then execute emergency response:

[0070]

[0071] Among them, F fall (t) is the fall detection result, D cam (t) is the data captured by the camera at time t. If a fall is detected, the system will automatically initiate the emergency response procedure, call the preset emergency contact number, and provide the elderly person’s location information and the emergency assistance that may be needed.

[0072] The beneficial effects of the present invention are:

[0073] (1) By integrating multiple data sources such as facial expressions, voice signals, physiological data and behavioral patterns of the elderly, the present invention can comprehensively assess the fall risk of the elderly from multiple perspectives, thereby increasing the accuracy of fall risk judgment in complex situations, compared with the traditional fall detection system that relies on a single data source.

[0074] (2) The present invention sets up the ElderSense model to monitor and analyze the emotional state and behavioral habits of the elderly in real time. At the same time, it not only stops at fall detection and early warning, but also provides intervention measures according to the specific situation of the elderly (such as emotional state and behavioral habits).

[0075] (3) When the present invention detects a high-risk fall event, it will not only issue an early warning to the elderly and caregivers, but also automatically dial emergency contact numbers, including police and ambulance numbers, after confirming the fall has occurred, thereby improving the efficiency of emergency medical rescue. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0077] Figure 1 A flowchart of a method for early warning and intervention of elderly falls based on emotion recognition and behavior pattern analysis proposed by the present invention;

[0078] Figure 2 A system structure schematic diagram of a fall warning and intervention method for the elderly based on emotion recognition and behavior pattern analysis proposed by the present invention. DETAILED DESCRIPTION

[0079] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0080] refer to Figure 1-Figure 2 , a fall warning and intervention method for the elderly based on emotion recognition and behavior pattern analysis, comprising the following steps:

[0081] S1, collect facial expression datasets, voice signal datasets and behavior datasets of the elderly through cameras, microphones and multi-category sensors set up indoors;

[0082] S2, using convolutional neural network model to analyze the collected facial expression dataset to identify the emotional state of the elderly;

[0083] S3, using the long short-term memory network model to analyze the speech signal dataset to identify the emotional state of the elderly and assess the intensity of emotions;

[0084] S4. Setting up the ElderSense model. The ElderSense model combines the analysis results of the convolutional neural network model and the long short-term memory network model, calculates the current fall risk score through a risk assessment layer based on a machine learning algorithm, integrates emotional state, behavior pattern and historical fall data, and gives a corresponding risk score.

[0085] S5. When the analysis result shows that the risk score is higher than the preset first threshold, a warning signal is sent to the elderly through the smart device. When the analysis result shows that the risk of falling is higher than the preset second threshold, a warning signal is sent to the elderly and their caregivers through the smart device, and the camera is used to confirm whether the elderly has fallen. If they have fallen, the police and ambulance will be automatically called.

[0086] In this embodiment, the facial expression dataset includes micro-expression data, eye tracking data and skin color data, the speech signal dataset includes emotional intonation data, breathing pattern data and sound change data, and the behavioral dataset includes subtle movement data, daily activity pattern data and environmental interaction data.

[0087] In this implementation, S1 specifically includes:

[0088] S11, micro-expression data E micro High-speed cameras capture subtle facial changes and eye tracking data eye Using infrared sensors to track eye movements, skin color data skin The spectral sensor analyzes the facial skin color changes and collects micro-expression data micro , Eye Tracking Data eye and skin color data C skin Integrate and construct facial expression dataset D face (T);

[0089] S12, emotional intonation data P intonation Identify emotional states by analyzing changes in the frequency and intensity of speech, breathing pattern data B breath Capture breathing rate and depth, voice change data through microphone variation The waveform analysis of the sound signal is used to detect subtle emotional fluctuations, and the collected emotional intonation data P intonation , breathing pattern data B breath and sound change data V variation Integrate and construct speech signal dataset D voice (T);

[0090] S13, subtle motion data M micro The accelerometer and gyroscope capture the body's tiny movements and daily activity pattern data. daily Identify behavioral patterns by analyzing motion sensor data, environmental interaction data interaction Environmental sensors are used to record the interaction between the elderly and their surroundings, and the collected subtle movement data is used to micro , Daily activity pattern data A daily Interaction Data with Environment I interaction Integration and construction of behavioral dataset D behavior (T).

[0091] In this embodiment, the micro-expression data E micro Detection involves capturing subtle changes in facial expressions over a short period of time. Using time series analysis to detect subtle changes:

[0092]

[0093] Among them, E micro (t) represents the amount of facial micro-expression change at time t, reflecting the slight changes in facial expressions of the elderly at a specific moment, which is related to emotional fluctuations. W and H represent the width and height of the analyzed image. I t (x,y),I t-1 (x, y) is the pixel intensity at position (x, y) at time t and t-1, respectively, which is used to capture the changes in facial expressions;

[0094] The small changes in facial expressions are quantified by calculating the average value of the pixel intensity difference between consecutive frames, capturing facial micro-expression movements in a short period of time;

[0095] Using eye tracking technology, we can analyze visual focus by calculating pupil position changes:

[0096]

[0097] Among them, T eye (t) represents the eye movement speed at time t, which indicates the speed of the elderly's line of sight movement, attention distribution and emotional state, The pupil position coordinates at time t reflect the specific position of the elderly's sight, which is related to their focus and interests. Represents the pupil position coordinates at time t-1, used to compare Compare and calculate eye movement velocity.

[0098] In this embodiment, the emotional intonation data P intonation The analysis extracts acoustic features related to emotions from speech signals and calculates the average pitch over a period of time by weighted average. The fundamental frequency is a key acoustic feature in emotion recognition, and different emotional states will cause changes in the fundamental frequency.

[0099] Breathing pattern data B breath Extracting breathing-related features from speech signals to reflect the speaker's emotional state. Changes in breathing rate reflect changes in emotional state, including tension or anxiety, which may lead to faster breathing. Analyzing breathing patterns provides additional clues to emotional state.

[0100] Sound Change Data V variation The analysis focuses on detecting small changes in speech signals that are associated with specific emotional states. The standard deviation of sound intensity over a period of time is calculated to reflect the degree of fluctuation of the sound signal. Changes in emotions, including the transition from calm to excitement, will lead to increased volatility of the sound. By analyzing the degree of sound variability, the sound characteristics associated with specific emotional states are captured.

[0101] In this embodiment, the fine motion data M microDetect potential fall risks or mood changes from subtle body movements, including tremors or unsteady stance:

[0102]

[0103] Among them, M micro (t) represents the intensity of subtle movements at time point t, reflecting the slight body movements of the elderly at that moment, including slight swaying or trembling, indicating emotional fluctuations or physical instability, and x i (t), (y i (t) and z i (t) represents the position coordinates of the ith sensor at time t on the x, y, and z axes in three-dimensional space, which are used to calculate the tiny movements of various parts of the body, and N represents the total number of sensors involved in the analysis;

[0104] Calculate the average value of the spatial changes of the position recorded by the sensor between two consecutive time points to reflect the intensity of the slight movement of the body parts. By monitoring the subtle movements, the precursors of falls or body movements caused by emotional changes can be identified in time.

[0105] Daily activity pattern data A daily Focus on identifying the behavioral habits and life rhythms of the elderly to predict risk behaviors:

[0106]

[0107] Among them, A daily (t) is the activity pattern score at time t, which comprehensively reflects the daily activity pattern of the elderly, including walking, sitting, resting, etc., as well as the frequency and regularity of activities, ω k represents the weight of a specific activity k. The parameter is set based on the contribution of the activity to the risk of falling, including that nighttime activities may have a higher weight due to insufficient light. k (t) refers to the frequency of performing a specific activity k at time t. A high frequency of certain activities, including frequent nighttime waking, indicates a high risk of falling. K represents the total number of activity types monitored;

[0108] Environmental Interaction Data I interaction Analyze how older adults interact with their living environment using data collected from environmental sensors such as light, temperature, or moving objects.

[0109] In this implementation, step S2 specifically includes:

[0110] S21, collect the facial expression data set D face (T) Perform standardization to reduce the influence of lighting, facial direction and expression intensity factors, and obtain the preprocessed facial expression dataset D′ face ;

[0111] S22. Apply the convolutional neural network model to the facial expression dataset D′ face Each data in (E micro (t),T eye (t),C skin (t)) to extract features and obtain the key feature vector F of facial expression face ,The process involves multiple convolutional and pooling layers to automatically learn and extract deep features of facial expressions;

[0112] S23, based on the feature vector E obtained from the feature extraction stage state (t), use the fully connected layer of the convolutional neural network model to classify the emotional state of the elderly and output the emotional state label E state (t), emotional states include basic emotion types such as happiness, sadness, anger, and surprise;

[0113] S24, use the labeled facial expression dataset to train and optimize the convolutional neural network model, using the cross entropy loss function L cross-entropy Measure the emotion recognition performance of the model and update the model weights through the back-propagation algorithm to optimize the accuracy and robustness of the model:

[0114]

[0115] Among them, L cross-entropy is the cross entropy loss function, C is the number of emotional state categories, y o,c Is an indication of whether the oth sample belongs to category c, 0 or 1, p o,c is the probability that the model predicts that sample o is of category c.

[0116] In this implementation, step S3 specifically includes:

[0117] S31, collect the speech signal data set D voice (T) is subjected to denoising, framing and windowing. The preprocessed data set is recorded as D' voice (T);

[0118] S32, apply long short-term memory network to the preprocessed speech signal D' voice (T) Perform feature extraction, process the time series data of speech signals, and capture the dynamic characteristics and emotional changes of speech;

[0119] S33. Based on the features extracted from the long short-term memory network, use one or more fully connected layers to classify the emotional state of the elderly, evaluate the emotional intensity, and output the emotional state label E emotion (t) and emotional intensity score I interaction (t);

[0120] S34. Use the labeled speech signal dataset to train the long short-term memory network:

[0121] L total =L cross-entropy (E emotion ,Y emotion )+λL intensity (I intensity ,Y intensity );

[0122] Among them, L total is the total loss function, which combines the cross entropy loss L of the emotional state cross-entropy and loss of emotional intensity intensity , Y emotion and Y intensity are the true labels of emotional state and emotional intensity, respectively, and λ is a hyperparameter that adjusts the contribution of the two losses.

[0123] In this implementation, step S4 specifically includes:

[0124] S41, integrated convolutional neural network model and long short-term memory network model analysis results, including emotional state label E emotion (t), emotional intensity score I intensity (t), facial expression feature vector F face and speech feature vector F voice , forming a comprehensive feature set F combined (t):

[0125]

[0126] S42, the comprehensive feature set F combined (t) and historical fall data D fall Combined with the machine learning-based risk assessment layer, the current fall risk score R is calculated fall (t):

[0127] R fall (t) = α·f(F combined (t),D fall )+β·g(E state (t),I intensity (t));

[0128] Among them, f is based on the comprehensive feature set F combined (t) and historical fall data D fall is the risk assessment function, g is the risk adjustment function based on emotional state and emotional intensity, and α and β are weight parameters for adjusting the contributions of the two risks.

[0129] In this implementation manner, S5 specifically includes:

[0130] S51, setting two preset risk thresholds, a first threshold T1 and a second threshold T2, where T2>T1, for determining the level of fall risk;

[0131] S52. When the fall risk score R calculated by the ElderSense model fall (t) exceeds the first threshold value T1 but does not exceed the second threshold value T2:

[0132]

[0133] Use smart devices to send warning signals to the elderly. The warning signals are sound, light or vibration to remind the elderly to pay attention to the risk of falling:

[0134] S53, when the fall risk score R fall (t) When exceeding the second threshold value T2:

[0135]

[0136] Not only does it send a stronger early warning signal to the elderly, but it also sends warning messages to the elderly’s caregivers through smart devices and activates the indoor camera system to detect whether the elderly actually fall;

[0137] S54. The camera system uses a motion detection algorithm to confirm whether a fall event has occurred:

[0138] F fall (t) = DetectFall(D cam (t));

[0139] If F fall (t) = True, then execute emergency response:

[0140]

[0141] Among them, F fall (t) is the fall detection result, D cam (t) is the data captured by the camera at time t. If a fall is detected, the system will automatically initiate the emergency response procedure, call the preset emergency contact number, and provide the elderly person’s location information and the emergency assistance that may be needed.

[0142] By integrating multiple data sources such as facial expressions, voice signals, physiological data and behavioral patterns of the elderly, the present invention can comprehensively assess the fall risk of the elderly from multiple angles compared to traditional fall detection systems that rely on a single data source, thereby increasing the accuracy of judging the risk of falls in complex situations.

[0143] The present invention sets up the ElderSense model to monitor and analyze the emotional state and behavioral habits of the elderly in real time. At the same time, it not only stops at fall detection and early warning, but also provides intervention measures according to the specific situation of the elderly (such as emotional state and behavioral habits).

[0144] When a high-risk fall event is detected, the present invention not only issues an early warning to the elderly and caregivers, but also automatically dials emergency contact numbers, including police and ambulance numbers, after confirming the fall has occurred, thereby improving the efficiency of emergency medical rescue.

[0145] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A fall warning and intervention method for the elderly based on emotion recognition and behavior pattern analysis, characterized in that: The following steps are involved: S1, collect facial expression datasets, voice signal datasets and behavior datasets of the elderly through cameras, microphones and multi-category sensors set up indoors; S2, using convolutional neural network model to analyze the collected facial expression dataset to identify the emotional state of the elderly; S3, using the long short-term memory network model to analyze the speech signal dataset to identify the emotional state of the elderly and assess the intensity of emotions; S4. Setting up the ElderSense model. The ElderSense model combines the analysis results of the convolutional neural network model and the long short-term memory network model, calculates the current fall risk score through a risk assessment layer based on a machine learning algorithm, integrates emotional state, behavior pattern and historical fall data, and gives a corresponding risk score. S5. When the analysis result shows that the risk score is higher than the preset first threshold, a warning signal is sent to the elderly through the smart device. When the analysis result shows that the risk of falling is higher than the preset second threshold, a warning signal is sent to the elderly and their caregivers through the smart device, and the camera is used to confirm whether the elderly has fallen. If they have fallen, the police and ambulance will be automatically called.

2. According to claim 1, a method for warning and intervening in elderly people's falls based on emotion recognition and behavior pattern analysis is characterized in that: The facial expression dataset includes micro-expression data, eye tracking data and skin color data, the speech signal dataset includes emotional intonation data, breathing pattern data and sound change data, and the behavior dataset includes subtle movement data, daily activity pattern data and environmental interaction data.

3. The method for warning and intervening in elderly people's falls based on emotion recognition and behavior pattern analysis according to claim 1, characterized in that: The S1 specifically includes: S11, micro-expression data E micro High-speed cameras capture subtle facial changes and eye tracking data eye Using infrared sensors to track eye movements, skin color data skin The spectral sensor analyzes the facial skin color changes and collects micro-expression data micro , Eye Tracking Data eye and skin color data C skin Integrate and construct facial expression dataset D face (T); S12, emotional intonation data P intonation Identify emotional states by analyzing changes in the frequency and intensity of speech, breathing pattern data B breath Capture breathing rate and depth, voice change data through microphone variation The waveform analysis of the sound signal is used to detect subtle emotional fluctuations, and the collected emotional intonation data P intonation , breathing pattern data B breath and sound change data V variation Integrate and construct speech signal dataset D voice (T); S13, subtle motion data M micro The accelerometer and gyroscope capture the body's tiny movements and daily activity pattern data. daily Identify behavioral patterns by analyzing motion sensor data, environmental interaction data interaction Environmental sensors are used to record the interaction between the elderly and their surroundings, and the collected subtle movement data is used to micro , Daily activity pattern data A daily Interaction Data with Environment I interaction Integration and construction of behavioral dataset D behavior (T).

4. The method for early warning and intervention of elderly falls based on emotion recognition and behavior pattern analysis according to claim 3 is characterized in that: The micro-expression data E micro Detection involves capturing subtle changes in facial expressions over a short period of time. Using time series analysis to detect subtle changes: Among them, E micro (t) represents the amount of facial micro-expression change at time t, reflecting the slight changes in facial expressions of the elderly at a specific moment, which is related to emotional fluctuations. W and H represent the width and height of the analyzed image. I t (x,y),I t-1 (x, y) is the pixel intensity at position (x, y) at time t and t-1, respectively, which is used to capture the changes in facial expressions; The small changes in facial expressions are quantified by calculating the average value of the pixel intensity difference between consecutive frames, capturing facial micro-expression movements in a short period of time; Using eye tracking technology, we can analyze visual focus by calculating pupil position changes: Among them, T eye (t) represents the eye movement speed at time t, which indicates the speed of the elderly's line of sight movement, attention distribution and emotional state, The pupil position coordinates at time t reflect the specific position of the elderly's sight, which is related to their focus and interests. Represents the pupil position coordinates at time t-1, used to compare Compare and calculate eye movement velocity.

5. The method for early warning and intervention of elderly falls based on emotion recognition and behavior pattern analysis according to claim 3 is characterized in that: The emotional intonation data P intonation The analysis extracts acoustic features related to emotions from speech signals and calculates the average pitch over a period of time by weighted average. The fundamental frequency is a key acoustic feature in emotion recognition, and different emotional states will cause changes in the fundamental frequency. Breathing pattern data B breath Extracting breathing-related features from speech signals to reflect the speaker's emotional state. Changes in breathing rate reflect changes in emotional state, including tension or anxiety, which may lead to faster breathing. Analyzing breathing patterns provides additional clues to emotional state. Sound Change Data V variation The analysis focuses on detecting small changes in speech signals that are associated with specific emotional states. The standard deviation of sound intensity over a period of time is calculated to reflect the degree of fluctuation of the sound signal. Changes in emotions, including the transition from calm to excitement, will lead to increased volatility of the sound. By analyzing the degree of sound variability, the sound characteristics associated with specific emotional states are captured.

6. The method for warning and intervening in elderly people's falls based on emotion recognition and behavior pattern analysis according to claim 3, characterized in that: The subtle motion data M micro Detect potential fall risks or mood changes from subtle body movements, including tremors or unsteady stance: Among them, M micro (t) represents the intensity of subtle movements at time point t, reflecting the slight body movements of the elderly at that moment, including slight swaying or trembling, indicating emotional fluctuations or physical instability, and x i (t), (y i (t) and z i (t) represents the position coordinates of the ith sensor at time t on the x, y, and z axes in three-dimensional space, which are used to calculate the tiny movements of various parts of the body. N represents the total number of sensors involved in the analysis; Calculate the average value of the spatial changes of the position recorded by the sensor between two consecutive time points to reflect the intensity of the slight movement of the body parts. By monitoring the subtle movements, the precursors of falls or body movements caused by emotional changes can be identified in time. Daily activity pattern data A daily Focus on identifying the behavioral habits and life rhythms of the elderly to predict risk behaviors: Among them, A daily (t) is the activity pattern score at time t, which comprehensively reflects the daily activity pattern of the elderly, including walking, sitting, resting, etc., as well as the frequency and regularity of activities, ω k represents the weight of a specific activity k. The parameter is set based on the contribution of the activity to the risk of falling, including that nighttime activities may have a higher weight due to insufficient light. k (t) refers to the frequency of performing a specific activity k at time t. A high frequency of certain activities, including frequent nighttime waking, indicates a high risk of falling. K represents the total number of activity types monitored; Environmental Interaction Data I interaction Analyze how older adults interact with their living environment using data collected from environmental sensors such as light, temperature, or moving objects.

7. The method for early warning and intervention of elderly falls based on emotion recognition and behavior pattern analysis according to claim 1, characterized in that: The step S2 specifically includes: S21, collect the facial expression data set D face (T) Perform standardization to reduce the influence of lighting, facial direction and expression intensity factors, and obtain the preprocessed facial expression dataset D′ face ; S22. Apply the convolutional neural network model to the facial expression dataset D′ face Each data in (E micro (t),T eye (t),C skin (t)) to extract features and obtain the key feature vector F of facial expression face ,The process involves multiple convolutional and pooling layers to automatically learn and extract deep features of facial expressions; S23, based on the feature vector E obtained from the feature extraction stage state (t), use the fully connected layer of the convolutional neural network model to classify the emotional state of the elderly and output the emotional state label E state (t), emotional states include basic emotion types such as happiness, sadness, anger, and surprise; S24, use the labeled facial expression dataset to train and optimize the convolutional neural network model, using the cross entropy loss function L cross-entropy Measure the emotion recognition performance of the model and update the model weights through the back-propagation algorithm to optimize the accuracy and robustness of the model: Among them, L cross-entropy is the cross entropy loss function, C is the number of emotional state categories, y o,c Is an indication of whether the oth sample belongs to category c, 0 or 1, p o,c is the probability that the model predicts that sample o is of category c.

8. The method for warning and intervening in elderly people's falls based on emotion recognition and behavior pattern analysis according to claim 1, characterized in that: The step S3 specifically includes: S31, collect the speech signal data set D voice (T) is subjected to denoising, framing and windowing. The preprocessed data set is recorded as D' voice (T); S32, apply long short-term memory network to the preprocessed speech signal D' voice (T) Perform feature extraction, process the time series data of speech signals, and capture the dynamic characteristics and emotional changes of speech; S33. Based on the features extracted from the long short-term memory network, use one or more fully connected layers to classify the emotional state of the elderly, evaluate the emotional intensity, and output the emotional state label E. emotion (t) and emotional intensity score I interaction (t); S34. Use the labeled speech signal dataset to train the long short-term memory network: THE total =L cross-entropy (AND emotion ,Y emotion )+λL intensity (THE intensity ,Y intensity ); Among them, L total is the total loss function, which combines the cross entropy loss L of the emotional state cross-entropy and loss of emotional intensity intensity , Y emotion and Y intensity are the true labels of emotional state and emotional intensity, respectively, and λ is a hyperparameter that adjusts the contribution of the two losses.

9. The method for warning and intervening in elderly people's falls based on emotion recognition and behavior pattern analysis according to claim 8, characterized in that: The ElderSense model build includes: S41, integrated convolutional neural network model and long short-term memory network model analysis results, including emotional state label E emotion (t), emotional intensity score I intensity (t), facial expression feature vector F face and speech feature vector F voice , forming a comprehensive feature set F combined (t): S42, the comprehensive feature set F combined (t) and historical fall data D fall Combined with the machine learning-based risk assessment layer, the current fall risk score R is calculated fall (t): R fall (t)=α·f(F combined (t),D fall )+β·g(E state (t),I intensity (t)); Among them, f is based on the comprehensive feature set F combined (t) and historical fall data D fall is the risk assessment function, g is the risk adjustment function based on emotional state and emotional intensity, and α and β are weight parameters for adjusting the contributions of the two risks.

10. The method for warning and intervening in elderly people's falls based on emotion recognition and behavior pattern analysis according to claim 9, characterized in that: The S5 specifically includes: S51, setting two preset risk thresholds, a first threshold T1 and a second threshold T2, where T2>T1, for determining the level of fall risk; S52. When the fall risk score R calculated by the ElderSense model fall (t) exceeds the first threshold value T1 but does not exceed the second threshold value T2: Use smart devices to send warning signals to the elderly. The warning signals are sound, light or vibration to remind the elderly to pay attention to the risk of falling: S53, when the fall risk score R fall (t) When exceeding the second threshold value T2: Not only does it send a stronger early warning signal to the elderly, but it also sends warning messages to the elderly’s caregivers through smart devices and activates the indoor camera system to detect whether the elderly actually fall; S54. The camera system uses a motion detection algorithm to confirm whether a fall event has occurred: F fall (t)=DetectFall(D cam (t)); If F fall (t) = True, then execute emergency response: Among them, F fall (t) is the fall detection result, D cam (t) is the data captured by the camera at time t. If a fall is detected, the system will automatically initiate the emergency response procedure, call the preset emergency contact number, and provide the elderly person’s location information and the emergency assistance that may be needed.

Images