A Smart Monitoring Method and System for Home-based Elderly Care Based on Multimodal Feature Attention Fusion

Abstract

The present invention discloses a multimodal feature attention fusion home-based elderly care intelligent monitoring method and system, belonging to the field of home-based intelligent elderly care technology. The present invention comprehensively collects multimodal data on the physical condition of elderly people at home, and adopts a deep learning model based on multimodal feature technology to realize the comprehensive health status level judgment of the elderly. It can make a comprehensive judgment and analysis of the physical condition of the elderly, reduce the misjudgment rate, the misjudgment rate and the missed judgment rate, improve the practicality and effectiveness of monitoring, and provide a good monitoring means for home-based elderly care. At the same time, it is also conducive to promoting the effective implementation of the national hierarchical diagnosis and treatment system, and has broad application prospects.

Description

Technical Field

[0001] The present invention relates to a multi-modal feature attention fusion home-based elderly care intelligent monitoring method and system, belonging to the technical field of home-based intelligent elderly care. Background Art

[0002] Community home-based elderly care is in line with the development trend of international elderly care. Most elderly people in the world enjoy home-based and community-based elderly care. Even in developed countries and high-welfare areas such as Europe and the United States, home-based and community-based elderly care is the main form of elderly care. The physical health and medical treatment of the elderly are the most basic life guarantees. In today's society, smart phones have become popular, and smart wearable devices such as smart bracelets and smart watches can monitor human body temperature, heart rate, blood pressure and other parameter information. Cameras, smart toilets and other devices have also entered many families. However, current smart wearable devices are only used by the wearer to check their physical condition, and have not been combined with other smart devices to effectively monitor and judge the physical health of the elderly at home.

[0003] Zhang Hui et al. (patent application number: CN202011101279.4) provided a method for detecting abnormal behavior of the elderly at home, which placed several cameras in each room to obtain various life videos of the elderly in the residence; trained a single-person behavior recognition CNN neural network, and extracted the feature values ​​of the elderly when they performed abnormal behaviors according to the changes in their character features when they performed abnormal behaviors; trained the network with various life videos obtained from different angles; fused and judged the feature values, and notified the elderly's family members. This method does not use smart wearable devices, smart toilets and other devices to collect data, and cannot fully reflect the physical condition of the elderly.

[0004] Ni Jiabin et al. (patent application number: CN201911388532.6) disclosed a smart home-based elderly care service system based on deep transfer learning, including: smart monitoring bracelet, information processing terminal and remote monitoring terminal. The invention provides a fingerprint alarm button on the smart monitoring bracelet, and the elderly can use the fingerprint alarm button to make a one-click alarm and transmit it to the remote monitoring terminal through the information processing terminal, so as to realize one-click alarm and timely rescue. This method does not use cameras, smart toilets and other equipment to collect data, and cannot fully reflect the physical condition of the elderly. Summary of the invention

[0005] In view of the problems of incomplete data collection and inaccurate abnormal judgment of the physical condition of the elderly at home in the prior art, the present invention proposes a multi-modal feature attention fusion home-based elderly care intelligent monitoring method and system. The solution uses cameras, smart wearable devices, smart toilets, etc. to comprehensively collect multi-modal data on the physical condition of the elderly at home, with high accuracy, simple and easy to implement, and has a good application prospect.

[0006] The present invention adopts the following technical solutions to solve the above technical problems:

[0007] A multi-modal feature attention fusion home-based elderly care intelligent monitoring method, the specific steps are as follows:

[0008] Step 1: Select several action videos from the open source HMDB dataset and classify the selected action videos into N1 levels according to the dangerousness of the action, where level 1 is the lowest level and level N1 is the highest level;

[0009] Step 2, using the action video selected in step 1 and the corresponding level as action training samples, training an N1 classifier that judges the action state level of the elderly according to the action, and obtaining the action model of the elderly; wherein the input of the N1 classifier is a frame sequence consisting of continuous action frames in the video, each frame is represented as three matrices in RGB mode, firstly, each frame is respectively extracted with the first convolution module to extract image features, and then the image features of each frame are input into the LSTM network in chronological order, and then the output of the LSTM network outputs the classification result through the output layer, and the output layer includes the first multi-layer perceptron of N1 units and the softmax activation function;

[0010] Step 3: Select several face images from the open source fer2013 dataset and classify the selected face images into N2 levels according to the degree of pain in their expressions, where level 1 is the lowest level and level N2 is the highest level.

[0011] Step 4, using the face image selected in step 3 and the corresponding level as expression training samples, training an N2 classifier that determines the psychological state level of the elderly based on the expression, and obtaining a psychological model of the elderly; wherein the input of the N2 classifier is a face image represented as three matrices in RGB format, firstly inputting the features extracted by the second convolution module, and then outputting the classification results of the extracted features through the output layer, and the output layer includes a second multi-layer perceptron of N2 units and a softmax activation function;

[0012] Step 5, collect some human physiological data, and divide the collected physiological data into N3 levels according to the degree of danger, where level 1 is the lowest level and level N3 is the highest level;

[0013] Step 6, using the physiological data collected in step 5 and the corresponding levels as physiological training samples, training an N3 classifier for judging the physiological state level of the elderly, and obtaining a physiological model of the elderly; wherein the N3 classifier is a third multi-layer perceptron;

[0014] Step 7: collect the activity video and physiological data of the elderly in real time, and extract continuous action frames and face images from the real-time activity video as inputs of the action model, psychological model and physiological model;

[0015] Step 8: Based on the attention mechanism, multimodal fusion is performed on the outputs of the action model, psychological model and physiological model in step 7;

[0016] Step 9, input the result of multimodal fusion in step 8 into the fourth multi-layer perceptron to obtain the comprehensive health level of the elderly, and issue an early warning according to the preset alarm method.

[0017] Furthermore, in step 2, the first convolution module includes a first convolution layer, a first pooling layer, a second convolution layer, and a second pooling layer connected in sequence, wherein a ReLU activation function is added between the first convolution layer and the first pooling layer, and between the second convolution layer and the second pooling layer, respectively.

[0018] Furthermore, the second convolution module in step 4 includes the 1st convolution layer, the 1st pooling layer, the 2nd convolution layer, the 2nd pooling layer, ..., the Kth convolution layer and the Kth pooling layer connected in sequence, K is a positive integer greater than or equal to 3, and a ReLU activation function is added between the kth convolution layer and the kth pooling layer, k = 1, 2, ..., K.

[0019] Furthermore, the result of multimodal fusion at the current moment in step 8 is:

[0020]

[0021] In the formula, the weight of the action model is The weight of mental models Weights of Physiological Models x1~x l is the l outputs of the action model before the current moment, y1~y l is the l outputs of the psychological model before the current moment, z1~z l is the l outputs of the physiological model before the current moment, a x is the initial weight of the action model, a y is the initial weight of the psychological model, a z is the initial weight of the physiological model.

[0022] Furthermore, the analytic hierarchy process was used to find out the relative importance of the action model, psychological model and physiological model to the comprehensive health level of the elderly, and used this as the initial weight of each model.

[0023] Furthermore, in step 9, the comprehensive health level of the elderly is divided into 4 levels. When the level is level 1, a warning is issued to the elderly himself; when the level is level 2, a warning is issued to the elderly himself and his guardian is notified; when the level is level 3, a warning is issued to the elderly himself, his guardian is notified, and the community elderly care service agency is reported at the same time; when the level is level 4, a warning is issued to the elderly himself, his guardian is notified, and the community elderly care service agency and designated hospitals are reported at the same time.

[0024] A multi-modal feature attention fusion home-based elderly care intelligent monitoring system, the system includes a high-definition camera, a physiological information collection module, and a cloud platform, wherein the high-definition camera and the wearable physiological information collection module are connected to the cloud platform via a wireless network;

[0025] High-definition cameras are set up in the elderly's activity areas to collect real-time videos of their activities;

[0026] The physiological information collection module collects the physiological data of the elderly;

[0027] The cloud platform uses the above method to obtain the comprehensive health level of the elderly and issue early warnings according to the preset alarm method.

[0028] Furthermore, the system also includes a smartphone connected to the cloud platform via a wireless network. The smartphone is held by the elderly's guardian, community elderly care service agency, or hospital, and the cloud platform issues an early warning via the smartphone.

[0029] Furthermore, the physiological information collection module includes a smart bracelet and a smart toilet. The smart bracelet collects the elderly's blood pressure, body temperature and heart rate information, and the smart toilet collects the elderly's urine volume, urination time, and urine flow rate.

[0030] Furthermore, the smart bracelet includes a wireless communication module and a physiological information collection module, a GPS positioning module, and an alarm module respectively connected thereto. The wireless communication module is connected to a cloud platform via a wireless network, and the cloud platform issues an early warning through the alarm module.

[0031] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0032] 1. The present invention uses cameras, smart wearable devices, smart toilets, etc. to comprehensively collect multimodal data on the physical condition of elderly people at home, and adopts a deep learning model based on multimodal feature technology to realize the comprehensive health status level judgment of the elderly. It can make a comprehensive judgment and analysis of the physical condition of the elderly, solve the problems of too little collected data, high recognition difficulty, low accuracy, etc., greatly reduce the misjudgment rate, misjudgment rate and missed judgment rate, and improve the practicality and accuracy of monitoring;

[0033] 2. The present invention adopts an improved method based on the attention mechanism to complete the fusion of multimodal information, divides the comprehensive health status of the elderly into M levels, and performs corresponding operations on the cloud platform based on the judged levels, which is conducive to the subsequent timely and reasonable diagnosis and treatment of the elderly, and is also conducive to promoting the effective implementation of the national tiered diagnosis and treatment system. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1It is a schematic diagram of the layout of the home-based elderly care intelligent monitoring system with multimodal feature attention fusion of the present invention; wherein: 1. cloud platform, 2. high-definition camera, 3. smart wearable device, 4. smart toilet, 5. smart phone.

[0035] Figure 2 This is the structure diagram of the action recognition model;

[0036] Figure 3 It is a diagram of the structure of the psychological recognition model;

[0037] Figure 4 It is the structure diagram of multi-layer perceptron;

[0038] Figure 5 This is a structural diagram of a deep learning model based on multimodal features. DETAILED DESCRIPTION

[0039] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings:

[0040] In one embodiment, Figure 1 As shown, a multimodal feature attention fusion home-based elderly care intelligent monitoring system is provided, which mainly includes a high-definition camera, a smart bracelet, a smart toilet, a cloud platform, and a smart phone. The high-definition camera, the smart bracelet, and the smart toilet are connected to the cloud platform via a wireless network, and the cloud platform is wirelessly connected to the smart phone via a wireless network.

[0041] High-definition cameras are set up in areas where the elderly often move, such as living rooms, bedrooms, kitchens, and bathrooms. At least three cameras are set up in each area to capture real-time images of the elderly's activities from multiple angles and transmit them to the cloud platform via wireless networks. The cloud platform extracts continuous action frames and facial images of people from the images captured by the high-definition cameras, and uses deep learning models to identify abnormal conditions of the elderly.

[0042] The smart bracelet includes a physiological information collection module, a GPS positioning module, an alarm module and a wireless communication module. The wireless communication module is connected to the cloud platform via a wireless network; its physiological information collection module collects the elderly's blood pressure, body temperature and heart rate information, and transmits it to the cloud platform via the wireless communication module; the alarm module is a one-button alarm.

[0043] The smart toilet includes a urodynamic detection module and a wireless communication module. The wireless communication module is connected to the cloud platform via a wireless network. The urodynamic detection module monitors and records urodynamic parameters such as the elderly's urine volume, urination time, and urine flow rate. The urodynamic detection data is transmitted to the cloud platform via the wireless communication module.

[0044] Smartphones are mainly owned by the elderly’s guardians, community elderly care service agencies, hospitals, etc.

[0045] The cloud platform inputs the collected multimodal data such as videos, pictures, physiological information data, urodynamic parameters, etc. into the deep learning classifier to determine the overall health status level of the elderly and perform corresponding operations.

[0046] In one embodiment, a multi-modal feature attention fusion home-based elderly care intelligent monitoring method is provided when the cloud platform judges the comprehensive health level of the elderly. Figure 5 As shown, the basic steps are as follows:

[0047] (1) From the open source HMDB dataset, select videos with labels of common actions in daily life such as "climbing", "falling", "walking", "running", "sitting", "standing up", "turning around", "pushing", and "pulling", and classify them into N1 levels according to the degree of danger of the actions, with level 1 being the lowest and level N1 being the highest. Use the selected action videos and their corresponding levels as action training samples, train an N1 classifier that judges the action state level of the elderly based on the actions, and obtain the action model of the elderly.

[0048] In one embodiment, Figure 2 As shown, the N1 classifier is a deep learning model including a convolution module, an LSTM network and a multi-layer perceptron. The convolution module includes two stacks of "convolution layer + ReLU + pooling layer". The input of the N1 classifier is a series of frame sequences consisting of continuous action frames in the video. Each frame is represented as three matrices in RGB format, that is, the number of input channels is 3; for each frame, image features are extracted by stacking two convolution layers + ReLU + pooling layers; then, the image features of each frame extracted by the convolution module are input into the LSTM network in chronological order; the LSTM network is used to grasp the temporal correlation of the input features, and the application of the network enables the model to grasp the posture of the person contained in each frame by linking it to an action; the output of the LSTM layer outputs the classification results of N1 categories through the output layer - a multi-layer perceptron of N1 units and a softmax activation function.

[0049] In one embodiment, in the above-mentioned convolution module, the convolution layer uses a convolution kernel of size 3×3 with a step size of 1; the pooling layer adopts the maximum pooling method, uses a filter of size 2×2 with a step size of 2; and a ReLU activation function is added between the convolution layer and the pooling layer to accelerate the convergence of the network.

[0050] (2) From the open source fer2013 dataset, we selected images with common facial expressions in daily life, such as "pain", "happy", "neutral", and "angry", and classified them into N2 levels according to the degree of pain in people's expressions, with level 1 being the lowest and level N2 being the highest. We used the selected images and their corresponding levels as expression training samples, trained an N2 classifier to judge the psychological state level of the elderly based on their expressions, and obtained a psychological model of the elderly.

[0051] In one embodiment, Figure 3 As shown in the figure, the N2 classifier is a deep learning model that includes a convolution module and a multi-layer perceptron. The convolution module includes a stack of several "convolution layers + ReLU + pooling layers". The input of the N2 classifier is a picture containing facial expressions of people, represented as three matrices in RGB format; after the input is stacked through several convolution layers + ReLU + pooling layers to extract image features, the output layer - a multi-layer perceptron with N2 units and a softmax activation function, the classification results of N2 categories are obtained.

[0052] (3) Collect physiological information data such as blood pressure, body temperature, heart rate, and urodynamic parameter data such as urine volume, urination time, and urine flow rate as total physiological data of the human body, and divide the degree of danger of total physiological data of the human body into N3 levels, with level 1 being the lowest and level N3 being the highest. Use the total physiological data of the human body and the corresponding levels as physiological training samples, train an N3 classifier to judge the level of the physiological state of the elderly, and obtain a physiological model of the elderly. The input of the N3 classifier is the measured physiological information data such as blood pressure, body temperature, heart rate, and urodynamic parameter data such as urine volume, urination time, and urine flow rate of the elderly, which are then passed through a multi-layer perceptron (such as Figure 4 As shown), the classification results of N3 categories are obtained.

[0053] (4) The cloud platform collects relevant data of the elderly in the target group, uses transfer learning, connects the output results of the models in steps (1), (2), and (3) above to the same multimodal feature fusion network, and uses the multimodal data extracted from the elderly group for fine-tuning. The multimodal feature fusion network uses an improved attention-based method to complete the fusion of multimodal information. The main method is as follows:

[0054] a. Based on the attention mechanism, the above three modalities are integrated. In order to find the initial weights of each modal model, the hierarchical analysis method can be used to find the relative importance of each modal model to the comprehensive health level of the elderly as the initial weight. The initial weight of the action model is set to a x , the initial weight of the psychological model is a y , the initial weight of the physiological model is a z .

[0055] b. Improve the above initial weights and introduce the concepts of mean and variance. Assume that within time t, the output of the action model is That is, l results are output, which are x1~x l ; Mental model output That is, l results are output, which are y1~y l ;Physiological model output That is, l results are output, which are z1~z l. Calculate the average and variance of the outputs of the three models respectively, and set the average value of the action model to be The variance is The average mental model is The variance is The average value of the physiological model is The variance is Right now:

[0056]

[0057]

[0058]

[0059] The above average values ​​reflect the average risk level of the elderly in three aspects of their health. The lower the average value, the healthier the elderly, and the higher the average value, the more dangerous the elderly. The variance reflects the credibility of the average risk level of the elderly. The smaller the variance, the higher the credibility of the average risk level of the elderly, and the larger the variance, the lower the credibility of the average risk level of the elderly.

[0060] c. To further improve the three weights in a, we can introduce a function that is proportional to the mean and inversely proportional to the variance. Since the highest risk levels of the three models are different, we can change the function to be proportional to N. i (i=1,2,3)-is inversely proportional to the mean value and the variance. Let the function be f1,f2,f3, that is:

[0061]

[0062]

[0063]

[0064] A x , a y , a z The three weights are multiplied by the corresponding functions mentioned above, and then normalized to get the final weight. Let the final weight of the action model be w1, the final weight of the psychological model be w2, and the final weight of the physiological model be w3, then:

[0065]

[0066] Assume that the feature of multimodal fusion based on attention mechanism is F', then:

[0067]

[0068] (5) The features fused by the above-mentioned attention mechanism are then input into the multi-layer perceptron to complete the M classification of the elderly’s status; and the comprehensive health level of the elderly is obtained.

[0069] (6) After the comprehensive health level of the elderly is determined, the cloud platform performs corresponding operations based on the determined level (M=4): when the danger level is 1, only the elderly person is reminded; when the danger level is 2, the elderly person is reminded and the guardian holding the smartphone is notified; when the danger level is 3, the elderly person is reminded, the guardian is notified, and the community elderly care service agency holding the smartphone is reported to obtain community assistance; when the danger level is 4, the operations of the previous level are completed and the hospital holding the smartphone is reported.

[0070] The above is a detailed introduction to a multimodal feature attention fusion home-based elderly care intelligent monitoring method and system provided by an embodiment of the present invention. For those skilled in the art, according to the ideas of the embodiments of the present invention, there may be changes in the specific implementation methods and application scopes, and the content of this specification should not be understood as limiting the present invention.

Claims

1. An intelligent home-based elderly care monitoring method based on multimodal feature attention fusion, characterized in that: The specific steps are as follows: Step 1: Select several action videos from the open source HMDB dataset and classify the selected action videos into N1 levels according to the dangerousness of the action, where level 1 is the lowest level and level N1 is the highest level; Step 2, using the action video selected in step 1 and the corresponding level as action training samples, training an N1 classifier that judges the action state level of the elderly according to the action, and obtaining the action model of the elderly; wherein the input of the N1 classifier is a frame sequence consisting of continuous action frames in the video, each frame is represented as three matrices in RGB mode, firstly, each frame is respectively extracted with the first convolution module to extract image features, and then the image features of each frame are input into the LSTM network in chronological order, and then the output of the LSTM network outputs the classification result through the output layer, and the output layer includes the first multi-layer perceptron of N1 units and the softmax activation function; Step 3: Select several face images from the open source fer2013 dataset and classify the selected face images into N2 levels according to the degree of pain in their expressions, where level 1 is the lowest level and level N2 is the highest level. Step 4, using the face image selected in step 3 and the corresponding level as expression training samples, training an N2 classifier that determines the psychological state level of the elderly based on the expression, and obtaining a psychological model of the elderly; wherein the input of the N2 classifier is a face image represented as three matrices in RGB format, firstly inputting the features extracted by the second convolution module, and then outputting the classification results of the extracted features through the output layer, and the output layer includes a second multi-layer perceptron of N2 units and a softmax activation function; Step 5, collect some human physiological data, and divide the collected physiological data into N3 levels according to the degree of danger, where level 1 is the lowest level and level N3 is the highest level; Step 6, using the physiological data collected in step 5 and the corresponding levels as physiological training samples, training an N3 classifier for judging the physiological state level of the elderly, and obtaining a physiological model of the elderly; wherein the N3 classifier is a third multi-layer perceptron; Step 7: collect the activity video and physiological data of the elderly in real time, and extract continuous action frames and face images from the real-time activity video as inputs of the action model, psychological model and physiological model; Step 8: Based on the attention mechanism, multimodal fusion is performed on the outputs of the action model, psychological model and physiological model in step 7; The result of multimodal fusion at the current moment in step 8 is: In the formula, the weight of the action model is The weight of mental models Weights of Physiological Models is the l outputs of the action model before the current moment, y1~y l is the l outputs of the psychological model before the current moment, z1~z l is the l outputs of the physiological model before the current moment, a x is the initial weight of the action model, a y is the initial weight of the psychological model, a z is the initial weight of the physiological model; Step 9, input the result of multimodal fusion in step 8 into the fourth multi-layer perceptron to obtain the comprehensive health level of the elderly, and issue an early warning according to the preset alarm method.

2. The multi-modal feature attention fusion home-based elderly care intelligent monitoring method according to claim 1 is characterized in that: In step 2, the first convolution module includes a first convolution layer, a first pooling layer, a second convolution layer, and a second pooling layer connected in sequence, wherein a ReLU activation function is added between the first convolution layer and the first pooling layer, and between the second convolution layer and the second pooling layer, respectively.

3. The multi-modal feature attention fusion home-based elderly care intelligent monitoring method according to claim 1 is characterized in that: The second convolution module in step 4 includes the 1st convolution layer, the 1st pooling layer, the 2nd convolution layer, the 2nd pooling layer, ..., the Kth convolution layer and the Kth pooling layer connected in sequence, K is a positive integer greater than or equal to 3, and a ReLU activation function is added between the kth convolution layer and the kth pooling layer, k = 1, 2, ..., K.

4. The multi-modal feature attention fusion home-based elderly care intelligent monitoring method according to claim 1 is characterized in that: The analytic hierarchy process is used to find out the relative importance of the action model, psychological model and physiological model to the comprehensive health level of the elderly, and use this as the initial weight of each model.

5. The multi-modal feature attention fusion home-based elderly care intelligent monitoring method according to claim 1 is characterized in that: In step 9, the comprehensive health level of the elderly is divided into 4 levels. When the level is level 1, a warning is issued to the elderly person; when the level is level 2, a warning is issued to the elderly person and his guardian is notified; When the level is Level 3, a warning will be issued to the elderly person himself, his guardian will be notified, and the community elderly care service agency will be reported at the same time; when the level is Level 4, a warning will be issued to the elderly person himself, his guardian will be notified, and the community elderly care service agency and designated hospitals will be reported at the same time.

6. The multi-modal feature attention fusion home-based elderly care intelligent monitoring system is characterized by: The system includes a high-definition camera, a physiological information collection module, and a cloud platform. The high-definition camera and the wearable physiological information collection module are connected to the cloud platform via a wireless network; High-definition cameras are set up in the elderly's activity areas to collect real-time videos of their activities; The physiological information collection module collects the physiological data of the elderly; The cloud platform adopts the method described in any one of claims 1 to 5 to obtain the comprehensive health level of the elderly and issue an early warning according to a preset alarm method.

7. The multi-modal feature attention fusion home-based elderly care intelligent monitoring system according to claim 6 is characterized in that: The system also includes a smartphone connected to the cloud platform via a wireless network. The smartphone is held by the elderly's guardian, community elderly care service agency, or hospital, and the cloud platform issues an early warning through the smartphone.

8. The multi-modal feature attention fusion home-based elderly care intelligent monitoring system according to claim 6 is characterized in that: The physiological information collection module includes a smart bracelet and a smart toilet. The smart bracelet collects the elderly's blood pressure, body temperature and heart rate information, and the smart toilet collects the elderly's urine volume, urination time and urine flow rate.

9. The multi-modal feature attention fusion home-based elderly care intelligent monitoring system according to claim 8 is characterized in that: The smart bracelet includes a wireless communication module and a physiological information collection module, a GPS positioning module, and an alarm module respectively connected thereto. The wireless communication module is connected to a cloud platform via a wireless network, and the cloud platform issues an early warning through the alarm module.

Images