Old people health monitoring and behavior recognition system and method based on multi-mode perception

Abstract

The invention discloses an old people health monitoring and behavior recognition system and method based on multi-modal perception, and the system comprises a monitoring recognition system, a user interface module, a data set module, an algorithm module, a real-time monitoring module, a human body posture detection module, a human body tracking module, a fall detection module, a trust area frame selection module, an alarm module, and a health evaluation module. The output end of the monitoring identification system is in one-way connection with the input ends of the user interface module and the data set module, the output end of the user interface module is in one-way connection with the input end of the data set module, and the output end of the data set module is in one-way connection with the input ends of the algorithm module and the real-time monitoring module. The output end of the real-time monitoring module is in one-way connection with the input end of the human body posture detection module. According to machine vision, an advanced machine vision algorithm is adopted, and a self-training model can accurately judge the human body posture.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of health monitoring and behavior recognition of the elderly, in particular to a health monitoring and behavior recognition system and method for the elderly based on multi-modal perception. BACKGROUND

[0002] With the acceleration of global population aging, the safety of the elderly has been increasingly concerned. According to the data published by the World Health Organization, more than 300,000 people die of falls every year, of which more than half are the elderly. The data of the National Health Commission of China shows that the incidence of falls among people over 65 years old is 58.03 per 100,000 people, which is the first cause of injury death among the elderly. Among the injuries caused by falls, a considerable part is the secondary injury caused by the delay of medical assistance after falling. How to reduce such secondary injuries and detect whether someone has fallen and timely notify the guardian or medical personnel has become a key problem.

[0003] Through our previous investigation of existing products and solutions, we found that there are generally two solutions for existing products, and they all have different defects. One is to design a wearable device with a gyroscope sensor as the core, which directly detects acceleration and angle data to judge the body posture. But this solution has too much contingency, which will produce more false positives and false negatives, and long-term wearing on the body will cause some inconvenience to the user. The second existing solution is to use a family smart network camera to continuously collect family monitoring pictures and upload them to the cloud server, and the visual model deployed on the cloud server identifies and reasons the images. The main defects of this solution are two, one is that the continuous uploading of video data will bring a very large occupation to the uplink network of the family network; the other is that the continuously transmitted family monitoring pictures may cause privacy leakage. SUMMARY

[0004] The purpose of the present application is to provide a health monitoring and behavior recognition system and method for the elderly based on multi-modal perception to solve the problems raised in the background.

[0005] To achieve the above purpose, the present application provides the following technical solution: a health monitoring and behavior recognition system and method for the elderly based on multi-modal perception, comprising: a monitoring and recognition system, a user interface module, a data set module, an algorithm module, a real-time monitoring module, a human posture detection module, a human tracking module, a falling detection module, a trust area frame selection module, an alarm module and a health assessment module.

[0006] The output ends of the monitoring identification systems are each unidirectionally connected with the input ends of the user interface module and the dataset module, the output end of the user interface module is unidirectionally connected with the input end of the dataset module, the output ends of the dataset module are each unidirectionally connected with the input ends of the algorithm module and the real-time monitoring module, the output end of the real-time monitoring module is unidirectionally connected with the input end of the human body posture detection module, the output end of the human body posture detection module is unidirectionally connected with the input end of the human body tracking module, the output end of the human body tracking module is unidirectionally connected with the input end of the fall detection module, the output ends of the fall detection module are each unidirectionally connected with the input ends of the trusted area framing module and the alarm module, and the alarm module is bidirectionally connected with the health assessment module.

[0007] Preferably, the user interface module is used for registering a user information account and logging into the system.

[0008] Preferably, the dataset module is used for determining a target user group, collecting fall pictures in various scenes and postures through the Internet and autonomous shooting, finally forming a dataset of more than 1,000 pictures, and dividing the pictures into a training set, a test set and a verification set according to a ratio of 8:2:1.

[0009] Preferably, the algorithm module is used for using a YOLOv5 deep learning image recognition neural network, and migrating a pre-trained model that has been pre-trained by a large dataset to human body posture fall recognition through a migration learning manner.

[0010] Preferably, the real-time monitoring module is used for using a 1080p high-definition camera to continuously capture a video stream and cover a high-risk area in a family.

[0011] Preferably, the human body posture detection module is used for selecting a mature deep convolutional neural network image recognition algorithm YOLOv5 to perform human body posture recognition.

[0012] The human body tracking module is used for visually detecting a relative position of a human body in a picture, adjusting a camera position through a PID algorithm control rudder gimbal, and realizing tracking of the camera on the human body movement.

[0013] Preferably, the fall detection module is used for training a model through a YOLOv5 algorithm, and the model can detect a human body posture in real time to make a fall determination.

[0014] The trusted area framing module is used for the device to interact with a user terminal APP through MQTT, and the user can frame a picture captured by the device, and a "fall-like" behavior in the framed area, such as lying on a bed or a sofa, will not be determined as a fall by the device.

[0015] Preferably, the alarm module is used when the model determines that the human body is in a falling posture. We designed an algorithm to filter the human body after falling, reduce the possibility of misjudgment, and finally upload the alarm information through the MQTT Internet of Things protocol. After the device sends a message to the platform through the MQTT Internet of Things protocol, the APP on the mobile phone can automatically obtain the alarm information of the platform and display it, including the falling photos detected. The guardian can choose to handle it himself or through the alarm information set in advance, such as address, one-key alarm to contact medical, public security, fire department, etc.

[0016] Preferably, the health assessment module is used to monitor the health status of the elderly in real time according to the collected physiological sign data, combine the preset health threshold or established health model, judge whether there is an abnormal situation or potential health risk, such as whether the heart rate, blood pressure is beyond the normal range, whether the blood sugar is too high, etc., consider the physiological sign data and behavior data comprehensively, build a health risk assessment model, assess and predict the disease probability, health trend of the elderly, provide basis for prevention and intervention, and cooperate the health assessment module with the alarm module, assess the body state of the user after falling through the health assessment module, and feedback the physiological sign data of the user in time through the alarm module.

[0017] Preferably, the method comprises the following contents:

[0018] S1: Determine the data collection target group, and use the data set module to perform data collection and storage work;

[0019] S2: Preprocess the collected data and extract feature data;

[0020] S3: Use the collected and processed data to build a falling detection model through the algorithm module;

[0021] S4: The user wears the device and then logs in through the system user interface module;

[0022] S5: Use the real-time monitoring module to capture the user continuously and in real time;

[0023] S6: Use the algorithm module and the built falling detection model to cooperate with the human body posture detection module to identify the human body posture of the user;

[0024] S7: Track the human body movement action through the human body tracking module, and then use the model trained by the falling detection module to detect the human body falling determination;

[0025] S8: Use the trust region frame selection module to frame the user's "falling-like" behavior;

[0026] S9: When the user falls, the system determines this through the trained model, uploads a photo of the fall and the location through the alarm module, and feeds back the user's physiological data through the health assessment module.

[0027] Compared with the prior art, the application has the following advantages:

[0028] 1. Machine vision: advanced machine vision algorithms are used, and the self-trained model can accurately judge the human posture;

[0029] 2. Local deployment: the vision model is deployed in a local AI chip, combined with the camera part, and is small in size. Image recognition is completed locally, and the entire process does not require networking, avoiding privacy information leakage;

[0030] 3. Internet of Things: the lightweight MQTT protocol is used, and the network occupation is small when uploading fall information and pictures to the cloud platform, saving network resources. The Internet of Things protocol supports multiple devices online at the same time, and a user can deploy multiple devices in multiple scenarios to achieve comprehensive monitoring;

[0031] 4. Front-end interface: a user-friendly front-end interface is developed using the vue framework, which can provide timely fall information pop-up alarms and fall pictures taken by the camera, providing a basis for medical personnel to diagnose the condition in advance. At the same time, users can set the address, age, blood type, and medical history of the elderly in advance, which is beneficial to medical assistance. When a fall occurs, the "one-key alarm" can notify the medical, fire, and public security departments in a timely manner. The specific advantages are as follows:

[0032] Lightweight design: reduces resource consumption, suitable for low-performance devices;

[0033] Component-based development: easy to extend and maintain functions;

[0034] Cross-platform compatibility: supports multiple devices, including Android, iOS, and x86 devices;

[0035] Multifunctional integration: provides functions such as receiving alarms, pictures, and monitoring switching;

[0036] Efficient image processing: uses Base64 transcoding to maintain image quality;

[0037] Performance optimization: the application size is small, only 4M, reducing memory burden;

[0038] 5. Human tracking: the PID algorithm is used to control the rotation of the servo gimbal, so that the human body is always in the center of the screen. This allows the device to have a larger monitoring area and more accurately obtain fall images;

[0039] 6. Trust area frame selection: users can frame the camera picture area by themselves through the mobile phone APP. The device will not perform fall detection on the area framed by the user. It better solves the misjudgment and false report in the home environment. BRIEF DESCRIPTION OF DRAWINGS

[0040] Fig. 1 A schematic diagram of the system of the present application;

[0041] Fig. 2 A schematic diagram of the method flow of the present application. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0043] Please refer to Figs. 1-2 The present application provides a technical solution: a multi-modal perception-based health monitoring and behavior recognition system and method for the elderly, which comprises a monitoring and recognition system, a user interface module, a data set module, an algorithm module, a real-time monitoring module, a human body posture detection module, a human body tracking module, a fall detection module, a trust area frame selection module, an alarm module and a health assessment module.

[0044] The output ends of the monitoring and recognition system are all connected to the input ends of the user interface module and the data set module, the output end of the user interface module is connected to the input end of the data set module, the output ends of the data set module are all connected to the input ends of the algorithm module and the real-time monitoring module, the output end of the real-time monitoring module is connected to the input end of the human body posture detection module, the output end of the human body posture detection module is connected to the input end of the human body tracking module, the output end of the human body tracking module is connected to the input end of the fall detection module, the output ends of the fall detection module are all connected to the input ends of the trust area frame selection module and the alarm module, and the alarm module is bidirectionally connected to the health assessment module.

[0045] The user interface module is used to register user information accounts and log in the system.

[0046] The data set module is used to determine the target user group, collect fall pictures in various scenes and postures through the Internet and self-shooting, finally form a data set of more than 1,000 pictures, and divide them into a training set, a test set and a verification set according to a ratio of 8:2:1.

[0047] The algorithm module is used for transferring a pre-trained model pre-trained by a large data set to human posture fall recognition by using a YOLOv5 deep learning image recognition neural network through transfer learning.

[0048] The real-time monitoring module is used for capturing a video stream in real time and continuously by using a 1080p high-definition camera to cover high-risk areas in a family.

[0049] The human posture detection module is used for human posture recognition by using a mature deep convolutional neural network image recognition algorithm YOLOv5.

[0050] The human tracking module is used for visual detection of the relative position of a human body in a picture, adjustment of the position of a camera by a PID algorithm control of a steering engine holder, and realization of tracking of the human body movement by the camera.

[0051] The fall detection module is used for model training by using a YOLOv5 algorithm, and the model can detect a human posture in real time to determine a fall.

[0052] The trust region frame selection module is used for interaction of the device with a user terminal APP, and the user can frame select a picture taken by the device, and a "fall-like" behavior in the frame selected region, such as lying on a bed or a sofa, will not be determined as a fall by the device.

[0053] The alarm module is used for filtering processing of a human body condition after a fall by using an algorithm designed by the module when the model determines that the human body is in a fall posture, reducing the possibility of misjudgment, and finally uploading alarm information through an MQTT Internet of Things protocol; after the device sends a message to a platform through the MQTT Internet of Things protocol, the APP of the mobile phone terminal can automatically obtain and display the alarm information of the platform, including a fall photo detected. A guardian can choose to handle it by himself or through alarm information such as an address set in advance, and contact medical, public security, fire and other departments by one-key alarm.

[0054] The health assessment module is used for real-time monitoring of the health status of the elderly according to collected physiological sign data, combining a preset health threshold or a health model established, judging whether there is an abnormal condition or a potential health risk, such as whether a heart rate, a blood pressure, blood sugar, etc. is out of a normal range, and comprehensively considering physiological sign data and behavior data to construct a health risk assessment model, assess and predict the disease probability and health trend of the elderly, provide a basis for prevention and intervention, and cooperate the health assessment module with the alarm module to assess the physical condition of the user after a fall through the health assessment module, and timely feedback physiological sign data of the user through the alarm module.

[0055] Specific, using the application, step one: determine the data collection target group, using data set module for data collection and storage work, step two: the collected data are pretreated, and the characteristic data are extracted, step three: with the data after collection and processing, through the algorithm module constructs the fall detection model, step four: the user wears the device, then through the system user interface module login operation, step five: use real-time monitoring module for uninterrupted real-time capture of users, step six: use algorithm module and the constructed fall detection model, cooperate with human body posture detection module, identify the user human body posture, step seven: through the human body tracking module tracks the human body movement, and then uses the model trained by the fall detection module to detect the human body fall determination, step eight: use the trust area frame selection module to frame the user's "fall-like" behavior, step nine: through the model constructed by the trained model to determine the user's fall, through the alarm module uploads the fall photo and positioning to alarm, and through the health assessment module feedbacks the user's physiological sign data.

[0056] Although embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, replacements and changes can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. Elderly health monitoring and behavior recognition system based on multimodal perception, characterized by: include: Monitoring and recognition system, user interface module, data set module, algorithm module, real-time monitoring module, human posture detection module, human tracking module, fall detection module, trust region selection module, alarm module and health assessment module; The output ends of the monitoring and identification system are unidirectionally connected to the input ends of the user interface module and the data set module, the output end of the user interface module is unidirectionally connected to the input end of the data set module, the output ends of the data set module are unidirectionally connected to the input ends of the algorithm module and the real-time monitoring module, the output end of the real-time monitoring module is unidirectionally connected to the input end of the human posture detection module, the output end of the human posture detection module is unidirectionally connected to the input end of the human tracking module, the output end of the human tracking module is unidirectionally connected to the input end of the fall detection module, the output ends of the fall detection module are unidirectionally connected to the input ends of the trust area selection module and the alarm module, and the alarm module is bidirectionally connected to the health assessment module.

2. The multimodal sensing-based elderly health monitoring and behavior recognition system according to claim 1 is characterized by: The user interface module is used to register a user information account and log in to the system.

3. The multimodal sensing-based elderly health monitoring and behavior recognition system according to claim 1 is characterized by: The dataset module is used to identify the target user group. Through the internet and self-photographing, it collects pictures of falls in various scenes and postures, ultimately forming a dataset of more than a thousand pictures, which are divided into training set, test set and validation set in a ratio of 8:2:

1.

4. The multimodal sensing-based elderly health monitoring and behavior recognition system according to claim 1 is characterized by: The algorithm module is used to utilize the YOLOv5 deep learning image recognition neural network to migrate a pre-trained model that has been pre-trained with a large data set to human posture fall recognition through transfer learning.

5. The elderly health monitoring and behavior recognition system based on multimodal perception according to claim 1 is characterized by: The real-time monitoring module is used to use a 1080p high-definition camera to continuously capture real-time video streams, covering high-risk areas in the home.

6. The elderly health monitoring and behavior recognition system based on multimodal perception according to claim 1 is characterized by: The human posture detection module is used to select the mature deep convolutional neural network image recognition algorithm YOLOv5 for human posture recognition; The human body tracking module is used to visually detect the relative position of the human body to the screen, and controls the servo gimbal to adjust the camera position through the PID algorithm, so that the camera can track the movement of the human body.

7. The multimodal sensing-based elderly health monitoring and behavior recognition system according to claim 1 is characterized by: The fall detection module is used to train a model using the YOLOv5 algorithm. The model can detect human posture in real time to determine falls. The trusted area selection module is used for the device to interact with the user-side APP through MQTT. The user can select the picture taken by the device. "Fall-like" behaviors within the selected area, such as lying on the bed or sofa, will not be judged as falls by the device.

8. The multimodal sensing-based elderly health monitoring and behavior recognition system according to claim 1 is characterized by: When the model determines a person has fallen, the alarm module uses an algorithm designed to filter the person's condition after the fall, reducing the possibility of misjudgment. The module then uploads the alarm information via the MQTT IoT protocol. After the device sends a message to the platform via the MQTT IoT protocol, the mobile app automatically retrieves and displays the platform's alarm information, including a photo of the detected fall. Guardians can choose to handle the situation themselves or contact medical, public security, fire, and other departments with a single click using pre-set alarm information such as the address.

9. The multimodal sensing-based elderly health monitoring and behavior recognition system according to claim 1 is characterized by: The health assessment module is used to monitor the health status of the elderly in real time based on the collected physiological sign data, combined with preset health thresholds or established health models, to determine whether there are abnormal conditions or potential health risks, such as whether the heart rate and blood pressure are outside the normal range, whether the blood sugar is too high, etc., and comprehensively consider the physiological sign data and behavioral data to construct a health risk assessment model to evaluate and predict the probability of illness and health trends of the elderly, providing a basis for prevention and intervention. The health assessment module is coordinated with the alarm module to evaluate the user's physical condition after a fall through the health assessment module, and promptly feedback the user's physiological sign data through the alarm module.

10. A method for elderly health monitoring and behavior recognition based on multimodal perception, characterized by: Based on the multimodal perception-based elderly health monitoring and behavior recognition system according to any one of claims 1 to 9, the method includes the following contents: S1: Determine the target group for data collection and use the dataset module to collect and store data; S2: preprocess the collected data and extract feature data; S3: It is conducive to collecting and processing data and building a fall detection model through algorithm modules; S4: The user wears the device and then logs in through the system user interface module; S5: Use the real-time monitoring module to continuously capture the user in real time; S6: Using the algorithm module and the constructed fall detection model, in conjunction with the human posture detection module, the user's human posture is recognized; S7: Tracking human movement through the human tracking module, and then using the model trained by the fall detection module to detect human falls; S8: Use the trust region selection module to select the user's "fall-like" behavior; S9: By building a trained model to determine when a user falls, the alarm module uploads the fall photo and location to issue an alarm, and the health assessment module feeds back the user's physiological signs data.

Images