Nursing home elder missing early warning system based on multi-modal artificial intelligence model

Abstract

The invention discloses a nursing home elder missing early warning system based on a multi-modal artificial intelligence model. The system comprises a data acquisition module, a multi-modal fusion module, an artificial intelligence analysis module, an early warning module and a background management module. The data acquisition module is used for acquiring video image data, wearable equipment data and environment data of old people in a nursing home; the multi-modal fusion module carries out fusion processing on the collected data of different modals to generate a multi-modal feature vector; the artificial intelligence analysis module analyzes the feature vector based on a multi-modal artificial intelligence model, and outputs the missing risk level and confidence of the elderly; when it is judged that the missing risk level of the old person reaches a preset early warning threshold value and the confidence degree reaches a preset requirement, an early warning module starts early warning; and the background management module configures, manages and monitors the system. According to the invention, the early warning of the missing risk of the old people in the nursing home is realized by fusing various modal data and utilizing a deep learning technology, so that powerful technical support is provided for the safety management of the nursing home, the missing risk of the old people is effectively reduced, and the safety and health of the old people are guaranteed; the method has important application value in old people centralized nursing institutions such as nursing homes.

Description

[0001] manual Technical Field

[0002] The present invention relates to the technical field of safety monitoring for elderly people in nursing homes, and in particular to a missing elderly people warning system in nursing homes based on a multimodal artificial intelligence model. Background Art

[0003] In nursing homes, elderly residents getting lost is a common problem, especially for those with cognitive impairments (such as Alzheimer's disease). Due to memory loss and disorientation, they may get lost or leave the home without permission. This not only poses a huge challenge to nursing home management but can also pose a serious threat to the lives of the elderly. Traditional methods of preventing elderly residents from getting lost rely primarily on manual care and access control systems. However, manual care is costly and prone to oversight, while simple access control systems only trigger an alarm when an elderly person attempts to leave the building and provide no early warning of potential wandering.

[0004] In recent years, with the development of artificial intelligence (AI) technology, systems have emerged that use video surveillance or wearable devices to monitor individual activities. However, these systems are mostly based on single-modality data and are susceptible to environmental factors (such as light changes and occlusion) or device performance limitations. This results in inaccurate and incomplete monitoring results, making it difficult to effectively predict the risk of elderly people getting lost. For example, relying solely on video surveillance may make it impossible to continue tracking an elderly person if they enter a blind spot, while relying solely on positioning data from wearable devices may result in errors due to signal interference and other issues. Therefore, a more intelligent and efficient nursing home elder loss warning system is needed that can integrate multi-modal data to detect the tendency of elderly people to get lost in advance and take timely measures to prevent them from actually getting lost. Summary of the Invention

[0005] The purpose of this invention is to provide a nursing home elderly missing warning system based on a multimodal artificial intelligence model, which integrates data from multiple modalities and uses artificial intelligence technology to achieve early warning of the risk of elderly people in nursing homes getting lost.

[0006] The present invention's nursing home elderly missing warning system includes the following modules:

[0007] Data acquisition module: used to collect multiple modal data of the elderly in the nursing home, including video image data, wearable device data (such as location data, gait data, heart rate data and other physiological and motion data) and environmental data (such as temperature, humidity, light and other data in the nursing home). Video image data can be collected through cameras installed in various areas of the nursing home (such as corridors, halls, room entrances and exits, gates, etc.) to observe the elderly’s behavior, movements, facial expressions, and location information; wearable device data is monitored and transmitted in real time by the devices worn by the elderly to capture the elderly’s real-time location, gait changes, heart rate and other physiological and motion characteristics; environmental data is obtained through corresponding environmental sensors to help determine whether the elderly’s environment may affect their behavior patterns, for example, whether an environment that is too noisy or the temperature is not suitable may cause the elderly to want to leave.

[0008] Multimodal Fusion Module: This module fuses collected data from different modalities. First, preprocessing the data, such as cleaning, normalization, and time synchronization, is performed to eliminate discrepancies and noise. Deep learning and other technologies are then used to organically integrate behavioral and facial expression features from video images, location, gait, and physiological characteristics from wearable devices, and environmental data such as temperature, humidity, and lighting. This generates a multimodal feature vector that comprehensively reflects the elderly person's behavioral status and risk of wandering.

[0009] Artificial Intelligence Analysis Module: This module builds an algorithm for identifying the risk of elderly individuals getting lost, based on a multimodal AI model. Using a fused multimodal feature vector as input, the model is trained on a large amount of labeled and unlabeled daily behavioral data from elderly individuals (including both normal and those with a tendency to get lost). This allows the model to accurately identify behavioral patterns that may indicate an elderly individual is getting lost (e.g., wandering around doorways, frequent attempts to open doors, abnormal gait, anxious expressions, and other combinations of features). The model then outputs the elderly individual's risk level (e.g., low, medium, high) and the corresponding confidence level.

[0010] Early warning module: When the artificial intelligence analysis module determines that the risk level of the elderly person getting lost reaches the preset warning threshold (such as medium risk or above) and the confidence level meets certain requirements, the early warning module will promptly activate the early warning mechanism. Early warning methods include local sound and light warnings to remind nursing home staff that the elderly person may be at risk of getting lost. At the same time, early warning information is sent to the terminal devices of preset guardians, family members or nursing home managers via the network. The early warning information may include the elderly person's identity information, current location, loss risk level, and related on-site images or video clips and wearable device data screenshots, so that relevant personnel can quickly obtain accurate information and take corresponding measures, such as strengthening care, promptly comforting the elderly, etc., to prevent the elderly person from actually getting lost.

[0011] Backend Management Module: This module is used to configure, manage, and monitor the system. It enables functions such as setting parameters for data acquisition devices (such as camera parameters and wearable device data collection frequency), adjusting multimodal fusion strategies, training and optimizing AI models, setting warning thresholds, and querying and compiling warning records. The backend management module also provides a user interface, allowing managers, family members, and medical staff to view the elderly's behavioral status and system warning status at any time.

[0012] The beneficial effects of the present invention are:

[0013] By integrating multiple modal information such as video images, wearable devices and environmental data, the limitations of single modal data such as susceptibility to interference and incomplete information are overcome, the accuracy and reliability of monitoring the tendency of elderly people to get lost are improved, and the behavioral patterns of elderly people who may be at risk of getting lost can be more comprehensively captured.

[0014] The multimodal artificial intelligence model based on deep learning can automatically learn the normal behavior patterns of the elderly and the behavioral characteristics of those who have a tendency to get lost. It has strong adaptability and generalization capabilities, and can effectively respond to the needs of elderly behavior monitoring in different scenarios and individual differences, reducing the occurrence of false alarms and missed reports.

[0015] A timely early warning mechanism can enable nursing home staff and family members to be informed in advance that the elderly may be at risk of getting lost, so that they can take preventive measures in time to prevent the elderly from actually getting lost, greatly improving the safety management level of the nursing home and ensuring the safety of the elderly.

[0016] The system's modular design facilitates expansion and optimization according to actual needs, such as adding new data acquisition modalities, improving fusion algorithms, or upgrading artificial intelligence models. It has good scalability and maintainability, and can adapt to the ever-changing management and monitoring needs of nursing homes. DETAILED DESCRIPTION

[0017] The following is a detailed description of the early warning system for missing elderly people in nursing homes of the present invention.

[0018] The nursing home elderly missing warning system of the present invention mainly includes a data acquisition module, a multimodal fusion module, an artificial intelligence analysis module, an early warning module and a background management module.

[0019] In the data acquisition module, multiple cameras are installed in key areas of the nursing home, such as corridors, halls, room entrances and exits, and gates, ensuring comprehensive and comprehensive capture of the elderly's behavior and location. These cameras can be high-definition and equipped with night vision capabilities to adapt to monitoring needs in various lighting conditions, capturing real-time video image data of the elderly, including their movements, expressions, and walking trajectories. Meanwhile, wearable devices worn by the elderly (such as smart bracelets or smart positioning badges) integrate multiple sensors, including positioning chips, accelerometers, and heart rate sensors. These devices monitor the elderly's location coordinates (using indoor positioning technologies such as Wi-Fi, Bluetooth, or ultra-wideband positioning), gait characteristics (such as walking speed, stride length, and cadence), heart rate, and other physiological and motion data in real time, transmitting this data to the system via wireless networks (such as Bluetooth or Wi-Fi). Furthermore, environmental sensors are installed throughout the nursing home's rooms and public areas to collect environmental data such as temperature, humidity, and light intensity. This data is collected via wired or wireless communication to the data acquisition module's central processing unit.

[0020] After receiving the various data transmitted from the data acquisition module, the multimodal fusion module first preprocesses the data. For video image data, this involves noise removal, resizing, and frame rate unification. For wearable device data, this involves data format conversion, unit unification, and missing value imputation. For environmental data, this involves data smoothing. Next, a deep learning-based multimodal feature extraction method is used to extract behavioral features (such as lingering at the door, attempting to open the door, and walking quickly) and facial expression features (such as anxiety and tension) from the video images. Position change features, gait abnormalities (such as unsteady gait and sudden increases in walking speed), and physiological features (such as abnormal heart rate fluctuations) are extracted from the wearable device data. Temperature, humidity, and lighting characteristics are extracted from the environmental data. These features from different modalities are then fused using a fusion strategy (e.g., using feature concatenation within a deep neural network combined with an attention mechanism to highlight key features). This generates a multimodal feature vector that comprehensively reflects the elderly person's behavioral status and risk of loss. This vector is then fed into the artificial intelligence analysis module.

[0021] The AI ​​analysis module constructs a multimodal AI model based on a deep neural network. During the model training phase, a large amount of daily behavioral data from nursing home residents is collected, including data on normal behaviors (such as walking, resting, and eating) and data on behaviors indicating a tendency to wander (such as lingering at the door for extended periods, frequently attempting to open the door, and aimlessly walking quickly within the nursing home). This data is annotated and divided into training, validation, and test sets. The model is trained using the training set, and its parameters are adjusted using an optimization algorithm to enable it to learn the associations between different modal data and the characteristic representations of the elderly's tendency to wander. The model is evaluated and tuned on the validation set to determine the optimal model structure and hyperparameters. Finally, the trained model is tested on the test set to verify its performance in identifying the risk of wandering residents, including accuracy and recall. When the system is running, the fused multimodal feature vector is input into the trained model, which then calculates and outputs the current risk level and confidence level of the elderly person. For example, when an elderly person wanders near the gate of a nursing home for more than a certain period of time, with a faster gait and an increased heart rate, and environmental data indicates that the weather is good that day (which may to some extent stimulate the elderly person's willingness to go out), the model may output a higher risk level of getting lost and a higher confidence level after integrating these multimodal feature information.

[0022] After receiving the risk assessment results from the AI ​​analysis module, the early warning module immediately activates the warning mechanism if it determines that the elderly person's risk of getting lost reaches the preset warning threshold (e.g., medium risk or above) and the confidence level exceeds 70%. A local audio and visual warning device emits flashing lights and audible alerts in a staff duty room or public area near the elderly person's location, alerting on-site staff to the elderly person. Simultaneously, the warning is sent via the network to the mobile phones, computers, and other devices of designated guardians, family members, and nursing home administrators. The warning information includes the elderly person's name, photo, current location, loss risk level, confidence level, as well as on-site images or video clips captured by cameras (e.g., footage of the elderly person wandering at the door) and screenshots of location change data recorded by wearable devices, allowing relevant personnel to quickly understand the elderly person's situation and respond. For example, upon receiving the warning, staff can quickly visit the elderly person's location to check on them. If they determine that the elderly person is only temporarily staying and has no intention of getting lost, the warning can be lifted or downgraded. If they determine that the elderly person is indeed showing signs of getting lost, they can take timely measures to prevent them from leaving and provide comfort and care.

[0023] The backend management module provides comprehensive management capabilities for system administrators. Administrators can use this module to configure parameters for data acquisition devices such as cameras, wearable devices, and environmental sensors. These parameters include adjusting the camera's shooting angle, resolution, and frame rate, and setting the data collection frequency and upload cycle for wearable devices. Furthermore, the multimodal fusion module's fusion algorithms and strategies can be adjusted to accommodate the specific circumstances, such as the layout of different areas within the nursing home and the varying behavioral characteristics of the residents. Furthermore, the backend management module supports the training and optimization of artificial intelligence models. Administrators can import new behavioral data (including emerging patterns of wandering tendencies) to incrementally train the models, continuously improving their recognition accuracy and generalization capabilities. Warning thresholds can also be set in the backend management module, allowing for flexible adjustment of warning sensitivity based on the nursing home's actual circumstances and management needs. The system also records all warning events and system operation logs. Administrators can use the backend management module to query historical warning records and analyze information such as the frequency, time distribution, and location distribution of the risk of wandering residents, providing data support for nursing home safety management and resident care arrangements. Family members and medical staff can also view the elderly's behavioral status information at any time through the user interface provided by the background management module, including video images, physiological data and location information, to understand the elderly's daily life and safety conditions in the nursing home.

[0024] In summary, the nursing home elderly missing warning system based on the multimodal artificial intelligence model of the present invention realizes early warning of the risk of elderly people in nursing homes missing by integrating multiple modal data and using advanced deep learning technology for intelligent analysis and judgment, providing strong technical support for the safety management of nursing homes, effectively reducing the risk of elderly people getting lost, and ensuring the safety and health of the elderly. It has important application value and promotion prospects in nursing homes and other centralized care institutions for the elderly.

Claims

1. A missing elderly warning system for nursing homes based on a multimodal artificial intelligence model, characterized by: include: Data acquisition module, used to collect video image data, wearable device data and environmental data of the elderly in the nursing home; The multimodal fusion module is used to fuse the collected data of different modalities and generate a multimodal feature vector; The AI ​​analysis module builds an elderly person loss risk identification algorithm based on a multimodal AI model. It uses the fused multimodal feature vector as input and outputs the elderly person's loss risk level and confidence level. Early warning module: When the artificial intelligence analysis module determines that the risk level of the elderly person getting lost reaches the preset early warning threshold and the confidence level meets the preset requirements, the early warning mechanism is activated and an early warning message is sent to relevant personnel; The backend management module is used to configure, manage and monitor the system.

2. The missing elderly warning system for nursing homes based on a multimodal artificial intelligence model according to claim 1 is characterized in that: The data acquisition module includes: cameras, installed in the corridors, halls, room entrances and exits, gates and other areas of the nursing home, used to capture video image data such as the elderly's behavior and facial expressions; wearable devices, worn by the elderly, used to monitor the elderly's location data, gait data, heart rate data and other physiological and motion data in real time; environmental sensors, distributed in various rooms and public areas of the nursing home, used to collect environmental data such as temperature, humidity, and light.

3. The missing elderly warning system for nursing homes based on a multimodal artificial intelligence model according to claim 1 is characterized in that: The multimodal fusion module adopts multimodal feature extraction and fusion technology based on deep learning, including the following steps: preprocessing data of different modalities, including data cleaning, normalization and time synchronization; extracting behavioral action features and facial expression features from video image data, extracting position features, gait features and physiological features from wearable device data, and extracting temperature, humidity and lighting features from environmental data; adopting feature splicing and fusion method, combined with attention mechanism to highlight key features, fusing features of different modalities and generating multimodal feature vectors.

4. The missing elderly warning system for nursing homes based on a multimodal artificial intelligence model according to claim 1 is characterized in that: The multimodal artificial intelligence model of the artificial intelligence analysis module adopts a deep neural network architecture to process the spatiotemporal information of video images, the temporal characteristics of wearable device data, and the environmental characteristics of environmental data.

5. The missing elderly warning system for nursing homes based on a multimodal artificial intelligence model according to claim 1 is characterized in that: The warning methods of the warning module include local sound and light warning and network warning. The local sound and light warning is used to remind the staff in the nursing home. The network warning sends warning information to the terminal devices of the preset guardians, family members or nursing home managers through the network. The warning information includes the identity information of the elderly, current location, risk level of getting lost, and related on-site images or video clips and wearable device data screenshots.

6. The missing elderly warning system for nursing homes based on a multimodal artificial intelligence model according to claim 1 is characterized in that: The background management module has the following functions: parameter setting of data acquisition equipment; adjustment of multimodal fusion strategy; training and optimization of artificial intelligence models; setting of warning thresholds; query and statistics of warning records.