A community old person trajectory prediction and early warning method based on HMM-LSTM and CNN multi-source data fusion

Abstract

This invention relates to a method for predicting and warning the trajectory of elderly people in the community by fusing HMM-LSTM and CNN multi-source data. The method includes: S1, generating a personalized life profile with tags using a PostGIS spatial database; S2, using an HMM to train a state transition probability matrix based on the personalized life profile to predict the probability of the elderly person's behavioral intention in the next stage; S3, using LSTM to process real-time GPS sequences and physiological feature data, outputting the predicted spatial coordinates at time T; S4, capturing images of the elderly person through a community monitoring system, building an analysis model using CNN, and outputting comprehensive behavioral tags; S5, aligning the visual tags with the predicted spatial coordinates. If a logical conflict occurs between the two or an electronic fence is abnormally triggered, the system will trigger a tiered warning after assessing the abnormal behavior; S6, matching the nearest volunteer based on a GIS shortest path algorithm, which has advantages such as deeper prediction logic, multi-dimensional risk perception, and transparent response process.

Description

Technical Field

[0001] This invention relates to the fields of intelligent health monitoring and computer vision technology, and in particular to a method for predicting and warning the trajectory of elderly people in the community based on the fusion of HMM-LSTM and CNN multi-source data. Background Technology

[0002] With the accelerating aging of the population, the need for elderly care under the community-based elderly care model is becoming increasingly prominent, and the most important thing is to accurately predict and provide real-time early warning of the elderly's movement trajectory.

[0003] However, the community environment is complex, and the flow of people has a high degree of randomness and spatiotemporal nonlinearity. Furthermore, data from a single sensor is easily interfered with. Traditional methods based on positioning or simple threshold judgment are difficult to cope with long-term and large-scale trajectory changes, resulting in problems such as low prediction accuracy and high false alarm and false negative rates.

[0004] Existing trajectory prediction technologies are mostly concentrated in the fields of transportation or security, with relatively few studies on modeling the behavioral characteristics of the elderly. On the one hand, while location-based services (LBS) or radio frequency identification (RFID) methods can obtain real-time location, they lack the ability to learn from historical movement patterns and cannot predict potential risks. On the other hand, methods based on physical models or traditional time series analysis have insufficient generalization ability when dealing with high-dimensional and heterogeneous data.

[0005] Meanwhile, how to effectively integrate multi-source heterogeneous data from wearable devices, visual sensors, etc., to uncover the hidden behavioral patterns and address the unique movement characteristics of the elderly, such as slow walking and lingering, remains a technical challenge.

[0006] Therefore, this application provides a method for predicting and warning the trajectory of elderly people in the community based on the fusion of multi-source data from HMM-LSTM (Hidden Markov Model-Long Short-Term Memory Network) and CNN (Convolutional Neural Network) to solve the above problems. Summary of the Invention

[0007] The main objective of this invention is to address the technical challenges in existing community-based elderly care monitoring systems, such as reporting only after an incident, limited positioning accuracy in complex environments, and a lack of behavioral logic understanding. Existing monitoring solutions (such as GPS wristbands or ordinary video surveillance) cannot provide logical, proactive warnings for abnormal elderly behavior (such as leaving home late at night or lingering by a river for extended periods); furthermore, data from single sensors is prone to shifting in indoor or high-rise building obstructions, leading to a high false alarm rate in risk assessments.

[0008] This invention provides a method for predicting and warning the trajectory of elderly people in communities based on the fusion of HMM-LSTM and CNN multi-source data. The method includes:

[0009] S1. Use the PostGIS spatial database to link community maps and the historical trajectories of the elderly to generate personalized life profiles with tags; S2. Based on the personalized life profile generated in step S1, use HMM to train the state transition probability matrix to predict the probability of the elderly’s behavioral intentions in the next stage. S3. Use LSTM to process real-time GPS sequence and physiological feature data, and output the predicted spatial coordinates at the next time T. S4. Capture images of elderly people through the community monitoring system, use CNN to build analysis models of walking posture, gestures, and standing posture, and output comprehensive behavioral labels. ; S5. Align the visual label from step S4 with the predicted spatial coordinates from step S3. If there is a logical conflict between the two or an abnormal triggering of the electronic fence, the system will trigger a graded warning after evaluating the abnormal behavior. S6. Based on the GIS shortest path algorithm, the nearest volunteer is matched, and service certificates are automatically generated through electronic fences.

[0010] Optionally, step S1 specifically includes: S11. Use PostGIS to perform spatiotemporal denoising on the original trajectory points collected by the elderly wristband, remove isolated points caused by positioning drift, and resample according to a uniform time step. S12. Call the PostGIS spatial clustering function ST_ClusterDBSCAN and set the search radius parameter. and minimum number of points parameter The system identifies spatial clusters in the trajectory sequence and calculates the centroid of each cluster as a candidate stopping point. S13. Using the centroid of the stopping point identified in S12 as the center, construct a polygonal buffer with radius R using the ST_Buffer function, perform spatial overlay analysis with the POI data in the community base map, and automatically assign semantic labels to the locations in the buffer. S14. Calculate the total historical dwell time of elderly people in each semantic functional area using the ST_Intersects operator. and access frequency Calculate the dwell time weight for each node. A dataset of lifestyle profiles based on preferences.

[0011] Optionally, step S2 specifically includes: S21. Define the semantic functional region determined in S13 as the hidden state space of the HMM. The real-time trajectory coordinate point sequence is defined as the observation space. ; S22. Using the Baum-Welch algorithm, perform unsupervised learning based on historical trajectories to calculate their forward / backward probabilities and expectations, and train the state transition probability matrix. We obtain the prior probability that the elderly person will move from the current position to the next semantic node; S23. Decode the current observation sequence using the Viterbi algorithm and output the next time-instance behavioral intent label with the highest confidence.

[0012] Optionally, the specific steps of step S3 include: S31. Standardize the real-time GPS sequence collected by the wristband and integrate real-time physiological features to construct a multi-dimensional input vector. ; S32. Map the behavioral intent label output by S23 into a feature vector, and inject it into the hidden layer state of LSTM through an attention mechanism to guide the prediction trajectory toward high-probability semantic nodes. S33. Predicting the future using a trained LSTM model. spatial coordinate sequence of time .

[0013] Optionally, the specific steps of step S4 include: S41. When an elderly person enters the community's surveillance coverage area, the system automatically calls up the video stream and extracts key points of the human skeleton to represent the human posture. S42. Use CNN to build three parallel classification branches: a walking posture analysis branch, a gesture recognition branch, and a standing / falling posture analysis branch. The walking posture analysis branch is used to output the probability of the walking posture model. The gesture recognition branch is used to output the probability of the gesture model. The standing / falling posture analysis branch is used to output the probability of the combat posture model. ; S43. Using the weighted fusion formula A comprehensive behavioral assessment value is calculated to determine whether the current behavior contains risk characteristics. Output probabilities for the walking posture model Corresponding weighting factors Output probability for gesture model Corresponding weighting factors Output probabilities for the standing posture model The corresponding weighting factor.

[0014] Optionally, the specific steps of step S5 include: S51. Calculate the logical residual between the predicted trajectory output by S3 and the real behavioral features identified by S4; S52. If the trajectory points to a dangerous area and S4 identifies features such as "body tilting" or "large hand waving", a high-risk warning will be triggered. S53. If the real-time observation deviates from the high-probability path predicted by the HMM, the system will automatically trigger the AI ​​voice interaction module for verification.

[0015] Optionally, step S6 may include the following specific steps: S61. After the warning is triggered, use Dijkstra's algorithm or A... The algorithm searches the community road network for the nearest volunteer who has first aid skills; S62. Real-time monitoring of volunteer attendance and recording of service trajectories are achieved through an electronic fence mechanism, enabling digital and automated service record keeping.

[0016] This invention, through the coupling of the HMM intent layer and the LSTM coordinate layer, enables the system to "understand what the elderly want to do," improving prediction accuracy by more than 20% compared to a purely data-driven model. This application introduces CNN visual features as "hard constraints" to compensate for the shortcomings of sensor data in judging sudden behaviors such as falls and waving for help. This application combines GIS path algorithms and electronic fences to realize full-process digital management from early warning to rescue, ensuring the timeliness and reliability of services. Attached Figure Description

[0017] Figure 1 This is a flowchart of the community elderly trajectory prediction and early warning method based on HMM-LSTM and CNN multi-source data fusion provided by the present invention; Figure 2 This is a flowchart of the correction and backtracking mechanism in this invention; Figure 3 This is a diagram of the coupling architecture of HMM-LSTM and CNN multi-source data fusion in this invention; Figure 4 This is a flowchart of semantic extraction based on DBSCAN clustering and buffer analysis in this invention. Detailed Implementation

[0018] This invention provides a method for predicting and warning the trajectory of elderly people in a community based on multi-source data fusion of HMM-LSTM and CNN, including: S1, using a PostGIS spatial database to associate community maps and the elderly's historical trajectories to generate personalized life profiles with tags; S2, based on the personalized life profiles generated in step S1, using HMM to train a state transition probability matrix to predict the probability of the elderly's behavioral intentions in the next stage; S3, using LSTM to process real-time GPS sequences and physiological feature data, outputting the predicted spatial coordinates at the future time T; S4, capturing images of the elderly through a community monitoring system, using CNN to build analysis models for walking posture, gestures, and standing posture, and outputting comprehensive behavioral tags. S5. Align the visual labels from step S4 with the predicted spatial coordinates from step S3. If a logical conflict occurs between the two or an abnormal electronic fence is triggered, the system will trigger a tiered warning after assessing the abnormal behavior. S6. Match the nearest volunteer based on the GIS shortest path algorithm and automatically generate service records through the electronic fence. The main purpose of this invention is to solve the technical problems existing in current community elderly care monitoring, such as "reporting only after an incident," limited positioning accuracy in complex environments, and lack of understanding of behavioral logic. Existing monitoring solutions (such as GPS wristbands or ordinary video surveillance) cannot provide logical advance warnings for abnormal behaviors of the elderly (such as leaving home late at night or staying by the river for a long time); and single sensor data is prone to offset in indoor or high-rise building obstruction areas, resulting in a high false alarm rate in risk assessment.

[0019] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0020] Example Reference manual attached Figure 1-4 This embodiment provides a method for predicting and warning the trajectory of elderly people in the community based on the fusion of HMM-LSTM and CNN multi-source data. The method includes: S1: Use the PostGIS spatial database to construct a personalized "life profile" of the elderly.

[0021] S2: Construct an HMM based on the PostGIS spatial database from the previous step, identify behavioral intentions, and generate preliminary trajectory predictions.

[0022] S3: Use LSTM to perform time-series modeling of physiological and spatial data collected by the wristband to generate high-precision predicted trajectories.

[0023] S4: The system calls the community monitoring system, uses CNN to extract the elderly’s gait, gestures and posture features, and calculates the comprehensive behavioral assessment value.

[0024] S5: Visual recognition results are injected into the trajectory model as a correction factor to achieve accurate early warning of abnormal behavior.

[0025] S6: Based on the above data, intelligent dispatching and rescue are achieved by combining the GIS shortest path algorithm.

[0026] The process of step S1 includes the following steps: Collect basic spatial data of the target area, including the geographic coordinates and specific markings of daily activity areas such as residential buildings, parks, and hospitals, as well as dangerous areas such as rivers, steep slopes, and underground garage entrances and exits.

[0027] Through a specific smart bracelet worn by the elderly, historical trajectory and three-axis data are collected. Data such as acceleration and physiological data such as heart rate and blood pressure are collected, and historical access records are also collected through the access control system of the residential area.

[0028] The collected data (including but not limited to: geographic coordinates, specific annotations, historical trajectories, and three axes) will be collected. Acceleration data, physiological data, and historical travel records are stored, indexed, and managed using the PostGIS spatial database, thus constructing a complete spatial database.

[0029] The collected data is preprocessed, through 3 The criteria remove outliers from trajectory and physiological data, and perform timestamp alignment and normalization on all data to ensure the consistency of the spatiotemporal dimensions of the data.

[0030] The behavioral intent label is mapped to a feature vector, and then injected into the hidden layer state of the LSTM through an attention mechanism to guide the prediction trajectory toward high-probability semantic nodes. Predicting the future using a trained LSTM model spatial coordinate sequence of time .

[0031] The process of step S2 includes the following steps: For each elderly person, a personalized life profile is created, and a suitable Hidden Model (HMM) is constructed. The elderly person's current spatial location, time of day, and physiological state are used as the observation sequence of the HMM, while the elderly person's behavioral intentions are used as the hidden states of the HMM.

[0032] Based on the historical behavior sequences of the elderly, the Baum-Welch algorithm was used to train the Hidden Markov Model (HMM). The initial state probability vector, state transition probability matrix, and observation probability matrix of the model were solved and optimized. The accuracy of the model's behavior intention recognition was verified through the test set to ensure that the recognition accuracy reached more than 90%.

[0033] The system collects the elderly person's current location, time period, and heart rate data in real time, generates the current observation sequence, inputs it into the trained HMM, and decodes it using the Viterbi algorithm to obtain the most likely hidden state of the current behavior intention. Combined with the historical spatiotemporal transition pattern corresponding to the behavior intention, it generates a preliminary trajectory prediction result for the next 30 minutes, clarifying the elderly person's behavior logic and activity range.

[0034] Step S3 includes the following process during implementation: Using a 5-minute sliding window, the time-series data collected by the smart bracelet was sliced ​​to construct a multivariate time-series dataset. Input features included timestamps, latitude and longitude coordinates, movement speed, heart rate, and triaxial acceleration. Labels were the trajectory coordinates data 10 minutes and 30 minutes after the sliding window ended. The dataset was divided into training and test sets in an 8:2 ratio for data preprocessing.

[0035] A multi-input LSTM model was constructed, with the following structure: input layer, first LSTM layer (128 neurons), second LSTM layer (64 neurons), dropout layer (dropout rate 0.2), fully connected layer, and output layer.

[0036] MSE was used as the loss function, Adam as the optimizer, and the learning rate was set to 0.001. The training set was input into the model to complete iterative training, and the trajectory prediction accuracy of the model was verified by the test set to ensure that the average position error of the trajectory prediction was less than 5 meters.

[0037] The 5-minute time-series data collected in real time is input into the trained LSTM model, which outputs a high-precision trajectory prediction sequence for the next 30 minutes. At the same time, based on the degree to which the heart rate and acceleration data deviate from the physiological baseline of the elderly, the probability of physiological abnormalities is calculated and output.

[0038] Step S4 includes the following process during implementation: Based on the elderly person's real-time location and predicted trajectory, the corresponding high-definition surveillance cameras in the residential area are accessed to obtain real-time video streams, and video frames are extracted at a frame rate of 2 frames per second.

[0039] A joint model based on CNN, YOLOv7 for object detection and OpenPose for pose estimation, was constructed. First, YOLOv7 was used to detect and continuously track the elderly person in the video frame. After locking the elderly person, 18 human key points of the elderly person were extracted using OpenPose.

[0040] Based on the extracted human body key point coordinates, gait features such as step frequency, stride length, and gait symmetry, as well as posture features such as body tilt angle and limb movements, are calculated and extracted to distinguish behavioral states such as standing, walking, falling, bending over, and prolonged stillness.

[0041] Based on the extracted visual features and combined with the elderly's activity level, a behavioral abnormality assessment system is constructed. Abnormal behaviors such as gait abnormalities, falls, and prolonged stillness are quantitatively scored, and a comprehensive behavioral assessment value is calculated. The higher the value, the higher the degree of behavioral abnormality. The probability of visual abnormality in the 0-1 range is output simultaneously.

[0042] Step S5 includes the following process during implementation: A weighted fusion framework for multi-source data is constructed, which unifies the behavioral intent probability output by HMM, the trajectory prediction sequence and physiological abnormality probability output by LSTM, and the comprehensive behavioral evaluation value and visual abnormality probability output by CNN to the same spatiotemporal dimension.

[0043] The comprehensive behavioral evaluation value output by CNN is used as a dynamic correction factor and injected into the HMM-LSTM joint trajectory prediction model to dynamically correct the coordinates and anomaly confidence of the predicted trajectory.

[0044] Construct multi-dimensional abnormal early warning rules, such as trajectory abnormality rules: leaving home between 22:00 and 6:00 the next day, entering dangerous areas such as rivers, deviating from the daily trajectory by more than a threshold, etc.; behavioral and posture abnormality rules: falling, severe gait abnormality, remaining still for more than 30 minutes without activity, etc.; physiological data abnormality rules: heart rate exceeding the baseline by more than 30%, blood pressure exceeding the safe threshold, etc.

[0045] Based on anomaly confidence and anomaly warning rules, a three-level anomaly classification is completed: Level 1 warning is for minor anomalies, such as slight deviations in trajectory, which are only recorded and continuously tracked in the background; Level 2 warning is for moderate anomalies, such as entering dangerous areas or abnormal gait, which pushes warning information to community grid workers; Level 3 warning is for severe anomalies, such as falls or critical physiological data, which are simultaneously pushed to community elderly care service centers, medical staff, and the elderly's families, triggering an emergency response.

[0046] Step S6 includes the following process during implementation: When a level 2 or higher warning is triggered, the real-time location and response status of on-duty security personnel and medical staff in the area can be obtained through the spatial query function of the PostGIS spatial database.

[0047] Using the elderly person's real-time location as the endpoint and the current location of a person who meets the rescue requirements as the starting point, and taking into account the road conditions, access control settings, and obstacle distribution within the community, the Dijkstra shortest path algorithm is used to plan the optimal rescue route that can be reached on foot.

[0048] The system completes intelligent emergency rescue dispatch, pushes dispatch information to the best-matched rescuers, and simultaneously includes the elderly person's real-time location, predicted trajectory, abnormality type, physiological data and on-site monitoring footage. It tracks the location and progress of rescuers in real time, and updates the warning status after the rescue is completed, thus completing the closed-loop management of the entire process from warning to rescue.

[0049] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for predicting and warning the trajectory of elderly people in the community based on multi-source data fusion of HMM-LSTM and CNN, characterized in that, include: S1. Use the PostGIS spatial database to link community maps and the historical trajectories of the elderly to generate personalized life profiles with tags; S2. Based on the personalized life profile generated in step S1, use HMM to train the state transition probability matrix to predict the probability of the elderly’s behavioral intentions in the next stage. S3. Use LSTM to process real-time GPS sequence and physiological feature data, and output the predicted spatial coordinates at the next time T. S4. Capture images of elderly people through the community monitoring system, use CNN to build analysis models of walking posture, gestures, and standing posture, and output comprehensive behavioral labels. ; S5. Align the visual label from step S4 with the predicted spatial coordinates from step S3. If there is a logical conflict between the two or an abnormal triggering of the electronic fence, the system will trigger a graded warning after evaluating the abnormal behavior. S6. Based on the GIS shortest path algorithm, the nearest volunteer is matched, and service certificates are automatically generated through electronic fences.

2. The method for predicting and warning the trajectory of elderly people in the community based on multi-source data fusion of HMM-LSTM and CNN as described in claim 1, characterized in that, Step S1 specifically includes: S11. Use PostGIS to perform spatiotemporal denoising on the original trajectory points collected by the elderly wristband, remove isolated points caused by positioning drift, and resample according to a uniform time step. S12. Call the PostGIS spatial clustering function ST_ClusterDBSCAN and set the search radius parameter. and minimum number of points parameter The system identifies spatial clusters in the trajectory sequence and calculates the centroid of each cluster as a candidate stopping point. S13. Using the centroid of the stopping point identified in S12 as the center, construct a polygonal buffer with radius R using the ST_Buffer function, perform spatial overlay analysis with the POI data in the community base map, and automatically assign semantic labels to the locations in the buffer. S14. Calculate the total historical dwell time of elderly people in each semantic functional area using the ST_Intersects operator. and access frequency Calculate the dwell time weight for each node. A dataset of lifestyle profiles based on preferences.

3. The method for predicting and warning the trajectory of elderly people in the community based on multi-source data fusion of HMM-LSTM and CNN according to claim 2, wherein step S2 specifically includes: S21. Define the semantic functional region determined in S13 as the hidden state space of the HMM. The real-time trajectory coordinate point sequence is defined as the observation space. ; S22. Using the generalized forward-backward (Baum-Welch) algorithm, perform unsupervised learning based on historical trajectories to calculate their forward / backward probabilities and expectations, and train the state transition probability matrix. We obtain the prior probability that the elderly person will move from the current position to the next semantic node; S23. Decode the current observation sequence using the Viterbi algorithm and output the next time-instance behavioral intent label with the highest confidence.

4. The method for predicting and warning the trajectory of elderly people in the community based on multi-source data fusion of HMM-LSTM and CNN as described in claim 3, is characterized in that, The specific steps of step S3 include: S31. Standardize the real-time GPS sequence collected by the wristband and integrate real-time physiological features to construct a multi-dimensional input vector. ; S32. Map the behavioral intent label output by S23 into a feature vector, and inject it into the hidden layer state of LSTM through an attention mechanism to guide the prediction trajectory toward high-probability semantic nodes. S33. Predicting the future using a trained LSTM model. spatial coordinate sequence of time .

5. The method for predicting and warning the trajectory of elderly people in the community based on multi-source data fusion of HMM-LSTM and CNN as described in claim 4, characterized in that, The specific steps of step S4 include: S41. When an elderly person enters the community's surveillance coverage area, the system automatically calls up the video stream and extracts key points of the human skeleton to represent the human posture. S42. Use CNN to build three parallel classification branches: a walking posture analysis branch, a gesture recognition branch, and a standing / falling posture analysis branch. The walking posture analysis branch is used to output the probability of the walking posture model. The gesture recognition branch is used to output the probability of the gesture model. The standing / falling posture analysis branch is used to output the probability of the combat posture model. ; S43. Using the weighted fusion formula A comprehensive behavioral assessment value is calculated to determine whether the current behavior contains risk characteristics. Output probabilities for the walking posture model Corresponding weighting factors Output probability for gesture model Corresponding weighting factors Output probabilities for the standing posture model The corresponding weighting factor.

6. The method for predicting and warning the trajectory of elderly people in the community based on multi-source data fusion of HMM-LSTM and CNN as described in claim 5, is characterized in that, The specific steps of step S5 include: S51. Calculate the logical residual between the predicted trajectory output by S3 and the real behavioral features identified by S4; S52. If the trajectory points to a dangerous area and S4 identifies features such as "body tilting" or "large hand waving", a high-risk warning will be triggered. S53. If the real-time observation deviates from the high-probability path predicted by the HMM, the system will automatically trigger the AI ​​voice interaction module for verification.

7. The method for predicting and warning the trajectory of elderly people in the community based on multi-source data fusion of HMM-LSTM and CNN as described in claim 6, is characterized in that, The specific steps of step S6 include: S61. After the warning is triggered, use Dijkstra's algorithm or A... The algorithm searches the community road network for the nearest volunteer who has first aid skills; S62. Real-time monitoring of volunteer attendance and recording of service trajectories are achieved through an electronic fence mechanism, enabling digital and automated service record keeping.

Images