Cognitive disorder old people directional force training and monitoring system based on multi-modal interaction

Abstract

The invention discloses a directional force training and monitoring system for old people with cognitive impairment based on multi-modal interaction, and belongs to the technical field of cognitive impairment of old people. The invention discloses a directional force training and monitoring system for cognitive impairment old people based on multi-modal interaction. The directional force training and monitoring system comprises a data acquisition module, a training and monitoring cooperation module, a multi-modal interaction module, a cognitive portrait construction module, a training scheme generation module and an emergency response module. The problems that in the prior art, directional force training is not high in pertinence, and linkage monitoring and early warning of directional force abnormity and the body state cannot be achieved are solved. According to the method, the accuracy of directional force training is remarkably improved, the core directional ability such as spatial positioning of old people is effectively enhanced, the incidence relation between directional force abnormity and body state abnormity is accurately recognized through multi-source data fusion, linkage early warning is achieved, the monitoring safety is greatly improved, accurate targeted intervention is achieved, and the method is suitable for popularization and application. And the cooperative efficiency of multi-modal interaction and the directional force anomaly recognition precision are improved.

Description

Technical Field

[0001] This invention relates to the field of cognitive impairment technology for the elderly, specifically to a multimodal interaction-based orientation training and monitoring system for elderly people with cognitive impairment. Background Technology

[0002] With the acceleration of population aging, the number of elderly people with cognitive impairment is increasing year by year. Disorientation is one of the typical symptoms of elderly people with cognitive impairment, mainly manifested as blurred spatial positioning, difficulty in scene recognition, and loss of path memory, which seriously affects their ability to take care of themselves in daily life, and at the same time greatly increases the safety risks such as getting lost and falling, bringing a heavy care burden to families and society.

[0003] Chinese patent CN119181482A discloses a virtual reality-based cognitive health monitoring system for the elderly. Through specialized working memory training tasks, the cognitive neurofeedback module can help the elderly improve their working memory ability. Targeting the common problem of attention deficit in the elderly, by improving working memory and basic cognitive functions of attention, the elderly will be more adept at performing daily activities, thereby improving their quality of life. The cognitive neurofeedback module not only focuses on cognitive function, but also helps the elderly better manage their emotions by regulating emotional networks, promoting social interaction, and reducing loneliness and anxiety.

[0004] The aforementioned patents, in practical use, are limited to visual and simple operational interactions within virtual reality scenarios, lacking specificity for orientation training and failing to accurately enhance the spatial positioning and other orientation abilities of the elderly. Furthermore, their monitoring functions focus on preliminary processing of health data and cannot achieve linked monitoring and early warning of orientation abnormalities and physical conditions. Therefore, they do not meet existing needs. To address this, we propose an orientation training and monitoring system for elderly people with cognitive impairment based on multimodal interaction. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal interaction-based orientation training and monitoring system for elderly people with cognitive impairment. This system enhances spatial cognition through lightweight AR overlay with orientation markers, fosters multisensory collaborative memory through differentiated tactile vibrations, and dynamically adapts interactive parameters based on environmental perception, significantly improving the accuracy of orientation training and effectively strengthening core orientation abilities such as spatial positioning in the elderly. During training, orientation-related behavioral and physiological data are collected simultaneously. Multi-source data fusion accurately identifies the correlation between orientation abnormalities and abnormal physical conditions, enabling linked early warning and significantly improving monitoring safety. Through initial assessment and real-time data updates, weak points in orientation, such as spatial positioning and path memory, are identified. Personalized training plans are generated based on profiles, and training parameters are dynamically adjusted using reinforcement learning to achieve precise targeted intervention. This improves the collaborative efficiency of multimodal interaction and the accuracy of orientation abnormality identification, thus solving the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multimodal interaction-based orientation training and monitoring system for elderly people with cognitive impairment, comprising: The data acquisition module is used to collect training data, location data, physiological data, behavioral data and environmental data of elderly people with cognitive impairment in real time. After preprocessing, the data is accurately transmitted to the training monitoring and collaboration module and the cognitive profile construction module. The training monitoring and coordination module is used to accurately distinguish between normal training and risk states based on the collected data. When the risk is low, it triggers the training adjustment strategy and when the risk is high, it activates the emergency response strategy to achieve organic coordination between training and monitoring. The multimodal interaction module is used to accurately match all forms of interaction with the core training requirements of spatial positioning and scene recognition orientation. Through lightweight AR overlay of orientation markers, multi-dialect voice guidance, differentiated tactile vibration and environmental adaptive adjustment, it can achieve precise training guidance and real-time risk intervention for elderly people with cognitive impairment. The cognitive profile building module is used to combine initial neuropsychological scale assessments with real-time perception data to determine the degree of impairment in each core dimension, dynamically update training feedback and behavioral habit data, and build a personalized cognitive profile. The training scheme generation module is used to generate personalized initial training schemes based on individual cognitive profiles, and dynamically adjust parameters through reinforcement learning algorithms by combining real-time training feedback data.

[0007] The emergency response module is used to immediately activate the emergency response mechanism when a high-risk abnormal state is identified, push alarm information and accurate data to the family's APP, and coordinate with community medical institutions.

[0008] Preferably, the data acquisition module specifically includes: Real-time location data of elderly people with cognitive impairment is collected, enabling seamless indoor and outdoor positioning, and wearable devices are used to collect physiological data of elderly people with cognitive impairment in real time. Real-time collection of behavioral data of elderly people with cognitive impairment, as well as light intensity, noise level, and obstacle information in the training environment; The collected real-time location data, physiological data, behavioral data, and environmental data are transmitted to the training and monitoring collaboration module, and the environmental data provides environmental auxiliary information for risk status identification.

[0009] Preferably, the training monitoring and collaboration module specifically includes: The system receives location data, physiological data, behavioral data, and environmental data transmitted from the data acquisition module. It then uses a multi-source data fusion algorithm to preprocess and fuse the data, removing redundant data and abnormal noise to generate standardized fused data. Based on the generated fused data, a deep learning algorithm is used to distinguish between normal training pauses and abnormal risk states. The temporal and spatial data in the fused data are correlated and modeled, and the weight of key risk features is strengthened through an attention mechanism; When a low-risk abnormal state is identified, the training process is not interrupted. The corresponding guidance strategy is triggered based on the multimodal interaction module, and the difficulty of subsequent training is adjusted synchronously. Upon identifying a high-risk abnormal state, an emergency trigger signal is immediately sent to the emergency response module, and the training mode is simultaneously terminated. The current training data and risk state data are then uploaded to the emergency response module.

[0010] Preferably, when a low-risk abnormal state is identified, the training process is not interrupted; instead, a corresponding guidance strategy is triggered based on the multimodal interaction module, and the subsequent training difficulty is adjusted synchronously. Specifically, this includes: When a deviation from the training route is detected, the voice interaction unit broadcasts a route correction prompt, while differentiated vibration guides the user back to the correct path, and an enhanced path guidance sign is superimposed. When the training task is detected to be progressing slowly, the difficulty of the current training task is reduced. When mild gait instability is detected, the current training progress is paused, and the elderly person is prompted to slow down through a combination of voice and tactile feedback. The smoothness requirements of the subsequent training path are also adjusted.

[0011] Preferably, the state recognition process specifically includes: Receive location data, physiological data, behavioral data, and environmental data, and perform data preprocessing; A deep learning-based multi-source data fusion algorithm is used to extract feature vectors from each data source, thereby achieving the fusion of multi-dimensional data. The fused feature data is input into the Transformer model, and the model outputs state recognition results, including normal training pauses, low-risk anomalies, and high-risk anomalies. The status recognition results are verified. If there is ambiguity in the recognition, multiple frames of data are collected for secondary recognition.

[0012] Preferably, the deep learning algorithm uses the Transformer model, and the specific implementation steps are as follows: The fused data is classified and extracted to separate time-series data and spatial data. The encoder structure of the Transformer model is used to model the association between the two types of data and establish the mapping relationship between time-series and spatial data.

[0013] In the attention mechanism layer of the Transformer model, risk feature weight coefficients are introduced to strengthen the weight of feature data corresponding to key risks such as falls and sudden changes in heart rate, thereby increasing the model's attention to high-value risk features and reducing the interference of irrelevant data on the recognition results.

[0014] Collect real training and monitoring data of elderly people with cognitive impairment, and manually label them according to normal state, low-risk abnormal state and high-risk abnormal state; Construct a diverse dataset that includes different scenarios and different groups of people, and divide it into training set, validation set and test set; The Transformer model is trained based on the constructed dataset, the hyperparameters of the Transformer model are adjusted in real time using the validation set, and the performance of the Transformer model is verified using the test set.

[0015] Preferably, the cognitive profile construction module specifically includes: When elderly people with cognitive impairment use the system for the first time, an initial assessment of the core dimensions of orientation is completed by combining neuropsychological scales and multimodal perception data. It receives training feedback data and behavioral habit data transmitted from the data acquisition module in real time, and dynamically updates the basic data of the cognitive profile. Based on the initial assessment results, training feedback data, and behavioral habit data, an individual cognitive profile is constructed.

[0016] Preferably, the training scheme generation module specifically includes: Based on the individual cognitive profile generated by the cognitive profile construction module, a personalized initial training plan is generated. Combining real-time training feedback data collected by the data acquisition module, the reinforcement learning algorithm is used with the dual objective functions of maximizing training effect and minimizing fatigue in the elderly. The training feedback data is used as the state input, the training parameter adjustment is used as the action output, and the training strategy is optimized through the reward function. When an elderly person completes a training task of the current difficulty level with high quality three times in a row, the training difficulty will be automatically increased. If there are too many errors or an abnormal increase in heart rate, the training difficulty will be reduced and the rest interval will be increased.

[0017] Preferably, the training scheme generation module includes: Based on the results of each training task, the standardized completion time, path deviation error integral, and heart rate stability of elderly people with cognitive impairment during the task are extracted from the training feedback data. The comprehensive completion index for a single task is calculated based on the standardized completion time of each task, the integral of the path deviation error, and the heart rate stability of elderly people with cognitive impairment during the task. The obtained comprehensive completion index is compared with the preset quality completion threshold. When the comprehensive completion index is greater than or equal to the preset quality completion threshold, the current task is determined to be validly completed, and a valid count is recorded based on the determination result. When the effective count reaches 3 consecutive times, the training difficulty increase mechanism is triggered, and a new training difficulty level is calculated based on the trigger result, according to the difficulty level of the current task and the average of the comprehensive completion index of the three consecutive tasks. Based on the new training difficulty level, the key parameters for the next training cycle are adjusted according to the preset mapping rules. Meanwhile, in the first training cycle after the training difficulty level is updated, if the calculated comprehensive completion index is lower than the preset downgrade threshold, the training difficulty level will be immediately restored to the original training difficulty level, and the effective count will be cleared to zero.

[0018] Preferably, the training scheme generation module includes: Acquire location data, physiological data, behavioral data and environmental data. At the same time, acquire physiological time-series data and spatial time-series data of elderly people with cognitive impairment, and input them into a pre-trained temporal prediction neural network. The pre-trained temporal prediction neural network processes localization data, physiological data, behavioral data, and environmental data to obtain the attention state vector and motion intention vector at each time point. Based on graph convolutional networks, key objects and path nodes are extracted from spatial temporal data. Key objects and path nodes are used as vertices and spatial adjacency relationships are used as edges to obtain the spatial saliency embedding vector of each vertex at the current time. Meanwhile, the instantaneous heart rate variability of elderly people with cognitive impairment was extracted based on physiological data, and the instantaneous heart rate variability was used as low-dimensional physiological time series data and concatenated with the obtained motion intention vector to obtain an enhanced physiological state vector. The enhanced physiological state vector and the spatial saliency embedding vector are temporally aligned to obtain a feature pair sequence. The feature pair sequence is then analyzed based on a multi-head attention fusion layer to obtain the cross-modal correlation weight between the enhanced physiological state vector and the spatial saliency embedding vector. A joint feature representation is then generated based on the cross-modal correlation weight. The pre-trained temporal prediction neural network performs predictive analysis on the joint feature representation to obtain the feature evolution trajectory at each prediction time point after a preset time period. Based on the feature evolution trajectory, the probability of path decision hesitation, spatial cognitive confusion, and gait instability are determined. At the same time, the offset vector of the user's current position is determined based on the feature evolution trajectory. The offset vector is added to the estimated user position coordinates corresponding to the current prediction time point obtained from the digital twin model of the location to obtain the predicted spatial position coordinates. When the predicted probability value of any risk event exceeds the dynamic adaptive threshold, a pre-intervention mechanism is triggered. Based on the triggering result, augmented reality semantic information of the surrounding area is extracted from the pre-built digital twin model of the location based on the predicted spatial location coordinates. Multi-sensory prompts are generated based on the augmented reality semantic information. Based on multi-sensory prompts, preset interactive devices are driven to provide collaborative guidance. After collaborative guidance, the actual behavior data of users within the predicted time window is continuously monitored and compared with the predicted behavior trajectory to generate intervention effectiveness evaluation parameters. The weights and dynamic adaptive thresholds of the pre-trained temporal prediction neural network are dynamically updated based on the intervention effectiveness evaluation parameters.

[0019] Compared with the prior art, the beneficial effects of the present invention are: This invention enhances spatial cognition by overlaying lightweight AR orientation markers, fosters multi-sensory collaborative memory through differentiated tactile vibrations, and dynamically adapts interactive parameters based on environmental perception, significantly improving the accuracy of orientation training and effectively strengthening core orientation abilities such as spatial positioning in the elderly. During training, orientation-related behavioral and physiological data are collected simultaneously. Multi-source data fusion accurately identifies the correlation between orientation abnormalities and abnormal physical conditions, enabling linked early warning and significantly improving monitoring safety. Through initial assessment and real-time data updates, weaknesses in orientation, such as spatial positioning and path memory, are identified. Personalized training plans are generated based on profiles, and training parameters are dynamically adjusted using reinforcement learning to achieve precise targeted intervention, improving the collaborative efficiency of multimodal interaction and the accuracy of orientation abnormality identification. Attached Figure Description

[0020] Figure 1 This is a diagram of the orientation training and monitoring system for cognitively impaired elderly people based on multimodal interaction, as described in this invention. Figure 2 This is a flowchart of the orientation training and monitoring system for cognitively impaired elderly people based on multimodal interaction according to the present invention; Figure 3 This is a flowchart of the training monitoring and collaboration module of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] To address the shortcomings of existing technologies in providing targeted orientation training, which fails to accurately enhance spatial positioning and other orientational abilities in the elderly, and cannot achieve linked monitoring and early warning of orientation abnormalities and physical condition, please refer to [link to relevant documentation]. Figures 1-3 This embodiment provides the following technical solution: A multimodal interaction-based orientation training and monitoring system for cognitively impaired elderly includes: The data acquisition module is used to collect training data, location data, physiological data, behavioral data, and environmental data of elderly people with cognitive impairment in real time. After preprocessing, the data is accurately transmitted to the training monitoring and collaboration module and the cognitive profile construction module, providing comprehensive data support for the realization of the system's core functions. The training monitoring and coordination module is used to accurately distinguish between normal training and risk states based on the collected data. When the risk is low, it triggers the training adjustment strategy and when the risk is high, it activates the emergency response strategy to achieve organic coordination between training and monitoring. The multimodal interaction module is used to accurately match all forms of interaction with the core training requirements of spatial positioning and scene recognition orientation. Through lightweight AR overlay of orientation markers, multi-dialect voice guidance, differentiated tactile vibration and environmental adaptive adjustment, it can achieve precise training guidance and real-time risk intervention for elderly people with cognitive impairment. The cognitive profile building module is used to combine initial neuropsychological scale assessments with real-time perception data to determine the degree of impairment in each core dimension of orientation, such as spatial localization and path memory, dynamically update training feedback and behavioral habit data, and build personalized cognitive profiles to provide a basis for precision training. The training plan generation module is used to generate personalized initial training plans based on individual cognitive profiles. Combined with real-time training feedback data, it dynamically adjusts parameters through reinforcement learning algorithms, incorporates interest elements to design scenarios, and achieves accurate adaptation and continuous optimization of training plans, thereby improving training effectiveness and compliance.

[0023] The emergency response module is used to immediately activate the emergency response mechanism when a high-risk abnormal state is identified. It enables local sound and light alarms through wearable devices, pushes alarm information and accurate data to the family's APP, and coordinates with community medical institutions to ensure the safety of the elderly and retain emergency data.

[0024] The data acquisition module specifically includes: Employing dual-mode positioning technology of UWB and BeiDou, the system collects real-time location data from elderly individuals with cognitive impairment, enabling seamless switching between indoor and outdoor positioning. Indoors, UWB positioning is used, communicating with the UWB tag worn by the elderly via an indoor UWB base station to achieve centimeter-level positioning. Outdoors, it automatically switches to BeiDou positioning technology for positioning in open areas. The positioning data acquisition unit transmits the collected real-time location data to the training and monitoring collaboration module for training path deviation judgment and lost-risk identification. Wearable devices are used to collect real-time physiological data of elderly people with cognitive impairment, including heart rate, blood oxygen saturation, and body temperature. The physiological data acquisition unit uses medical-grade sensors to ensure the accuracy of the collected data. The physiological data is used for risk identification of sudden illnesses and for determining the appropriate training intensity. Real-time collection of behavioral data from elderly individuals with cognitive impairment, as well as light intensity, noise levels, and obstacle information in the training environment. Behavioral data includes gait data, dwell time, and completion status of training actions. Gait data includes cadence, stride length, and gait stability parameters for fall risk identification. Dwell time refers to the duration of time an elderly person stays in a specific area on the training path, used to distinguish between normal training pauses and disorientation. Completion status of training actions includes the completion time of training tasks and the number of errors, used for training feedback data statistics. The collected real-time location data, physiological data, behavioral data, and environmental data are transmitted to the training and monitoring collaboration module, and the environmental data provides environmental auxiliary information for risk status identification.

[0025] The training monitoring and collaboration module specifically includes: The system receives location data, physiological data, behavioral data, and environmental data transmitted from the data acquisition module. It then uses a multi-source data fusion algorithm to preprocess and fuse the data, removing redundant data and abnormal noise to generate standardized fused data. Based on the generated fusion data, a deep learning algorithm is used to distinguish between normal training pauses and abnormal risk states. Abnormal risk states are divided into low-risk and high-risk states. Low-risk states include deviation from the training route, slow completion of training tasks, and mild gait instability. High-risk states include falls, sudden changes in heart rate, blood oxygen saturation below 90%, prolonged stay or getting lost. The model associates temporal and spatial data in the fused data and strengthens the weight of key risk features through an attention mechanism. During model training, real training and monitoring data of elderly people with cognitive impairment are used for annotation to construct a dataset that includes normal state, low-risk abnormal state, and high-risk abnormal state. Transfer learning is used to improve the model's recognition accuracy and generalization ability. When a low-risk abnormal state is identified, the training process is not interrupted. The corresponding guidance strategy is triggered based on the multimodal interaction module, and the difficulty of subsequent training is adjusted synchronously. Upon identifying a high-risk abnormal state, an emergency trigger signal is immediately sent to the emergency response module, and the training mode is simultaneously terminated. The current training data and risk state data are then uploaded to the emergency response module.

[0026] When a low-risk abnormal state is identified, the training process is not interrupted. Instead, a corresponding guidance strategy is triggered based on the multimodal interaction module, and the difficulty of subsequent training is adjusted synchronously. Specifically, this includes: When a deviation from the training route is detected, the voice interaction unit broadcasts a route correction prompt, while differentiated vibration guides the user back to the correct path, and an enhanced path guidance sign is superimposed. When the training task is detected to be progressing slowly, the difficulty of the current training task is reduced (e.g., the path length is shortened, the number of markers is increased); when mild gait instability is detected, the current training progress is paused, and the elderly person is prompted to slow down through a combination of voice and tactile feedback, and the smoothness requirements of the subsequent training path are adjusted.

[0027] The status recognition process specifically includes: Receive location data, physiological data, behavioral data, and environmental data, and perform data preprocessing, including data cleaning and data standardization; A deep learning-based multi-source data fusion algorithm is used to extract feature vectors from each data source, thereby achieving the fusion of multi-dimensional data. The fused feature data is input into the Transformer model, and the model outputs state recognition results, including normal training pauses, low-risk anomalies, and high-risk anomalies. The status recognition results are verified. If there is ambiguity in the recognition (such as the inability to distinguish between a normal pause and being lost), multiple frames of data are collected for secondary recognition to ensure recognition accuracy. The deep learning algorithm uses the Transformer model, and the specific implementation steps are as follows: The fused data is classified and extracted to separate time-series data (such as heart rate change curves, blood oxygen time-series data, and positioning trajectory time-series data) and spatial data (such as real-time location coordinates and spatial distribution data of environmental obstacles). The encoder structure of the Transformer model is used to model the association between the two types of data and establish the mapping relationship between time-series and spatial data.

[0028] In the attention mechanism layer of the Transformer model, risk feature weight coefficients are introduced to strengthen the weight of feature data corresponding to key risks such as falls and sudden changes in heart rate, thereby increasing the model's attention to high-value risk features and reducing the interference of irrelevant data on the recognition results.

[0029] We collected real training and monitoring data of elderly people with cognitive impairment, manually labeled them according to normal state, low-risk abnormal state and high-risk abnormal state, and constructed a diverse dataset containing different scenarios and different groups of people. At the same time, we divided the dataset into training set, validation set and test set with a ratio of 7:2:1. The Transformer model is trained based on the constructed dataset. During the training process, a transfer learning method is introduced to initialize the Transformer model using the parameters of the already trained general behavior recognition model, thereby shortening the training cycle. The hyperparameters of the Transformer model are adjusted in real time using the validation set, and the performance of the Transformer model is verified using the test set.

[0030] The cognitive profile building module specifically includes: When elderly people with cognitive impairment use the system for the first time, an initial assessment of the core dimensions of orientation is completed by combining neuropsychological scales and multimodal perception data. The neuropsychological scales include an orientation assessment scale, which is used to obtain the basic orientation assessment results of the elderly. The multimodal perception data includes localization data, behavioral data, and interactive feedback data from the initial test training, which are used to supplement the assessment of the degree of impairment of the elderly in core dimensions such as spatial localization, path memory, and scene recognition. The initial assessment process specifically includes: obtaining basic information about elderly individuals with cognitive impairment, including age, gender, type and degree of cognitive impairment, and past medical history; conducting a basic orientation assessment using neuropsychological scales, with assessors asking questions of the elderly and recording the assessment results; initiating initial testing training by pushing simple orientation test tasks through a multimodal interaction module; collecting positioning data, behavioral data, and interactive feedback data during the initial testing training; and integrating and analyzing the basic information, scale assessment results, and initial test data to determine the degree of impairment in each core dimension of orientation and generate an initial individual cognitive profile. The system receives training feedback data and behavioral habit data transmitted from the data acquisition module in real time, and dynamically updates the basic data of the cognitive profile. The training feedback data includes the completion time of training tasks, the number of errors, and the degree of dependence on voice / tactile interaction. The behavioral habit data includes the training preference time of the elderly, their acceptance of different training scenarios, and their preferences for daily activity trajectories. This is used to construct an individual cognitive profile based on initial evaluation results, training feedback data, and behavioral habit data. Among them, the orientation impairment dimension indicators include the degree of spatial localization impairment, the degree of path memory impairment, and the degree of scene recognition impairment; the training feedback indicators include training efficiency score and interaction dependency score; the behavioral habit indicators include training time preference and scene preference; the profile generation unit determines the weight of each indicator through the analytic hierarchy process and uses the fuzzy comprehensive evaluation method to generate the final individual cognitive profile.

[0031] The training scheme generation module specifically includes: Based on the individual cognitive profile generated by the cognitive profile construction module, a personalized initial training plan is generated. By combining real-time training feedback data collected by the data acquisition module, training parameters are dynamically adjusted using a reinforcement learning algorithm. These parameters include training difficulty (path length, number of turns, number of markers) and training pace (training duration, rest interval). The reinforcement learning algorithm takes maximizing training effectiveness and minimizing fatigue in the elderly as its dual objective functions. It uses training feedback data (completion time, number of errors, and heart rate changes) as state input and training parameter adjustment as action output. The training strategy is optimized through a reward function (higher reward value for high completion quality and lower reward value for fatigue signal). When an elderly person completes a training task of the current difficulty level with high quality three times in a row, the training difficulty will be automatically increased. When there are too many errors or an abnormally high heart rate, the training difficulty will be reduced and the rest interval will be increased. Personalized initial training programs; tailored training content to address different weaknesses in orientation impairment: for those with weak path memory, priority is given to familiar environment repetitive navigation training, with training content involving fixed path navigation in the elderly's daily living environment, gradually increasing path length and number of turns; for those with weak scene recognition, unfamiliar environment sign recognition training is conducted, with training content involving sign recognition and path planning in simulated public scenes such as supermarkets and parks; for those with weak spatial positioning, orientation judgment training is conducted, with training content involving direction recognition and location confirmation based on AR signs.

[0032] Working principle: When using the orientation training and monitoring system for cognitively impaired elderly based on multimodal interaction of this invention, according to... Figures 1-3 This includes the following steps: S1: The cognitive profile building module completes the initial orientation assessment of elderly people with cognitive impairment and generates an individual cognitive profile; S2: The training plan generation module generates a personalized initial training plan based on an individual's cognitive profile, and pushes training instructions to the elderly through the multimodal interaction module; S3: During training, the data acquisition module collects location data, physiological data, behavioral data and environmental data in real time and transmits them to the training monitoring and collaboration module and the cognitive profile construction module. S4: The training monitoring and collaboration module integrates and processes the collected data to distinguish between normal training status and abnormal risk status. S5: If the training state is identified as normal, the training plan generation module dynamically adjusts the training parameters and optimizes the training plan based on real-time training feedback data and reinforcement learning algorithm, and the cognitive profile building module updates the individual cognitive profile in sync. S6: If a low-risk abnormal state is identified, a guidance strategy is triggered through the multimodal interaction module to guide the elderly back to a normal training state and adjust the difficulty of subsequent training. S7: If a high-risk abnormal state is identified, the training mode will be terminated immediately and an emergency trigger signal will be sent to the emergency response module. The emergency response module will activate a three-level emergency response, including local audible and visual alarms, push notifications to family members' apps, and linkage with community medical institutions, and push precise location and real-time physiological data. S8: After the emergency response is completed, record the emergency response data. The training plan generation module further optimizes the training plan based on the emergency response data and the updated cognitive profile.

[0033] This embodiment provides a system for orientation training and monitoring of elderly people with cognitive impairment based on multimodal interaction. The training program generation module includes: Based on the results of each training task, the standardized completion time, path deviation error integral, and heart rate stability of elderly people with cognitive impairment during the task are extracted from the training feedback data. The comprehensive completion index for a single task is calculated based on the standardized completion time of each task, the integral of the path deviation error, and the heart rate stability of elderly people with cognitive impairment during the task. ;

[0034] in, This indicates the overall completion index of a single task; The preset baseline completion time indicates the training difficulty level of a single task; Indicates the standardized completion time of a single task; This represents the balance coefficient for configuring historical data based on spatial navigation accuracy in an individual's cognitive profile. This represents a balance coefficient based on historical data configuration of physiological regulation ability and endurance level in an individual's cognitive profile; This represents the integral of the path deviation error; This indicates the stability of heart rate in older adults with cognitive impairment during the task. The obtained comprehensive completion index is compared with the preset quality completion threshold. When the comprehensive completion index is greater than or equal to the preset quality completion threshold, the current task is determined to be validly completed, and a valid count is recorded based on the determination result. When the effective count reaches 3 consecutive times, the training difficulty increase mechanism is triggered, and a new training difficulty level is calculated based on the trigger result, according to the difficulty level of the current task and the average of the comprehensive completion index of the three consecutive tasks. ; in, This indicates a new training difficulty level; Indicates the original training difficulty level of the current task; This represents the average overall completion index of three consecutive tasks. This indicates the preset quality completion threshold; This represents the exponential increment required to increase the difficulty level by one unit. Indicates rounding down; Based on the new training difficulty level, the key parameters for the next training cycle are adjusted according to the preset mapping rules. Meanwhile, in the first training cycle after the training difficulty level is updated, if the calculated comprehensive completion index is lower than the preset downgrade threshold, the training difficulty level will be immediately restored to the original training difficulty level, and the effective count will be cleared to zero.

[0035] In this embodiment, the comprehensive completion index refers to a numerical indicator that quantifies the quality of a single training task completion, which is calculated by combining the task completion efficiency, path following accuracy, and physiological load stability.

[0036] In this embodiment, the preset baseline completion time refers to the standard time reference value that is pre-set for a specific training difficulty level, which is expected to take the user to complete the training task.

[0037] In this embodiment, the standardized completion time refers to the actual time spent by the user to complete the task in the actual training task, which has been standardized according to the specific conditions of the task.

[0038] In this embodiment, the balance coefficient refers to a parameter used to adjust the weights of path deviation error and heart rate stability in the calculation of the comprehensive completion index. Its value is personalized according to the individual's exercise capacity and cognitive function, specifically as follows: From individual cognitive profiles, historical average path deviation error integral and historical average heart rate stability are extracted as configuration benchmark values. The extracted historical average path deviation error integral and historical average heart rate are then normalized to obtain normalized benchmark values. and Design configuration function ,in, and As the preset global adjustment factor, respectively The output is mapped to a preset coefficient range to obtain the final balance coefficient. and The specific value is given by k, where k is the penalty coefficient applied to the integral of the path deviation error. Increasing its value will lead to a stronger negative impact of path deviation behavior on the overall completion index. The configuration is primarily linked to historical data reflecting spatial navigation accuracy within the individual's cognitive profile. It is a gain coefficient that affects heart rate stability; increasing its value will increase the positive contribution weight of physiological stability to the overall performance index. The configuration is mainly related to historical data in the individual's cognitive profile that reflects physiological regulation ability and endurance level.

[0039] In this embodiment, the path deviation error integral refers to the cumulative measure of the degree of deviation between the user's actual movement trajectory and the preset correct path during the execution of the training task.

[0040] In this embodiment, heart rate stability refers to a quantitative indicator of the magnitude of heart rate fluctuations during the training task, which is used to reflect the stability of the user's physiological load.

[0041] In this embodiment, the new training difficulty level refers to the task difficulty level that the system determines through calculation and will be implemented in the next training cycle.

[0042] In this embodiment, the original training difficulty level refers to the difficulty level corresponding to the current or recently completed training task.

[0043] In this embodiment, the mean comprehensive completion index refers to the arithmetic mean of the comprehensive completion indices corresponding to the three most recent training tasks that were judged as completed with quality.

[0044] In this embodiment, the preset quality completion threshold refers to the minimum comprehensive completion index score that the system pre-sets to determine whether a single training task has reached the "high-quality completion" standard.

[0045] In this embodiment, the exponential increment required for a unit increase in difficulty refers to the additional increment required for the average comprehensive completion index of the user to exceed the preset quality completion threshold for each level increase in training difficulty.

[0046] The working principle and beneficial effects of the above technical solution are as follows: By quantitatively evaluating the quality of each training task, an objective and continuous task completion index is formed, thereby dynamically adjusting the training difficulty. It can adapt to the changes in the abilities of the elderly in real time, automatically increasing the challenge when the performance is steadily improving, ensuring that the training is always at an effective difficulty level, thus effectively promoting the rehabilitation of orientation ability. At the same time, a safe reverse adjustment mechanism is introduced. When the performance declines significantly or physiological indicators are abnormal after the difficulty is increased, the difficulty can be quickly adjusted back and the count reset to prevent frustration or accidental risks caused by difficulty mismatch. It achieves accurate and automatic matching between training intensity and the user's actual cognitive-motor state, improving rehabilitation efficiency while ensuring the safety of the training process.

[0047] This embodiment provides a system for orientation training and monitoring of elderly people with cognitive impairment based on multimodal interaction. The training program generation module includes: Acquire location data, physiological data, behavioral data and environmental data. At the same time, acquire physiological time-series data and spatial time-series data of elderly people with cognitive impairment, and input them into a pre-trained temporal prediction neural network. The pre-trained temporal prediction neural network processes localization data, physiological data, behavioral data, and environmental data to obtain the attention state vector and motion intention vector at each time point. Based on graph convolutional networks, key objects and path nodes are extracted from spatial temporal data. Key objects and path nodes are used as vertices and spatial adjacency relationships are used as edges to obtain the spatial saliency embedding vector of each vertex at the current time. Meanwhile, the instantaneous heart rate variability of elderly people with cognitive impairment was extracted based on physiological data, and the instantaneous heart rate variability was used as low-dimensional physiological time series data and concatenated with the obtained motion intention vector to obtain an enhanced physiological state vector. The enhanced physiological state vector and the spatial saliency embedding vector are temporally aligned to obtain a feature pair sequence. The feature pair sequence is then analyzed based on a multi-head attention fusion layer to obtain the cross-modal correlation weight between the enhanced physiological state vector and the spatial saliency embedding vector. A joint feature representation is then generated based on the cross-modal correlation weight. The pre-trained temporal prediction neural network performs predictive analysis on the joint feature representation to obtain the feature evolution trajectory at each prediction time point after a preset time period. Based on the feature evolution trajectory, the probability of path decision hesitation, spatial cognitive confusion, and gait instability are determined. At the same time, the offset vector of the user's current position is determined based on the feature evolution trajectory. The offset vector is added to the estimated user position coordinates corresponding to the current prediction time point obtained from the digital twin model of the location to obtain the predicted spatial position coordinates. When the predicted probability value of any risk event exceeds the dynamic adaptive threshold, a pre-intervention mechanism is triggered. Based on the triggering result, augmented reality semantic information of the surrounding area is extracted from the pre-built digital twin model of the location based on the predicted spatial location coordinates. Multi-sensory prompts are generated based on the augmented reality semantic information. Based on multi-sensory prompts, preset interactive devices are driven to provide collaborative guidance. After collaborative guidance, the actual behavior data of users within the predicted time window is continuously monitored and compared with the predicted behavior trajectory to generate intervention effectiveness evaluation parameters. The weights and dynamic adaptive thresholds of the pre-trained temporal prediction neural network are dynamically updated based on the intervention effectiveness evaluation parameters.

[0048] In this embodiment, spatial temporal data refers to a data sequence that is continuously collected over time and reflects the spatial structure and characteristic information of the user's physical environment. It may include scene image streams captured by cameras or point cloud sequences scanned by lidar.

[0049] In this embodiment, the attention state vector refers to a high-dimensional mathematical vector generated by analyzing user behavior data through a neural network, which is used to quantitatively represent the distribution state of visual attention or cognitive focus at the current moment.

[0050] In this embodiment, the motion intention vector refers to a high-dimensional mathematical vector that represents the direction of movement or intention of action that the user is about to perform, inferred by a neural network through analysis of user behavior data (such as gait and limb orientation).

[0051] In this embodiment, the spatial saliency embedding vector refers to the mathematical vector generated for each entity after modeling objects and path nodes in the scene through a graph convolutional network. This vector represents the importance of the entity's current spatial positioning and navigation to the user.

[0052] In this embodiment, the enhanced physiological state vector refers to the fusion feature vector formed by combining the key physiological indicator of instantaneous heart rate variability, which reflects the state of autonomic nervous activity, with the motion intention vector, which represents behavioral intention, to more comprehensively depict the user's physical and mental readiness state.

[0053] In this embodiment, the feature pair sequence refers to a series of feature combinations that are paired at the same time point and represent the relationship between the user's internal state and external environment, formed by aligning the enhanced physiological state vector and the spatial saliency embedding vector according to strict timestamps.

[0054] In this embodiment, joint feature representation refers to a unified and dense mathematical representation generated by fusing feature pairs through an attention mechanism, which can simultaneously encode the complex interactive relationships between user physiological and behavioral intentions and environmental spatial cues.

[0055] In this embodiment, the feature evolution trajectory refers to the prediction sequence output by the joint feature representation after it is input into the prediction neural network, which predicts how the feature representation will change at a series of consecutive time points in the future, and is used to infer the user's future state.

[0056] In this embodiment, the predicted spatial location coordinates refer to the geographical location coordinates that the user is most likely to appear at a specific time in the future, which are calculated by combining the location offset decoded from the feature evolution trajectory and the basic positioning estimate.

[0057] In this embodiment, collaborative guidance refers to rendering the gradient outline of the key inflection point or landmark that is about to be reached in advance in the peripheral vision area of ​​the user's current field of vision with a transparency lower than that of explicit perception in the lightweight AR overlay. At the same time, background ambient sound or short poem fragments that are unrelated to the current training task but contain directional semantics are played to activate the spatial perception brain region in a non-task manner. The wearable device initiates a set of micro-tactile sequences that are sub-perceptual threshold and synchronized with the user's gait frequency harmonics to unconsciously adjust the user's gait rhythm and direction of travel.

[0058] In this embodiment, the intervention effectiveness evaluation parameter refers to one or a set of numerical indicators calculated after the implementation of pre-intervention by comparing the difference between the user's actual behavior and the predicted behavior trajectory, which is used to quantitatively measure the effectiveness of this intervention measure.

[0059] The working principle and beneficial effects of the above technical solution are as follows: By performing in-depth time-series modeling and fusion analysis on the user's real-time physiological, behavioral, and environmental multimodal data, it achieves accurate prediction of potential cognitive and behavioral risks in the short term. It can initiate non-invasive, multi-sensory collaborative pre-intervention guidance before the risk actually occurs, transforming the traditional passive response into proactive prevention, and significantly reducing the probability of accidents such as getting lost and falling. Secondly, by continuously comparing the prediction results with actual feedback, a self-evaluation and optimization closed loop is formed, enabling the prediction model and intervention strategy to dynamically adapt to changes in the user's individual behavioral patterns and abilities. Thus, in long-term application, it continuously improves the predictability, safety, and personalized adaptation level of monitoring and rehabilitation training.

[0060] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0061] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention.

Claims

1. A multimodal interaction-based orientation training and monitoring system for elderly people with cognitive impairment, characterized in that, include: The data acquisition module is used to collect training data, location data, physiological data, behavioral data and environmental data of elderly people with cognitive impairment in real time. After preprocessing, the data is accurately transmitted to the training monitoring and collaboration module and the cognitive profile construction module. The training monitoring and coordination module is used to accurately distinguish between normal training and risk states based on the collected data. When the risk is low, it triggers the training adjustment strategy and when the risk is high, it activates the emergency response strategy to achieve organic coordination between training and monitoring. The multimodal interaction module is used to accurately match all forms of interaction with the core training requirements of spatial positioning and scene recognition orientation. Through lightweight AR overlay of orientation markers, multi-dialect voice guidance, differentiated tactile vibration and environmental adaptive adjustment, it can achieve precise training guidance and real-time risk intervention for elderly people with cognitive impairment. The cognitive profile building module is used to combine initial neuropsychological scale assessments with real-time perception data to determine the degree of impairment in each core dimension, dynamically update training feedback and behavioral habit data, and build a personalized cognitive profile. The training scheme generation module is used to generate personalized initial training schemes based on individual cognitive profiles, and dynamically adjust parameters through reinforcement learning algorithms in combination with real-time training feedback data. The emergency response module is used to immediately activate the emergency response mechanism when a high-risk abnormal state is identified, push alarm information and accurate data to the family's APP, and coordinate with community medical institutions.

2. The orientation training and monitoring system for elderly people with cognitive impairment based on multimodal interaction according to claim 1, characterized in that, The data acquisition module specifically includes: Real-time location data of elderly people with cognitive impairment is collected, enabling seamless indoor and outdoor positioning, and wearable devices are used to collect physiological data of elderly people with cognitive impairment in real time. Real-time collection of behavioral data of elderly people with cognitive impairment, as well as light intensity, noise level, and obstacle information in the training environment; The collected real-time location data, physiological data, behavioral data, and environmental data are transmitted to the training and monitoring collaboration module, and the environmental data provides environmental auxiliary information for risk status identification.

3. The orientation training and monitoring system for cognitively impaired elderly based on multimodal interaction according to claim 2, characterized in that, The training monitoring and collaboration module specifically includes: The system receives location data, physiological data, behavioral data, and environmental data transmitted from the data acquisition module. It then uses a multi-source data fusion algorithm to preprocess and fuse the data, removing redundant data and abnormal noise to generate standardized fused data. Based on the generated fused data, a deep learning algorithm is used to distinguish between normal training pauses and abnormal risk states. The temporal and spatial data in the fused data are correlated and modeled, and the weight of key risk features is strengthened through an attention mechanism; When a low-risk abnormal state is identified, the training process is not interrupted. The corresponding guidance strategy is triggered based on the multimodal interaction module, and the difficulty of subsequent training is adjusted synchronously. Upon identifying a high-risk abnormal state, an emergency trigger signal is immediately sent to the emergency response module, and the training mode is simultaneously terminated. The current training data and risk state data are then uploaded to the emergency response module.

4. The orientation training and monitoring system for elderly people with cognitive impairment based on multimodal interaction according to claim 3, characterized in that, When a low-risk abnormal state is identified, the training process is not interrupted. Instead, a corresponding guidance strategy is triggered based on the multimodal interaction module, and the difficulty of subsequent training is adjusted synchronously. Specifically, this includes: When a deviation from the training route is detected, the voice interaction unit broadcasts a route correction prompt, while differentiated vibration guides the user back to the correct path, and an enhanced path guidance sign is superimposed. When the training task is detected to be progressing slowly, the difficulty of the current training task is reduced. When mild gait instability is detected, the current training progress is paused, and the elderly person is prompted to slow down through a combination of voice and tactile feedback. The smoothness requirements of the subsequent training path are also adjusted.

5. The orientation training and monitoring system for cognitively impaired elderly based on multimodal interaction according to claim 3, characterized in that, The state recognition process specifically includes: Receive location data, physiological data, behavioral data, and environmental data, and perform data preprocessing; A deep learning-based multi-source data fusion algorithm is used to extract feature vectors from each data source, thereby achieving the fusion of multi-dimensional data. The fused feature data is input into the Transformer model, and the model outputs state recognition results, including normal training pauses, low-risk anomalies, and high-risk anomalies. The status recognition results are verified. If there is ambiguity in the recognition, multiple frames of data are collected for secondary recognition.

6. The orientation training and monitoring system for cognitively impaired elderly based on multimodal interaction according to claim 3, characterized in that, The deep learning algorithm uses the Transformer model, and the specific implementation steps are as follows: The fused data is classified and extracted to separate time-series data and spatial data. The encoder structure of the Transformer model is used to model the relationship between the two types of data and establish the mapping relationship between time-series and spatial data. In the attention mechanism layer of the Transformer model, risk feature weight coefficients are introduced to strengthen the weight of feature data corresponding to key risks such as falls and sudden changes in heart rate, thereby increasing the model's attention to high-value risk features and reducing the interference of irrelevant data on the recognition results. Collect real training and monitoring data of elderly people with cognitive impairment, and manually label them according to normal state, low-risk abnormal state and high-risk abnormal state; Construct a diverse dataset that includes different scenarios and different groups of people, and divide it into training set, validation set and test set; The Transformer model is trained based on the constructed dataset, the hyperparameters of the Transformer model are adjusted in real time using the validation set, and the performance of the Transformer model is verified using the test set.

7. The orientation training and monitoring system for cognitively impaired elderly based on multimodal interaction according to claim 1, characterized in that, The cognitive profile construction module specifically includes: When elderly people with cognitive impairment use the system for the first time, an initial assessment of the core dimensions of orientation is completed by combining neuropsychological scales and multimodal perception data. It receives training feedback data and behavioral habit data transmitted from the data acquisition module in real time, and dynamically updates the basic data of the cognitive profile. Based on the initial assessment results, training feedback data, and behavioral habit data, an individual cognitive profile is constructed.

8. The orientation training and monitoring system for elderly people with cognitive impairment based on multimodal interaction according to claim 1, characterized in that, The training scheme generation module specifically includes: Based on the individual cognitive profile generated by the cognitive profile construction module, a personalized initial training plan is generated. Combining real-time training feedback data collected by the data acquisition module, the reinforcement learning algorithm is used with the dual objective functions of maximizing training effect and minimizing fatigue in the elderly. The training feedback data is used as the state input, the training parameter adjustment is used as the action output, and the training strategy is optimized through the reward function. When an elderly person completes a training task of the current difficulty level with high quality three times in a row, the training difficulty will be automatically increased. If there are too many errors or an abnormal increase in heart rate, the training difficulty will be reduced and the rest interval will be increased.

9. The orientation training and monitoring system for cognitively impaired elderly based on multimodal interaction according to claim 8, characterized in that, The training scheme generation module includes: Based on the results of each training task, the standardized completion time, path deviation error integral, and heart rate stability of elderly people with cognitive impairment during the task are extracted from the training feedback data. The comprehensive completion index for a single task is calculated based on the standardized completion time of each task, the integral of the path deviation error, and the heart rate stability of elderly people with cognitive impairment during the task. The obtained comprehensive completion index is compared with the preset quality completion threshold. When the comprehensive completion index is greater than or equal to the preset quality completion threshold, the current task is determined to be validly completed, and a valid count is recorded based on the determination result. When the effective count reaches 3 consecutive times, the training difficulty increase mechanism is triggered, and a new training difficulty level is calculated based on the trigger result, according to the difficulty level of the current task and the average of the comprehensive completion index of the three consecutive tasks. Based on the new training difficulty level, the key parameters for the next training cycle are adjusted according to the preset mapping rules. Meanwhile, in the first training cycle after the training difficulty level is updated, if the calculated comprehensive completion index is lower than the preset downgrade threshold, the training difficulty level will be immediately restored to the original training difficulty level, and the effective count will be cleared to zero.

10. The orientation training and monitoring system for cognitively impaired elderly based on multimodal interaction according to claim 1, characterized in that, The training scheme generation module includes: Acquire location data, physiological data, behavioral data and environmental data. At the same time, acquire physiological time-series data and spatial time-series data of elderly people with cognitive impairment, and input them into a pre-trained temporal prediction neural network. The pre-trained temporal prediction neural network processes localization data, physiological data, behavioral data, and environmental data to obtain the attention state vector and motion intention vector at each time point. Based on graph convolutional networks, key objects and path nodes are extracted from spatial temporal data. Key objects and path nodes are used as vertices and spatial adjacency relationships are used as edges to obtain the spatial saliency embedding vector of each vertex at the current time. Meanwhile, the instantaneous heart rate variability of elderly people with cognitive impairment was extracted based on physiological data, and the instantaneous heart rate variability was used as low-dimensional physiological time series data and concatenated with the obtained motion intention vector to obtain an enhanced physiological state vector. The enhanced physiological state vector and the spatial saliency embedding vector are temporally aligned to obtain a feature pair sequence. The feature pair sequence is then analyzed based on a multi-head attention fusion layer to obtain the cross-modal correlation weight between the enhanced physiological state vector and the spatial saliency embedding vector. A joint feature representation is then generated based on the cross-modal correlation weight. The pre-trained temporal prediction neural network performs predictive analysis on the joint feature representation to obtain the feature evolution trajectory at each prediction time point after a preset time period. Based on the feature evolution trajectory, the probability of path decision hesitation, spatial cognitive confusion, and gait instability are determined. At the same time, the offset vector of the user's current position is determined based on the feature evolution trajectory. The offset vector is added to the estimated user position coordinates corresponding to the current prediction time point obtained from the digital twin model of the location to obtain the predicted spatial position coordinates. When the predicted probability value of any risk event exceeds the dynamic adaptive threshold, a pre-intervention mechanism is triggered. Based on the triggering result, augmented reality semantic information of the surrounding area is extracted from the pre-built digital twin model of the location based on the predicted spatial location coordinates. Multi-sensory prompts are generated based on the augmented reality semantic information. Based on multi-sensory prompts, preset interactive devices are driven to provide collaborative guidance. After collaborative guidance, the actual behavior data of users within the predicted time window is continuously monitored and compared with the predicted behavior trajectory to generate intervention effectiveness evaluation parameters. The weights and dynamic adaptive thresholds of the pre-trained temporal prediction neural network are dynamically updated based on the intervention effectiveness evaluation parameters.

Images