Home monitoring system and method for cognitive impairment of the elderly based on instant messaging video call

Abstract

The application discloses a kind of old cognitive impairment home monitoring system and method based on instant messaging video call, system embeds the video call component of instant messaging software, with old daily conversation, through the guide prompt on video picture, the professional evaluation item in professional scale is converted into daily communication technique, and scale score is completed naturally in daily care communication by family member;Meanwhile, system real-time acquisition call voice stream and video frame, extract semantic understanding ability, semantic features such as attention dispersion and gait posture parameter;Multi-modal fusion model comprehensive scale score, semantic feature and gait feature, output current cognitive risk score;Establish individual longitudinal baseline database, detect cognitive function change trend by time series analysis, when risk exceeds threshold value or trend is abnormal, send early warning to family member and medical institution.The application realizes "evaluation is communication" non-sensitization home cognitive monitoring, with high compliance and universality while protecting privacy.

Description

Technical Field

[0001] This invention relates to the fields of medical and health monitoring and artificial intelligence technology, and in particular to a home monitoring system and method for elderly cognitive impairment based on instant messaging video calls. Background Technology

[0002] Early identification and intervention can slow the progression of Alzheimer's disease (AD) and reduce the social medical burden. Current cognitive impairment screening technologies have the following problems: (1) Traditional medical assessment model: using scales such as MMSE and MoCA, and being tested face-to-face by doctors or social workers. This approach requires a high degree of cooperation from the subjects and cannot achieve unobtrusive monitoring; the screening cost is high and it is difficult to cover a large population in the community; the assessment is a single snapshot test and cannot dynamically capture the progressive decline of cognitive function; and the assessment results are easily affected by the subjective judgment of the test taker.

[0003] (2) Home monitoring solutions based on independent devices: such as smart speaker products that analyze cognitive status through voice interaction. This type of solution requires the elderly to actively interact with the device, resulting in low compliance; moreover, most solutions rely on semantic content analysis, which is not robust to dialects and accents, and there is also a risk of voice privacy leakage.

[0004] (3) Behavioral analysis solution based on video surveillance: continuously collect behavioral data through cameras. This type of solution infringes on family privacy and carries high legal risks; it is also sensitive to changes in lighting and partial obstruction, resulting in a high false detection rate in community settings.

[0005] (4) Existing speech recognition screening technologies generally have the problem of "semantic bias": Existing speech recognition technologies are mainly designed to understand "semantic content". Their preprocessing filters out "non-standard pronunciations" such as pauses, stutters, and vocal tremors. However, the core early signs of AD are hidden in these discarded prosodic features.

[0006] As can be seen from the above, existing technologies lack a cognitive impairment assessment scheme that can be embedded in daily family communication scenarios, requires no active cooperation from the elderly, protects privacy, and allows for long-term longitudinal monitoring. This invention addresses this technological gap. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a home-based monitoring and early warning system and method for cognitive impairment in the elderly based on instant messaging video calls, which can realize non-intrusive home-based cognitive monitoring of "assessment as communication", protect privacy, and have high compliance and universality.

[0008] To address the aforementioned technical problems, this invention provides a home-based monitoring and early warning method for cognitive impairment in the elderly based on instant messaging video calls, comprising the following steps: S1, activating the monitoring function embedded in the video call component of instant messaging software, displaying cognitive assessment guidance prompts on the call interface, wherein the guidance prompts convert each item of the cognitive assessment scale into everyday communication scripts, guiding family members to complete the scale scoring in a natural conversational manner; S2, during the call, simultaneously acquiring audio streams and video frames, performing real-time speech-to-text and semantic analysis on the audio stream, extracting semantic coherence, attention duration, information density, and abnormal repetition rate as semantic features; and performing video frame analysis... S1. Pedestrian posture estimation: Step length, gait frequency, gait symmetry, and trunk sway are extracted as gait features. S2. Input the scale scores obtained in step S1, the semantic features obtained in step S2, and the gait features into a multimodal fusion model to output the comprehensive cognitive risk score of the current call. S3. Establish a longitudinal baseline database for each elderly person bound with an anonymous ID, storing the comprehensive cognitive risk scores and sub-feature vectors of each call. Calculate the dynamic deviation rate between the current score and the historical baseline, and output the risk probability value in combination with time decay weights. When the risk probability value exceeds a preset threshold or an abnormal feature trend is detected, trigger an alert and send a notification to the family member and / or medical institution.

[0009] Furthermore, the cognitive assessment scale in step S1 is the ABC dementia scale, which includes 13 items across three dimensions: daily living abilities, mental and behavioral symptoms, and cognitive function. The guiding script for each item is pre-generated based on the assessment content of that item and is displayed at the edge of the video call interface in the form of text bubbles or voice prompts. Based on the elderly person's response to the guiding script, the family member slides the scoring bar on the call interface to select a score from 1 to 9 to complete the scoring for that item.

[0010] Furthermore, the mapping relationship between the guiding dialogue and the scale scores is as follows: For the dressing ability item, the guiding dialogue is "Dad / Mom, did it go smoothly to put on this outfit this morning? Did you have any trouble buttoning it?" The family member's description of "I put it on quickly without any help" is mapped to 9 points, "I put it on slowly but can do it myself" is mapped to 5-7 points, "I need help buttoning / putting on sleeves" is mapped to 3-4 points, and "I need help putting it on and taking it off completely" is mapped to 1-2 points; other items have corresponding guiding dialogues and scoring mapping rules.

[0011] Furthermore, in step S2, the semantic analysis uses a pre-trained Chinese language model as the semantic encoder. The input is the dialogue text transcribed from the call speech by ASR. The embedding vector of the [CLS] token is extracted as the semantic feature, and the output dimension is 384-768 dimensions. The semantic features are further used to calculate the following quantitative indicators: average speech rate, lexical diversity index, grammatical integrity score, topic maintenance duration, number of abnormal silences and silence duration.

[0012] Furthermore, the gait feature extraction in step S2 adopts a two-stage method: the first stage extracts 2D human key points from video frames using OpenPose or MediaPipe human pose estimation algorithms; the second stage inputs the pose sequence of consecutive frames into a spatiotemporal Transformer encoder, outputs a 256-512 dimensional gait feature vector, and decodes stride length, stride frequency, stride speed, gait periodicity, double support phase ratio, and torso tilt angle from this vector.

[0013] Furthermore, the multimodal fusion model in step S3 adopts a late fusion strategy, including: semantic features are projected to 512 dimensions through a fully connected layer, gait features are projected to 512 dimensions through a fully connected layer, and scale scores are projected to 512 dimensions through a linear embedding layer; the three 512-dimensional vectors are input into a cross-attention fusion module, where the first attention output is calculated using semantic features as the query, gait features as the key, and scale scores as the value, while the query, key, and value roles are alternated to calculate the other two attention outputs; the three attention outputs are concatenated and input into a multi-layer Transformer encoder, and finally output as a continuous risk score of 0-100 points through two fully connected layers, where 0-20 points indicate normal cognitive function, 21-40 points indicate mild cognitive impairment risk, 41-60 points indicate moderate cognitive impairment risk, and 61-100 points indicate severe cognitive impairment risk.

[0014] Furthermore, the dynamic deviation rate calculation formula in step S4 is: ΔV = (|V t - B t-1 | / B t-1 )× W decay V t For the overall risk score of the t-th call, B t-1 W is the baseline of the exponentially weighted moving average of the previous t-1 calls. decay = e -λ·Δt λ is the time decay factor, Δt is the number of days since the last call; the risk probability P(Risk) = sigmoid(ΔV); when P(Risk) < 0.3, it is considered healthy; when 0.3 ≤ P(Risk) ≤ 0.7, it is considered a warning of mild cognitive impairment; when P(Risk) > 0.7, it is considered a high risk of Alzheimer's disease, triggering a clinical warning.

[0015] Furthermore, the instant messaging software is WeChat, and the video call function is embedded by calling the WeChat Mini Program audio and video call component wx.join1v1Chat or Tencent Cloud TUICallKit; the monitoring function is provided in the form of a WeChat Mini Program plugin, and the cognitive assessment guidance panel can be activated on the call interface after the user authorizes it.

[0016] Furthermore, the processing of the audio stream and video frames is completed locally on the user device, and only the extracted anonymized semantic feature vector, gait feature vector and scale score are uploaded to the cloud vertical database; the original audio and video data are immediately overwritten and cleared from memory after processing and are not persistently stored.

[0017] To address the aforementioned technical problems, this invention also provides a home-based monitoring and early warning system for cognitive impairment in the elderly based on instant messaging video calls, used in the aforementioned method. The system includes: a guidance and interaction module: embedded in an instant messaging video call interface, used to display cognitive assessment guidance scripts, receive scores input by family members, and generate a 13-item ABC scale score vector; a speech and semantic acquisition and analysis module: used to acquire the call speech stream in real time, perform speech activity detection, ASR transcription, and semantic feature extraction, and output a semantic feature vector; a video posture acquisition and analysis module: used to acquire call video frames in real time, perform human posture estimation and gait feature extraction, and output a gait feature vector; a multimodal fusion assessment module: containing a pre-trained multimodal fusion model, used to receive scale score vectors, semantic feature vectors, and gait feature vectors, and output a comprehensive cognitive risk score; a longitudinal baseline management module: used to store historical assessment data for each elderly person, calculate the index-weighted moving average baseline, perform dynamic deviation rate calculation, and trend anomaly detection; and an early warning notification module: used to send early warning notifications to family members and / or the linked medical institution via WeChat template messages or mini-program subscription messages when the risk probability exceeds a threshold or a trend anomaly occurs.

[0018] Compared with existing technologies, this invention offers the following advantages: The home-based monitoring and early warning system and method for elderly cognitive impairment based on instant messaging video calls provided by this invention seamlessly embeds cognitive assessment into daily video calls between family members and the elderly, achieving high compliance, privacy protection, and long-term longitudinal monitoring with multimodal fusion. Specific effects are as follows: 1. Zero learning cost and high compliance: The assessment function is fully integrated into daily family video calls. Family members do not need to perform any additional operations, and the elderly are unaware of it. Compliance has increased from less than 30% in traditional scales to more than 85%.

[0019] 2. Multi-dimensional fusion assessment improves accuracy: By fusing semantic, gait and scale three modalities, the three-class Macro F1 score reached 0.79 on the internal validation set (300 elderly people, including 120 with HC, 120 with MCI and 60 with AD diagnosed by the hospital), which is better than single modality (semantic 0.71, gait 0.68, scale 0.74) and bimodal fusion.

[0020] 3. Long-term longitudinal monitoring to capture early changes: Through dynamic baseline and trend anomaly detection, the inflection point of MCI to AD can be identified on average 4.2 months earlier than a single assessment (p<0.01).

[0021] 4. Privacy Protection: The original audio and video data does not leave the local machine; only anonymized feature vectors are uploaded, fundamentally eliminating the risk of privacy leaks.

[0022] 5. Universality across dialects: Semantic feature extraction uses a pre-trained language model, which maintains good feature consistency for common dialects (Cantonese, Wu, and Minnan); gait and scale scores are completely unaffected by dialect. Attached Figure Description

[0023] Figure 1 This is a flowchart of the home-based monitoring process for cognitive impairment in the elderly based on instant messaging video calls, as described in this invention. Detailed Implementation

[0024] The present invention will now be further described with reference to the accompanying drawings and embodiments.

[0025] Figure 1 This is a flowchart of the home-based monitoring process for cognitive impairment in the elderly based on instant messaging video calls, as described in this invention.

[0026] Please see Figure 1 The present invention provides a home-based monitoring method for cognitive impairment in the elderly via instant messaging video calls, comprising the following steps: S1. Activate the monitoring function of the video call component embedded in the instant messaging software, and display cognitive assessment guidance prompts on the call interface. The guidance prompts will transform each item of the cognitive assessment scale into daily communication scripts to guide family members to complete the scale scoring in a natural dialogue manner. S2. During the call, the voice stream and video frames are collected simultaneously. The voice stream is converted to text in real time and semantic analysis is performed to extract semantic features. The video frames are used to estimate human posture and extract gait features. S3. Input the scale score obtained in step S1, the semantic features and gait features obtained in step S2 into the multimodal fusion model, and output the comprehensive cognitive risk score of the current call. S4. Establish a longitudinal baseline database for each elderly person, calculate the dynamic deviation rate between the current score and the historical baseline, and trigger an early warning when the risk exceeds the threshold or the trend is abnormal.

[0027] Furthermore, the cognitive assessment scale in step S1 is the ABC dementia scale, which contains 13 items covering three dimensions: activities of daily living (ADL), behavioral and psychotic symptoms (BPSD), and cognitive impairment (CF). The assessment can be completed in approximately 10 minutes and possesses high reliability and validity. The guiding script for each item is shown in the table below: Furthermore, the semantic analysis in step S2 uses the pre-trained Chinese language model BERT-base-Chinese as the semantic encoder. The input is the dialogue text transcribed by ASR (Automatic Speech Recognition) from the spoken speech (combining questions from family members and answers from the elderly, with a maximum length of 512 tokens). The embedding vector of the [CLS] token (Classification Token) is extracted as the semantic feature, and the output dimension is 768. The system also calculates the following quantitative indicators: average speech rate (words / second), lexical diversity index (number of different words / total number of words), grammatical integrity score (based on dependency parsing), topic duration (seconds), number of abnormal silences (number of silences > 1.5 seconds), and total silence duration.

[0028] Furthermore, the gait feature extraction in step S2 employs a two-stage method: the first stage extracts 25 2D human keypoints from video frames using the MediaPipe human pose estimation algorithm; the second stage inputs a continuous 30-frame pose sequence (1-second time window, 0.5-second step) into a spatiotemporal Transformer encoder, which consists of 4 layers of multi-head self-attention layers (8 heads) and a feedforward network, outputting a 512-dimensional gait feature vector. The following parameters are decoded from this vector: stride length (meters, estimated using keypoint pixel coordinates combined with preset camera intrinsic parameters), cadence (steps / minute), gait speed (meters / second), gait cycle variability (standard deviation of stride length), dual-stance ratio (the proportion of time both feet are on the ground simultaneously within the cycle), and torso tilt angle (the angle of the shoulder line relative to the horizontal line).

[0029] Furthermore, the structure of the multimodal fusion model in step S3 is as follows: Semantic encoding branch: BERT outputs 768-dimensional features → LayerNorm → Dropout(0.1) → linear layer projection to 512 dimensions.

[0030] Gait coding branch: Spatiotemporal Transformer outputs 512-dimensional features → LayerNorm → Dropout(0.1) → Linear layer maintains 512 dimensions.

[0031] Scale encoding branch: 13-dimensional rating → linear embedding layer (13×512) → average pooling → 512-dimensional.

[0032] Cross-attention fusion module: Three 512-dimensional vectors are used as Q, K, and V respectively. Three sets of cross-attention are calculated: Attention 1 = CrossAttn(Q=semantics, K=gait, V=scale); Attention 2 = CrossAttn(Q=gait, K=scale, V=semantics); Attention 3 = CrossAttn(Q=scale, K=semantics, V=gait). The three outputs are concatenated to obtain a 1536-dimensional vector.

[0033] Joint coding layer: 1536 dimensions → Multi-layer Transformer encoder (2 layers, 8 heads, feedforward dimension 2048) → Output 1536 dimensions.

[0034] Prediction Header: 1536-dimensional → LayerNorm → Dropout(0.2) → Linear Layer(1536→256) → ReLU → Dropout(0.2) → Linear Layer(256→1) → Sigmoid → Output 0-100 points (multiplied by 100).

[0035] The model is trained using a multi-task joint loss function: L total = λ1·MSE (predicted score, physician score) +λ2·CrossEntropy (tri-class HC / MCI / AD) + λ3·TripletLoss (ensuring higher scores for samples with more severe cognitive impairment). The training dataset includes longitudinal call data from 500 elderly individuals (10-50 calls per elderly individual, totaling approximately 10,000 samples), as well as speech and gait data from the publicly available datasets ADReSS and Pitt Corpus.

[0036] Furthermore, the dynamic baseline algorithm in step S4 is specifically implemented as follows: the system maintains an anonymous ID for each elderly person and locally encrypts and stores historical feature vectors. Let the comprehensive risk score of the t-th call be V. t (0-100), Exponentially weighted moving average baseline B t = α·V t + (1-α)·B t-1 Initially, B0 = V0, α = 0.2. Dynamic deviation rate ΔV = (|V t - B t-1 | / B t-1 ) × W decay W decay = e -λ·Δt λ=0.03, Δt is the number of days since the last call (Δt=1 if calls are made daily). Risk probability P(Risk) = 1 / (1 + e^(-1 / 2)) -k·ΔV), where k=5 is the sensitivity coefficient. Output judgment: P(Risk)<0.3 is healthy (green light), 0.3≤P(Risk)≤0.7 is MCI warning (yellow light), P(Risk)>0.7 is AD high risk (red light).

[0037] In addition, the system calculates Mahalanobis distance for semantic feature vectors and gait feature vectors respectively. When any feature deviates from the baseline by more than 2 standard deviations in 3 consecutive calls, a "specific warning" (such as "gait stability continues to decline") is triggered even if the overall score does not exceed the threshold.

[0038] Example 1: ABC Scale Guidance and Multimodal Assessment in WeChat Video Calls This embodiment uses a WeChat mini-program to achieve cognitive monitoring during video calls between family members and the elderly.

[0039] Step 1: Startup and Authorization The child opens the "Cognitive Care Assistant" mini-program in WeChat and clicks the "Start a Call with Assessment" button. The mini-program calls wx.authorize to request microphone and camera permissions, and after obtaining authorization, calls wx.join1v1Chat to initiate a video call. After the call is connected, the mini-program overlays a semi-transparent floating panel on top of the video screen, displaying the current assessment progress (13 items in total, item 1 / clothing ability).

[0040] Step 2: Guided Interaction A floating panel displays guiding speech bubbles: "Dad / Mom, did it go smoothly getting into this outfit this morning? Did you have any trouble buttoning it?" The children, seeing this, naturally ask their parents these questions. After the elderly person answers, the children select the corresponding score from a 1-9 scale on the panel below, based on the actual response. The system records the score for that item and automatically moves to the next item, completing all 13 items in sequence. Throughout the process, the elderly person is unaware that they are being assessed and perceives it as a casual conversation.

[0041] Step 3: Seamless Data Acquisition During the 60-second call (automatically timed by the system; after 60 seconds, the call can continue but the evaluation ends), the following data is collected in the background: Speech stream: Speech was captured at a 16kHz sampling rate. Voice activity detection (VAD) within WeChat was used to separate the speech segments of the children and the elderly, with ASR transcription performed only on the elderly's speech segments. The transcribed text was then fed into a locally deployed BERT-base-Chinese model (quantized to INT8) to extract 768-dimensional semantic features. Simultaneously, semantic quantization metrics such as average speech rate, number of silences, and topic duration within the 60-second duration were calculated.

[0042] Video stream: Video frames are captured at 15fps, and one frame is extracted from every five frames and fed into the MediaPipe pose estimation model to extract the coordinates of 25 key points. The pose sequence of 30 consecutive frames (i.e., a 2-second window with a 1-second step) is input into the spatiotemporal Transformer encoder, which outputs 512-dimensional gait features and decodes parameters such as stride length, stride frequency, and torso sway.

[0043] Step 4: Multimodal fusion evaluation The 13-dimensional ABC score vector obtained in step 2, the 768-dimensional semantic features obtained in step 3 (reduced to 512 dimensions), and the 512-dimensional gait features are input into the multimodal fusion model. The model runs on a local edge computing node (such as a home smart gateway or the user's mobile phone NPU), with an inference time of approximately 50ms. The output is a comprehensive cognitive risk score, for example, 72 points (indicating a moderate to severe cognitive impairment risk).

[0044] Step 5: Longitudinal Baseline Update and Early Warning The elderly person's anonymous ID already has a history of 30 phone calls. The system calculates the dynamic deviation rate ΔV between the current score of 72 and the historical baseline of 55: ΔV = (|72-55| / 55) × e -0.03×1 ≈ 0.309 × 0.97 ≈ 0.30. Risk probability P(Risk) = sigmoid(5×0.30) ≈ sigmoid(1.5) ≈ 0.82>0.7, triggering AD high-risk warning.

[0045] Step 6: Early Warning Notification The system sends a notification to the children via a WeChat mini-program subscription message: "[Cognitive Care] Your father's recent speech and gait data show a significant decline in cognitive function compared to last month (risk score 72 / 100). It is recommended that you take your father to the community hospital for a professional scale re-examination as soon as possible." At the same time, the system pushes the anonymized and encrypted assessment report to the community doctor's workbench linked to the elderly man.

[0046] Example 2: Pure semantic feature warning (no gait anomaly triggered) In another scenario, an elderly person with mobility issues is confined to a wheelchair, making it impossible to extract effective gait features. In this case, the system only uses semantic features and ABC scale scores for fusion (gait features are set as a 0 vector, and the model ignores this modality through a masking mechanism). During a call, semantic features showed that the number of abnormal silences increased from an average of 2 times / minute to 8 times / minute, and the lexical diversity index decreased from 0.65 to 0.42. Although the ABC scale score only changed from 32 to 38 points (still within the MCI range), the Mahalanobis distance of the semantic sub-features exceeded 2 standard deviations, and this was the case in three consecutive calls. The system triggered a "Special Warning: Language function continues to decline; language rehabilitation training recommended."

[0047] Example 3: Identity Recognition in Multi-User Home Scenarios When multiple elderly people live in the same household (such as grandparents), the system verifies the identity of the person being called within the first 10 seconds after the call is connected through both voiceprint recognition (based on i-vector) and facial recognition (based on face detection in video frames). The voiceprint model registers each family member in an unsupervised manner, with an accuracy rate >95%. The identified identity ID is used to associate with the correct longitudinal baseline database.

[0048] The system of this invention can be deployed in two modes: 1. Pure Software Mode: Runs as a WeChat mini-program plugin on the user's mobile phone. All data collection and processing are completed locally on the phone, with only anonymized feature vectors (size <2KB / time) uploaded to the cloud. Suitable for most smartphone users.

[0049] 2. Home Edge Gateway Mode: For homes with poor network conditions or limited mobile phone performance, a home smart gateway (such as one based on a Raspberry Pi 4 or RK3588 development board) can be configured. The gateway connects to the mobile phone via Bluetooth / LAN. The mobile phone is only responsible for audio and video acquisition, and all model inference is completed on the gateway. The gateway has a built-in NPU (computing power ≥ 1 TOPS), and the power consumption per inference is < 2W.

[0050] The system of the present invention includes: Guiding Interaction Module: Embedded in an instant messaging video call interface, used to display cognitive assessment guidance scripts, receive scores input by family members, and generate a 13-item ABC scale rating vector. The speech and semantic acquisition and analysis module is used to acquire call speech streams in real time, perform speech activity detection, ASR transcription and semantic feature extraction, and output semantic feature vectors. Video pose acquisition and analysis module: used to acquire call video frames in real time, perform human pose estimation and gait feature extraction, and output gait feature vector; Multimodal fusion assessment module: contains a pre-trained multimodal fusion model, which receives scale rating vectors, semantic feature vectors and gait feature vectors, and outputs a comprehensive cognitive risk score; Longitudinal baseline management module: used to store historical assessment data for each elderly person, calculate the index-weighted moving average baseline, and perform dynamic deviation rate calculation and trend anomaly detection; Early warning notification module: Used to send early warning notifications to family members and / or the linked medical institutions via WeChat template messages or mini-program subscription messages when the risk probability exceeds the threshold or the trend is abnormal.

[0051] This invention strictly complies with the Personal Information Protection Law and medical data security regulations. Specific privacy protection mechanisms include: The original audio and video data never leaves the user's device. ASR transcription, semantic feature extraction, and pose estimation are all performed locally.

[0052] The extracted feature vectors are encrypted using AES-256 before transmission, and stored in a dedicated database after uploading. They are decoupled from user identities, retaining only the anonymous ID.

[0053] Users can view, export, or delete all their historical data at any time through the "Data Management" page of the mini-program.

[0054] The system has passed the Level 3 certification of the National Information Security Protection System and has undergone a third-party privacy audit.

[0055] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications and improvements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be defined by the claims.

Claims

1. A home-based monitoring method for cognitive impairment in the elderly based on instant messaging video calls, characterized in that, Includes the following steps: S1. Activate the monitoring function of the video call component embedded in the instant messaging software, and display cognitive assessment guidance prompts on the call interface. The guidance prompts will transform each item of the cognitive assessment scale into daily communication scripts to guide family members to complete the scale scoring in a natural dialogue manner. S2. During the call, the voice stream and video frames are collected simultaneously. The voice stream is converted to text in real time and semantically analyzed. Semantic coherence, attention duration, information density and abnormal repetition rate are extracted as semantic features. Human pose estimation is performed on video frames, and stride length, stride frequency, gait symmetry, and trunk sway are extracted as gait features; S3. Input the scale score obtained in step S1, the semantic features and gait features obtained in step S2 into the multimodal fusion model, and output the comprehensive cognitive risk score of the current call. S4. Establish a longitudinal baseline database for each elderly person with an anonymous ID, storing the comprehensive cognitive risk score and sub-feature vectors of each call. Calculate the dynamic deviation rate between the current score and the historical baseline, and output the risk probability value by combining the time decay weight; When the risk probability value exceeds the preset threshold or an abnormal characteristic trend is detected, an early warning is triggered and a notification is sent to the family member and / or medical institution.

2. The home-based monitoring method for cognitive impairment in the elderly based on instant messaging video calls as described in claim 1, characterized in that, The cognitive assessment scale in step S1 is the ABC dementia scale, which includes 13 items across three dimensions: daily living abilities, mental and behavioral symptoms, and cognitive function. The guiding script for each item is pre-generated based on the assessment content of that item and is displayed at the edge of the video call interface in the form of text bubbles or voice prompts. Based on the elderly person's response to the guiding script, the family member slides the scoring bar on the call interface to select a score from 1 to 9 to complete the scoring for that item.

3. The home-based monitoring method for cognitive impairment in the elderly based on instant messaging video calls as described in claim 2, characterized in that, The mapping relationship between the guiding dialogue and the scale scores is as follows: For the dressing ability item, the guiding dialogue is "Dad / Mom, did it go smoothly to put on this outfit this morning? Did you have any trouble buttoning it?" The family member's description of "I put it on quickly without any help" is mapped to 9 points, "I put it on slowly but I can do it by myself" is mapped to 5-7 points, "I need help buttoning / putting on the sleeves" is mapped to 3-4 points, and "I need help putting it on and taking it off completely" is mapped to 1-2 points; other items have corresponding guiding dialogues and scoring mapping rules.

4. The home-based monitoring method for cognitive impairment in the elderly based on instant messaging video calls as described in claim 1, characterized in that, The semantic analysis in step S2 uses a pre-trained Chinese language model as the semantic encoder. The input is the dialogue text transcribed from the call speech by ASR. The embedding vector of the [CLS] token is extracted as the semantic feature, and the output dimension is 384-768 dimensions. The semantic features are further used to calculate the following quantitative indicators: average speech rate, lexical diversity index, grammatical integrity score, topic maintenance duration, number of abnormal silences and silence duration.

5. The home-based monitoring method for cognitive impairment in the elderly based on instant messaging video calls as described in claim 1, characterized in that, The gait feature extraction in step S2 adopts a two-stage method: the first stage extracts 2D human key points from video frames using OpenPose or MediaPipe human pose estimation algorithms; the second stage inputs the pose sequence of consecutive frames into a spatiotemporal Transformer encoder, outputs a 256-512 dimensional gait feature vector, and decodes stride length, stride frequency, stride speed, gait periodicity, double support phase ratio, and torso tilt angle from the vector.

6. The home-based monitoring method for cognitive impairment in the elderly based on instant messaging video calls as described in claim 1, characterized in that, The multimodal fusion model in step S3 employs a late-stage fusion strategy, including: projecting semantic features to 512 dimensions via a fully connected layer, projecting gait features to 512 dimensions via a fully connected layer, and projecting scale scores to 512 dimensions via a linear embedding layer; inputting the three 512-dimensional vectors into a cross-attention fusion module, where the first attention output is calculated using semantic features as the query, gait features as the key, and scale scores as the value, while simultaneously calculating the other two attention outputs by alternating the roles of query, key, and value; concatenating the three attention outputs and inputting them into a multi-layer Transformer encoder, which finally outputs a continuous risk score of 0-100 points via two fully connected layers, where 0-20 points indicate normal cognitive function, 21-40 points indicate mild cognitive impairment risk, 41-60 points indicate moderate cognitive impairment risk, and 61-100 points indicate severe cognitive impairment risk.

7. The home-based monitoring method for cognitive impairment in the elderly based on instant messaging video calls as described in claim 1, characterized in that, The formula for calculating the dynamic deviation rate in step S4 is: ΔV = (|V t - B t-1 | / B t-1 ) × W decay V t For the overall risk score of the t-th call, B t-1 W is the baseline of the exponentially weighted moving average of the previous t-1 calls. decay = e -λ·Δt λ is the time decay factor, and Δt is the number of days since the last call; Risk probability P(Risk) = sigmoid(ΔV); when P(Risk) < 0.3, it is considered healthy; when 0.3 ≤ P(Risk) ≤ 0.7, it is considered a warning of mild cognitive impairment; when P(Risk) > 0.7, it is considered a high risk of Alzheimer's disease, triggering a clinical warning.

8. The home-based monitoring method for cognitive impairment in the elderly based on instant messaging video calls as described in claim 1, characterized in that, The instant messaging software is WeChat, and the video call function is embedded by calling the WeChat Mini Program audio and video call component wx.join1v1Chat or Tencent Cloud TUICallKit; the monitoring function is provided in the form of a WeChat Mini Program plugin, and the cognitive assessment guidance panel can be activated in the call interface after the user authorizes it.

9. The home-based monitoring method for cognitive impairment in the elderly based on instant messaging video calls as described in claim 1, characterized in that, The processing of the audio stream and video frames is completed locally on the user device. Only the extracted anonymized semantic feature vectors, gait feature vectors, and scale scores are uploaded to the cloud-based longitudinal database. The original audio and video data are immediately overwritten and cleared from memory after processing and are not persistently stored.

10. A home-based monitoring and early warning system for cognitive impairment in the elderly based on instant messaging video calls, used to implement the method as described in any one of claims 1 to 9, characterized in that, The system includes: Guiding Interaction Module: Embedded in an instant messaging video call interface, used to display cognitive assessment guidance scripts, receive scores input by family members, and generate a 13-item ABC scale rating vector. The speech and semantic acquisition and analysis module is used to acquire call speech streams in real time, perform speech activity detection, ASR transcription and semantic feature extraction, and output semantic feature vectors. Video pose acquisition and analysis module: used to acquire call video frames in real time, perform human pose estimation and gait feature extraction, and output gait feature vector; Multimodal fusion assessment module: contains a pre-trained multimodal fusion model, which receives scale rating vectors, semantic feature vectors and gait feature vectors, and outputs a comprehensive cognitive risk score; Longitudinal baseline management module: used to store historical assessment data for each elderly person, calculate the index-weighted moving average baseline, and perform dynamic deviation rate calculation and trend anomaly detection; Early warning notification module: Used to send early warning notifications to family members and / or the linked medical institutions via WeChat template messages or mini-program subscription messages when the risk probability exceeds the threshold or the trend is abnormal.

Images