A Multimodal Fusion-Based Cognitive Ability Assessment System and Method for the Elderly

Abstract

This application discloses a multimodal fusion-based cognitive ability assessment system and method for the elderly, relating to the field of cognitive assessment for the elderly. It first acquires the user's current multimodal data and fuses it into a current feature vector. Subsequently, instead of comparing it with a universal standard, it retrieves the user's personal baseline profile and calculates the difference score between the current state and their historical baseline. Specifically, it introduces the core indicator of drift velocity, which combines the current difference with the historical rate of change, thereby quantitatively modeling the speed of change. By simultaneously analyzing the magnitude and speed of change, it can effectively distinguish between gradual, low-speed normal aging and rapid, drastic pathological decline, thus accurately solving the key technical problem in the prior art where these two types of aging are easily confused.

Description

Technical Field

[0001] This application relates to the field of cognitive assessment of the elderly, and more specifically, to a cognitive ability assessment system and method for the elderly based on multimodal fusion. Background Technology

[0002] With the increasing global trend of population aging, the incidence of cognitive impairment diseases such as Alzheimer's disease is rising year by year, placing a heavy burden on society and families. Clinical practice has proven that early screening and continuous monitoring of cognitive abilities in the elderly are key to slowing disease progression and improving quality of life. Traditional cognitive assessments rely heavily on questionnaires, which are not only subjective and time-consuming but also difficult to track continuously at high frequencies. Therefore, developing an objective, convenient, and non-invasive automated assessment technology has become an urgent need in this field. Utilizing sensors such as cameras and microphones to collect multimodal data on the elderly, including facial expressions, speech, and gait, and then conducting comprehensive analysis through artificial intelligence models, provides a promising technological path to achieving the above goals.

[0003] To address this, several cognitive assessment schemes based on multimodal data fusion have emerged in existing technologies. These schemes typically collect video and audio data from users during specific tasks or daily life, extract relevant features, and input them into a pre-trained classification or regression model to output a single cognitive state score. However, these existing methods generally suffer from a core flaw: they mostly employ static, group-based assessment models. These one-size-fits-all models learn average patterns extracted from large-scale datasets, aiming to compare all individuals with a fixed, universal health standard. This approach ignores the inherent and significant differences between individuals and treats each assessment as an isolated event, lacking modeling and analysis of individual longitudinal data. This static assessment model leads to a key technical challenge: the system struggles to effectively distinguish normal individual baseline drift, i.e., normal aging, from true pathological cognitive decline. Normal aging and early cognitive impairment show similarities in many external characteristics (such as slower speech and postural changes), but their core differences often lie in the rate and pattern of change. Pathological decline often manifests as an accelerated or precipitous decline in certain cognitive abilities, while normal aging is a more gradual and progressive process.

[0004] Existing technologies are unable to establish and track personalized health baselines for each user, nor can they model the rate of change. Therefore, they are prone to misjudgment when faced with natural changes in individuals over time, which significantly reduces the clinical credibility and practical value of the assessment system. Summary of the Invention

[0005] To address the problems mentioned above, according to one aspect of this application, a method for assessing the cognitive abilities of the elderly based on multimodal fusion is provided, comprising: acquiring the original video stream, original audio stream, and original gait data of the elderly subject to be assessed; performing parallel modal feature extraction on the original video stream, original audio stream, and original gait data to obtain a current feature vector for multimodal fusion; retrieving and loading a personal baseline profile based on the user ID of the elderly subject to be assessed to obtain a personal baseline profile, the personal baseline profile including the last updated baseline feature vector, the number of times the baseline has been updated, and a historical drift velocity sequence; calculating a difference score based on the current feature vector for multimodal fusion and the last updated baseline feature vector in the personal baseline profile; calculating a drift velocity based on the difference score and the historical drift velocity sequence; and generating a cognitive assessment report based on the difference score and the drift velocity.

[0006] According to another aspect of this application, a multimodal fusion-based cognitive ability assessment system for the elderly is provided, comprising: an elderly subject raw data acquisition module for acquiring raw video streams, raw audio streams, and raw gait data of the elderly subject to be assessed; a raw data multimodal feature fusion module for performing parallel modal feature extraction on the raw video streams, raw audio streams, and raw gait data to obtain a multimodal fusion current feature vector; a personal baseline profile loading module for retrieving and loading a personal baseline profile based on the user ID of the elderly subject to be assessed to obtain a personal baseline profile, the personal baseline profile including the last updated baseline feature vector, the number of times the baseline has been updated, and a historical drift velocity sequence; a difference score calculation module for calculating a difference score based on the multimodal fusion current feature vector and the last updated baseline feature vector in the personal baseline profile; a drift velocity calculation module for calculating a drift velocity based on the difference score and the historical drift velocity sequence; and a cognitive assessment report generation module for generating a cognitive assessment report based on the difference score and the drift velocity.

[0007] Compared with existing technologies, this application provides a multimodal fusion-based cognitive ability assessment system and method for the elderly. By establishing and continuously updating a unique personal baseline profile for each elderly person, the assessment reference standard shifts from comparing with the group average to comparing with the individual's historical level. Specifically, it first acquires the user's current multimodal data and fuses it into a current feature vector. Then, instead of comparing it with a universal standard, it retrieves the user's personal baseline profile and calculates the difference score between the current state and their historical baseline. More importantly, it introduces the core indicator of drift speed, which combines the current difference with the historical rate of change, thereby quantitatively modeling the speed of change. By simultaneously analyzing the magnitude and speed of change, it can effectively distinguish between gradual, low-speed normal aging and rapid, drastic pathological decline, thus accurately solving the key technical problem of easy confusion between the two in the prior art. Attached Figure Description

[0008] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings.

[0009] Figure 1 This is a flowchart of a method for assessing cognitive abilities of the elderly based on multimodal fusion, according to an embodiment of this application.

[0010] Figure 2 This is a schematic diagram of data flow in a multimodal fusion-based cognitive ability assessment method for the elderly according to an embodiment of this application.

[0011] Figure 3 This is a flowchart of step 2 in the multimodal fusion-based cognitive ability assessment method for the elderly according to an embodiment of this application.

[0012] Figure 4 This is a flowchart of step 6 in the multimodal fusion-based cognitive ability assessment method for the elderly according to an embodiment of this application.

[0013] Figure 5 This is a block diagram of a multimodal fusion-based cognitive ability assessment system for the elderly according to an embodiment of this application. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. It should be understood that the drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] In response to the main problems mentioned in the technical background, this application proposes a method for assessing the cognitive abilities of the elderly based on multimodal fusion. Figure 1 This is a flowchart of a method for assessing cognitive abilities of the elderly based on multimodal fusion, according to an embodiment of this application. Figure 2 This is a schematic diagram of data flow in a multimodal fusion-based cognitive ability assessment method for the elderly according to an embodiment of this application. Figure 1 and Figure 2 As shown, the cognitive ability assessment method for the elderly based on multimodal fusion according to an embodiment of this application includes: Step 1, acquiring the original video stream, original audio stream, and original gait data of the elderly subject to be assessed; Step 2, performing parallel modal feature extraction on the original video stream, original audio stream, and original gait data to obtain a current feature vector for multimodal fusion; Step 3, retrieving and loading a personal baseline profile based on the user ID of the elderly subject to be assessed to obtain a personal baseline profile, which includes the last updated baseline feature vector, the number of times the baseline has been updated, and a historical drift velocity sequence; Step 4, calculating a difference score based on the current feature vector for multimodal fusion and the last updated baseline feature vector in the personal baseline profile; Step 5, calculating a drift velocity based on the difference score and the historical drift velocity sequence; Step 6, generating a cognitive assessment report based on the difference score and the drift velocity.

[0016] In step 1, the raw video stream, raw audio stream, and raw gait data of the elderly subjects to be assessed are acquired. It should be understood that cognitive ability, as a high-level neural activity, is not determined by a single physiological or behavioral indicator, but is comprehensively reflected in multiple dimensions such as an individual's language expression, facial micro-expressions, emotional responses, and motor coordination. Data from any single modality, such as analyzing only speech or observing only gait, can only provide a one-sided perspective and is easily influenced by short-term individual emotions, fatigue, or environmental factors, leading to biased and unstable assessment results. To construct a comprehensive, objective, and robust cognitive assessment model, this application first captures complementary cues reflecting cognitive states from multiple information sources. Therefore, in the initial stage of the assessment, the raw video stream, raw audio stream, and raw gait data of the elderly subjects to be assessed are acquired in parallel to construct an information-rich, multi-dimensional, and high-fidelity raw dataset. This provides a solid data foundation for subsequent accurate extraction of multimodal features, establishment of personalized baselines, and keen capture of subtle changes, thereby improving the accuracy and reliability of the assessment.

[0017] In one alternative implementation, step 1 is carried out as follows: The implementation process takes place in a standardized evaluation environment, such as a quiet room with an area of ​​at least 15 square meters, a simple background, uniform and bright lighting, and ambient noise below 40 decibels. Data acquisition equipment is pre-deployed in the room, including a high-definition digital camera, a high-fidelity microphone array, and a 6-meter-long pressure-sensing walkway. The camera records at a resolution of 1920x1080 pixels and a frame rate of 30 frames per second, positioned approximately 1.5 meters directly in front of the subject's seat, at eye level, to ensure complete and clear capture of their face, upper body posture, and hand movements. The microphone array is placed on a table in front of the subject, with a sampling frequency set to 48 kHz and a quantization bit depth of 24 bits for lossless recording of speech signals. The pressure-sensing walkway incorporates thousands of high-precision pressure sensors capable of acquiring plantar pressure distribution data at a frequency of 100 Hz. All acquisition devices are connected to the same high-precision clock source via a central control unit to ensure nanosecond-level synchronization of all subsequently generated data streams in terms of timestamps. Specifically, the parameters in this acquisition environment were determined according to best practice standards in clinical research and signal processing.

[0018] When an elderly person with user ID USER_001 enters the assessment process, they are first guided by a professional to sit in a designated seat. At the start of the assessment, the central control unit simultaneously triggers the recording functions of all data acquisition devices and begins executing a pre-set cognitive stimulation task protocol. This protocol contains multiple sub-tasks designed to stimulate the subject's cognitive responses from different dimensions.

[0019] First, a resting-state observation task was performed, requiring the subject to sit naturally, look straight ahead, and remain silent for 30 seconds. During this time, a continuous sequence of image frames recorded by the camera constituted part of the raw video stream, which included the subject's baseline facial expression, blink frequency, and minor head postures; background noise recorded by the microphone and the subject's breathing sounds constituted part of the raw audio stream; at this time, there was no data input from the pressure-sensing walkway.

[0020] Next, a speech fluency task was performed, instructing the subject to name as many animals as possible within one minute. During this minute, the camera continuously recorded the subject's facial expressions, eye movements, and body language as they processed cognitive retrieval and speech organization; these continuous video frames formed the raw video stream for this task. Simultaneously, the microphone recorded every word spoken by the subject, pauses between words, speech rate, intonation, and any hesitant or repetitive non-fluent expressions; this constituted a high-quality raw audio stream. For example, the audio stream would contain specific speech content such as "tiger, lion, um... elephant..." and its accompanying acoustic features.

[0021] Next, an image description task is performed. A standard image with complex content, such as a picture of a cookie theft, is presented on the screen, and the subject is asked to describe in detail what happened in the picture. The raw video stream generated by this process captures the subject's complete visual behavior during observation, understanding, language organization, and expression, such as the scanning trajectory of the gaze across different areas of the image and specific facial expressions resulting from recall or thought. The raw audio stream records the syntactic complexity, lexical richness, logical coherence, and semantic accuracy of the description.

[0022] Finally, a timed walking task was executed. The subject was instructed to stand up from the chair and walk along the pressure-sensing track at a comfortable, normal pace to the 6-meter finish line, turn around, walk back to the starting point, and sit down. During this process, the pressure-sensing track was activated, continuously collecting and outputting a series of timestamps and corresponding plantar pressure distribution matrices. This series of time-series pressure matrix data accurately recorded the spatiotemporal information of each foot contact with and departure from the track, constituting the subject's raw gait data. For example, the data would include detailed information such as the left heel strike time, the right toe lift-off time, the stride length, stride width, stance duration, and the pressure center trajectory of both feet. Simultaneously, the camera also recorded the subject's torso swing, gait posture, and stability during turns; these video frames also became part of the raw video stream.

[0023] After the entire evaluation protocol was completed, the central control unit stopped data acquisition from all devices. At this point, a complete data package associated with user IDUSER_001's evaluation was obtained. This data package contained three raw data files, precisely timestamped and unprocessed: an MP4 file containing video footage from all task phases (the raw video stream), a WAV file containing audio recordings from all task phases (the raw audio stream), and a CSV or binary file recording all pressure sensor readings during the walking task (the raw gait data).

[0024] In step 2, parallel modal feature extraction is performed on the original video stream, original audio stream, and original gait data to obtain a multimodal fusion current feature vector. Correspondingly, after acquiring the original video stream, audio stream, and gait data, heterogeneous and unstructured raw signals are obtained. These raw data, such as pixel matrices in videos, waveform samples in audio, and pressure readings in gait data, while containing rich cognitive state information, are too high-dimensional, contain a large amount of redundancy and noise, and cannot be directly used for quantitative, longitudinal comparative analysis. Therefore, these raw data need to be deeply processed and refined, transforming them from the signal layer to the semantic feature layer. Based on this, this application performs parallel feature extraction on the data of each modality, transforming the high-dimensional, heterogeneous raw input into a low-dimensional, isomorphic numerical vector with clear clinical or behavioral significance, ultimately fusing them into a feature vector that can characterize an individual's current cognitive state, providing data support for subsequent processing.

[0025] In one optional implementation Figure 3 This is a flowchart of step 2 in the multimodal fusion-based cognitive ability assessment method for the elderly according to an embodiment of this application. Figure 3 As shown, step 2 involves parallel modal feature extraction of the original video stream, original audio stream, and original gait data to obtain the current multimodal fusion feature vector, including: step 21, inputting the original video stream into a pre-trained face and pose analysis model to obtain video feature sub-vectors; step 22, inputting the original audio stream into a speech feature extraction module to obtain audio feature sub-vectors; step 23, after performing speech recognition on the original audio stream, extracting text features from its speech recognition results; step 24, extracting gait feature sub-vectors from the original gait data; and step 25, fusing the video feature sub-vectors, audio feature sub-vectors, text features, and gait feature sub-vectors to obtain the current multimodal fusion feature vector.

[0026] In the above optional implementation, step 2 is implemented as follows: First, the processing in step 21 is performed. The pre-trained face and pose analysis model mainly consists of two parallel processing branches: a face analysis branch and a pose analysis branch. The face analysis branch accurately captures all visual information of the face. When the original video stream is read in frame by frame, each frame image first passes through a face detector based on a cascaded convolutional neural network, which is a multi-task cascaded convolutional network MTCNN, to accurately locate and crop the face region. This MTCNN is a cascaded structure composed of three sub-networks that are processed sequentially with increasing complexity, to achieve fast and accurate localization from coarse to fine. Before processing the original video stream, an image pyramid needs to be constructed for each frame image. The full-resolution image of the original frame is used as the bottom layer of the pyramid, and the image is iteratively downsampled using a preset scaling factor, such as 0.709. Each downsampling generates a new image layer with a smaller size. This scaling process continues until the minimum side length of the image reaches or falls below the size of the receptive field of the detection network. Ultimately, the original image and all its scaled-down versions together form the image pyramid. Then, a shallow, fully convolutional proposal network, P-Net, with its fixed 12x12 pixel receptive field, performs efficient sliding scans on the image pyramid formed by scaling down the original image. This P-Net contains only three convolutional layers, and its task is to perform fast binary classification (face / non-face) for each window and output an initial bounding box regression vector. This stage generates a large number of candidate windows that may contain faces. Next, the image regions corresponding to all candidate windows filtered by P-Net are cropped and uniformly scaled to 24x24 pixels, serving as input to the second-stage refining network, R-Net. R-Net has a deeper convolutional neural network structure than P-Net, containing three convolutional layers and two max-pooling layers, with a fully connected layer at the end. This structure performs a more rigorous examination of each input candidate window, filtering out a large number of false positive windows misclassified in the P-Net stage and performing more accurate bounding box regression calibration. Finally, the remaining high-confidence candidate windows, after being filtered again by R-Net, are cropped and scaled to 48x48 pixels before being fed into the final output network of the cascaded structure, O-Net. O-Net consists of four convolutional layers, three max-pooling layers, and a larger fully connected layer. O-Net performs the final discrimination on the input candidate regions, outputting a precise bounding box with extremely high confidence. Specifically, in addition to performing face classification and bounding box regression, O-Net also performs facial landmark localization in parallel, outputting the precise coordinates of five landmarks, such as the centers of the eyes, the tip of the nose, and the corners of the mouth. Based on the final bounding box coordinates output by O-Net, the face region can be cropped from the original frame.The cropped face image is aligned using keypoint information from the O-Net output, in-plane rotation is corrected via affine transformation, and its size is normalized to a fixed size required by subsequent modules, such as a standardized size of 112x112 pixels. Specifically, the weight parameters of each of the above networks are obtained through pre-training. Subsequently, the normalized face image is fed into a deep residual network, ResNet-50, whose network structure includes an initial convolutional layer, four residual block stages, and a final global average pooling layer, which compresses the high-dimensional feature map into a 2048-dimensional feature vector. The weights of this ResNet-50 are pre-trained on a large identity recognition dataset containing millions of face images. To adapt it to the specific task of cognitive assessment, the top layer of this pre-trained model is replaced and connected to three different output heads, used to predict facial behavior encoding units (AU), head 3D pose, and gaze direction, respectively. The weights of these three output heads are fine-tuned on an internally constructed dataset of cognitive assessment videos of the elderly, annotated frame-by-frame by clinical experts. Specifically, the AU prediction head is a multi-label classification layer consisting of two fully connected layers. The first layer reduces the 2048-dimensional feature vector output by ResNet-50 to 256 dimensions and performs ReLU non-linear activation. The second layer maps these 256-dimensional intermediate features to a 17-dimensional output, representing 17 core AUs, such as AU01 - inner eyebrow lift and AU12 - continuous activation values ​​of 0-5 for the corner of the mouth. The head pose prediction head is a regression layer that maps the 2048-dimensional input to a 128-dimensional intermediate layer through a fully connected layer, and then directly regresses the 6-degree-of-freedom head rotation angles (pitch, yaw, roll, and translation) through an output fully connected layer. The gaze direction prediction head is also a regression layer containing a fully connected layer that maps the backbone features to 128 dimensions, and then outputs a 6-dimensional vector through a final fully connected layer. This vector is then split into two 3-dimensional vectors and subjected to L2 norm normalization to obtain a normalized 3D gaze vector representing the gaze direction. Therefore, for each video frame, the face analysis branch outputs a 29-dimensional frame-level feature vector, composed of 17-dimensional AU intensity, 6-dimensional head pose parameters, and 6-dimensional gaze vector. In parallel, each original frame image is also fed into the pose analysis branch. This branch employs a heatmap-based convolutional neural network model, similar in structure to a stacked hourglass network. This network consists of multiple hourglass modules connected in series, each performing a feature extraction process from high resolution to low resolution and back to high resolution. This structure allows it to simultaneously capture both local and global body structure information. The network's weight parameters are pre-trained on a large human pose dataset. After obtaining the keypoint coordinate sequence for each frame, further kinematic calculations are performed to extract more dynamically meaningful features.For example, the angle between the line connecting the shoulders and the horizontal line is calculated to characterize the torso tilt; the movement speed, acceleration, and total trajectory length of the wrist keypoints in space are calculated; and the swaying amplitude and frequency of the body's center of mass (estimated from the torso keypoints) are calculated. This extracts approximately 40 dynamic posture features with clear physical meaning from the original coordinates of 33 keypoints. For each frame in the video stream, a 29-dimensional facial feature vector and a 40-dimensional posture feature vector are obtained. These two vectors are concatenated to form a comprehensive 69-dimensional frame-level feature vector. This vector sequence completely records the user's visual behavior at every moment during the evaluation process. Finally, to capture the evolutionary patterns of these behaviors over time, the time sequence of this 69-dimensional feature vector is input into a Gated Recurrent Unit (GRU) network. This GRU network has a two-layer structure, with each layer containing 128 hidden units, and employs a bidirectional processing mechanism to simultaneously capture past and future contextual information. The weight parameters of the GRU network are obtained through end-to-end training on the aforementioned internal elderly cognitive assessment dataset using a supervised learning task. After processing the entire video sequence, the hidden state vectors output by the last time step of the bidirectional GRU network on the forward and backward paths are concatenated to form a 256-dimensional (128+128) floating-point vector. This vector is the video feature sub-vector, which is a highly condensed spatiotemporal information representation that comprehensively summarizes all key visual behaviors related to cognitive evaluation in the original video stream.

[0027] Next, proceed to step 22. First, the raw audio stream is input to the speech feature extraction module. The first stage of the module is signal preprocessing. The raw audio signal is first pre-emphasized by a first-order high-pass filter, the transfer function of which is set as H(z) = 1 - 0.97z. -1To enhance the energy of the high-frequency components of the signal and compensate for the natural spectral tilt during speech production, the pre-emphasized signal is then segmented into continuous, overlapping short frames. Each frame is set to a length of 25 milliseconds, a widely adopted duration in speech signal processing that ensures short-term stationarity within the frame. The frame shift is set to 10 milliseconds, meaning a 15-millisecond overlap between adjacent frames ensures a smooth transition of information between frames. Finally, a Hamming window function is applied to each segmented frame to reduce potential spectral leakage during subsequent Fourier transforms. Specifically, the values ​​for these parameters are set according to widely adopted and empirically validated industry standards in the field of speech signal processing. The second stage of the module is frame-level acoustic feature extraction, which involves calculating a set of low-level acoustic descriptors (LLDs) for each pre-processed frame. This set of descriptors mainly includes the following categories: firstly, Mel-frequency cepstral coefficients (MFCCs), used to characterize timbre. For each frame of the signal, a Fast Fourier Transform (FFT) is performed to obtain its spectrum. The spectral energy is then passed through a Mel filter bank containing 40 triangular filters, the logarithm is taken, and finally a Discrete Cosine Transform is performed. The first 13 coefficients are taken as the MFCCs for that frame. Next are prosodic features, mainly the fundamental frequency and energy. The fundamental frequency is calculated using an algorithm based on the autocorrelation function, reflecting the pitch; the energy is the sum of the squares of the sample values ​​of the signal in that frame. Thirdly, there are sound quality features, including the harmonic noise ratio, and perturbations and jitter that characterize small disturbances in the fundamental frequency and amplitude. The third stage of the module is feature aggregation. Since frame-level features are time-series sequences, to obtain a single vector that can represent the entire audio file or a meaningful segment (such as a single response), the time series of these LLDs need to be statistically aggregated. For the time series of each type of LLD, such as the fundamental frequency value sequence of all frames in the entire audio file, a set of preset statistical functions are applied for calculation. This set of statistical functions includes, but is not limited to, mean, standard deviation, kurtosis, skewness, maximum value, minimum value, and extreme value range. For example, by calculating the mean and standard deviation of the fundamental frequency sequence, the average pitch and intonation of the speech segment can be obtained. In addition, some macroscopic features are calculated, such as speech rate, which is determined by using a speech activity detector (VAD) to identify effective vocal segments and calculate the number of syllables per unit time; and the silent period ratio, i.e., the percentage of silent time to total duration. Finally, the features obtained by applying all statistical functions to all LLDs are aggregated to form a high-dimensional feature set. To generate a feature vector with moderate dimensionality and high information density, this high-dimensional set is input into a pre-trained principal component analysis (PCA) model for dimensionality reduction. The weights of this PCA model are trained on a large internal corpus containing the speech of thousands of elderly people.The PCA model linearly projects the high-dimensional feature vectors into a 128-dimensional latent space, thus obtaining the audio feature sub-vectors that characterize the core acoustic information of the original audio stream.

[0028] Next, step 23 is performed. The first stage is automatic speech recognition. The original audio stream file is input into a pre-trained, end-to-end acoustic model based on the Transformer architecture. This model consists of an encoder and a decoder. The encoder consists of multiple stacked convolutional layers and self-attention layers. Its function is to receive the Mel spectrogram of the audio signal as input and learn to extract a high-dimensional acoustic representation sequence that can represent the audio content. The decoder also consists of multiple stacked self-attention layers and cross-attention layers. It uses the encoder's output as contextual information and autoregressively generates the most probable sequence of text characters. The model's weight parameters are pre-trained on a large corpus containing tens of thousands of hours of diverse speech data. For example, when the input is audio recorded in a speech fluency task containing content such as "tiger, lion, um... elephant...", the model outputs the corresponding speech recognition result, i.e., the string: "tiger lion um elephant". The second stage is text feature extraction. After obtaining the text transcript, it is processed to extract two types of key features: linguistic features and deep semantic features. First, linguistic features are extracted. The transcribed text undergoes preprocessing, including word segmentation and punctuation removal. Subsequently, a series of quantifiable linguistic metrics are calculated. For example, lexical richness is calculated by the ratio of unique words to the total vocabulary (type-tag ratio); lexical density is calculated as the proportion of content words (nouns, verbs, etc.) to the total number of words; and speech non-fluency metrics are calculated by precisely counting the frequency of filler words such as "um," "that," repetitions, and self-correcting phrases. These features directly reflect an individual's cognitive efficiency in vocabulary extraction and language organization. Next, deep semantic features are extracted. To capture the semantic coherence and logic hidden behind the text, the preprocessed text is input into a pre-trained Chinese Bidirectional Transformer Encoder (BERT) model. This model is a deep neural network consisting of 12 stacked Transformer encoder layers. Its weights are pre-trained on a massive Chinese text corpus containing billions of words through a self-supervised task. During processing, the text sequence is tagged with special [CLS] and [SEP] tags and converted into corresponding word vectors before being input into the BERT model. The transformation process involves segmenting the text sequence into a series of tokens using a predefined vocabulary. Each token corresponds to a word embedding vector obtained from a lookup table in a pre-trained model. Secondly, to enable the model to understand word order, a unique, learned positional embedding vector is added to each token. Finally, a segment embedding vector to distinguish different sentences is superimposed. The element-wise summation of these three embedding vectors constitutes the word vector. The model outputs a 768-dimensional high-dimensional vector representation of each word within its specific context. The 768-dimensional output vector corresponding to its special [CLS] token is taken as the aggregate semantic representation of the entire input text sequence.Finally, the vector composed of the linguistic features (such as lexical richness, number of non-fluent times, etc.) calculated above is concatenated with the semantic representation vector output by the BERT model to serve as a text feature. This vector integrates the surface statistical characteristics and deep semantic information of language.

[0029] Next, step 24 is performed. First, the raw gait data is loaded, containing a series of pressure matrices with precise timestamps. The first step in processing is footprint separation and calculation of the center of plantar pressure (CoP). For each timestamp of the pressure matrix, a connected component labeling algorithm is used to cluster all non-zero pressure sensor readings into one or two independent regions, i.e., footprints. By analyzing the centroid position of each footprint, it can be stably labeled as the left or right foot. Based on this, for each identified footprint, the precise CoP coordinates and total pressure value of that foot at that moment are calculated by weighted averaging of all sensor pressure values ​​within it. Thus, the original pressure matrix sequence is transformed into two parallel time series representing the CoP trajectories and total pressure values ​​of the left and right feet, respectively. Based on this, precise gait event detection is performed to construct a gait event sequence. Key events are identified by analyzing the first derivative of the total pressure value time series of each foot, i.e., the rate of change of pressure. Specifically, heel strike is precisely defined as the moment when the total pressure value starts from zero and its rate of change first exceeds a preset positive threshold, while the CoP position at this moment is within the last 25% of the footprint area to ensure heel-first landing. Conversely, toe-off is defined as the moment just before the total pressure value drops to zero, when its rate of change first falls below a preset negative threshold, and the CoP is within the first 25% of the footprint area. By scanning the data of the entire walking process, a precise event sequence can be generated, which records in detail the timestamps of all left and right heel strikes and toe-off events and their corresponding CoP spatial coordinates. In particular, the specific values ​​of the aforementioned preset thresholds are pre-calibrated based on statistical analysis of a large number of expert-annotated reference gait datasets, with the goal of optimizing event detection accuracy. After obtaining this structured gait event sequence, standardized gait parameters can then be calculated. These parameters are calculated entirely based on the temporal and spatial information in the event sequence. The first category is time parameters. For example, left-side stepping is obtained by calculating the difference between the timestamp of a right foot's heel strike and the immediately following timestamp of the left foot's heel strike; left-side gait cycle is obtained by calculating the difference between the timestamps of two consecutive left foot heel strikes. The duration of the left foot's stance phase is obtained by subtracting the heel strike timetamp of the same landing from its toe-off timetamp, while the duration of the bipedal stance phase is obtained by subtracting the heel strike timetamp of the other foot from the toe-off timetamp of the previous landing. The second category is spatial parameters. For example, left-side stride length is obtained by calculating the absolute value of the difference between the CoP ordinate of a right foot's heel strike and the CoP ordinate of the immediately following left foot heel strike; left-side stride width is the absolute value of the difference between the CoP ordinates of two consecutive left foot heel strikes. Stride width is the lateral distance between the CoP points of the left and right feet at the time of a heel strike event.The third category consists of derived parameters, such as gait speed, calculated by dividing the average of all stride lengths by the average of all gait cycles; cadence is converted to steps per minute by dividing 120 by the average gait cycle duration. Finally, to capture gait stability and symmetry, variability and symmetry parameters also need to be calculated. Variability is quantified by calculating the coefficient of variation (standard deviation) of a series of core parameters, such as gait cycle and stride length, across multiple repeated measurements, divided by the mean. A lower coefficient of variation indicates a more stable gait. Symmetry is measured by calculating the ratio of corresponding parameters between the left and right legs. For example, the stride length symmetry index is defined as the ratio of the left stride length to the right stride length; a value close to 1 indicates high symmetry. All calculated core gait parameters, including temporal, spatial, derived, variability, and symmetry parameters, are organized into a vector in a predetermined order, which is the gait feature sub-vector. This vector comprehensively and quantitatively describes the elderly individual's walking ability and pattern in this assessment.

[0030] Finally, step 25 is performed. In an optional implementation, step 25, which involves fusing the video feature vector, audio feature vector, text feature vector, and gait feature vector to obtain the current multimodal fusion feature vector, includes: step 251, concatenating the video feature vector, audio feature vector, text feature vector, and gait feature vector to obtain the current multimodal concatenated feature vector; and step 252, inputting the current multimodal concatenated feature vector into a feature dimensionality reduction layer to obtain the current multimodal fusion feature vector.

[0031] First, step 251 is executed. The four feature sub-vectors are arranged sequentially at the vector level according to a pre-defined order that remains constant across all evaluations, such as video-audio-text-gait, forming a new vector with a higher dimension, which is the multimodal concatenation current feature vector. Then, step 252 is executed. Since the concatenated multimodal fusion current feature vector has a high dimension, it may contain redundant cross-modal information and is not conducive to efficient subsequent similarity calculations; therefore, it needs to be refined and compressed. Structurally, this feature dimensionality reduction layer is a single-layer fully connected feedforward neural network without a non-linear activation function. The neurons in the input layer of this network perfectly match the dimension of the input concatenation vector; the number of neurons in its output layer is set to a predefined, lower dimension, such as 512. The core of this layer is a weight matrix and a bias vector. The parameters of the weight matrix and bias vector are obtained through supervised learning on a large, labeled multimodal cognitive assessment dataset of the elderly. When the current feature vector from the multimodal concatenation is input into this layer, it undergoes matrix multiplication with the weight matrix and is supplemented with a bias vector, thus being linearly mapped into a 512-dimensional latent semantic space. Following this linear transformation, an L2 norm normalization operation is performed, adjusting the magnitude of the output vector to 1. This 512-dimensional floating-point vector, after dimensionality reduction, refinement, and normalization, is the current feature vector from the multimodal fusion.

[0032] In step 3, a personal baseline profile is retrieved and loaded based on the user ID of the elderly person to be assessed. This profile includes the last updated baseline feature vector, the number of times the baseline has been updated, and a historical drift rate sequence. It is understandable that static, one-off assessments cannot capture the dynamic trajectory of individual cognitive changes, making it difficult to distinguish between normal physiological aging and pathological decline. Any meaningful longitudinal analysis presupposes the establishment of a dynamically evolving reference system specific to the individual. Therefore, accurately retrieving the aggregate of all past assessment information of the elderly person to be assessed from storage—the personal baseline profile—provides the necessary, personalized historical reference data for subsequent calculations of the deviation between the current state and the individual's own history, as well as for quantifying the rate of change.

[0033] In one alternative implementation, step 3 is implemented as follows: the unique identifier of the elderly person to be evaluated, i.e., the user ID, such as USER_001.

[0034] The user ID is used as a primary key for querying within a centralized, persistent database. This database contains a table named the Personal Baseline Profile, whose structure includes at least the following fields: user ID, the last updated baseline feature vector, the number of times the baseline has been updated, and the historical drift velocity sequence.

[0035] Upon receiving the user ID "USER_001", a query command is sent to the database. If the query is successful, the database returns a complete record associated with that ID, which constitutes the user's personal baseline profile. The personal baseline profile loaded into memory is a data structure containing the following specific data: First, a 512-dimensional floating-point vector, which is the baseline feature vector updated in the last time. This vector is a dynamically weighted average representation of the user's historical state. It is initialized during the first evaluation as the multimodal fusion current feature vector generated in that evaluation. In each subsequent update, it is obtained by weighting and fusing the old baseline feature vector with the newly generated multimodal fusion current feature vector using a preset update rule, such as exponential moving average, thus smoothly reflecting the individual's long-term trend. Second, an integer, such as 5, representing that the user's baseline profile has undergone 5 valid updates. This value is set to 1 during the initial evaluation. Afterward, each time the baseline feature vector is successfully updated according to preset conditions, this counter is automatically incremented by one, recording the number of iterations of the baseline model. Finally, there is a sequence containing four floating-point numbers, such as [0.012, 0.015, 0.013, 0.018], which is the historical drift velocity sequence. This sequence is empty during the first evaluation. Subsequently, whenever a new drift velocity is calculated in an evaluation and the baseline profile is updated, the newly calculated velocity value is appended to the end of this sequence, thus forming a time series recording the rate of change in an individual's cognitive state during each evaluation. It is worth noting that this personal baseline profile is not static but is created during the user's first evaluation. In the first evaluation, since there is no record for this user ID in the database, the query will return a null value. At this time, the current multimodal fusion feature vector generated in the first evaluation is used as the initial baseline feature vector, the baseline update count is recorded as 1, and the historical drift velocity sequence is empty, thus completing the initial creation of the personal baseline profile and storing it in the database. In each subsequent evaluation, this profile is loaded first and updated in later steps of the evaluation process based on the new evaluation results.

[0036] In step 4, a dissimilarity score is calculated based on the current feature vector fused from the multimodal fusion and the previously updated baseline feature vector in the individual baseline profile. It should be understood that after obtaining the fused feature vector representing the current state and the baseline feature vector representing the individual's historical norm, what is possessed are two coordinate points in a high-dimensional space. However, these two vectors themselves do not intuitively reveal the degree of change; they are merely two mathematical expressions. To achieve accurate quantification of cognitive state changes, the complex relationship between these two high-dimensional vectors needs to be transformed into a single scalar value with clear physical meaning and clinical interpretability. This value needs to be able to stably and reliably measure the deviation of the current state from the individual's own historical baseline. Therefore, this application introduces a standardized distance or dissimilarity metric to calculate the difference between the two feature vectors as a normalized dissimilarity score, thereby providing a core, quantifiable basic indicator for subsequent assessment of the stability and rate of change of cognitive states.

[0037] In an optional implementation, step 4, calculating the difference score based on the current feature vector fused from the multimodal fusion and the previously updated baseline feature vector in the personal baseline profile, includes: calculating the difference score based on the current feature vector fused from the multimodal fusion and the previously updated baseline feature vector in the personal baseline profile using the following formula: ;in, To fuse the current feature vector in a multimodal manner, This is the baseline feature vector from the last update. Let be the magnitude of the vector. This is the difference score. The core of the formula is to calculate the difference between the cosine similarity of 1 and two vectors. Cosine similarity measures the similarity in direction between two vectors by calculating the cosine of the angle between them in multidimensional space, and its value ranges from -1 to 1. The specific calculation process is broken down as follows: First, calculate the numerator of the formula, that is, the difference between the two vectors... and dot product ( This calculation is obtained by multiplying the corresponding elements of the two vectors one by one, and then summing all the multiplications. Secondly, the denominator of the formula is the product of the magnitudes of the two vectors. The magnitude of a single vector is obtained by calculating the square root of the sum of the squares of all its elements. It is worth noting that in the embodiment of step 2, the current feature vector from multimodal fusion undergoes L2 norm normalization before output, therefore its magnitude... It is always 1. Similarly, the baseline feature vector in the individual baseline profile also remains normalized with each update, therefore its magnitude is always 1. The denominator is also 1. In this preferred embodiment, the result of the calculation is 1 multiplied by 1, which equals 1. Next, the result of the dot product is divided by the result of the product of the moduli to obtain the cosine similarity between the two vectors. Since the denominator is 1, this value is equal to the value of the dot product. For example, if the calculated dot product is 0.998, the cosine similarity is also 0.998. This value is close to 1, indicating that the two vectors are very close in direction in the 512-dimensional space. Finally, the cosine similarity calculated in the previous step is subtracted from 1 to obtain the final difference score. Continuing with the previous example, =1 - 0.998 = 0.002. This score is a scalar value between 0 and 2. When A score of 0 indicates that the current feature vector is exactly the same as the baseline feature vector, and the cognitive state has not changed at all; the closer the score is to 0, the smaller the change. As the difference in the directions of the two vectors increases, The value of also increases accordingly.

[0038] In particular, when assessing the cognitive abilities of older adults, simply concatenating feature vectors from multiple modalities such as video, audio, and gait and directly comparing them with historical baselines, while reflecting changes in overall condition, overlooks a crucial dimension: the intrinsic coordination between different modalities. The health of higher cognitive functions, such as language fluency, emotional expression, and the consistency of body posture, is precisely reflected in the subtle and complex coordination patterns between these modalities. Early cognitive decline often does not manifest as a sharp drop in a single indicator, but rather as a disruption or mismatch in these coordination patterns—for example, positive verbal content but a blank facial expression. This subtle inconsistency is difficult to capture with simple vector distance calculations. Therefore, this application introduces a cross-modal collaborative attention mechanism that goes beyond simply comparing surface-level numerical values ​​of the state; instead, it deeply mines and quantifies the differences between the intrinsic coordination relationships of the current multimodal feature distribution and the stable coordination patterns in the individual's historical baseline. By generating a current feature representation deeply calibrated to the baseline, this method significantly improves the sensitivity to subtle early signs of cognitive decline and the accuracy of difference score calculations.

[0039] In a preferred implementation, step 4, based on the multimodal fusion current feature vector and the previously updated baseline feature vector in the individual baseline profile, calculates a difference score, including: performing cross-modal distribution attention association on the multimodal fusion current feature vector and the previously updated baseline feature vector to obtain an attention affinity matrix, i.e.: ;in, It is a multimodal fusion of the current feature vector. These are the baseline feature vectors from the last update; all vectors are in column vector form. It is matrix multiplication. It is a transpose operation. Represents one-dimensional convolution. It is subtracted based on position. This is the attention affinity matrix. It should be understandable that this is used to quantify the changes in multimodal coordination between the current state and the baseline state. Directly comparing two high-dimensional feature vectors cannot explicitly reveal how the statistical correlations between their internal modal features evolve. Therefore, a mathematical entity is needed to describe this structural change. First, the multimodal fusion current feature vector is calculated separately. And the baseline feature vector updated last time Covariance in column vector form This is used to extract the distributional correlations between features within each multimodal vector. Then, element-wise subtraction is performed. Calculate the difference between these two covariance vectors. This difference vector precisely captures how the coordination patterns within the multimodal features have changed from the baseline state to the current state. Finally, combine this difference vector with... The transpose of the matrix is ​​used for an outer product operation, projecting this structural change onto the feature space of the baseline to generate an attention affinity matrix. This matrix is ​​not a simple measure of difference, but a complex operator in which each element encodes the change in the strength of the association between the current feature and the baseline feature in a certain dimension.

[0040] The attention weight vector is obtained by calculating the attention weights of the current feature vector relative to the previously updated baseline feature vector based on the attention affinity matrix, i.e.: ;in, It is a dot product by position. To calculate the reciprocal of each eigenvalue, This is the attention weight vector. Correspondingly, after obtaining the attention affinity matrix describing the changes in the relationship, this matrix needs to be used for evaluation. The degree to which each dimension contributes to this deviation in relationship. In other words, it is necessary to determine which feature dimensions' changes are the key factors causing the current coordination pattern to deviate from the baseline pattern, and to give these dimensions greater attention. First, the attention affinity matrix... respectively with and Perform matrix multiplication. The result can be understood as highlighting the dimensions most relevant to changes in the relationship within the current feature vector. Similarly, This provides a reference value for this salience effect at the baseline state. Then, by performing element-wise Hadamard product multiplication of the former and the latter (element-1), the salience score of the current feature vector is normalized or scaled. The resulting attention weight vector... Each element value represents The importance or guiding score of the corresponding dimensional features in this baseline calibration process.

[0041] The attention weight vector is weighted by a learnable baseline coordination matrix shift to obtain an optimized multimodal fusion current feature vector, i.e.: ;in, It is a learnable baseline coordination matrix. This is the optimized multimodal fusion of the current feature vector. It should be understandable that the attention weight vector generated in the previous step... While indicating the focus, it is calculated entirely based on two specific vectors from this assessment and may contain noise or randomness. To make the weighting process more robust and clinically applicable, these on-the-fly calculated weights need to be projected into a more stable and generalizable reconciliation representation space.

[0042] In the above formula It is a key, pre-trained, learnable baseline coordination matrix, obtained by training on longitudinal datasets containing a large number of healthy individuals and patients at different cognitive stages. It learns and solidifies universal, healthy multimodal coordination patterns. In implementation, it is first achieved through matrix multiplication. The attention weight vector calculated in this operation Transfer or projection to by Within the defined general coordination space, a new set of weights, calibrated using expert knowledge, is obtained. Subsequently, this optimized set of weights is... Apply to the original multimodal fusion current feature vector Above, an optimized multimodal fusion current feature vector was obtained. This vector can be understood as the current state calibrated by the baseline coordination pattern. While retaining the current state information, the weights of its internal features have been adjusted to highlight those dimensions that deviate from the individual's health baseline coordination pattern in a clinically meaningful way. In other words, to enable direct measurement of the deviation through calculations such as distance, the current features need to lie within the projection distribution space of the baseline features under the baseline coordination pattern, based on attention weights, thus effectively measuring the difference between the baseline state and the actual current state. Therefore, a difference score is calculated between the optimized multimodal fusion current feature vector and the previously updated baseline feature vector in the individual baseline profile. This difference score can essentially be understood as how much the multimodal fusion features in the current state deviate from the baseline feature coordination pattern.

[0043] Based on the optimized multimodal fusion of the current feature vector and the previously updated baseline feature vector, the difference score is calculated, i.e.: ;in, This is the difference score. In other words, after the aforementioned transformation, we have obtained two vectors that can be directly and meaningfully compared within the same coordinated representation space: a baseline feature vector representing the stable, normal state of an individual. And the optimized feature vector representing the current state after baseline calibration. At this point, a final metric is needed to quantify the differences between them. This is achieved through calculation... and The final difference score is obtained by comparing the similarity between the vectors. Cosine similarity is often used because it is insensitive to the absolute magnitude of the vectors and focuses more on their orientation in high-dimensional space, which aligns well with the goal of comparing patterns rather than intensity. The range of cosine similarity is [-1, 1], with values ​​closer to 1 indicating greater similarity. Subtracting this similarity value from 1 yields a difference score within the range [0, 2]. A higher score indicates a greater deviation of the current state's multimodal coordination pattern from the individual's baseline pattern. The final result is an output scalar value. This score profoundly reflects the degree of dysfunction in higher-level integration functions. Compared to direct vector distance, it is more sensitive and accurate to subtle early signs of cognitive decline. Thus, the calculation of the difference score shifts from state comparison to relational comparison, measuring the difference between the coordination relationship between current modalities and the coordination relationship between baseline modalities. This is closer to the essence of cognitive decline—the dysfunction of higher-level integration functions. Furthermore, since early signs of cognitive impairment are often not a sharp decline in a single indicator, but rather subtle mismatches, such as speaking with positive content but lacking facial expression, cross-modal collaborative attention can capture these subtle cross-modal inconsistencies.

[0044] In step 5, the drift velocity is calculated based on the difference score and the historical drift velocity sequence. Correspondingly, while the difference score can quantify the instantaneous deviation of the current state from the historical baseline, it is essentially a static and isolated measure. It is sensitive to accidental fluctuations that may occur in a single assessment, such as performance decline due to temporary fatigue or poor mood, but cannot reveal the long-term trend and rate of change. A truly effective indicator to distinguish between normal aging and pathological decline requires not only knowing how much deviation has occurred, but also knowing at what rate the deviation is continuing. Therefore, this application calculates a smoother and more robust drift velocity by weighting and fusing the current difference score with the individual's historical rate of change, thereby upgrading the assessment from a static comparison of a point to a dynamic measure of a trend, providing a core basis for more accurately judging cognitive trajectories.

[0045] In an optional implementation, step 5, calculating the drift velocity based on the difference score and the historical drift velocity sequence, includes: calculating the drift velocity based on the difference score and the historical drift velocity sequence using the following formula: ;in, For the difference score, For historical drift velocity sequences, The weight hyperparameters used to control the current degree of dissimilarity. For calculation The mean, The drift velocity is the input to this embodiment. The input consists of two key data points: one is the difference score. According to the aforementioned example, its value is 0.002; the second is the historical drift velocity sequence loaded from the personal baseline profile of user USER_001, which, according to the aforementioned example, is a sequence containing four floating-point numbers: [0.012, 0.015, 0.013, 0.018]. In the above formula... This is the current drift velocity to be determined. The formula is essentially a variant of exponential smoothing, designed to utilize a weighted hyperparameter. To balance the current new information Historical trend information The impact of this. The specific calculation process is broken down as follows: First, it is necessary to determine the weight hyperparameters. The value of β, ranging from 0 to 1, is a pre-defined parameter that controls the weight of the current difference score in calculating the drift velocity. This value is set based on experience or through optimization on a validation dataset, aiming to ensure that the calculated drift velocity both sensitively reflects real changes and effectively filters out short-term noise. In this embodiment, It is set to 0.3. This means that when calculating the new drift velocity, the current difference score contributes 30%, while the historical average drift velocity contributes 70%, reflecting a greater emphasis on long-term trend stability. Secondly, the historical drift velocity sequence is calculated. The mean, i.e. This calculation is performed by summing all elements in the sequence and dividing by the number of elements. Based on the input historical drift velocity sequence [0.012, 0.015, 0.013, 0.018], its mean is (0.012 + 0.015 + 0.013 + 0.018) / 4 = 0.0145. Finally, the difference score is calculated. The value of the historical drift velocity sequence and the mean of the drift velocity sequence. and weight hyperparameters Substitute the value of into the formula to perform the calculation. The calculation process is as follows: =0.3*0.002+(1-0.3)*0.0145. This can be decomposed into a weighted sum of two parts: the first part is the contribution of the current difference, i.e., 0.3*0.002=0.0006; the second part is the contribution of the historical trend, i.e., 0.7*0.0145=0.01015. Adding these two parts together gives the final drift velocity. =0.01075. This calculated scalar value of 0.01075 is the drift velocity in this assessment. This value, by integrating the instantaneous deviation of the current state with the average trend of historical changes, ultimately represents the long-term evolutionary rate of an individual's cognitive state deviating from its stable baseline as a smooth and quantifiable rate index.

[0046] In particular, an individual's life is a dynamic process, and their physiological and cognitive states naturally evolve over time—a process known as normal aging. A cognitive baseline established at age 65, without adjustment, will no longer accurately reflect their current normal level by age 70. If an outdated, fixed baseline is continuously used as a reference, even the smallest changes caused by normal aging may be erroneously amplified, leading to misjudgments of the risk of cognitive decline. To ensure the long-term effectiveness and accuracy of longitudinal assessments, the reference standard itself must possess the ability to adaptively evolve. Therefore, further, when a significant and non-accidental deviation is detected, the individual's baseline profile should be carefully and smoothly updated, allowing it to dynamically and gradually follow the individual's real, long-term trajectory of change, thus maintaining its value as a valid reference point.

[0047] In one alternative implementation, when the difference score is lower than the normal fluctuation threshold, the individual baseline profile is updated based on the multimodal fusion of the current feature vector using the following formula: ;in, For smoothing factor, This is the baseline feature vector from the last update. To fuse the current feature vector in a multimodal manner, This is the currently updated baseline feature vector. To illustrate the specific process of updating the personal baseline profile in this invention, a preferred embodiment is provided below. The core of this update process is a conditional judgment logic, which first considers the currently calculated difference score... The result is compared to a preset threshold for normal fluctuations. This threshold is a key hyperparameter used to define the boundary between normal physiological fluctuations and potential abnormal changes. Its value is set based on statistical analysis of longitudinal data from a large-scale elderly population; for example, it can be set as the 95th percentile of the distribution of difference scores between two consecutive assessments in a healthy, stable elderly population, ensuring that the vast majority of daily fluctuations are included. In this embodiment, the normal fluctuation threshold is set to 0.05. The calculated difference score is then compared... =0.002 is compared with the threshold of 0.05. Since 0.002 is much smaller than 0.05, the condition is true, indicating that the changes reflected in this assessment are within the expected normal fluctuation range. Therefore, the baseline eigenvector update procedure will be initiated. The baseline eigenvector update adopts an exponentially weighted moving average (EWMA) strategy, strictly following the above formula. Here... It is a smoothing factor that determines the weight of the impact of new assessment data on the existing baseline. The value of is a decimal between 0 and 1. The smaller the value, the smoother the baseline update and the lower the dependence on the result of a single evaluation. In this embodiment, the smoothing factor... Setting it to 0.1 is a conservative choice, designed to ensure the long-term stability of the baseline. The specific update calculation is a vector operation. =0.1, and and Substitute these two vectors into the formula. The calculation process is as follows: Multiply each element of the vector by 0.1, and simultaneously... Multiply each element at the corresponding position in the vector by (1-0.1)=0.9, then add the corresponding elements of the two resulting new vectors to generate a new vector. This is the updated baseline feature vector, which largely retains historical baseline information while gently incorporating new information from this assessment. In contrast, the difference score calculated in another assessment... The value is 0.08. Since 0.08 is greater than the normal fluctuation threshold of 0.05, the condition judgment is false. This indicates that an abnormal event may have occurred. To prevent this potential, atypical state from contaminating the long-term stable baseline, the baseline feature vector will not be updated this time. That is, the new baseline feature vector... This will be directly equal to the old baseline feature vector. After the baseline feature vector is updated, other components of the individual baseline profile also need to be updated accordingly. In this embodiment where the baseline is below the threshold, the number of baseline updates is incremented by 1, changing from 5 to 6. Simultaneously, the current drift velocity calculated in step 5... The value 0.01075 will be appended to the end of the historical drift velocity sequence. Finally, this will contain the updated baseline feature vector. The updated number 6 and the expanded historical drift velocity sequence complete personal baseline profile are persistently stored back into the database, overwriting the original record associated with user ID "USER_001" for loading and use in the next evaluation.

[0048] In step 6, a cognitive assessment report is generated based on the difference score and drift velocity. That is, after the complex preceding calculations, although two core numerical indicators that can accurately quantify instantaneous changes (difference score) and long-term trends (drift velocity) have been obtained, these values ​​themselves are abstract and lack intuitive meaning for clinicians, caregivers, and even the elderly individuals themselves. They are mathematical symbols containing information, rather than conclusions that can be directly used for decision-making. To transform these objective, quantitative analysis results into qualitative judgments with clear clinical guidance that are easy to understand and communicate, a mapping bridge from numerical values ​​to semantics needs to be established. Therefore, this application ultimately translates these quantitative indicators into a standardized, hierarchical assessment language, ultimately generating a cognitive assessment report with a clear structure and explicit conclusions.

[0049] In one optional implementation Figure 4 This is a flowchart of step 6 in the multimodal fusion-based cognitive ability assessment method for the elderly according to an embodiment of this application. Figure 4 As shown, step 6 generates a cognitive assessment report based on the difference score and drift speed, including: step 61, converting the difference score into the current state risk level; step 62, converting the drift speed into a decline trend.

[0050] In the above feasible embodiment, step 6 is implemented as follows: This implementation process first executes step 61. This transformation process is completed through a preset, threshold-based rule mapping. This rule maps continuous difference scores... This is mapped to three discrete risk levels: low risk, intermediate risk, and high risk. These thresholds are determined based on statistical analysis of large-scale longitudinal clinical data, combined with the knowledge of clinical experts. For example, a specific threshold setting could be: when A value below 0.05 is mapped to low risk, indicating that the current state is highly consistent with the individual's baseline and is within the normal fluctuation range; when... When the value is between 0.05 and 0.15, it is mapped to medium risk, indicating a certain degree of deviation that requires attention; when... A score greater than or equal to 0.15 is mapped to high risk, indicating a significant deviation that may foreshadow an important state change. In this embodiment, the calculated difference score... The value is 0.002, which is less than the threshold of 0.05. Therefore, the corresponding current state risk level is determined to be low risk.

[0051] Then, step 62 is executed. This transformation process employs a rule-based mapping system that combines absolute thresholds with relative changes, aiming to provide more dynamic insights. This system first bases its conclusions on drift velocity. The absolute value of this value can be categorized into three basic trend types: stable, slow decline, and accelerating decline. For example, a specific threshold setting could be: when... When the baseline value is below 0.02, the baseline category is stable, indicating that the individual's cognitive state changes at an extremely low rate, consistent with a typical healthy aging pattern; when... When the value is between 0.02 and 0.08, the underlying class exhibits slow decay; when... A value greater than or equal to 0.08 indicates accelerated decay. However, this system further increases the current drift rate. This is compared to its historical average drift velocity, thus providing a more refined description of the base category. For example, if an individual's historical average drift velocity is 0.01, while the calculated drift velocity in this study... It rose slightly to 0.015. Although its absolute value remains within the stable range, considering its slight increase compared to its historical levels, the final downward trend will be described as stable with slight fluctuations. Conversely, if this... A value below its historical average can be described as stable, indicating a positive trend. In this embodiment, the calculated drift velocity... The value is 0.01075, which is less than the absolute threshold of 0.02, therefore its base category is stable. Furthermore, comparing this value with the historical average drift velocity of 0.0145 obtained from the individual baseline profile reveals that the current velocity is below the historical average. This indicates a slowdown in the rate of state change, a positive sign. Therefore, its corresponding final decline trend is determined to be stable, with a positive trend.

[0052] Finally, the qualitative conclusions obtained from the above transformation are integrated with the original quantitative indicators to generate the final cognitive assessment report. This report is a structured document, and its core content includes: basic user information such as user ID "USER_001", the date of this assessment, a summary of the quantitative results, and clearly lists the difference scores. =0.002, drift speed =0.01075), and the most important comprehensive assessment conclusion. The comprehensive assessment conclusion section will be presented in clear natural language text. This text presentation is achieved through a rule-based text generation module. This module has multiple preset text templates, each corresponding to a specific assessment result composed of a combination of the current state risk level and the decline trend. When generating the report, the corresponding text template will be precisely matched and called based on the two qualitative conclusions of low risk and stability, and a positive trend determined in steps 61 and 62. These templates are pre-written and reviewed by clinical experts based on the specific clinical significance of each combination of risk level and decline trend. Subsequently, specific placeholders in the template will be dynamically filled by the specific quantitative indicators calculated in this assessment, thereby ensuring that the final output assessment conclusion is consistent and professional in language expression, and accurately reflects the personalized results of each assessment in content. For example, for the assessment result of this embodiment, the called template will generate the following text after being filled: The results of this cognitive ability assessment show that the current state risk level is low risk, indicating that your comprehensive performance this time is highly consistent with your personal historical norms and is within the normal fluctuation range. Meanwhile, your cognitive status trend was assessed as stable and positive. This indicates that your rate of cognitive change is not only within a very low, stable range consistent with a healthy aging pattern, but also slower than your historical average rate of change, presenting a positive signal. Overall, no clinically significant signs of cognitive decline were found, and your overall condition shows a steady improvement. This report will serve as the final output, providing direct and clear evidence for subsequent health management and clinical decision-making.

[0053] In summary, the cognitive ability assessment method for the elderly based on multimodal fusion, as described in this application, is elucidated. It establishes and continuously updates a unique personal baseline profile for each elderly person, shifting the assessment reference standard from comparison with the group average to comparison with the individual's historical level. Specifically, firstly, the user's current multimodal data is acquired and fused into a current feature vector. Subsequently, instead of comparing it with a universal standard, the user's personal baseline profile is retrieved, and the difference score between the current state and their own historical baseline is calculated. More importantly, a core indicator, drift speed, is introduced, combining the current difference with the historical rate of change to quantitatively model the speed of change. By simultaneously analyzing the magnitude and speed of change, it is possible to effectively distinguish between gradual, low-speed normal aging and rapid, drastic pathological decline, thus accurately solving the key technical problem in the prior art where these two types of aging are easily confused.

[0054] Figure 5 This is a block diagram of a multimodal fusion-based cognitive ability assessment system for the elderly according to an embodiment of this application. Figure 5 As shown, the cognitive ability assessment system 100 for the elderly based on multimodal fusion according to an embodiment of this application includes: an elderly subject raw data acquisition module 110, used to acquire the raw video stream, raw audio stream, and raw gait data of the elderly subject to be assessed; a raw data multimodal feature fusion module 120, used to perform parallel modal feature extraction on the raw video stream, raw audio stream, and raw gait data to obtain a multimodal fusion current feature vector; a personal baseline profile loading module 130, used to retrieve and load a personal baseline profile based on the user ID of the elderly subject to be assessed to obtain a personal baseline profile, the personal baseline profile including the last updated baseline feature vector, the number of times the baseline has been updated, and a historical drift speed sequence; a difference score calculation module 140, used to calculate a difference score based on the multimodal fusion current feature vector and the last updated baseline feature vector in the personal baseline profile; a drift speed calculation module 150, used to calculate a drift speed based on the difference score and the historical drift speed sequence; and a cognitive assessment report generation module 160, used to generate a cognitive assessment report based on the difference score and the drift speed.

[0055] Here, those skilled in the art will understand that the specific operations of each step in the above-described multimodal fusion-based cognitive ability assessment system for the elderly have been referenced above. Figures 1 to 4 The description of the multimodal fusion-based cognitive ability assessment method for older adults is detailed here, and therefore, its repeated description will be omitted.

Claims

1. A method for assessing cognitive abilities in older adults based on multimodal fusion, characterized in that, include: Acquire the raw video stream, raw audio stream, and raw gait data of the elderly subjects to be assessed; Parallel modal feature extraction is performed on the original video stream, original audio stream, and original gait data to obtain the current feature vector for multimodal fusion; Based on the user ID of the elderly person to be assessed, a personal baseline profile is retrieved and loaded to obtain a personal baseline profile. The personal baseline profile includes the last updated baseline feature vector, the number of times the baseline has been updated, and the historical drift velocity sequence. Based on the current feature vector of the multimodal fusion and the previously updated baseline feature vector in the individual baseline profile, a difference score is calculated, including: performing cross-modal distribution attention association on the current feature vector of the multimodal fusion and the previously updated baseline feature vector to obtain an attention affinity matrix; calculating the attention weight of the current feature vector of the multimodal fusion relative to the previously updated baseline feature vector based on the attention affinity matrix to obtain an attention weight vector; weighting the current feature vector of the multimodal fusion based on a learnable baseline coordination matrix to obtain an optimized current feature vector of the multimodal fusion, wherein the learnable baseline coordination matrix is ​​obtained by training on a longitudinal dataset containing a large number of healthy individuals and patients at different cognitive stages; and calculating the difference score based on the optimized current feature vector of the multimodal fusion and the previously updated baseline feature vector. The drift velocity is calculated based on the difference score and the historical drift velocity sequence; A cognitive assessment report is generated based on the difference score and drift speed.

2. The method for assessing cognitive abilities in the elderly based on multimodal fusion according to claim 1, characterized in that, Parallel modal feature extraction is performed on the original video stream, original audio stream, and original gait data to obtain a multimodal fusion current feature vector. This includes: inputting the original video stream into a pre-trained face and pose analysis model to obtain video feature sub-vectors; inputting the original audio stream into a speech feature extraction module to obtain audio feature sub-vectors; extracting text features from the speech recognition results after performing speech recognition on the original audio stream; extracting gait feature sub-vectors from the original gait data; and fusing the video feature sub-vectors, audio feature sub-vectors, text features, and gait feature sub-vectors to obtain the multimodal fusion current feature vector.

3. The method for assessing cognitive abilities in the elderly based on multimodal fusion according to claim 2, characterized in that, The process of fusing video feature vectors, audio feature vectors, text features, and gait feature vectors to obtain the current multimodal fusion feature vector includes: concatenating the video feature vectors, audio feature vectors, text features, and gait feature vectors to obtain the current multimodal concatenated feature vector; and inputting the current multimodal concatenated feature vector into a feature dimensionality reduction layer to obtain the current multimodal fusion feature vector.

4. The method for assessing cognitive abilities of the elderly based on multimodal fusion according to claim 1, characterized in that, The drift velocity is calculated based on the difference score and the historical drift velocity sequence, including: calculating the drift velocity using the following formula based on the difference score and the historical drift velocity sequence, wherein the formula is: ； in, For the difference score, For historical drift velocity sequences, The weight hyperparameters used to control the current degree of dissimilarity. For calculation The mean, This represents the drift speed.

5. The method for assessing cognitive abilities of the elderly based on multimodal fusion according to claim 1, characterized in that, Based on the difference score and drift velocity, a cognitive assessment report is generated, including: converting the difference score into the current state risk level; and converting the drift velocity into a decline trend.

6. The method for assessing cognitive abilities of the elderly based on multimodal fusion according to claim 4, characterized in that, Also includes: When the difference score is lower than the normal fluctuation threshold, the individual baseline profile is updated based on the multimodal fusion of the current feature vector using the following formula: ； in, For smoothing factor, This is the baseline feature vector from the last update. To fuse the current feature vector in a multimodal manner, This is the currently updated baseline feature vector.

7. A cognitive ability assessment system for the elderly based on multimodal fusion, characterized in that, include: The module for acquiring raw data of elderly subjects is used to acquire the raw video stream, raw audio stream, and raw gait data of the elderly subjects to be assessed. The raw data multimodal feature fusion module is used to extract modal features in parallel from the raw video stream, raw audio stream, and raw gait data to obtain the current multimodal fusion feature vector; The personal baseline profile loading module is used to retrieve and load personal baseline profiles based on the user ID of the elderly subject to be assessed in order to obtain a personal baseline profile. The personal baseline profile includes the baseline feature vector of the last update, the number of times the baseline has been updated, and the historical drift velocity sequence. The difference score calculation module is used to calculate the difference score based on the current feature vector of multimodal fusion and the previously updated baseline feature vector in the individual baseline profile. This includes: performing cross-modal distribution attention association on the current feature vector of multimodal fusion and the previously updated baseline feature vector to obtain an attention affinity matrix; calculating the attention weight of the current feature vector of multimodal fusion relative to the previously updated baseline feature vector based on the attention affinity matrix to obtain an attention weight vector; weighting the current feature vector of multimodal fusion based on a learnable baseline coordination matrix to obtain an optimized current feature vector of multimodal fusion, wherein the learnable baseline coordination matrix is ​​obtained by training on a longitudinal dataset containing a large number of healthy individuals and patients at different cognitive stages; and calculating the difference score based on the optimized current feature vector of multimodal fusion and the previously updated baseline feature vector. The drift velocity calculation module is used to calculate the drift velocity based on the difference score and the historical drift velocity sequence. The cognitive assessment report generation module is used to generate cognitive assessment reports based on difference scores and drift speed.

Images