Multi-modal objective scoring method, system and device for cognitive scale

Abstract

The invention provides a multi-modal objective scoring method, system and device for a cognitive scale, and relates to the technical field of cognitive evaluation. The multi-modal objective scoring method for the cognitive scale comprises the following steps: continuously acquiring audio stream, video frame, touch / handwriting trajectory and hand action trajectory data; nTP / system clock alignment and soft time mark fusion are adopted, audio, video and touch control are unified to the same time axis, different feature data are extracted from data of different modes, task objective scores and comprehensive scores are output, weighted fusion is carried out through a learnable weight matrix according to the hierarchical structure of task-dimension-scale total score, and the scale total score is obtained. According to the method, through a complete technical system of multi-modal data acquisition, time synchronization, feature extraction, task regression scoring, scale fusion and full-process auditing, fine quantification and objective evaluation of behaviors of subjects are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cognitive assessment, and in particular to a multi-modal objective scoring method, system and device for cognitive scales. BACKGROUND

[0002] In clinical cognitive assessment, paper or semi-electronic standardized scales (such as MMSE, MoCA, ADAS-Cog, etc.) are commonly used in primary care and hospitals in China to screen and re-evaluate subjects. The evaluation process relies on the real-time observation and subjective judgment of the evaluators, and there are problems of large subjective scoring, low consistency between evaluators, unplayable process and difficult post-recheck. Due to the limited data dimensions (mostly oral questions and simple operations), it is difficult to form quantifiable indicators for complex tasks such as drawing, connecting lines and action execution, resulting in insufficient clinical repeatability.

[0003] With the development of speech recognition, computer vision, multi-modal learning and edge inference technology, electronic evaluation systems have begun to introduce speech recognition and simple automatic scoring. However, existing systems generally remain at the "single modal" or "weak fusion" stage, lacking an engineering solution for time synchronization, robust feature extraction and hierarchical fusion of multi-modal data (speech, image / video, touch / penmanship). There is also a lack of auditable, interpretable scoring logic and confidence interval evaluation mechanism.

[0004] In terms of regulation and application demands, medical institutions have higher requirements for data traceability, quality control and compliance traceability, and the government and industry need a standardized, replicable and scalable objective scoring technology for cognitive screening in community, elderly care and family settings, which can maintain stable performance and repeatability in different populations (dialects, low education level, mobility difficulties, etc.). SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides a multi-modal objective scoring method, system and device for cognitive scales. The present application is aimed at clinical cognitive impairment screening, primary medical health management, long-term care evaluation in old-age institutions, and self-cognitive monitoring in families, etc. In a hospital environment, the present method can be embedded in the outpatient departments of neurology, geriatrics, psychiatry, etc. for preliminary screening, re-evaluation, and longitudinal tracking, to improve the objectivity, consistency and traceability that paper quality scales cannot achieve. In a community health service center or home environment, the present method can be run through an all-in-one machine, a tablet or an embedded device to assist medical staff in carrying out rapid screening of cognitive function for a large number of elderly groups, to realize high-throughput and low-labor-cost standardized evaluation. In an old-age institution, the present method can be used to perform quarterly or annual health evaluation of hospitalized elderly people, to realize visualization of the trend of cognitive ability changes and to provide a basis for adjusting the care plan. In a family scenario, the present method can support the elderly to monitor their own cognitive condition and prompt them to seek medical help when the deviation exceeds a threshold. Through multi-modal behavior perception, robust feature extraction, objective scoring and interpretable output, the present application can maintain stable technical performance in different populations and different scenarios, and significantly improve the scientific nature, efficiency and continuity of cognitive screening and evaluation.

[0006] In a first aspect, the present application provides a multi-modal objective scoring method for cognitive scales, comprising the following steps:

[0007] S1, multi-source data acquisition: under the triggering of a scale question controller, continuously acquire audio stream, video frame, touch / pen trajectory, and hand action trajectory, and record unified reference clock timestamp and question event to form a task event log;

[0008] S2, time synchronization and segmentation: adopt NTP / system clock alignment + soft timestamp fusion to establish a multi-modal mapping function, unify audio, video and touch to the same time axis to ensure alignment of voice, action, image and trajectory; form question segments and key frame points through question event slicing;

[0009] S3, feature extraction, extract different feature data for different modal data;

[0010] S4, intra-task objective scoring: input the normalized feature vector into a robust scorer to output a task objective score, then perform comprehensive scoring to generate a comprehensive score; use confidence to screen the comprehensive score, and automatically reduce the weight of low-quality features or trigger re-measurement;

[0011] S5, scale domain hierarchical fusion: according to the hierarchical structure of task→dimension→scale total score, use a learnable weight matrix for weighted fusion to perform comprehensive scoring in a progressive manner, and finally obtain a scale total score; use expectation filling / feature weight reduction strategy when encountering missing features to ensure scoring stability and fairness;

[0012] S6. Report Generation: Outputs doctor / patient version reports, which can be connected to HL7 / FHIR interfaces or HIS / EMR systems;

[0013] S7. Audit Data Recording: Save the process data recorded in the scale for review, playback and quality control purposes.

[0014] As a further improvement to the present invention, the data collected in step S1 includes:

[0015] Video data: facial key points, hand key points, ROI (Region of Interest) for drawing, and outline of the drawing trajectory lines;

[0016] Audio data: speech rate, phrasing, keywords, logical coherence;

[0017] Touch / handwriting data: coordinate sequence, pressure, speed, pen lift / put down events;

[0018] Task event data: Question start / end, number of prompts, skipped questions, and subject response time.

[0019] As a further improvement to the present invention, step S2 includes:

[0020] S21. Time Synchronization: Based on the hardware clock, NTP assisted and / or PTP, or NTP assisted and / or hard trigger, are used to align the clocks of voice, video, and touch data; AI alignment is used to match soft timescales using sound abrupt changes and motion keyframes.

[0021] S22. Segmentation: The problem is segmented according to the events in the problem; the events in the problem include: drawing segment, recall segment, language retelling segment, and command execution segment. Each segment has a timestamp interval.

[0022] As a further improvement to the present invention, step S21 specifically includes: first, using AI to find the start and end / abrupt point of the sound and the key frame of the action as soft time stamps, then matching these soft time stamps with the prompts in the question as nearest neighbors, then using weighted straight line correction to eliminate abnormal matches and retain high-confidence matches, aligning the audio, video, and touch to the same time line; at the same time, correcting the drift time through timed detection and realignment to ensure that the synchronization time deviation does not exceed the threshold.

[0023] As a further improvement to the present invention, the segmentation method in step S22 includes:

[0024] Drawing segment: By setting a threshold for the time interval between putting down and lifting the pen, the continuous operation process of the user drawing / connecting / tracing on the screen is determined;

[0025] Recall Fragments: Based on the simultaneous capture of thinking facial expressions / postures and hand / head movements, determine the silent thinking process of the user in order to answer the question;

[0026] Language repetition segment: Determine the continuous speech process of a user's verbal response / repetition by using a speech pause time threshold;

[0027] Command execution segment: By detecting the movement / stability of key points of the hand / upper limb and the achievement of the command target, the process of the user performing the operation according to the command is determined.

[0028] As a further improvement to the present invention, the feature data extracted in step S3 includes:

[0029] Speech feature data: speech rate, pause rate and pause length, proportion of silent segments, pitch and pitch stability, ASR recognition accuracy, keyword hit rate, pronunciation clarity, MFCC, and speech rate deviation percentage;

[0030] Drawing / touch feature data: contour similarity, stroke order consistency, pen speed curve, offset rate, handwriting continuity, overlap rate, spatial offset heatmap, and completion time;

[0031] Action / gesture / facial feature data: hand / fingertip / facial key point trajectory, trajectory completeness, hand trajectory smoothness, posture change range, blink and gaze deviation rate, micro-expression frequency.

[0032] As a further improvement to the present invention, the feature data extracted in step S3 specifically includes:

[0033] Contour similarity: IoU figure contour overlap, or Hausdorff / Chamfer / Fréchet distance;

[0034] Pen speed curve: A curve formed by the time series of pen tip movement speed, and the mean / peak / standard deviation of the speed series are calculated;

[0035] Offset rate: the difference in distance from the center line;

[0036] Handwriting continuity: number of skipped strokes and breaks in the pen;

[0037] Hand trajectory smoothness: acceleration variance;

[0038] Gaze deflection rate: The frequency of gaze direction deviation angles greater than 15°.

[0039] As a further improvement to the present invention, step S4 includes:

[0040] S41. Obtain normalized feature vector: Extract the feature data obtained in step S3. Each feature corresponds to a one-dimensional feature vector. Perform unit and data direction unification, quality screening and noise reduction, missing / low-quality processing, and robust standardization on each feature vector. Then, concatenate them to obtain the normalized feature vector.

[0041] S42. Generate individual scores for each task category: Build robust scorers using Huber regression / quantile regression / lightweight Transformer / random forest / GBDT;

[0042] S43. Generate a comprehensive score: Use Huber Regression / L1-L2 hybrid regression to obtain a comprehensive score based on individual scores; reduce the impact of outliers by using Huber loss or truncated averaging; use SNR estimation and feature weighting to deal with noise.

[0043] S44. Confidence Level Filtering: Use confidence level to filter the comprehensive score. If the comprehensive score does not fall into the confidence interval, it is judged as low quality, and its corresponding features are automatically downweighted or a retest is triggered. If the result of the retest is judged as low quality again, the result is retained.

[0044] S45. Threshold control: By setting and controlling thresholds, judgments are made on threshold-related items or weight reduction / retesting operations are initiated.

[0045] As a further improvement to the present invention, the method for generating individual scores in step S42 is as follows:

[0046] Obtain the normalized feature vector corresponding to the question event, and select the feature subset corresponding to the task type family;

[0047] For each task type family, train the robust scorer and output a normalized continuous score;

[0048] The normalized continuous score is proportionally restored to the original continuous score value of the question, or the corresponding score level is obtained by segmentation.

[0049] Quality control is performed using confidence level / CI, and low-quality features are downweighted by masking, or retested or re-verified.

[0050] As a further improvement to the present invention,

[0051] The calculation method of Huber Regression / L1-L2 mixed regression:

[0052] ;

[0053] Where: Score is the overall rating. For eigenvalues, Let b be the learning weight and b be the regression constant.

[0054] Use Huber loss to fit the weights ω i With a constant term b, the influence of outliers is automatically reduced; at the same time, L1–L2 regularization is used to prevent overfitting.

[0055] Confidence interval calculation method: Based on a preset sample, the calculation is performed using a bootstrap or Bayesian posterior distribution with at least 1000 iterations.

[0056] ;

[0057] Among them, CI 95 This represents the 95% confidence interval, where μ is the sample mean and σ is the sample variance.

[0058] As a further improvement to the present invention, the method for handling outliers in step S43 includes:

[0059] Eigenvalue anomaly detection criteria: The absolute value of the standardized residual is greater than the outlier threshold of 3, or exceeds the threshold of the training set. The quantile band, or the quality weight obtained by mapping SNR / ASR confidence / IoU, is less than 0.3;

[0060] For outlier sample values, for samples where the Huber loss is greater than 1.5–2.0 times the residual standard deviation, a linear penalty is applied to automatically reduce the impact of the sample on weight learning.

[0061] As a further improvement to the present invention, in the retesting step S44, abnormal question tracing is performed, and the specific method is as follows:

[0062] The topics are sorted in descending order of influence based on the product of the weighted feature value and the quality assessment value. The current topics are excluded in order, and a new comprehensive technical score is given. If the change in the comprehensive score is greater than the threshold or the quality weight of the current topic is less than the threshold, the current topic is marked as an abnormal topic.

[0063] For abnormal questions, the weight is reduced or the test is retested. If the result of the retest is again judged as low quality, the result is retained and marked with low confidence.

[0064] As a further improvement to the present invention, the threshold control in step S45 specifically includes:

[0065] ASR recognition accuracy ≥ 0.85 → judged as correct;

[0066] ASR recognition accuracy < 0.6 → trigger retesting or downgrading;

[0067] IoU < 0.4 → Indicates a severe rendering anomaly;

[0068] IoU ≥ 0.8 → indicates that the plotted curves have consistent heights;

[0069] If DTW ≤ threshold, the order is deemed reasonable.

[0070] Voice pause threshold setting: 300–500ms;

[0071] Pause time > 600ms → Indicates abnormal speech fluency;

[0072] Silent segment percentage > 30% → Trigger retesting or demotion.

[0073] As a further improvement to the present invention, in step S5,

[0074] A task is the smallest assessment unit in a scale, including a question or a sub-section within a question;

[0075] A dimension is a cognitive domain defined by the scale, consisting of several tasks that point to the same cognitive domain.

[0076] The total score of the scale is the final score of the entire scale.

[0077] As a further improvement to the present invention, step S5 includes:

[0078] Weighted fusion uses a linear weighting method:

[0079] ;

[0080] Where σ is the normalization function, and Score final For the total score of the scale, W task Score represents the task weight. task Let b be the task score and b be the linear regression constant.

[0081] The method for feature weight reduction is as follows:

[0082] ;

[0083] in, Wtask represents the weight after weight reduction, Wtask represents the weight before weight reduction, and MissingRate represents missing data.

[0084] As a further improvement to the present invention, the report in step S6 includes: total score, dimensional score, key indicators, confidence interval, abnormal behavior prompts, time usage statistics and task duration analysis, and evaluation quality score.

[0085] As a further improvement to the present invention, the audit data in step S7 includes: question sequence, number of prompts, reason for jump, complete audio and video trajectory, feature value and weight of each step, heatmap of feature weight and importance of each modality, timeline events and logs.

[0086] In a second aspect, the present invention provides a multimodal objective scoring system for cognitive scales to implement the steps of the method described in the first aspect; the system consists of a terminal data acquisition and edge inference subsystem and a central scoring and reporting subsystem; the hardware includes a high-definition camera / depth camera, a microphone array, and a touch screen or stylus input module; the software includes a data acquisition service, a time synchronization and segment annotation service, a voice / graphics / action feature extraction engine, an in-task scorer, a scale domain fusion processor, a reporting and auditing service, and a privacy and access control module.

[0087] Thirdly, the present invention provides an embodiment of a multimodal objective scoring device for a cognitive scale to implement the steps of the method described in the first aspect; the device includes: a data acquisition module, a synchronization and segment annotation module, a speech / drawing / action feature extraction module, a task scoring module, a scale fusion module, a report generation module, and an auditing and security module; the device is deployed on an all-in-one machine or adopts an end + cloud collaborative architecture.

[0088] Fourthly, the present invention provides a computer program product that, when executed by a processor, implements the steps of the method described in the first aspect.

[0089] This invention proposes a comprehensive solution for "multimodal objective scoring" for cognitive scale tasks, achieving: (1) multi-source data acquisition and millisecond-level time synchronization; (2) robust feature extraction of voice / image / action / touch; (3) in-task objective scoring and confidence interval estimation; (4) scale domain-level fusion and automatic reporting; and (5) auditable and interpretable output throughout the entire process. This solution significantly improves scoring consistency, efficiency, and verifiability without altering the medical connotations of the scales.

[0090] Specifically, this invention proposes a complete technical system encompassing "multimodal data acquisition + time synchronization + feature extraction + task regression scoring + scale fusion + full-process auditing," achieving precise quantification and objective assessment of subject behavior. By fusing multi-source information such as images, voice, actions, and trajectories, this invention significantly improves assessment accuracy and stability; through 25ms synchronization, 95% confidence interval estimation, and anomaly triggering mechanisms, it enhances the reliability of output results; through a hierarchical fusion model, it maps task performance to scale scores, improving cross-scenario consistency; and through auditing links and interpretability mechanisms, it meets the healthcare industry's requirements for quality control, traceability, and compliance.

[0091] This invention addresses the problems of existing cognitive assessment methods, which mainly rely on manual observation and paper-and-pen records, resulting in strong subjectivity, low cross-assessor consistency, difficulty in quantification, lack of traceability, and the fact that existing electronic solutions generally only use single-modal data, such as speech recognition or simple image comparison, which is difficult to support multimodal complex tasks in MoCA / MMSE involving drawing, performing actions, naming, and language repetition. It solves the core technical challenges of cognitive assessment being "unquantifiable, unverifiable, and lacking objectivity." Attached Figure Description

[0092] Figure 1 This is a flowchart of a multimodal objective scoring method for a cognitive scale disclosed in this invention.

[0093] Figure 2 This is a diagram illustrating the overall architecture of a multimodal objective scoring system for cognitive scales, as disclosed in this invention. Detailed Implementation

[0094] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Steps S1, S2… in the described embodiments of the present invention do not limit the scope of execution of the present invention; the various models, simulation environments, and software described in the present invention are not considered as the only limiting methods of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0095] In this invention, computer device / equipment / system refers to a related entity applied to a computer, such as hardware, a combination of hardware and software, software, or software in execution. More specifically, for example, software includes, but is not limited to, a process running on a processor, a processor, an object, executable software, an execution thread, a program, and / or a computer. Furthermore, an application program or script running on a server, and the server itself, can also be software. One or more software programs may be in an execution process and / or thread, and the software may be localized on one computer and / or distributed across two or more computers, and may be run on various computer-readable media.

[0096] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0097] In a first aspect, the present invention provides an embodiment of a multimodal objective scoring method for cognitive scales, such as... Figure 1 As shown, the specific process can be as follows:

[0098] In the context of conducting scale tests on subjects, the designed scale consists of several items, which are divided into different modalities, including those requiring verbal responses, writing, drawing, and action.

[0099] S1. Multi-source data acquisition: Under the trigger of the question controller, continuously acquire audio stream (16–48kHz), video frames (≥25fps), touch / handwriting trajectory (≥60Hz), and hand movement trajectory, and record a unified reference clock timestamp and question events (start / end / prompt) to form a task event log.

[0100] The specific steps and data collection requirements / content include:

[0101] S11. Video capture:

[0102] Resolution: ≥1080p; Frame rate: ≥25fps;

[0103] Acquisition angle: The camera and screen are at a fixed angle of 20°–35°;

[0104] Camera distance: 40–70 cm;

[0105] Lighting requirement: ≥300 lx;

[0106] The collected content includes:

[0107] Facial key points (68 points); Hand key points (21 points);

[0108] The region of interest (ROI) for drawing; the outline of the drawing trajectory lines;

[0109] S12, Audio Acquisition:

[0110] Sampling rate 16kHz, 24bit; microphone array SNR ≥ 60 dB;

[0111] Automatic noise reduction (DNS / Wiener) to remove background noise;

[0112] The data collected includes: speech rate, phrasing, keywords, and logical coherence;

[0113] S13, Touch / Handwriting Capture:

[0114] Sampling frequency ≥ 60Hz;

[0115] The collected data includes: coordinate sequence, pressure, speed, and pen lift / put down events;

[0116] Accuracy: ≤2mm;

[0117] S14, Task Event Collection:

[0118] The task is executed in a flow according to the scale, with each scale item corresponding to a task event.

[0119] Record the start / end of the question, the number of prompts, skipped questions, and the subject's response time.

[0120] S2. Time Synchronization and Segmentation: NTP / system clock alignment + soft time stamp fusion is adopted to establish a multimodal mapping function, unifying audio, video, and touch on the same timeline to ensure the alignment of voice, action, image, and trajectory; "question fragments" and "key frames" are formed by slicing question events.

[0121] The specific steps include:

[0122] S21. Time Synchronization: Employs a multi-source synchronization mechanism, including:

[0123] Under the condition of a unified reference clock (system clock), auxiliary NTP alignment is used to align the clocks of multimodal data such as voice and video with the hardware clock.

[0124] Optionally, regarding the synchronization mechanism settings, the NTP alignment mechanism can be replaced or superimposed with PTP / hard trigger lines.

[0125] AI alignment: matching soft timescales using abrupt changes in sound and keyframes of motion;

[0126] The specific method is as follows: First, use AI to find the start and end / abrupt points of the sound and the key frames of the action as soft time markers. Then, perform nearest neighbor pairing between these soft time markers and the prompts in the question. Next, use weighted straight line correction to correct the time axis to remove anomalies and retain high confidence, so that the audio, video and touch are all aligned to the same timeline.

[0127] To control the synchronization error to ≤25ms, the specific steps include:

[0128] (1) Table alignment: Align the table with the standard time before the test begins; the standard time is the time calibrated with NTP and / or PTP / hard trigger.

[0129] (2) Capture positioning points: During the test, the system automatically finds obvious nodes such as the start and stop of sound and the start / end of action and uses them as positioning points.

[0130] (3) Measurement and calibration: The time is fine-tuned approximately every half minute using these positioning points to correct the drift time.

[0131] (4) Automatic over-threshold correction: Once the synchronization time deviation is found to exceed the set threshold (e.g., 25ms), the system will automatically re-align and prompt manual inspection if necessary.

[0132] Throughout the entire process of synchronization error control, the time anchors used, the number of calibrations performed, and the maximum deviation were recorded.

[0133] In one embodiment of the present invention, the average error throughout the process is ≤25ms, and in most cases it does not exceed 30ms.

[0134] S22. Segmentation: The problem is segmented based on the events in the problem. The events in the problem include: drawing segment, recall segment, language retelling segment, and command execution segment. Each segment has a timestamp interval.

[0135] Specific segmentation methods include:

[0136] (1) Drawing fragment:

[0137] Definition: The continuous process of a subject drawing / connecting lines / tracing on a screen.

[0138] Start: After the question instruction is read out, a pen / touch button is detected or the stylus touches the screen for the first time, accompanied by handwriting.

[0139] End: No new handwriting appears for ≥500ms after the pen is lifted / touch is released; or the subject clicks "Complete / Next Question".

[0140] Note: If there is a brief moment when the pen is lifted and then immediately put down (<200ms), it is considered as the same segment and will not be segmented.

[0141] (2) Fragments of memory:

[0142] Definition: The silent thinking stage undertaken in order to answer a question.

[0143] Start: After the prompt is finished, the user exhibits either of the following two combinations of signals and has not yet spoken.

[0144] Visual: shifting gaze, furrowed brows, changes in blinking frequency, slight mouth preparation, and other expressions / postures indicating thought.

[0145] Behavior: Hesitant movements such as pausing with the hands, slight nodding or tilting of the head.

[0146] End: The game ends when the first recognizable voice (speaking) is heard; or if no one speaks within the set time limit (e.g., 10 seconds), the game ends due to timeout.

[0147] Note: Once you enter the "language retelling segment", this segment will immediately terminate (the transition from recall to speaking is a one-way conversion).

[0148] (3) Language retelling segments:

[0149] Definition: The continuous speech process in which a subject responds or repeats aloud.

[0150] Start: After the question instruction is finished, the first speech start point is detected (silence turns into speech, or the first character / word appears in ASR).

[0151] End: The audio ends and the subject remains silent for ≥500ms; or the subject clicks "Complete / Next Question".

[0152] Note: If there is a short pause of less than 300ms before continuing to speak, it will be considered as the same segment.

[0153] (4) Execute command fragment:

[0154] Definition: The process of performing visible operations such as clicking an option, pointing to a target, or using gestures, in accordance with instructions.

[0155] Start: After the instruction is broadcast, a rapid movement of key points of the hand / upper limb or a target click / selection event related to the task is detected (clearly different from a static state).

[0156] End: The objective is detected (e.g., an option is selected, or a gesture is completed) and the system stabilizes within ≥300ms; or click "Complete / Next Question".

[0157] Note: If the action is not completed and there is a pause of ≥2 seconds, the segment can be considered finished and labeled "Incomplete".

[0158] S3. Feature extraction: Extract different feature data for data of different modalities;

[0159] Speech features include: speech rate (words / minute), pause rate (>300ms) and pause length, proportion of silent segments, pitch F0 and pitch stability (F0 jitter), ASR recognition accuracy (recognition confidence ≥ 0.85 is considered correct), keyword hit rate, pronunciation clarity, MFCC, and speech rate offset percentage (|speech rate - norm| / norm).

[0160] Drawing / touch features include: contour similarity (IoU graphic contour overlap), stroke order consistency (DTW), pen speed curve (mean / peak / standard deviation), offset rate (centerline distance difference), handwriting continuity (number of skips and breaks), overlap rate, spatial offset heatmap, and completion time.

[0161] The pen speed curve is a time series of the pen tip's movement speed from the moment the pen touches the paper to the moment it is lifted. The speed is obtained by removing bits from each frame and calculating the time difference. The average, maximum, and standard deviation of this series of speeds are then calculated to quantify the speed and stability of writing.

[0162] Optionally, contour similarity can be evaluated using the Hausdorff / Chamfer / Fréchet distance.

[0163] Action / gesture / facial features include: hand / fingertip / facial key point trajectories, trajectory completeness, hand trajectory smoothness (acceleration variance), posture change amplitude, blinking and gaze deviation rate (gaze direction deviation ≥15° is considered abnormal), and micro-expression frequency.

[0164] S4. Objective scoring within the task: Input the normalized feature vector into the robust scorer (Huber regression / quantile regression / lightweight Transformer) and output the objective score of the task ∈ [0,100]. Use bootstrap (≥1000 times) or Bayesian method to estimate the 95% confidence interval, automatically reduce the weight of low-quality features (such as insufficient ASR confidence) or trigger retesting and update the objective score.

[0165] The specific steps include:

[0166] S41. Obtain the normalized feature vector. The specific steps include:

[0167] S411, Fragment Location:

[0168] Based on the S22 segmentation, select the time interval for this task (or question). Only numbers are taken within that interval.

[0169] S412. Feature Data Extraction: Extract the feature data obtained in step S3, with each feature forming a one-dimensional feature vector. .

[0170] S413. Units and directions are consistent:

[0171] Perform unit conversions for feature data, such as pixel to millimeter, millisecond to second; unify the direction of error-related features (such as error rate and offset) to ensure the same direction of numerical meaning. For example, unify different types of error indicators to: the larger the indicator value, the smaller the error.

[0172] S414. Quality Screening and Noise Reduction:

[0173] Set a baseline threshold (e.g., ASR confidence ≥ 0.6, IoU ≥ 0.4), and mark feature data below the threshold as low quality / missing; perform slight smoothing or truncation (e.g., 3σ truncation) on obvious peaks.

[0174] S415, Missing / Low-Quality Handling:

[0175] Missing feature data is imputed using the number of bits in the training set or the median of historical data for the same task; a quality mask is generated simultaneously. Record whether the feature data of this dimension is reliable.

[0176] S416, Normalization / Standardization:

[0177] For each eigenvector Obtained through robust standardization : ;

[0178] in To train a concentrated number of bits, The value is MAD (median absolute deviation); a regular z-score (mean / standard deviation) or Min–Max to [-1,1] can also be used, but this step requires a consistent training / inference method.

[0179] S417, Assembly and Output:

[0180] Each dimension in a fixed order Concatenate into normalized feature vectors It also outputs the quality mask m and the timestamp / version number of this task for use by the scoring model and for auditing.

[0181] S42. Generate individual scores: Generate objective scores for each type of task;

[0182] Preferred task types, categorized by question response method and scoring rules, include:

[0183] Drawing / touchscreen applications (connect-the-dots, tracing, clock drawing, etc.);

[0184] Language repetition / response type (naming, repetition, forward and backward recitation, etc.);

[0185] Recall / Memory (Immediate / Delayed Recall);

[0186] Execute commands / operations (click, point, gesture actions).

[0187] Task types correspond one-to-one with scale dimensions (such as executive / attention / language / memory / orientation in MoCA), maintaining consistency in the mapping table.

[0188] The method for generating individual scores is as follows:

[0189] (1) Feature extraction: Obtain the normalized feature vector of the question based on the segment of S22. (Units have been standardized, noise has been removed, and missing units have been processed).

[0190] (2) Selecting a subset: Predefine a feature subset for the task type family (e.g., IoU, DTW, and pen speed statistics for drawing; ASR accuracy, speech rate, and pause rate for language), and obtain .

[0191] (3) Scorer: Train a dedicated scorer for each task type family. (e.g., linear / logistic regression, Huber regression, or lightweight tree models), output continuous scores: Where w is the weight matrix, b is the regression constant, and σ is the normalization function.

[0192] Optionally, the task scorer can use Random Forest / GBDT / Lightweight Transformer.

[0193] (4) Mapping scales for consecutive score questions: Mapping the original score range of the question to a single score:

[0194] ; where MaxScore is the full score for the question, and round() is the rounding function.

[0195] (5) Divide discrete score questions into score tiers: learn or verify a set of thresholds. Will Segment to the corresponding level, such as 0 / 1 / 2; (the threshold is optimal on the training set with the consistency κ value).

[0196] (6) Quality control: For low-quality features (such as low ASR confidence), reduce the weight by mask or trigger retesting; at the same time, provide confidence / CI for review, and mark feature data that is below the set standard as pending review.

[0197] S43. Generate a comprehensive score:

[0198] Huber Regression / L1-L2 hybrid regression was used to synthesize the individual scores, resulting in a comprehensive score:

[0199] ;

[0200] in: The feature value is the synthesizer input feature derived from the i-th individual item score. Let b be the learning weight and b be the regression constant.

[0201] When training the comprehensive scorer, the Huber loss is used to fit the weights ω. i It is used in conjunction with a constant term b to automatically reduce the impact of outliers; it also works with L1–L2 regularization to prevent overfitting.

[0202] outliers The influence of eigenvalues ​​is automatically reduced using Huber loss, or processed using a truncated averaging method. Noise is addressed using SNR estimation and eigenvalue weighting.

[0203] Among them, anomalies and their determination in the eigenvalue layer:

[0204] Standardize and weight each item according to its quality: Any of the following conditions must be met for an item to be considered abnormal / low-quality:

[0205] Outlier threshold or beyond the training set Subdivision band;

[0206] Quality threshold: <0.3 (obtained by mapping from SNR / ASR confidence / IoU, for example: ASR<0.6, IoU<0.4, silent segment>30%, etc.).

[0207] deal with:

[0208] Demotion: Already reflect;

[0209] Truncation: Cut to [-3,3] or [P5,P95];

[0210] Mark for review / retest: when When it is very low (e.g., <0.2).

[0211] Sample layer anomalies and judgment:

[0212] When training the synthesizer, the Huber loss is used to automatically reduce the weights of samples with large residuals:

[0213] Criteria for judgment: ( Typically, the residual standard deviation is taken as 1.5–2.0 times.

[0214] Solution: Huber The interval uses a linear penalty to automatically reduce the impact of the sample on weight learning, eliminating the need for manual removal.

[0215] S44. Confidence Calculation:

[0216] The confidence level is calculated using a pre-set sample, and the specific method is as follows:

[0217] Use bootstrap (≥1000 times) or Bayesian posterior distribution to filter the comprehensive score:

[0218] ;

[0219] Among them, CI 95 This represents the 95% confidence interval, where μ is the mean and σ is the variance.

[0220] For objective scores that do not fall within the confidence interval, they are judged as low quality, and their corresponding features are automatically downweighted or a retest is triggered. Then, a comprehensive score is generated again through step S43. If the retest result still does not fall within the confidence interval, the retest result is retained.

[0221] Optionally, confidence intervals can be calculated using variants such as DeLong / BCa.

[0222] In the evaluation of this step, when In this case, the origins are traced in the following order:

[0223] Influence Ranking: Calculation (Contribution × Low Quality), select questionable questions from high to low.

[0224] Elimination by elimination (LOO): Eliminating the i-th question in sequence, we get... ;like Large and / or If the value is low, it is marked as an abnormal item.

[0225] Handling: Reduce the weight of question with doubt or retest → return to S43 for recalculation; if it still does not qualify for CI95, retain the result and add a "low confidence" mark.

[0226] It should be noted that while this method performs a quality / confidence assessment upon obtaining the individual score in step S42, it does not use this as the final assessment conclusion. This design aims to achieve the following objectives:

[0227] (1) Overall consistency: Individual questions are scored normally, but may still conflict with each other or be biased by low-quality questions after combination; a comprehensive evaluation is required.

[0228] (2) Weighting effect: The weighting and standardization of S43 will change the impact of each question on the total score, and need to be verified once at the comprehensive level.

[0229] (3) Process closed loop: This invention controls the quality of questions at the question level through S42, performs weighted fusion through S43, and performs CI evaluation on the total score and checks for problems through S44, forming an executable correction and retesting path.

[0230] S45, Threshold Control:

[0231] ASR recognition accuracy ≥ 0.85 → judged as correct;

[0232] ASR recognition accuracy < 0.6 → Automatic downgrading or triggering retest;

[0233] IoU < 0.4 → Indicates a severe rendering anomaly;

[0234] IoU ≥ 0.8 → considered highly consistent;

[0235] If DTW ≤ threshold, the order is considered reasonable;

[0236] Voice pause threshold: 300–500ms;

[0237] Pause time > 600ms → Abnormal speech fluency;

[0238] Silent segment percentage > 30% → Trigger retesting or demotion.

[0239] S5. Scale Domain Hierarchical Fusion: Following the hierarchical structure of "task → dimension → total scale score", a learnable weight matrix W is used for weighted fusion; when encountering missing features, an expected imputation / feature weight reduction strategy is adopted to ensure the stability and fairness of the scoring.

[0240] The specific steps include:

[0241] S51. Integration Level: Task → Dimension → Total Score of Scale; Specifically:

[0242] (1) Task:

[0243] Meaning: The smallest assessment unit / item or sub-item within a scale, such as: matching, naming, forward and backward recitation, delayed recall, clicking according to instructions, etc.

[0244] Data: Fragments corresponding to S22, such as drawing / recalling / language / execution, etc.

[0245] Score source: S42 generates the individual objective score for this task. .

[0246] (2) Dimensions:

[0247] Meaning: The cognitive domain defined by the scale consists of several tasks; for example, the executive / visuospatial, attention, language, memory, and orientation domains of the MoCA.

[0248] Dimensional score: Score based on the task being assigned In-dimensional fusion using the S43 method:

[0249] ;

[0250] in For quality weight, This is a standardized function.

[0251] (3) Total score of the scale:

[0252] Meaning: The final score of the entire scale; the score range can be 0–30 or 0–100.

[0253] Finally, map / round the scale to the score setting range.

[0254] It is worth noting that:

[0255] (1) Step S43 provides a general fusion tool that can be used to generate dimension scores within dimensions or to generate total scale scores between dimensions.

[0256] (2) If all task scores are directly fused at once using S43, the total score can also be obtained; however, fusion using a hierarchical structure of task → dimension → total score is more in line with clinical structure and makes it easier to issue dimension reports and locate the source of problems.

[0257] S52. Fusion Method: Using a linear weighting method:

[0258] ;

[0259] Where σ is the normalization function, and Score final For the total score of the scale, W task Score represents the task weight. task Let b be the task score and b be the linear regression constant.

[0260] In one embodiment of the present invention, the specific steps for fusion according to the layered structure include:

[0261] Layer 1: Task → Dimensional Segmentation;

[0262] The objective score for the i-th question / sub-stage is obtained from S42;

[0263] • The quality weight of this question is mapped from ASR / IoU / SNR to 0–1, and the higher the quality, the greater the quality weight.

[0264] Features input to the fusion unit (standardized and then weighted by quality).

[0265] The weight of this question in dimension d (obtained through learning or set by experts).

[0266] : Dimensional layer bias term;

[0267] Normalization / scale mapping function (e.g., Sigmoid → linear mapping to the score of this dimension);

[0268] Calculation formula: .

[0269] Layer 2: Dimension → Total Scale Score;

[0270] Scores for each cognitive dimension;

[0271] Dimension weights (learning / setting, can be normalized according to importance or number of questions);

[0272] Total hierarchical bias term;

[0273] Map the results to a scale (e.g., 0–30 / 0–100), rounding to the nearest integer if necessary;

[0274] formula: .

[0275] Optionally, the fusion strategy can use an attention mechanism or a hierarchical Bayesian model.

[0276] S53. Missing data strategy: For missing features, use expected imputation or feature weight reduction.

[0277] The method for feature weight reduction is as follows:

[0278] ;

[0279] in, Wtask represents the weight after weight reduction, Wtask represents the weight before weight reduction, and MissingRate represents missing data.

[0280] S6. Report Generation: Output doctor / patient version reports, including total score, dimension score (cognitive dimension radar chart), key indicators, confidence interval (confidence bar), abnormal prompts (such as behavioral abnormality labels such as "incomplete drawing / excessive speech pause"), time statistics and task duration analysis, and quality of assessment (QoE).

[0281] The report supports PDF export and interfaces with HL7 / FHIR or custom interfaces of in-hospital systems (such as HIS / EMR).

[0282] S7. Audit and Explainability: Records question sequence, number of prompts, and reasons for jumps; saves complete audio and video trajectories; saves feature values ​​and weights for each step; outputs heatmaps of feature weights and importance for each modality; records timeline events and logs; and supports review, playback, and quality control.

[0283] Secondly, the present invention provides an embodiment of a multimodal objective scoring system for cognitive scales, such as... Figure 2 As shown, it consists of a terminal data acquisition and edge inference subsystem and a central scoring and reporting subsystem; the hardware includes a high-definition camera / depth camera, a microphone array, and a touch screen or stylus input module; the software includes data acquisition services, time synchronization and segment annotation services, a feature extraction engine (speech / drawing / action), an in-task scorer, a scale domain fusion unit, reporting and auditing services, and a privacy and access control module.

[0284] Thirdly, this invention provides an embodiment of a multimodal objective scoring device for cognitive scales, comprising: a data acquisition module, a synchronization and segment annotation module, a feature extraction module (voice / drawing / action), a task scoring module, a scale fusion module, a report generation module, and an auditing and security module (logging, encryption, access control). This device can be deployed on an all-in-one machine or adopt a "device + cloud" collaborative architecture.

[0285] Fourthly, the present invention provides a computer program product embodiment, which, when executed by a processor, implements the steps of the method described in the first aspect.

Claims

1. A multimodal objective scoring method for cognitive scales, characterized in that, Includes the following steps: S1. Multi-source data acquisition: Under the trigger of the scale item controller, continuously acquire audio streams, video frames, touch / handwriting trajectories, and hand movement trajectories, and record unified reference clock timestamps and item events to form a task event log; S2. Time Synchronization and Segmentation: Employing NTP / system clock alignment + soft timescale fusion, a multimodal mapping function is established to unify audio, video, and touch input onto the same timeline, ensuring alignment of voice, action, image, and trajectory; through question event slicing, question segments and keyframes are formed. S3. Feature extraction: Extract different feature data for data of different modalities; S4. Objective scoring within the task: Input the normalized feature vector into the robust scorer, output the objective score of the task, and then synthesize it to generate a comprehensive score; use confidence to filter the comprehensive score, and automatically reduce the weight of low-quality features or trigger a retest. S5. Scale Domain Hierarchical Fusion: Following the hierarchical structure of task → dimension → total scale score, a learnable weight matrix is ​​used for weighted fusion, and a comprehensive score is progressively obtained to finally obtain the total scale score; when encountering missing features, an expected imputation / feature weight reduction strategy is adopted to ensure the stability and fairness of the score. S6. Report Generation: Outputs doctor / patient version reports, which can be connected to HL7 / FHIR interfaces or HIS / EMR systems; S7. Audit Data Recording: Save the process data recorded in the scale for review, playback and quality control purposes.

2. The method according to claim 1, characterized in that, The data collected in step S1 includes: Video data: facial key points, hand key points, ROI (Region of Interest) for drawing, and outline of the drawing trajectory lines; Audio data: speech rate, phrasing, keywords, logical coherence; Touch / handwriting data: coordinate sequence, pressure, speed, pen lift / put down events; Task event data: Question start / end, number of prompts, skipped questions, and subject response time.

3. The method according to claim 1, characterized in that, Step S2 includes: S21. Time Synchronization: Based on the hardware clock, NTP assisted and / or PTP, or NTP assisted and / or hard trigger, are used to align the clocks of voice, video, and touch data; AI alignment is used to match soft timescales using sound abrupt changes and motion keyframes. S22. Segmentation: The problem is segmented according to the events in the problem; the events in the problem include: drawing segment, recall segment, language retelling segment, and command execution segment. Each segment has a timestamp interval.

4. The method according to claim 3, characterized in that, Step S21 specifically includes: first, using AI to find the start and end / abrupt points of the sound and the key frames of the action as soft time stamps; then, matching these soft time stamps with the prompts in the question to nearest neighbors; then, using weighted straight lines to correct the time axis to eliminate abnormal matches and retain high-confidence matches, aligning the audio, video, and touch to the same timeline; at the same time, correcting drift time through timed detection and realignment to ensure that the synchronization time deviation does not exceed the threshold.

5. The method according to claim 3, characterized in that, The segmentation method in step S22 includes: Drawing segment: By setting a threshold for the time interval between putting down and lifting the pen, the continuous operation process of the user drawing / connecting / tracing on the screen is determined; Recall Fragments: Based on the simultaneous capture of thinking facial expressions / postures and hand / head movements, determine the silent thinking process of the user in order to answer the question; Language repetition segment: Determine the continuous speech process of a user's verbal response / repetition by using a speech pause time threshold; Command execution segment: By detecting the movement / stability of key points of the hand / upper limb and the achievement of the command target, the process of the user performing the operation according to the command is determined.

6. The method according to claim 1, characterized in that, The feature data extracted in step S3 includes: Speech feature data: speech rate, pause rate and pause length, proportion of silent segments, pitch and pitch stability, ASR recognition accuracy, keyword hit rate, pronunciation clarity, MFCC, and speech rate deviation percentage; Drawing / touch feature data: contour similarity, stroke order consistency, pen speed curve, offset rate, handwriting continuity, overlap rate, spatial offset heatmap, and completion time; Action / gesture / facial feature data: hand / fingertip / facial key point trajectory, trajectory completeness, hand trajectory smoothness, posture change range, blink and gaze deviation rate, micro-expression frequency.

7. The method according to claim 6, characterized in that, The feature data extracted in step S3 specifically includes: Contour similarity: IoU figure contour overlap, or Hausdorff / Chamfer / Fréchet distance; Pen speed curve: A curve formed by the time series of pen tip movement speed, and the mean / peak / standard deviation of the speed series are calculated; Offset rate: the difference in distance from the center line; Handwriting continuity: number of skipped strokes and breaks in the pen; Hand trajectory smoothness: acceleration variance; Gaze deflection rate: The frequency of gaze direction deviation angles greater than 15°.

8. The method according to claim 1, characterized in that, Step S4 includes: S41. Obtain normalized feature vector: Extract the feature data obtained in step S3. Each feature corresponds to a one-dimensional feature vector. Perform unit and data direction unification, quality screening and noise reduction, missing / low-quality processing, and robust standardization on each feature vector. Then, concatenate them to obtain the normalized feature vector. S42. Generate individual scores for each task category: Build robust scorers using Huber regression / quantile regression / lightweight Transformer / random forest / GBDT; S43. Generate a comprehensive score: Use Huber Regression / L1-L2 hybrid regression to obtain a comprehensive score based on individual scores; reduce the impact of outliers by using Huber loss or truncated averaging; use SNR estimation and feature weighting to deal with noise. S44. Confidence Level Filtering: Use confidence level to filter the comprehensive score. If the comprehensive score does not fall into the confidence interval, it is judged as low quality, and its corresponding features are automatically downweighted or a retest is triggered. If the result of the retest is judged as low quality again, the result is retained. S45. Threshold control: By setting and controlling thresholds, judgments are made on threshold-related items or weight reduction / retesting operations are initiated.

9. The method according to claim 8, characterized in that, The method for generating individual scores in step S42 is as follows: Obtain the normalized feature vector corresponding to the question event, and select the feature subset corresponding to the task type family; For each task type family, train the robust scorer and output a normalized continuous score; The normalized continuous score is proportionally restored to the original continuous score value of the question, or the corresponding score level is obtained by segmentation. Quality control is performed using confidence level / CI, and low-quality features are downweighted by masking, or retested or re-verified.

10. The method according to claim 8, characterized in that, The calculation method of Huber Regression / L1-L2 mixed regression: ； Where: Score is the overall rating. For eigenvalues, Let b be the learning weight and b be the regression constant. Use Huber loss to fit the weights ω i With a constant term b, the influence of outliers is automatically reduced; at the same time, L1–L2 regularization is used to prevent overfitting. Confidence interval calculation method: Based on a preset sample, the calculation is performed using a bootstrap or Bayesian posterior distribution with at least 1000 iterations. ； Among them, CI 95 This represents the 95% confidence interval, where μ is the sample mean and σ is the sample variance.

11. The method according to claim 8, characterized in that, The method for handling outliers in step S43 includes: Eigenvalue anomaly detection criteria: The absolute value of the standardized residual is greater than the outlier threshold of 3, or exceeds the threshold of the training set. The quantile band, or the quality weight obtained by mapping SNR / ASR confidence / IoU, is less than 0.3; For outlier sample values, for samples where the Huber loss is greater than 1.5–2.0 times the residual standard deviation, a linear penalty is applied to automatically reduce the impact of the sample on weight learning.

12. The method according to claim 11, characterized in that, In the retesting step S44, abnormal questions are traced back to their source. The specific method is as follows: The topics are sorted in descending order of influence based on the product of the weighted feature value and the quality assessment value. The current topics are excluded in order, and a new comprehensive technical score is given. If the change in the comprehensive score is greater than the threshold or the quality weight of the current topic is less than the threshold, the current topic is marked as an abnormal topic. For abnormal questions, the weight is reduced or the test is retested. If the result of the retest is again judged as low quality, the result is retained and marked with low confidence.

13. The method according to claim 8, characterized in that, The threshold control in step S45 specifically includes: ASR recognition accuracy ≥ 0.85 → judged as correct; ASR recognition accuracy < 0.6 → trigger retesting or downgrading; IoU < 0.4 → Indicates a severe rendering anomaly; IoU ≥ 0.8 → indicates that the plotted curves have consistent heights; If DTW ≤ threshold, the order is deemed reasonable. Voice pause threshold setting: 300–500ms; Pause time > 600ms → Indicates abnormal speech fluency; Silent segment percentage > 30% → Trigger retesting or demotion.

14. The method according to claim 1, characterized in that, In step S5 A task is the smallest assessment unit in a scale, including a question or a sub-section within a question; A dimension is a cognitive domain defined by the scale, consisting of several tasks that point to the same cognitive domain. The total score of the scale is the final score of the entire scale.

15. The method according to claim 1, characterized in that, Step S5 includes: Weighted fusion uses a linear weighting method: ； Where σ is the normalization function, and Score final For the total score of the scale, W task Score represents the task weight. task Let b be the task score and b be the linear regression constant. The method for feature weight reduction is as follows: ； in, Wtask represents the weight after weight reduction, Wtask represents the weight before weight reduction, and MissingRate represents missing data.

16. The method according to claim 1, characterized in that, The report in step S6 includes: total score, dimensional score, key indicators, confidence interval, abnormal behavior alerts, time usage statistics and task duration analysis, and evaluation quality score.

17. The method according to claim 1, characterized in that, The audit data in step S7 includes: question sequence, number of prompts, reason for jump, complete audio and video trajectory, feature value and weight of each step, heatmap of feature weight and importance of each modality, timeline events and logs.

18. A multimodal objective rating system for cognitive scales, characterized in that, The system components work together to implement the steps of the method described in any one of claims 1-17; the system consists of a terminal acquisition and edge inference subsystem and a central scoring and reporting subsystem; the hardware includes a high-definition camera / depth camera, a microphone array, and a touch screen or stylus input module; the software includes a data acquisition service, a time synchronization and segment annotation service, a voice / graphics / motion feature extraction engine, an in-task scorer, a scale domain fusion unit, a reporting and auditing service, and a privacy and access control module.

19. A multimodal objective scoring device for a cognitive scale, characterized in that, The modules work together to implement the steps of the method described in any one of claims 1-17; the device includes: a data acquisition module, a synchronization and segment annotation module, a speech / graphics / action feature extraction module, a task scoring module, a scale fusion module, a report generation module, and an audit and security module; the device is deployed on an all-in-one machine or adopts an end + cloud collaborative architecture.

20. A computer program product, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1-17.

Images