A traditional Chinese and western medicine fusion cognitive auxiliary evaluation and intervention system based on a multi-modal model and a personal memory atlas

Abstract

The application belongs to the field of artificial intelligence and digital medical technology, and specifically relates to a traditional Chinese and western medicine fusion cognitive auxiliary evaluation and intervention system based on a multi-modal model and a personal memory graph. The system comprises a multi-source data acquisition module, a traditional Chinese and western medicine fusion cognitive auxiliary evaluation module, a personal memory graph construction module and a personalized intervention module. Text or voice question and answer, drawing results and trajectories, action videos and face-tongue images formed when a patient completes a traditional Chinese and western medicine scale are collected, traditional western medicine cognitive scale evaluation results, traditional Chinese medicine constitution identification results and face-tongue diagnosis structured features are fused, a patient cognitive level auxiliary evaluation result is generated, and constraint information is generated; according to patient input, a multi-hop search is performed on the personal memory graph, and personalized intervention content is generated in combination with the search result, the auxiliary evaluation result and the constraint information. The application can improve the multi-modal fusion degree of cognitive auxiliary evaluation and the traceability of intervention content, and reduce the risk of fact bias and privacy leakage.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and digital medical technology, specifically involving a cognitive auxiliary assessment and intervention system integrating traditional Chinese and Western medicine based on multimodal models and personal memory maps. Background Technology

[0002] As the population ages, cognitive decline-related issues such as mild cognitive impairment and Alzheimer's disease are receiving increasing attention. Current cognitive assessments largely rely on standardized scales such as the Mini-Mental State Examination and the Montreal Cognitive Assessment Scale, and are scored manually by professional physicians. This approach is time-consuming, dependent on professional physicians, and difficult to conduct frequently in home or community settings.

[0003] Currently, some digital cognitive screening systems have migrated traditional scales to electronic terminals, but most still rely mainly on text-based or voice-based question-and-answer sessions. They do not make sufficient use of multi-source heterogeneous data such as drawing trajectories, motion videos, voice expressions, facial images, and tongue images, making it difficult to uniformly collect, structure, and comprehensively analyze non-textual behavioral information in items such as visuospatial and executive functions, and complex command execution.

[0004] Existing cognitive assessment systems are mostly based on Western medical cognitive scales, and rarely incorporate information from traditional Chinese medicine (TCM) diagnosis, constitution identification, and other methods into a unified auxiliary assessment process. TCM information such as facial complexion, tongue appearance, constitution, sleep, and emotions can reflect changes in the physical and mental functions of the elderly from an overall perspective, but in practical applications, this information often relies on human experience and judgment, making it difficult to form stable, calculable, and integrated structured features.

[0005] In terms of cognitive intervention, existing non-pharmacological interventions mostly employ general cognitive training, picture recall, or general conversational companionship, which are difficult to establish deep connections with patients' real life experiences, family relationships, and emotional memories; ordinary folder-style photo management or keyword retrieval are also difficult to express the complex relationships between people, events, places, times, and multimedia resources.

[0006] With the development of large language models and multimodal models, it has become possible to use artificial intelligence to generate assessment interpretations and intervention scripts. However, general-purpose large models are prone to generating content inconsistent with real experiences when lacking the constraint of patients' personal facts; uploading sensitive data such as patients' family photos, videos, voice recordings, medical history, and kinship relationships to the cloud for processing may also pose a risk of privacy leaks. Therefore, there is an urgent need for an improved cognitive-assisted assessment and personalized intervention system to address the problems of a single assessment system, lack of patient-specific memory support for intervention content, and susceptibility to factual bias. Summary of the Invention

[0007] The purpose of this invention is to provide a cognitive auxiliary assessment and intervention system based on multimodal models and personal memory maps that integrates traditional Chinese and Western medicine. On the one hand, it integrates and reasones about the characteristics of traditional Chinese and Western medicine to improve the comprehensiveness and interpretability of the auxiliary assessment results. On the other hand, it uses generation constraint information and personal memory maps to constrain the generation of personalized intervention content, thereby improving the personalization, authenticity and traceability of the intervention content.

[0008] To achieve the above objectives, the present invention provides a multimodal model-based integrated traditional Chinese and Western medicine cognitive auxiliary assessment system, comprising: a multi-source data acquisition module and an integrated traditional Chinese and Western medicine cognitive auxiliary assessment module; the multi-source data acquisition module is used to collect Western medicine cognitive scale data and traditional Chinese medicine assessment-related data; the traditional Chinese medicine assessment-related data includes traditional Chinese medicine constitution identification scale data, facial image data, and tongue image data; The integrated traditional Chinese and Western medicine cognitive auxiliary assessment module includes: The Western Medicine Cognition Scale Assessment Unit is used to score items based on the data of the Western Medicine Cognition Scale and generate the Western Medicine Cognition Scale Assessment Results. The Traditional Chinese Medicine (TCM) constitution identification unit is used to score items based on the TCM constitution identification scale data and generate TCM constitution identification results. The TCM facial and tongue diagnosis feature extraction unit is used to extract features from the facial image data and tongue image data to obtain TCM facial and tongue diagnosis structured features. The fusion reasoning unit is used to perform fusion reasoning on the assessment results of the Western medicine cognitive scale, the results of constitution identification, and the structured features of facial and tongue diagnosis according to preset reasoning rules, and generate auxiliary assessment results of the patient's cognitive level.

[0009] Furthermore, the Western Medicine Cognition Scale assessment unit first divides the items of the Western Medicine Cognition Scale data into rule-based scoring items and multimodal auxiliary scoring items. The rule-based scoring items are items with clear standard answers or scoring rules, while the multimodal auxiliary scoring items are items that require judgment by combining text responses, voice expressions, drawing results, drawing trajectories, action videos, or video frame sequences. The rule-based scoring items are scored according to preset rules. The multimodal auxiliary scoring items are preprocessed and structured auxiliary features are extracted. The extracted structured auxiliary features and the corresponding original Western medicine cognitive scale data are input into the multimodal large model along with the item requirements, scoring rules and output format constraints to generate the Western medicine cognitive scale evaluation results, which include item scores, total scores, cognitive domain scores, reasons for deductions, scoring basis and abnormal item prompts.

[0010] Furthermore, the TCM constitution identification scale data includes the selection results input by the patient or caregiver based on the questionnaire items, as well as the scale response information related to symptom perception, sleep status, diet, emotional state, cold and heat manifestations, and physical condition. The TCM constitution identification unit calculates and standardizes the scores of each constitution dimension according to the preset item score mapping rules, thereby obtaining the patient's TCM constitution identification results. The TCM constitution identification results include the main constitution, the tendency constitution, and related constitution characteristics. The constitution dimensions include balanced constitution, qi deficiency constitution, yang deficiency constitution, yin deficiency constitution, phlegm-dampness constitution, damp-heat constitution, blood stasis constitution, qi stagnation constitution, and special constitution.

[0011] Furthermore, the TCM facial and tongue diagnosis feature extraction unit first preprocesses the facial image data and tongue image data, then performs target region localization, key region segmentation, and feature classification. It extracts structured facial diagnosis feature labels, including complexion, luster, lip color, and local color changes, from the facial image data; and extracts structured tongue diagnosis feature labels, including tongue color, coating color, tongue shape, thickness of tongue coating, degree of moisture, teeth marks, and cracks, from the tongue image data.

[0012] Furthermore, the fusion reasoning unit converts the Western medicine cognitive scale assessment results, constitution identification results, and facial and tongue diagnosis structured features into a fusion feature set. Then, according to preset cognitive domain mapping rules, feature weight rules, abnormal prompting rules, and natural language generation rules, it processes the relationship between various features to generate the patient's cognitive level auxiliary assessment results. The cognitive domain mapping rules include: the correspondence between items of the Western Medicine Cognitive Scale and cognitive domains, and the association between intervention intention and cognitive domains; wherein, the correspondence between items of the Western Medicine Cognitive Scale and cognitive domains is used to calculate the scores of each cognitive domain, and the association between intervention intention and cognitive domains is used to determine the cognitive domain associated with the current intervention intention and its associated weight. The cognitive level auxiliary assessment results include: cognitive level auxiliary assessment conclusions, cognitive risk classification prompts, performance of major cognitive domains, explanation of assessment basis, prompts of abnormal items, prompts related to traditional Chinese medicine constitution and facial and tongue diagnosis, and non-diagnostic conditioning and intervention suggestions.

[0013] This invention also provides a cognitive intervention system integrating traditional Chinese and Western medicine based on a multimodal model and a personal memory map, comprising the multimodal model-based cognitive auxiliary assessment system integrating traditional Chinese and Western medicine, a personal memory map construction module, and a personalized intervention module based on graph retrieval enhancement as described above; the multi-source data acquisition module is also used to collect information related to the patient's personal experience; The integrated traditional Chinese and Western medicine cognitive auxiliary assessment module also includes a generation constraint information construction module connected to the fusion reasoning unit; the generation constraint information construction module is used to construct generation constraint information based on the patient cognitive level auxiliary assessment results generated by the fusion reasoning unit, and the generation constraint information is used to constrain the process of the personalized intervention module generating memory recall scripts, prompting questions or companion-style answers. The personal memory graph construction module is used to extract entities and relationships from information related to the patient's personal experiences to construct a personal memory graph. The personalized intervention module is used to perform multi-hop retrieval of the personal memory map based on the information input by the patient, and then generate personalized intervention content under the constraints of the generated constraint information to achieve interactive intervention with the patient.

[0014] Furthermore, the generated constraint information is generated based on one or more of the following: cognitive risk grading prompts, cognitive domain performance, abnormal question prompts, task completion rate, TCM constitution, and facial and tongue diagnosis-related prompts. The "generated constraint information" referred to in this invention is structured information constructed by the generated constraint information construction module based on the patient's cognitive level auxiliary assessment results, used to control the subsequent content generation process of the multimodal large model. The generated constraint information includes one or more of the following: question difficulty, prompt intensity, language complexity, waiting time, and multimedia display method, and can be adjusted at the round level based on the patient's current round of interaction performance during personalized intervention.

[0015] Furthermore, the methods for obtaining the problem difficulty, hint strength, language complexity, and waiting time include: The cognitive deficit normalization coefficient is determined based on the actual score, highest score and associated weight of the cognitive domain associated with the current intervention intention. Then, the cognitive deficit normalization coefficient, the proportion of abnormal items in the Western Medicine Cognitive Scale, the task completion rate and the overall state auxiliary adjustment coefficient obtained based on the TCM constitution identification results and the TCM facial and tongue diagnosis structured features are weighted and summed to obtain the cognitive support demand coefficient. The interaction difficulty coefficient is obtained based on the patient's errors, lack of response, and reaction time in the current round of responses. Then, the cue intensity, question difficulty, language complexity, and waiting time for the next round of interaction are determined based on the cognitive support need coefficient and the interaction difficulty coefficient. In the first round of interaction, the difficulty coefficient of the interaction is set to 0; when the number of completed items on the Western Medicine Cognition Scale is 0, the cognitive support demand coefficient takes a preset initial value.

[0016] Furthermore, the cognitive support demand coefficient is obtained according to the following formula:

[0017] In the formula, AFor cognitive support demand coefficient, Rc The normalized coefficient for the cognitive deficit is... Ra The proportion of abnormal questions, C This is the ratio of the number of tasks completed to the number of tasks that should be completed. Z is the term Overall status auxiliary adjustment coefficient; , , and The weights are non-negative and their sum is 1; The interaction difficulty coefficient is obtained according to the following formula:

[0018] In the formula, Let be the difficulty coefficient of the interaction in round t. For the patient in the first The indicator value is 1 if the answer is incorrect or the answer is significantly deviating from the current question; otherwise, it is 0. The indicator value is 1 if the patient does not respond or requests a repeat prompt within the preset longest reference response time, and 0 otherwise. For the first Actual reaction time; The maximum reference response time is preset, and ; , and The weights are non-negative and their sum is 1; Then the first Wheel's cue intensity Difficulty of the problem Language complexity and waiting time We obtain the following formula:

[0019] , , and These are the benchmark hint strength, benchmark problem difficulty, benchmark language complexity, and benchmark wait time, among which... , and The value range is from 0 to 1. , and satisfy ; , , , , and It is a dimensionless, non-negative adjustment coefficient; and It is a non-negative adjustment coefficient with the same time dimension as the waiting time.

[0020] Furthermore, the information related to the patient's personal experience includes pictures, videos, audio, text materials, and their descriptive information; The entity nodes of the personal memory map include one or more of the following: person nodes, time nodes, location nodes, event nodes, item nodes, emotional attribute nodes, and multimedia resource nodes; the edge relationships include one or more of the following: person relationships, event participation relationships, location relationships, time relationships, item association relationships, emotional association relationships, resource association relationships, and event sequence relationships. The multimedia resource nodes are associated with corresponding person nodes, time nodes, location nodes, event nodes, or item nodes, so that information related to the patient's personal experience can be used as fact nodes in the personal memory map for retrieval.

[0021] Furthermore, the personalized intervention module includes: The intent parsing unit is used to parse the information input by the patient, identify people, places, times, events, items, kinship terms or question intents, and form query information for multi-hop retrieval of the graph; The evaluation result and generation constraint information reading unit is used to read the cognitive level auxiliary evaluation result and generation constraint information and transmit them to the generation context construction unit; The multi-hop memory retrieval unit is used to perform multi-hop memory retrieval in the personal memory graph based on the query information, and to recall memory paths, graph node relationships and associated multimedia resources related to the information input by the patient. A context building unit is used to construct a context from the cognitive level auxiliary assessment results, the generation constraint information, and the results of the multi-hop graph retrieval. A multimodal generation unit is used to generate personalized intervention content based on the generation context, so that the generated content is constrained by real nodes, edge relationships and associated resources in the personal memory graph; The multimedia synchronous output unit is used to synchronously output the generated personalized intervention content and the recalled images, videos, audio or text materials to the user terminal. The user terminal presents the intervention content to the patient through screen display, voice broadcast, image display, video playback or text display.

[0022] In summary, compared with the prior art, the above-described technical solutions conceived by this invention mainly possess the following technical advantages: (1) This invention constructs a multimodal information fusion mechanism for cognitive auxiliary assessment. The system comprehensively collects text or voice question-and-answer data, drawing results and continuous trajectory data, action video data, and facial and tongue images used for TCM observation during the patient's completion of the TCM and Western medicine scales, so that cognitive assessment is no longer limited to single text question-and-answer or manual scale scoring. The results of Western medicine cognitive scale assessment, TCM constitution identification results, and TCM facial and tongue diagnosis structured features are uniformly input into the fusion reasoning process to form auxiliary assessment results of the patient's cognitive level, cognitive risk classification prompts, and non-diagnostic conditioning and intervention suggestions, thereby improving the comprehensiveness and interpretability of the auxiliary assessment results.

[0023] (2) This invention forms a closed-loop mechanism between cognitive auxiliary assessment and personalized intervention. The system calculates the cognitive support need coefficient based on the auxiliary assessment results of the patient's current cognitive level, and determines the initial generation constraint information accordingly; the system also calculates the interaction difficulty coefficient based on the correctness of the answers, the absence of responses, and the reaction time during the interaction process, and dynamically adjusts the prompt intensity, question difficulty, language complexity, and waiting time of the next round of interaction based on the cognitive support need coefficient and the interaction difficulty coefficient, so that the auxiliary assessment results and interaction performance can quantitatively affect the subsequent generation process.

[0024] (3) This invention improves the authenticity and traceability of intervention content through a personal memory map and graph retrieval enhancement generation mechanism. The system constructs a personal memory map from the patient’s personal experiences, family relationships, time and place, events and multimedia resources, and recalls relevant memory paths, node relationships and multimedia materials as factual constraints when generating intervention scripts, which facilitates the generation of personalized memory evoking scripts, prompting questions or companion-style answers.

[0025] (4) The present invention uses a local or controlled private environment to process patient cognitive data, TCM image data, family multimedia materials and personal memory maps, reducing the need to upload sensitive data to the public cloud, and helping to reduce the risk of leakage of patient personal information and family memory data. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the overall architecture of the system described in this invention; Figure 2 This is a flowchart of the multi-source data acquisition and Western medicine cognitive scale-assisted assessment process of the present invention; Figure 3 This is a flowchart of the structured feature extraction process for TCM constitution identification and facial and tongue diagnosis in this invention. Figure 4 This is a flowchart of the process for integrating traditional Chinese and Western medicine in reasoning, generating auxiliary evaluation results, and constructing constraint information in this invention. Figure 5 This is a flowchart of the personal memory map construction process of the present invention; Figure 6 This is a flowchart of the graph retrieval enhancement generation intervention based on personal memory graphs in this invention.

[0027] In all the accompanying drawings, the same reference numerals are used to denote the same elements or structures, wherein: 101-User terminal; 110-Multi-source data acquisition module; 120-Integrated traditional Chinese and Western medicine cognitive auxiliary assessment module; 130-Personal memory map construction module; 140-Personalized intervention module based on graph retrieval enhancement; 150-Private data processing environment; 160-Data storage area; 121-Western Medicine Cognitive Scale Assessment Unit; 1210-Item Type Judgment; 1211-Rule Scoring; 1212-Multimodal Auxiliary Scoring; 1213-Western Medicine Cognitive Scale Results Summary; 122-Traditional Chinese Medicine Constitution Identification Unit; 1221-Item Processing; 1222-Score Calculation; 1223-Standardization; 1224-Constitution Determination; 123-Traditional Chinese Medicine Facial and Tongue Diagnosis Feature Extraction Unit; 1231-Image Preprocessing; 1232-Target Region Localization; 1233-Target Region Segmentation; 1234-Multi-Label Classification; 1235-Traditional Chinese Medicine Assessment Results Summary; 124-Fusion Reasoning Unit; 125-Patient Cognitive Level Auxiliary Assessment Results; 126-Constraint Information Generation Module; 127-Constraint Information Generation; 131-Data preprocessing; 132-Entity extraction; 133-Relation extraction; 134-Personal memory graph; 141-Intent parsing; 142-Evaluation results and generation constraint information reading; 143-Graph multi-hop retrieval; 144-Generation context construction; 145-Multimodal generation; 146-Multimedia synchronous output. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0029] Example 1: System Overall Architecture Reference Figure 1This embodiment provides a cognitive aid assessment and intervention system integrating traditional Chinese and Western medicine based on a multimodal model and a personal memory map. It includes a multi-source data acquisition module 110, a cognitive aid assessment module integrating traditional Chinese and Western medicine 120, a personal memory map construction module 130, and a personalized intervention module 140 based on graph retrieval enhancement. These modules can be deployed on local devices, home servers, or private data processing environments 150, and data input, data upload, and interactive output are achieved through a user terminal 101. Storage or processing is performed on the local device, home server, or private data processing environment to reduce the risk of leakage of patient personal information and family memory data.

[0030] The multi-source data acquisition module 110 is used to receive multimodal data input or uploaded by the patient, family members, or caregivers through the user terminal 101, and distribute it to the corresponding processing modules according to the different uses of the data. Among them, the text or voice question-and-answer data generated by the patient in completing the Chinese and Western medicine scale, the drawing results and continuous trajectory data generated when completing the drawing items of the Western medicine cognitive scale, the action video data generated when completing the instruction execution items of the Western medicine cognitive scale, and the facial and tongue image data used for Chinese medicine observation are sent to the integrated Chinese and Western medicine cognitive auxiliary assessment module 120; the patient's personal experience-related materials (pictures, videos, audio, text materials and their descriptive information) are sent to the personal memory map construction module 130; and the voice or text information input by the patient in the personalized intervention process is sent to the personalized intervention module 140 based on graph retrieval enhancement generation.

[0031] The integrated traditional Chinese and Western medicine cognitive auxiliary assessment module 120 is used to integrate the assessment results of Western medicine cognitive scales, the results of traditional Chinese medicine constitution identification, and the structured features of traditional Chinese medicine facial and tongue diagnosis to generate an auxiliary assessment result 125 of the patient's cognitive level. The integrated traditional Chinese and Western medicine cognitive auxiliary assessment module 120 also includes a generation constraint information construction module 126 connected to the fusion reasoning unit 124. The generation constraint information construction module 126 is used to receive the auxiliary assessment result 125 of the patient's cognitive level and determine generation constraint information 127 based on the auxiliary assessment result. The auxiliary assessment result 125 of the patient's cognitive level and the generation constraint information 127 determined therebased can be directly sent to the personalized intervention module 140 based on graph retrieval enhancement generation, or written into the data storage area 160 and then read and used by the personalized intervention module 140.

[0032] The personal memory map construction module 130 is used to construct a personal memory map reflecting the patient's personal experiences, family relationships, and life events based on multimedia materials and descriptive information uploaded by the patient, family members, or caregivers, and to provide the personalized intervention module 140 with searchable memory paths, map node relationships, and associated multimedia resources.

[0033] The personalized intervention module 140 based on graph retrieval enhancement is used to perform multi-hop retrieval of the personal memory map according to the patient's current voice or text input, and generate personalized memory waking speech, prompt questions or companion-style answers based on the retrieval recall results, the patient's cognitive level auxiliary assessment results and generation constraint information. At the same time, the generated content and corresponding pictures, videos, audio or text materials are synchronously output to the user terminal 101.

[0034] The system also includes a data storage area 160. The data storage area 160 is used to store one or more of the following: patient cognitive level auxiliary assessment results, generated constraint information, multimedia materials, personal memory maps, map nodes, edge relationships, multimedia resource indexes, and interaction records. The integrated traditional Chinese and Western medicine cognitive auxiliary assessment module 120, the personal memory map construction module 130, and the personalized intervention module 140 can write to or read from the data storage area 160 as needed.

[0035] Example 2: Multi-source data acquisition and Western medicine cognitive scale-assisted assessment process Reference Figure 2 This embodiment illustrates the process of multi-source data acquisition and auxiliary assessment using the Western Medicine Cognition Scale. The multi-source data acquisition module 110 collects multi-source cognitive data generated by the patient in the Western Medicine Cognition Scale task through the user terminal 101 and sends it to the Western Medicine Cognition Scale assessment unit 121; the Western Medicine Cognition Scale assessment unit 121 performs auxiliary scoring on different types of items to obtain the Western Medicine Cognition Scale assessment results, and sends them to the fusion reasoning unit 124.

[0036] The Western medicine cognitive scale tasks include items related to memory, orientation, attention, calculation, language, visuospatial and executive functions, and command execution ability. The "cognitive domain" referred to in this invention is a functional dimension used to characterize a patient's cognitive function, consisting of one or more items related to that cognitive function in the Western medicine cognitive scale. These include one or more of the following domains: memory, orientation, attention, calculation, language, and visuospatial and executive functions. The specific division is determined according to the Western medicine cognitive scale used and its preset scoring rules. Each item belongs to one or more cognitive domains according to preset cognitive domain mapping rules. The system summarizes or standardizes the scores of the corresponding items to obtain the corresponding cognitive domain score. Patients complete text-based question-and-answer, voice-based responses, selection-based answers, drawing operations, or action execution tasks through user terminal 101; the system correspondingly collects text responses, voice transcription, selection results, drawing results, drawing trajectories, action videos, or video frame sequences.

[0037] The Western Medicine Cognition Scale assessment unit 121 can determine the item type 1210 based on the item scoring method, dividing the Western Medicine Cognition Scale items into rule-based scoring items and multimodal auxiliary scoring items. For multiple-choice questions, true / false questions, directed question-and-answer questions, simple memory and retelling questions, calculation questions, or items with clear standard answers, the Western Medicine Cognition Scale assessment unit 121 adopts rule-based scoring 1211, that is, it generates corresponding item scores and rule-based scoring criteria based on preset standard answers, keyword matching rules, score mapping rules, or manually set scoring rules.

[0038] For items that require judgment based on non-textual information, the Western Medicine Cognitive Scale Assessment Unit 121 employs multimodal auxiliary scoring 1212. It can first preprocess and extract structured auxiliary features from the drawing results, drawing trajectories, speech expression content, action videos, or video frame sequences. Then, the structured auxiliary features, original multimodal data, or combinations thereof, along with the item requirements, scoring rules, and output format constraints, are input into the multimodal large model. Combined with preset scoring rules, auxiliary analysis is performed to generate item scores, reasons for deductions, scoring basis, and abnormal prompts.

[0039] For drawing-related questions, the auxiliary features may include key components, relative positions, drawing order, time taken, pauses, repeated drawing, or spatial positional deviations. For action execution-related questions, the auxiliary features may include action keyframes, step sequence, completion status, reaction time, and prompts for missing steps or incomplete execution. For speech expression-related questions, the auxiliary features may include speech-to-text transcription, keyword matching results, semantic similarity, answer completeness, and accuracy of pauses or restatements.

[0040] In one specific implementation, for the multimodal auxiliary scoring 1212, the input multimodal large model prompt words can adopt the format shown in Table 1 below: Table 1 Multimodal Assisted Scoring Prompt Template

[0041] The Western Medicine Cognitive Scale assessment unit 121 finally summarizes the scores of the rule-based scoring items and the multimodal auxiliary scoring items to form the Western Medicine Cognitive Scale assessment result. The assessment result includes the score for each item, the total score, the cognitive domain score, the basis for deductions, abnormal item prompts, and scoring explanations. The assessment result can be written to the data storage area 160 and used as input for subsequent integrated Chinese and Western medicine reasoning and assessment result generation processes.

[0042] This embodiment combines rule-based scoring, structured auxiliary feature extraction, and multimodal large model-assisted scoring to integrate text question answering, voice response, drawing tasks, and action execution tasks into the auxiliary evaluation process, and avoids limiting scoring capabilities entirely to a single model structure.

[0043] Example 3: Process of Structured Feature Extraction for Traditional Chinese Medicine Constitution Identification and Facial and Tongue Diagnosis Reference Figure 3 This embodiment illustrates the process of TCM constitution identification and structured feature extraction for facial and tongue diagnosis. The TCM constitution identification unit 122 and the TCM facial and tongue diagnosis feature extraction unit 123 process TCM constitution identification scale data, facial image data, and tongue image data, respectively, and output TCM assessment results that can be called by the fusion inference unit 124.

[0044] The TCM constitution identification unit 122 receives the answer data of the TCM constitution identification scale, which includes the selection results input by the patient or caregiver based on the scale items, as well as scale answer information related to symptom perception, sleep status, diet, emotional state, cold and heat manifestations, and physical condition. The TCM constitution identification unit 122 performs item processing 1221, that is, according to the preset item score mapping rules, it performs score calculation 1222 and standardization 1223 on each constitution dimension.

[0045] The TCM constitution identification unit 122 determines the patient's main constitution, tendency constitution, and related constitution characteristics (constitution determination 1224) based on the standardized scores of each constitution dimension. The constitution dimensions may include balanced constitution, qi deficiency constitution, yang deficiency constitution, yin deficiency constitution, phlegm-dampness constitution, damp-heat constitution, blood stasis constitution, qi stagnation constitution, and special constitution, etc.

[0046] The TCM facial and tongue diagnosis feature extraction unit 123 performs image quality detection and preprocessing on facial image data and tongue image data. The preprocessing includes one or more of the following: image clarity judgment, shooting angle judgment, illumination correction, target area cropping, and size standardization, in order to reduce the impact of shooting environment differences on subsequent feature extraction.

[0047] For facial image data, the TCM facial and tongue diagnosis feature extraction unit 123 extracts structured facial diagnosis feature labels such as complexion, luster, lip color, and local color changes through facial region localization, key region segmentation, and feature classification. For tongue image data, the TCM facial and tongue diagnosis feature extraction unit 123 extracts structured tongue diagnosis feature labels such as tongue color, coating color, tongue shape, thickness and moisture of the tongue coating, teeth marks, and cracks through tongue region localization, tongue region segmentation, and multi-label feature classification.

[0048] The TCM facial and tongue diagnosis feature extraction unit 123 can sequentially perform structured processing using an image preprocessing unit 1231, a target region localization unit 1232, a target region segmentation unit 1233, and a multi-label classification unit 1234. Specifically, the image preprocessing unit 1231 performs quality detection and standardization on the input image; the target region localization unit 1232 determines the facial or tongue region; the target region segmentation unit 1233 reduces background interference; and the multi-label classification unit 1234 outputs facial or tongue diagnosis feature labels. These units can be implemented using deep learning models, image processing algorithms, or rule-based models.

[0049] The TCM constitution identification results output by the TCM constitution identification unit 122, and the structured feature labels for facial diagnosis and tongue diagnosis output by the TCM facial and tongue diagnosis feature extraction unit 123, together form the TCM assessment result summary 1235. The TCM assessment results can be written to the data storage area 160 and used as input for subsequent TCM-Western medicine integrated reasoning and assessment result generation processes. The TCM assessment results are used to help reflect the patient's overall condition and constitution characteristics, and are not considered as independent clinical diagnostic conclusions.

[0050] This embodiment transforms the answer data of the TCM constitution identification scale, facial image data, and tongue image data into structured results, enabling constitution characteristics, facial diagnosis characteristics, and tongue diagnosis characteristics to participate in subsequent fusion reasoning in a computable and callable manner.

[0051] Example 4: Process of Integrating Traditional Chinese and Western Medicine Reasoning, Generating Auxiliary Assessment Results, and Constructing Constraint Information Reference Figure 4 This embodiment illustrates the process of integrating traditional Chinese and Western medicine inference, generating auxiliary assessment results, and constructing constraint information. The fusion inference unit 124 receives the assessment results from the Western medicine cognitive scale, the results from traditional Chinese medicine constitution identification, and the structured features from traditional Chinese medicine facial and tongue diagnosis. It then performs unified expression and fusion processing on these results to generate the patient's cognitive level auxiliary assessment result 125. The constraint information construction module 126 is located within the integrated traditional Chinese and Western medicine cognitive auxiliary assessment module 120 and is connected to the fusion inference unit 124. It receives the patient's cognitive level auxiliary assessment result 125 and determines the constraint information 127 based on this result. The patient's cognitive level auxiliary assessment result 125 and the constraint information 127 can be written to the data storage area 160. The constraint information 127 can also be input into the personalized intervention module 140 based on graph retrieval enhancement to constrain the generation of subsequent personalized intervention content.

[0052] The fusion reasoning unit 124 converts evaluation results from different sources into a fusion feature set. The fusion feature set includes item scores, total scores, cognitive domain scores, deduction criteria, and abnormal item prompts from the Western Medicine Cognitive Scale; scores for each constitution dimension, main constitution, and predisposing constitution from the constitution identification results; and structured feature labels from facial and tongue diagnosis. In one specific embodiment, the fusion feature set is represented in JSON format to uniformly express evaluation features from different sources. Table 2. Representation of the fused feature set

[0053]

[0054] Among them, the relevant features of the Western medicine cognitive scale serve as the main basis for auxiliary assessment of cognitive level, while the results of TCM constitution identification, facial diagnosis structured features, and tongue diagnosis structured features serve as auxiliary basis for overall status indication, non-diagnostic conditioning suggestions, and intervention strategy adjustments; the above JSON only lists the tongue diagnosis structured features as an example, and the facial diagnosis structured features can be represented using the same field structure.

[0055] The fusion reasoning unit 124 performs comprehensive reasoning based on the fusion feature set to generate an auxiliary assessment result 125 of the patient's cognitive level. The comprehensive reasoning processes the correspondence, supplementary relationships, and prompting relationships between various features according to preset cognitive domain mapping rules, feature weight rules, anomaly prompting rules, and natural language generation rules. The preset cognitive domain mapping rules include the correspondence between items of the Western medicine cognitive scale and cognitive domains, as well as the association between intervention intention and cognitive domains; wherein, the correspondence between items of the Western medicine cognitive scale and cognitive domains is used to calculate the scores of each cognitive domain, and the association between intervention intention and cognitive domains is used to determine the cognitive domain associated with the current intervention intention and its associated weight. The assessment results of the Western medicine cognitive scale serve as the primary basis, while the results of traditional Chinese medicine constitution identification and the structured features of traditional Chinese medicine facial and tongue diagnosis serve as auxiliary basis; when data quality is insufficient or results from different sources conflict, the system can generate a review prompt, reduce the weight of relevant features, or mark low-confidence results.

[0056] The patient's cognitive level auxiliary assessment result 125 includes the cognitive level auxiliary assessment conclusion, cognitive risk classification indication, performance in major cognitive domains, explanation of assessment basis, abnormal item indication, TCM constitution and facial and tongue diagnosis related indications, and non-diagnostic conditioning and intervention suggestions. The patient's cognitive level auxiliary assessment result 125 is intended to provide reference for patients, families, caregivers, or professionals and should not be considered as an independent clinical diagnostic conclusion.

[0057] The patient's cognitive level auxiliary assessment results 125 and the generated constraint information 127 can be written into the data storage area 160 for subsequent querying, display, or use in the personalized intervention process. The data storage area 160 can store information such as assessment time, assessment items, fusion feature set, assessment results, assessment basis explanation, and non-diagnostic suggestions.

[0058] The constraint information generation module 126 is connected to the fusion reasoning unit 124 and is used to receive the patient's cognitive level auxiliary assessment result 125 and construct constraint information 127 based on the patient's cognitive level auxiliary assessment result 125. The constraint information 127 can be determined based on cognitive risk grading prompts, performance of major cognitive domains, abnormal question prompts, task completion rate, TCM constitution and facial and tongue diagnosis related prompts, including one or more of the following: question difficulty, prompt intensity, language complexity, waiting time, and multimedia display method. The patient's cognitive level auxiliary assessment result 125 and the constraint information 127 can be written into the data storage area 160 for subsequent querying, display, or use in the personalized intervention process. The data storage area 160 can store information such as assessment time, assessment items, fusion feature set, patient's cognitive level auxiliary assessment result, assessment basis explanation, constraint information, and non-diagnostic suggestions.

[0059] Based on the above-mentioned cognitive domain division and mapping relationship, in order to quantitatively transform the auxiliary assessment results of the patient's cognitive level into constraint information, the constraint information generation module 126 first determines the cognitive domain associated with the current intervention intention based on the correlation between the current intervention intention and the cognitive domain, and calculates the cognitive deficit normalization coefficient based on the actual score, the highest score and the corresponding correlation weight of the cognitive domain. The calculation formula is: (1) In equation (1), This represents the number of cognitive domains that have been completed and whose results are valid in this calculation. For the first The actual score of each cognitive domain; For the first The highest score in each cognitive domain (i.e., the theoretical highest score), among which ,and ; For the first The normalized comprehensive weights of each cognitive domain; the normalized comprehensive weights Determined according to the following formula: (2) in, For the first Each cognitive domain corresponds to a preset cognitive domain influence weight. Non-negative weights; For the first The association weights between each cognitive domain and the current intervention intention The value range is from 0 to 1, and it is guaranteed that the corresponding cognitive domains are... and The sum of the products is greater than 0; when no current intervention intention is specified, each All are set to 1. After normalization, all The sum is 1. The value range is from 0 to 1. The larger the value, the higher the degree of cognitive deficit the patient has in the cognitive domain associated with the current intervention intention, and the higher the level of cognitive support required for the current interaction.

[0060] Furthermore, the constraint information generation module 126 normalizes the cognitive deficit coefficients. The proportion of abnormal questions Task completion rate and overall state auxiliary adjustment coefficient Integrating into the cognitive support demand coefficient The calculation formula is: (3) In formula (3), the number of items in the Western Medicine Cognition Scale that have been completed and have valid results is... When greater than 0, , The number of abnormal items, and ; This is the ratio of the number of completed tasks to the number of tasks that should be completed, and its value ranges from 0 to 1, with the number of tasks that should be completed being greater than 0. The overall state auxiliary adjustment coefficient is obtained by pre-mapping and normalization based on the abnormal constitution dimension, main constitution and tendency constitution of the TCM constitution identification scale, as well as the prompts related to the current interaction state in the structured features of facial diagnosis and tongue diagnosis. Its value ranges from 0 to 1. , , and The weights are non-negative and their sum is 1, where Greater than This approach uses Western medical cognitive scale assessment results as the primary basis, with Traditional Chinese Medicine (TCM) constitution and facial / tongue diagnosis as supplementary evidence. The cutoff function used in the formula is defined as follows: (4) The truncation function is used to truncate variables. Limit to the lower limit and upper limit Between. In equation (3), Take 0, Take 1; in the calculation of the waiting time in equation (6), Pick , Pick When the number of completed questions... When the value is 0, the personalized cognitive support demand coefficient is not calculated according to equation (3). The cognitive support demand coefficient Set to preset initial value This is to determine the initial generation constraint information required for the first personalized intervention interaction. The preset initial value... This can be preset by professionals, for example, set to 0.5, so that the initial personalized intervention interaction uses moderate prompt intensity, moderate problem difficulty, and moderate language complexity; if an effective auxiliary assessment result of the patient's cognitive level is obtained, then the cognitive support need coefficient calculated based on the auxiliary assessment result is used. Replace the preset initial value The preset mapping can encode no relevant prompts, slightly relevant prompts, and obviously relevant prompts as 0, 0.5, and 1, respectively, and then sum them up according to the importance of each prompt.

[0061] After the patient completed the first After each round of responses, the personalized intervention module 140 calculates the interaction difficulty coefficient based on the response results and reactions. In order to determine the first constraint based on the initial generated constraint information. The generation constraint information of the wheel is calculated as follows: (5) In equation (5), For the patient in the first The indicator value is 1 if the answer is incorrect or the answer is significantly deviating from the current question; otherwise, it is 0. The indicator value is 1 if the patient does not respond or requests a repeat prompt within the preset longest reference response time, and 0 otherwise. For the first Actual reaction time; The maximum reference response time is preset, and ; , and Each has a non-negative weight and their sum is 1. When the patient is... If there is no response, order , And order From this, we obtain The value ranges from 0 to 1. The larger the value, the better the patient's condition at the time of diagnosis. The higher the difficulty of the interaction in each round, the greater the difficulty. At the start of the first personalized intervention interaction, because feedback from the previous round of interaction has not yet been obtained, the initial interaction difficulty coefficient is set higher. Set to 0. Personalized intervention module 140 calculates the [number]th [intervention] according to the following formula. Wheel's cue intensity Difficulty of the problem Language complexity and waiting time : (6) In equation (6), , , and These are the benchmark hint strength, benchmark problem difficulty, benchmark language complexity, and benchmark wait time, among which... , and The value range is from 0 to 1. , and satisfy The benchmark values ​​can be preset by professionals according to the intervention scenario. , , , , and Here, is a dimensionless, nonnegative adjustment coefficient, where , used to indicate the first The degree to which the difficulty of one round of interaction modifies the language complexity of the next round; and It is a non-negative adjustment coefficient with the same time dimension as the waiting time. The larger the value, the higher the value. The more specific the people, time, place, or event facts included in the prompts; The smaller the value, the higher the value. The fewer elements of memory involved in the round question; The smaller the value, the higher the value. The shorter the sentence structure of the generated content and the less information per sentence; The larger the value, the more likely it is to enter the [number]th [stage]. The longer the waiting time before the wheel, the stronger the warning. Difficulty of the problem Language complexity and waiting time All based on the cognitive support needs coefficient and the Wheel interaction difficulty coefficient To be determined jointly.

[0062] The system can be based on the cognitive support demand coefficient. Set the initial generation constraint level: When the difficulty level is below the first threshold, use low-intensity prompts and medium-to-high difficulty questions. When the question is between the first and second thresholds, use medium-intensity prompts and medium-difficulty questions. When the threshold is higher than the second threshold, high prompt intensity, low-difficulty short sentences, extended waiting time, and image priority display are used, and based on the initial generated constraint level, according to the first... Wheel interaction difficulty coefficient Adjustments are made at the round-level. The first threshold can be set to 0.35, and the second threshold can be set to 0.65. The thresholds, weights, and adjustment coefficients can be preset by professionals or calibrated based on confirmed historical interaction records. For example, , , and The values ​​were taken as 0.65, 0.15, 0.15, and 0.05 in sequence; when It is 0.68 and When the value is 0.25, take , , ,available Approximately 0.67; take , , ,available Approximately 0.24; take , , ,available Approximately 0.24, corresponding to high hint intensity, low difficulty, and low language complexity modes; (take) For 5 seconds For 4 seconds Given a time interval of 1 second, and assuming the cutoff interval contains 7.97 seconds, we can obtain... It takes approximately 7.97 seconds; the example uses 8 seconds.

[0063] Through the above methods, the system integrates the assessment results of Western medicine cognitive scales, the results of traditional Chinese medicine constitution identification, and the structured features of traditional Chinese medicine facial and tongue diagnosis into the fusion reasoning process, and transforms the auxiliary assessment results of the patient's cognitive level into the generation constraint information required for subsequent personalized intervention. After the patient completes each round of responses, the system dynamically adjusts the generation constraints for the next round based on the difficulty coefficient of that round of interaction, so that the prompting level, question difficulty, language complexity, and waiting time of the generated content match the patient's current cognitive level and interaction performance.

[0064] Example 5: Personal Memory Map Construction Process Reference Figure 5This embodiment describes the specific construction process of the personal memory map construction module 130. The personal memory map construction module 130 is used to receive pictures, videos, audio, text materials and their descriptive information uploaded by the patient, family members or caregivers, and construct a personal memory map reflecting the patient's personal experiences, family relationships and life events based on the above materials.

[0065] The personal memory map construction module 130 first performs data preprocessing 131 on the uploaded data, including multimedia data storage, numbering, format recognition, content organization and descriptive information association, so that each picture, video, audio or text data has a resource identifier and can be associated with the subsequently extracted information on people, time, place, event or item.

[0066] The personal memory graph construction module 130 performs entity extraction 132 based on the multimedia materials and their descriptive information to obtain graph nodes such as people, time, place, event, item, emotional attributes, and multimedia resources. Multimedia resources participate in graph construction as resource nodes, enabling images, videos, audio, or text materials to be associated with nodes such as people, events, places, and time. This facilitates the use of images, videos, audio, or text materials as fact nodes in the personal memory graph for retrieval and context generation.

[0067] The personal memory graph construction module 130 further extracts relationships from the graph nodes 133, forming edge relationships including relationships between people, relationships between participants in events, relationships between locations, relationships between times of occurrence, relationships between items, relationships between emotions, relationships between resources, and relationships between the order of events.

[0068] During the entity extraction and relationship extraction process, the patient, family members, or caregivers can confirm, supplement, or correct the extraction results. This confirmation, supplementation, or correction includes verifying or supplementing information such as person identity, kinship, event name, time of occurrence, location of occurrence, resource description, and node relationships to improve the accuracy of factual information in the personal memory map.

[0069] The personal memory graph construction module 130 writes the confirmed graph nodes and edge relationships into the data storage area 160, forming a personal memory graph 134 that supports multi-hop retrieval. The personal memory graph 134 can express the associations between the patient's personal experiences, family relationships, life events, and multimedia resources, and provides a factual basis for subsequent retrieval enhancement generation.

[0070] The personal memory map construction module 130 provides a retrieval interface for the personalized intervention module 140. When the patient mentions people, places, times, events, objects, or related intentions in subsequent interactions, the personal memory map 134 can be used to recall relevant memory paths, map node relationships, and associated multimedia resources, providing traceable factual evidence for subsequent generation.

[0071] Example 6: Graph Retrieval Enhancement Generation Intervention Process Based on Personal Memory Graph Reference Figure 6 This embodiment illustrates the graph retrieval enhancement generation intervention process based on a personal memory map. The personalized intervention module 140 receives voice or text information input by the patient through the user terminal 101, performs multi-hop retrieval on the personal memory map 134 based on the voice or text information, and constructs the generation context by combining the retrieval results, the patient's cognitive level auxiliary assessment results 125, and the generation constraint information 127. The multimodal generation unit directly generates personalized memory recall dialogue, prompt questions, or companion-style answers, and simultaneously outputs relevant multimedia materials.

[0072] The personalized intervention module 140 is used to respond to interactive requests initiated by patients via voice or text, and to perform intent parsing on the patient's voice or text input. For voice input, the system can first convert the voice into text; for text input, the system can directly perform semantic analysis. The intent parsing includes recognizing people, places, times, events, objects, kinship terms, or question intents, and forming query information for graph multi-hop retrieval 143.

[0073] The personalized intervention module 140 performs a multi-hop graph retrieval 143 in the personal memory graph 134 based on the query information. The multi-hop graph retrieval 143 revolves around person nodes, event nodes, location nodes, time nodes, item nodes, and multimedia resource nodes, recalling memory paths, graph node relationships, and associated multimedia resources related to the patient's input. In one specific embodiment, the patient, family member, or caregiver can pre-upload images and input corresponding descriptive information for the images. The personal memory graph construction module 130 performs entity extraction and relationship extraction based on the descriptive information, forming graph nodes and edge relationships related to people, time, location, events, and items, and establishing associations between the multimedia resource nodes corresponding to the images and the graph nodes. After the personalized intervention module 140 completes intent parsing of the patient's input information, the multi-hop graph retrieval 143 retrieves the corresponding graph nodes, node relationships, and associated multimedia resources based on the formed query information. For example, when the query information contains a person entity related to "daughter", the graph multi-hop retrieval 143 can recall event nodes, location nodes, time nodes and associated images related to that person entity, so that the recalled graph facts and associated images can be used for subsequent generation of context construction and multimedia synchronous output.

[0074] Simultaneously, the personalized intervention module 140 reads the patient's cognitive level auxiliary assessment results 125 and generates constraint information 127. The generated constraint information 127 includes one or more of the following: question difficulty, prompt intensity, language complexity, waiting time, and multimedia display method. The personalized intervention module 140 controls the expression of subsequent generated content based on the generated constraint information 127, ensuring that the generated memory-awakening dialogue, prompting questions, or supportive responses match the patient's current cognitive level.

[0075] The personalized intervention module 140 uses the patient's input, the patient's cognitive level auxiliary assessment results 125, the generated constraint information 127, the recalled memory path, the graph node relationships, and the associated multimedia resources as the generation context. The generation context can be organized in JSON format or using prompt word templates.

[0076] In one specific implementation, the generation context of the graph retrieval enhancement generation stage can adopt the following JSON format, or be converted from this format into multimodal large model cue words: Table 3. Generation context format for the graph retrieval enhancement generation stage.

[0077] In the JSON example above, `support_demand` represents the cognitive support demand coefficient A; `question_difficulty`, `prompt_strength`, `language_complexity`, and `waiting_time` represent the question difficulty, prompt strength, language complexity, and waiting time, respectively, determined based on the cognitive support demand coefficient and the interaction difficulty coefficient; `media_mode` represents the multimedia display mode. This JSON format is merely an exemplary organization of the generation context, and each field can be selected and set according to the actual generated constraint information.

[0078] The multimodal generation unit in the personalized intervention module 140, based on the generation context and according to the question difficulty, prompt intensity, language complexity, waiting time, and multimedia display method defined by the generation constraint information 127, directly generates personalized memory recall scripts, prompting questions, or companion-style responses, corresponding to multimodal generation 145. The template fields of the generation context include one or more of patient_input, cognitive_state, generation_constraints, retrieved_graph_facts, media_resources, generation_rules, and output_format. The generated content includes gentle prompts for familiar figures, recall guidance based on real-life events, question prompts that complement associated multimedia materials, and encouraging expressions suitable for the patient's current cognitive state.

[0079] Finally, the personalized intervention module 140 can synchronously output the generated text content and the recalled images, videos, audio or text materials (corresponding to multimedia synchronous output) to the user terminal 101. The user terminal 101 can present the intervention content to the patient through screen display, voice broadcast, image display, video playback or text display.

[0080] The data storage area 160 is used to store one or more of the following: patient cognitive level auxiliary assessment results, generated constraint information, multimedia materials, personal memory map, map nodes, edge relationships, multimedia resource index, and interaction records; the integrated traditional Chinese and Western medicine cognitive auxiliary assessment module, the personal memory map construction module, and the personalized intervention module based on graph retrieval enhancement can write or read data from the data storage area as needed.

[0081] Through the above methods, the system can combine the patient's current cognitive level auxiliary assessment results 125, generated constraint information 127, and personal memory map 134 to achieve retrieval enhancement generation intervention constrained by real personal memory, thereby improving the pertinence and traceability of personalized memory evoked intervention.

[0082] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A cognitive assessment system integrating traditional Chinese and Western medicine based on a multimodal model, characterized in that, include: Multi-source data acquisition module and integrated traditional Chinese and Western medicine cognitive auxiliary assessment module; The multi-source data acquisition module is used to collect data from Western medicine cognitive scales and related data from traditional Chinese medicine assessments. The TCM assessment data includes TCM constitution identification scale data, facial image data, and tongue image data. The integrated traditional Chinese and Western medicine cognitive auxiliary assessment module includes: The Western Medicine Cognition Scale Assessment Unit is used to score items based on the data of the Western Medicine Cognition Scale and generate the Western Medicine Cognition Scale Assessment Results. The Traditional Chinese Medicine (TCM) constitution identification unit is used to score items based on the TCM constitution identification scale data and generate TCM constitution identification results. The TCM facial and tongue diagnosis feature extraction unit is used to extract features from the facial image data and tongue image data to obtain TCM facial and tongue diagnosis structured features. The fusion reasoning unit is used to perform fusion reasoning on the assessment results of the Western medicine cognitive scale, the results of constitution identification, and the structured features of facial and tongue diagnosis according to preset reasoning rules, and generate auxiliary assessment results of the patient's cognitive level.

2. The integrated traditional Chinese and Western medicine cognitive assessment system based on a multimodal model according to claim 1, characterized in that, The Western Medicine Cognition Scale assessment unit first divides the items of the Western Medicine Cognition Scale data into rule-based scoring items and multimodal auxiliary scoring items. The rule-based scoring items are items with clear standard answers or scoring rules, while the multimodal auxiliary scoring items are items that require judgment by combining text answers, voice expressions, drawing results, drawing trajectories, action videos, or video frame sequences. The scoring items of the rules are scored according to preset rules to generate item scores; The multimodal auxiliary scoring items are preprocessed and structured auxiliary features are extracted. The extracted structured auxiliary features and the corresponding original Western medicine cognitive scale data are input into the multimodal large model along with the item requirements, scoring rules and output format constraints to generate the Western medicine cognitive scale evaluation results, which include item scores, total scores, cognitive domain scores, reasons for deductions, scoring basis and abnormal item prompts.

3. The integrated traditional Chinese and Western medicine cognitive assessment system based on a multimodal model according to claim 1, characterized in that, The data from the TCM constitution identification scale includes the selection results entered by the patient or caregiver based on the items of the constitution identification scale, as well as the scale response information related to symptom perception, sleep, diet, emotional state, cold and heat manifestations, and physical condition. The TCM constitution identification unit calculates and standardizes the scores of each constitution dimension according to the preset question score mapping rules, thereby obtaining the TCM constitution identification results of the patient; the TCM constitution identification results include the main constitution, the tendency constitution, and related constitution characteristics; the constitution dimensions include balanced constitution, qi deficiency constitution, yang deficiency constitution, yin deficiency constitution, phlegm-dampness constitution, damp-heat constitution, blood stasis constitution, qi stagnation constitution, and special constitution; And / or, the TCM facial and tongue diagnosis feature extraction unit first preprocesses the facial image data and tongue image data, then performs target region localization, key region segmentation, and feature classification, extracting facial diagnosis structured feature labels including complexion, luster, lip color, and local color changes from the facial image data; and extracting tongue diagnosis structured feature labels including tongue color, coating color, tongue shape, thickness of tongue coating, degree of moisture, teeth marks, and cracks from the tongue image data.

4. The integrated traditional Chinese and Western medicine cognitive assessment system based on a multimodal model according to any one of claims 1-3, characterized in that, The fusion reasoning unit converts the Western medicine cognitive scale assessment results, constitution identification results, and facial and tongue diagnosis structured features into a fusion feature set. Then, according to preset cognitive domain mapping rules, feature weight rules, abnormal prompting rules, and natural language generation rules, it processes the relationship between various features to generate the patient's cognitive level auxiliary assessment results. The cognitive domain mapping rules include: the correspondence between items of the Western Medicine Cognitive Scale and cognitive domains, and the association between intervention intention and cognitive domains; wherein, the correspondence between items of the Western Medicine Cognitive Scale and cognitive domains is used to calculate the scores of each cognitive domain, and the association between intervention intention and cognitive domains is used to determine the cognitive domain associated with the current intervention intention and its associated weight. The cognitive level auxiliary assessment results include: cognitive level auxiliary assessment conclusions, cognitive risk classification prompts, performance of major cognitive domains, explanation of assessment basis, prompts of abnormal items, prompts related to traditional Chinese medicine constitution and facial and tongue diagnosis, and non-diagnostic conditioning and intervention suggestions.

5. A cognitive intervention system integrating traditional Chinese and Western medicine based on multimodal models and personal memory maps, characterized in that, Includes the integrated traditional Chinese and Western medicine cognitive auxiliary assessment system based on a multimodal model, the personal memory map construction module, and the personalized intervention module based on graph retrieval enhancement as described in any one of claims 1-4; the multi-source data acquisition module is also used to collect information related to the patient's personal experience; The integrated traditional Chinese and Western medicine cognitive auxiliary assessment module also includes a generation constraint information construction module connected to the fusion reasoning unit; the generation constraint information construction module is used to construct generation constraint information based on the patient cognitive level auxiliary assessment results generated by the fusion reasoning unit, and the generation constraint information is used to constrain the process of the personalized intervention module generating memory recall scripts, prompting questions or companion-style answers. The personal memory graph construction module is used to extract entities and relationships from information related to the patient's personal experiences to construct a personal memory graph. The personalized intervention module is used to perform multi-hop retrieval of the personal memory map based on the information input by the patient, and then generate personalized intervention content under the constraints of the generated constraint information to achieve interactive intervention with the patient.

6. The integrated traditional Chinese and Western medicine cognitive intervention system based on multimodal models and personal memory maps according to claim 5, characterized in that, The generated constraint information is generated based on one or more of the following: cognitive risk level prompts, cognitive domain performance, abnormal question prompts, task completion rate, TCM constitution and facial and tongue diagnosis related prompts; the generated constraint information includes one or more of the following: question difficulty, prompt intensity, language complexity, waiting time and multimedia display method.

7. The integrated traditional Chinese and Western medicine cognitive intervention system based on multimodal models and personal memory maps according to claim 6, characterized in that, The methods for obtaining the problem difficulty, hint strength, language complexity, and waiting time include: The cognitive deficit normalization coefficient is determined based on the actual score, highest score and associated weight of the cognitive domain associated with the current intervention intention. Then, the cognitive deficit normalization coefficient, the proportion of abnormal items in the Western Medicine Cognitive Scale, the task completion rate and the overall state auxiliary adjustment coefficient obtained based on the TCM constitution identification results and the TCM facial and tongue diagnosis structured features are weighted and summed to obtain the cognitive support demand coefficient. The interaction difficulty coefficient is obtained based on the patient's errors, lack of response, and reaction time in the current round of responses. Then, the cue intensity, question difficulty, language complexity, and waiting time for the next round of interaction are determined based on the cognitive support need coefficient and the interaction difficulty coefficient. In the first round of interaction, the difficulty coefficient of the interaction is set to 0; when the number of completed items on the Western Medicine Cognition Scale is 0, the cognitive support demand coefficient takes a preset initial value.

8. The integrated traditional Chinese and Western medicine cognitive intervention system based on multimodal models and personal memory maps according to claim 7, characterized in that, The cognitive support demand coefficient is obtained according to the following formula: In the formula, A For cognitive support demand coefficient, Rc The normalized coefficient for the cognitive deficit is... Ra The proportion of abnormal questions. C This is the ratio of the number of tasks completed to the number of tasks that should be completed. Z is the term Overall status auxiliary adjustment coefficient; , , and The weights are non-negative and their sum is 1; The interaction difficulty coefficient is obtained according to the following formula: In the formula, Let be the difficulty coefficient of the interaction in round t. For the patient in the first The indicator value is 1 if the answer is incorrect or the answer is significantly deviating from the current question; otherwise, it is 0. The indicator value is 1 if the patient does not respond or requests a repeat prompt within the preset longest reference response time, and 0 otherwise. For the first Actual reaction time; The maximum reference response time is preset, and ; , and The weights are non-negative and their sum is 1; Then the first Wheel's cue intensity Difficulty of the problem Language complexity and waiting time We obtain the following formula: , , and These are the benchmark hint strength, benchmark problem difficulty, benchmark language complexity, and benchmark wait time, among which... , and The value range is from 0 to 1. , and satisfy ; , , , , and It is a dimensionless, non-negative adjustment coefficient; and It is a non-negative adjustment coefficient that uses the same time dimension as the waiting time.

9. The integrated traditional Chinese and Western medicine cognitive intervention system based on multimodal models and personal memory maps according to claim 5, characterized in that, The patient's personal experience-related information includes pictures, videos, audio, text materials, and descriptive information; The entity nodes of the personal memory map include one or more of the following: person nodes, time nodes, location nodes, event nodes, item nodes, emotional attribute nodes, and multimedia resource nodes; the edge relationships include one or more of the following: person relationships, event participation relationships, location relationships, time relationships, item association relationships, emotional association relationships, resource association relationships, and event sequence relationships. The multimedia resource nodes are associated with corresponding person nodes, time nodes, location nodes, event nodes, or item nodes, so that information related to the patient's personal experience can be used as fact nodes in the personal memory map for retrieval.

10. The integrated traditional Chinese and Western medicine cognitive intervention system based on multimodal models and personal memory maps according to any one of claims 5-9, characterized in that, The personalized intervention module includes: The intent parsing unit is used to parse the information input by the patient, identify people, places, times, events, items, kinship terms or question intents, and form query information for multi-hop retrieval of the graph; The evaluation result and generation constraint information reading unit is used to read the cognitive level auxiliary evaluation result and generation constraint information and transmit them to the generation context construction unit; The multi-hop memory retrieval unit is used to perform multi-hop memory retrieval in the personal memory graph based on the query information, and to recall memory paths, graph node relationships and associated multimedia resources related to the information input by the patient. A context building unit is used to construct a context from the cognitive level auxiliary assessment results, the generation constraint information, and the results of the multi-hop graph retrieval. A multimodal generation unit is used to generate personalized intervention content based on the generation context, so that the generated content is constrained by real nodes, edge relationships and associated resources in the personal memory graph; The multimedia synchronous output unit is used to synchronously output the generated personalized intervention content and the recalled images, videos, audio or text materials to the user terminal. The user terminal presents the intervention content to the patient through screen display, voice broadcast, image display, video playback or text display.

Images