Multimodal alzheimer's disease early screening algorithm based on large language model

Abstract

The application relates to the technical field of medical information, and specifically discloses a multi-modal Alzheimer's disease early screening algorithm based on a large language model. The algorithm first collects speech data and demographic information of participants. The speech data is processed through noise suppression, voice enhancement and speaker separation, and is transcribed into descriptive text. Then, the eGeMAPS feature set is extracted from the speech data and time-series spliced, and combined with the demographic information to form a multi-modal feature matrix. Key features are obtained using an attention pooling module, and feature classification is performed using a support vector machine to distinguish potential Alzheimer's disease patients from healthy individuals. The classification results, descriptive text and demographic information are input into the large language model, and zero-shot thinking chains are used to analyze four language barriers: reduced information content, lexical semantic defects, poor referential coherence and language loss of psychological state, and a comprehensive screening report is finally generated. The algorithm improves the accuracy of early diagnosis by combining audio and semantic features.

Description

Technical Field

[0001] This invention relates to the field of medical information technology, and in particular to a multimodal Alzheimer's disease early screening algorithm based on a big oracle model. Background Technology

[0002] Alzheimer's disease (AD) is one of the most common types of dementia and, in fact, the most prevalent cause of dementia. Dementia is a broad term used to describe a group of symptoms involving memory loss, cognitive decline, and social skill impairment to the point of interfering with daily life. Alzheimer's disease is the specific disease that causes these symptoms, while dementia itself is not a disease, but rather a collective term for a group of symptoms caused by a variety of different diseases. In short, Alzheimer's disease is a form of dementia and one of the leading causes of dementia symptoms. As AD progresses, patients' cognitive function gradually declines, eventually manifesting as typical dementia symptoms.

[0003] Compared to the average aging population, Alzheimer's disease (AD) patients exhibit significant differences in semantic comprehension, speech production, and memory function. To diagnose and assess AD symptoms, current technologies primarily rely on analyzing the speech patterns of patients describing pictures to explore subtle changes in their speech and semantics. Based on feature type, these technologies can be broadly categorized into three types: audio-based feature analysis, semantic feature analysis based on paraphrased text, and comprehensive analysis combining textual and semantic features.

[0004] Alzheimer's disease (AD) patients typically exhibit speech disfluency, such as slow speech, prolonged pauses, and difficulty searching for words. To address these characteristics, Longbin Jin et al., in their 2023 CONSEN system, innovatively combined multiple audio features, including VGGish, wav2vec, and i-vector, with 18 disfluency features extracted from the original audio, such as the frequency and number of pauses. This comprehensive analytical method can effectively identify AD patients speaking different languages.

[0005] Furthermore, Utkarsh Sarawgi et al. explored a method combining non-fluency feature sets and emotion detection audio feature sets (ComparE2013), and introduced speaker identity change sequences in dialogues as simple semantic features reflecting the patient's logical thinking ability. By comprehensively judging the results of these three feature models, they provided a more comprehensive basis for the diagnosis of AD patients.

[0006] Zih-Jyun Lin et al. proposed a new approach for AD detection using existing, mature Automatic Speech Recognition (ASR) models. Through transfer learning, they enabled the ASR model to extract specific audio features for AD patients and classify them accordingly.

[0007] However, it should be noted that current methods primarily focus on audio feature analysis. While they have shown some effectiveness in detecting AD patients, they neglect the patients' semantic features, making it difficult to deeply capture the thought processes behind their language. Considering that early-stage AD patients often exhibit more significant memory decline and semantic comprehension problems, relying solely on audio may be insufficient for accurate patient identification in the early stages.

[0008] Furthermore, these techniques face another challenge: they typically rely on large amounts of labeled data for training. However, currently available AD datasets are primarily focused on English, while English-speaking AD patients globally comprise less than 50% of the total patient population. For other languages, there is a lack of corresponding datasets for model training, which limits the practical application of these methods.

[0009] Alzheimer's disease (AD), a degenerative neurodegenerative disorder, significantly affects patients' language comprehension, speech production, and memory abilities, especially in the early stages where symptoms such as disordered speech logic and difficulty finding words appear. To effectively detect AD patients, researchers have attempted to extract information from the semantic features of patients' descriptions of pictures.

[0010] In existing research, Sarawgi et al. used the GloVe model to extract lexical features from image description text and then processed them uniformly using LSTM. Ben Ammar et al. proposed an AD detection model that uses the KNN algorithm for feature selection based on the linguistic features of patient speech transcriptions, and then trains machine learning classifiers such as Support Vector Machines (SVM) for diagnosis. Yamanki et al. further introduced a contrastive learning model based on Siamese BERT, aiming to extract discriminative features from patient speech transcriptions and other features (such as demographics, vocabulary, and semantic information) to support AD diagnosis.

[0011] Recently, Yi Wang et al. proposed a Prompt-based pre-trained language model fine-tuning method for AD detection in 2023. This method utilizes pre-trained language models such as Roberta and Bert to extract semantic features from speech-recorded text and transforms the classification problem into a label generation problem through prompting engineering techniques. However, these methods face several challenges in practical applications.

[0012] First, these methods rely on large amounts of labeled data for training, which are often limited to the content described in a specific dataset and fail to delve into changes in patients' specific logical thinking and memory abilities. Therefore, these methods have limited generalization ability, especially when faced with patients from different images or backgrounds.

[0013] Secondly, most of the currently available datasets focus on the single image "cookie theft," which limits the applicability of these models to people with similar life experiences. However, this group represents a relatively small proportion of the total AD patient population, restricting the applicability of the models in real-world clinical settings.

[0014] Finally, these semantic feature-based models lack interpretability in their output, making it difficult for clinicians to understand and trust the model's diagnostic results. This further hinders the widespread application of these models in clinical practice.

[0015] Since only the participants' voice data can be obtained initially, it needs to be converted into text information through an automatic speech recognition (ASR) model. Therefore, in practical applications, the fusion of audio and semantic features mostly occurs at the decision-making level.

[0016] In recent years, researchers have proposed a series of multimodal AD detection methods. Among them, Xuchu Chen et al. proposed an algorithm that uses the sub-language feature sets IS10 and XLSR-50 for audio features, and the Roberta model for semantic features for feature extraction and classification. Finally, they achieved multimodal comprehensive detection by fusing audio and semantic features at the decision layer through multilayer perceptron (MLP) and SVM.

[0017] Similarly, Kangdi Mei et al. proposed a multimodal AD detection algorithm with a structure similar to that of Xuchu Chen et al., but they experimented with more audio feature sets, such as ComParE and eGeMAPS, and supplemented it with a non-fluency feature set. This combination of multimodal features further improved the accuracy of the diagnosis.

[0018] Although the aforementioned methods utilize both speech and semantic features and achieve good detection results, they still face some challenges. First, these methods require large labeled datasets for model training, which may limit their application in practice. Second, while audio features are somewhat unaffected by language content, the semantic feature extraction methods used are still limited by the dataset content, thus their generalization ability needs improvement.

[0019] In 2023, Wei et al. proposed the "Chain-of-Thought" technique, which significantly improves the performance of large language models in solving complex reasoning problems by providing them with multi-step reasoning processes. However, these reasoning processes generally require manual customization, which is time-consuming and labor-intensive, and the performance improvement depends on the quality of the manually customized reasoning process. To address the problem of manual annotation of reasoning processes, Kojima et al. proposed the Zero-Shot Chain-of-Thought technique. Through the inductive instruction "Let's think step by step," the model automatically generates reasoning steps without providing them, ultimately achieving excellent results.

[0020] While existing diagnostic technologies for Alzheimer's disease (AD) have shown advantages in some aspects, they still face numerous challenges: reliance on large amounts of labeled data, lack of linguistic and cultural diversity, insufficient model generalization ability, immature multimodal data fusion, lack of interpretability of diagnostic models, and privacy and ethical issues. These problems limit the widespread application and accuracy of the technologies and urgently require improvement through technological innovation and data diversification. Summary of the Invention

[0021] The purpose of this invention is to overcome several major drawbacks of existing technologies: First, traditional methods rely on large amounts of labeled data for model training, which limits the versatility and practicality of the algorithm when data is scarce; second, screening relying solely on semantic features is easily influenced by the patient's subjective perception and lacks comprehensiveness; third, traditional screening lacks interpretability, making it difficult for doctors to understand and trust the model's diagnostic results. To address these issues, this invention proposes a multimodal AD early screening method based on a large language model. This method combines audio and semantic features, utilizing a general large language model to perform effective detection in zero-sample and small-sample scenarios, and provides interpretable diagnostic results. By using a zero-sample thought process, this invention can achieve complex reasoning and higher diagnostic accuracy without requiring large amounts of labeled data. Furthermore, the fusion of audio and semantic features enhances the comprehensiveness of the detection, making early screening more accurate and reliable.

[0022] To address the above situation, this invention provides a multimodal Alzheimer's disease early screening algorithm based on a large language model, specifically including the following steps:

[0023] S1. Collect participants' voice data and demographic information; the demographic information includes gender, age, and education level; convert the voice data into descriptive text;

[0024] S2. Extract the eGeMAPS feature set of the speech data; concatenate different time series data in the eGeMAPS feature set to generate a feature matrix with time series information, and integrate demographic information into this feature matrix to obtain a multimodal feature matrix containing audio features and demographic information.

[0025] S3. Process the multimodal feature matrix using the attention pooling module to obtain the pooled features;

[0026] S4. Use a support vector machine to classify and detect the pooled features, and obtain the audio-based prediction results and corresponding confidence scores.

[0027] S5. The descriptive text, demographic information, audio-based prediction results, and confidence scores are used as inputs to the prompting engineering of the large language model, and the output is a report on whether the participant has Alzheimer's disease screening. The semantic similarity of positive and negative training samples is introduced as a reference for the prompting engineering. The prompting engineering uses zero-shot thinking chain to analyze the multimodal data based on language barriers. The language barriers are reduced information content, lexical semantic defects, poor referential coherence, and lack of mental state language.

[0028] Preferably, step S1 further performs noise suppression, voice enhancement, and speaker separation operations on the speech data to obtain processed speech data; and converts the processed speech data into descriptive text.

[0029] Preferably, step S1 further uses the Koala model of PICOVOICE to suppress noise in the speech data to preserve human voice and enhance human voice; uses the Falcon model of PICOVOICE to separate the speaker to accurately extract the patient's speech; and uses automatic speech recognition technology to convert the patient's speech into the descriptive text and calibrate the descriptive text.

[0030] Preferably, the processed speech data is sliced ​​to extract the eGeMAPS feature set of each slice; the slices have the same duration to ensure the consistency of the time series information of the eGeMAPS feature set.

[0031] Preferably, step S3 further includes:

[0032] a. Divide the processed speech data into x segments on average, and extract the eGeMAPS feature set corresponding to each speech segment; the eGeMAPS feature set contains y acoustic parameters in y dimensions; the acoustic parameters are represented in float type;

[0033] b. Concatenate the x segments of the eGeMAPS feature set to form a matrix A of [x, y]; where matrix A ∈ R x×y Where R is a matrix in which all elements are real numbers;

[0034] c. Convert the demographic information of the current sample to float type to obtain the demographic information matrix B; the matrix B∈R n Where n represents the number of types of demographic information; the demographic information includes gender, age, and education level, etc.

[0035] d. Fill matrix B with matrix B' along the time dimension to form matrix B' of [x+n]; wherein matrix B' ∈ R x×n ;

[0036] e. Concatenate matrix A and matrix B′:

[0037] C = [A|B′] ∈ R x×(y+n) ,

[0038] Where C is the characteristic matrix.

[0039] Preferably, step 5 further includes calculating the semantic similarity between positive and negative samples using the TF-IDF method.

[0040]

[0041] TF-IDF(t,d)=TF(t,d)×IDF(t)

[0042] Where TF represents term frequency, IDF represents inverse document frequency, (t, d) represents the frequency of character t in description text d, t′ represents traversing every character t in description text d; N represents the total number of description texts, and |{d∈D:t∈d}| represents the number of description texts containing character t.

[0043] Preferably, the design of the prompting process in step 5 is as follows:

[0044] Model(f(x, a1, ..., a) n ))=p

[0045] Where Model represents the large language model, f is the prompt, p is the output of the prompt project, x is the descriptive text, and a1, ..., a2 are the descriptive texts. n This represents the thinking dimensions of n types of prompting engineering; the thinking dimensions include the gender, the age, the education level, the audio-based prediction result, and the confidence level.

[0046] Preferably, the language barrier mentioned in step S5 is defined as follows:

[0047] Reduced information content refers to a decrease in both the quantity and quality of information conveyed in language expression;

[0048] Lexical semantic defects refer to difficulties in acquiring and using vocabulary;

[0049] Poor referential coherence refers to problems with coherence and consistency in the use of pronouns and other referential expressions.

[0050] The lack of psychological state language refers to the absence of language in the descriptive text that describes a person's beliefs, desires, intentions, and emotions.

[0051] Preferably, the analysis of language barriers further includes:

[0052] The information richness and relevance of the descriptive text are analyzed and quantified. Natural language processing techniques are used to compare the differences between the descriptive text and the image being described to assess whether there is a reduction in information content.

[0053] The presence of semantic defects is assessed by detecting unconventional or erroneous word usage in the descriptive text through semantic analysis and lexical networks; preferably, the words include nouns and verbs.

[0054] Analyze the referential relationship between antecedents and corresponding pronouns in the descriptive text. When there is inappropriate use of pronouns or a lack of the antecedents, the reference coherence is assessed as poor.

[0055] Identify and analyze the psychological state language in the descriptive text. When there is a lack of language expressing emotional understanding or psychological motivation, it is assessed as a lack of psychological state language. The psychological state language includes language describing a person's beliefs, desires, intentions, and emotions.

[0056] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0057] 1. Reduced dependence on labeled data: Existing technologies typically rely on large amounts of labeled data for model training, which is limited in the AD field due to data scarcity. This invention, by using a general large language model, can work effectively even with zero or very few samples, significantly improving its practicality and applicability.

[0058] 2. Comprehensive Detection Through Multimodal Features: Existing technologies often fail to capture subtle changes in a patient's cognitive and memory abilities solely through audio features. This invention integrates audio and semantic features, ensuring more comprehensive analysis and improving the accuracy and reliability of early AD detection.

[0059] 3. Improved Interpretability and Trustworthiness: Traditional diagnostic models often lack interpretability, making them difficult for medical professionals to trust and understand. This invention addresses this issue by providing interpretable diagnostic results, utilizing a zero-shot thought chain approach to simulate human reasoning, thereby enhancing the transparency and trustworthiness of the diagnosis. Attached Figure Description

[0060] Figure 1 This is a diagram of the large language model analysis framework described in Embodiment 2 of the present invention;

[0061] Figure 2 This is a flowchart of the voice data preprocessing process described in Embodiment 2 of the present invention;

[0062] Figure 3 This is a diagram of the multimodal data fusion strategy described in Embodiment 2 of the present invention;

[0063] Figure 4 This is a schematic diagram of the feature splicing and attention pooling model structure described in Embodiment 2 of the present invention;

[0064] Figure 5 This is a schematic diagram of the prompting engineering and zero-sample thinking chain described in Embodiment 2 of the present invention;

[0065] Figure 6 This is a schematic diagram of the early Alzheimer's disease screening report of participant 202 as described in Embodiment 2 of the present invention;

[0066] Figure 7 This is a schematic diagram of the optimized prompting engineering and zero-sample thinking chain described in Embodiment 2 of the present invention;

[0067] Figure 8 This refers to the semantic similarity analysis between participant 202 and positive and negative training samples as described in Embodiment 2 of the present invention.

[0068] Figure 9 The images described are those of Embodiments 2 and 3 of the present invention;

[0069] Figure 10 This refers to the input information of participant 203 in the large language model described in Embodiment 3 of the present invention;

[0070] Figure 11 This refers to the semantic similarity analysis between participant 203 and positive and negative training samples as described in Embodiment 3 of the present invention.

[0071] Figure 12 This is a schematic diagram of the early screening report of Alzheimer's disease for participant 203 as described in Embodiment 3 of the present invention. Detailed Implementation

[0072] The present invention will be further described in detail below with reference to experimental examples and specific embodiments. However, this should not be construed as limiting the scope of the above-mentioned subject matter of the present invention to the following embodiments; all technologies implemented based on the content of the present invention fall within the scope of the present invention.

[0073] This embodiment provides a multimodal Alzheimer's disease early screening algorithm based on a large language model. The specific steps are as follows:

[0074] S1. Collect participants' voice data and demographic information. The demographic information includes gender, age, and education level, which is crucial for subsequent analysis and helps improve the diagnostic accuracy of the model. Preprocess the collected voice data, converting it into descriptive text.

[0075] S2. Extract the eGeMAPS feature set of the speech data; concatenate different time series data in the eGeMAPS feature set to generate a feature matrix with time series information, and integrate demographic information into this feature matrix to obtain a multimodal feature matrix containing audio features and demographic information.

[0076] S3. Use the attention pooling module to process the multimodal feature matrix and obtain the pooled features.

[0077] S4. Use a support vector machine to classify and detect the pooled features, and obtain audio-based prediction results and corresponding confidence scores to distinguish between potential Alzheimer's patients and healthy individuals.

[0078] S5. The descriptive text, demographic information, audio-based prediction results, and confidence level are used as inputs to the prompting engineering of the large language model, and the output is a screening report on whether the participant has Alzheimer's disease. The prompting engineering uses zero-shot thinking chain to analyze the multimodal data based on language barriers. The language barriers are reduced information content, lexical semantic defects, poor referential coherence, and lack of mental state language.

[0079] The generated screening reports will provide clinicians with crucial diagnostic information to help identify and intervene in Alzheimer's disease at an early stage.

[0080] Through this series of steps, this embodiment not only improves the accuracy and efficiency of early screening for Alzheimer's disease, but also increases the interpretability of the diagnostic process, providing strong decision support for medical professionals and patients' families.

[0081] Example 2

[0082] This embodiment proposes a multimodal Alzheimer's disease (AD) early screening algorithm based on a large language model. This method not only utilizes the excellent task reasoning and general task solving capabilities of the large language model, but also integrates audio and semantic features to improve diagnostic accuracy.

[0083] Alzheimer's disease is the most common form of dementia, characterized by symptoms such as memory loss, cognitive decline, and decreased social skills, severely impacting daily life. In this embodiment, the terms 'Alzheimer's disease' and 'dementia' are used to describe the symptoms of dementia caused by AD, emphasizing that AD is the leading cause of dementia.

[0084] A significant advantage of large language models is their ability to learn effectively even in the absence of task-specific data, which is particularly crucial in early AD screening. AD patients exhibit clear language impairments in the early stages, including semantic, pragmatic, and grammatical aspects. Large language models can effectively capture these linguistic features by leveraging their powerful contextual learning and semantic understanding capabilities.

[0085] However, because semantic content is easily influenced by subjective factors, relying solely on semantic analysis may be insufficient to ensure accuracy. Therefore, this embodiment uses a multimodal detection strategy, combining audio and semantic features to comprehensively assess the patient's language performance, thereby improving detection accuracy. Figure 1 The diagram illustrates this multimodal data processing flow.

[0086] S1. Preprocessing of voice data

[0087] like Figure 2 As shown, this embodiment performs three preprocessing operations on the acquired speech data: noise suppression, speaker separation, and speech transcription. The Koala model of PICOVOICE is used to suppress noise in the speech data to preserve human voice and enhance it; the Falcon model of PICOVOICE is used for speaker separation to accurately extract the patient's speech; then, the ASR model is used to transcribe the patient's speech, and the transcribed text information is calibrated.

[0088] S2. Feature Extraction and Concatenation

[0089] In the AD detection method targeting audio features, this embodiment uses the eGeMAPS feature set (extended Geneva Minimalistic Acoustic Parameter Set) to classify speech. This is an efficient acoustic parameter set that includes the temporal concatenation of various audio features. This concatenation method not only covers the basic features of sound but also includes some more complex acoustic measurements, such as fundamental frequency variation, formant features, and multi-dimensional acoustic parameters such as sound quality.

[0090] First, the speech data is precisely sliced ​​to ensure each slice has the same duration, thus maintaining the consistency of time-series information. Next, for each slice, its corresponding eGeMAPS feature set is extracted. The eGeMAPS feature sets from different time series are concatenated to obtain a matrix sequence containing time-series information. These audio features are combined into a feature vector using a time-series concatenation method, which will be used as input data for subsequent model training and analysis. This time-series concatenation method not only preserves the time-series information in the audio but also enhances the correlation between features. This feature vector contains multi-dimensional acoustic parameters, each represented by a float type, ensuring data processing consistency. Next, patient demographic information is filled along the time-series dimension and concatenated into the matrix sequence containing time-series information to obtain a multimodal feature matrix containing audio features and demographic information. The specific steps are as follows:

[0091] S21. Divide an audio segment into x segments on average, and extract the eGeMAPS feature set corresponding to each segment. The eGeMAPS feature set contains y acoustic parameters, all represented as float type.

[0092] S22. Concatenate the x segments of the eGeMAPS feature set to form a two-dimensional matrix A of [x, y].

[0093] A∈R x× y

[0094] In this matrix R, all elements are real numbers.

[0095] S23. Convert the demographic information of the current sample to float type and obtain the demographic information matrix B:

[0096] B∈R n

[0097] Here, n represents the number of demographic data types. When the demographic data consists of three types—gender, age, and education level—n = 3. The higher the correlation between demographic data and Alzheimer's disease, and the more diverse the types of demographic data, the more accurate the prediction results.

[0098] S24. Fill matrix B with [x+n] matrix B' according to the time dimension:

[0099] B′∈R x×n

[0100] S25. Concatenate matrix A and matrix B′ to obtain the multimodal feature matrix C:

[0101] C = [A|B′] ∈ R x×(y+n)

[0102] S3, Attention Pooling Processing

[0103] The attention pooling module is used to process the multimodal feature matrix. Figure 4 The diagram below illustrates the structure of the feature concatenation and attention pooling model. The specific steps for performing attention pooling on the multimodal feature matrix are as follows:

[0104] (1) Calculation of attention weights: Using the attention pooling module, the model calculates an attention weight for each feature in the multimodal feature matrix. These weights reflect the importance of each feature in a specific task. Features with higher weights will be given greater influence in the final output.

[0105] (2) Weighted Pooling: Based on the calculated attention weights, the model will perform a weighted pooling operation, that is, use the weights to perform a weighted average or weighted summation on the multimodal feature matrix. In this way, the final feature representation will focus on the most critical features.

[0106] S4 and SVM classification detection

[0107] Next, SVM was used to classify and detect the attention-pooled features, obtaining audio-based prediction results and corresponding confidence scores to initially distinguish between AD patients and healthy individuals. In the prediction results, "dementia" represents AD patients and "control" represents healthy individuals.

[0108] To further improve the accuracy and robustness of classification, this embodiment uses 10 different random seeds to randomly initialize the parameters of the model, then trains them separately, then uses small samples for k-fold fine-tuning, and finally sums and averages the parameters of the 10 fine-tuned models.

[0109] S5. Multimodal Alzheimer's Disease Early Screening Based on Large Language Model

[0110] S51. Input Data Preparation

[0111] In this embodiment, the application of SVM classification detection and a large language model constitutes a multi-layered diagnostic strategy, aiming to improve the accuracy and reliability of AD diagnosis by leveraging the advantages of different methods. In the initial stage of the processing flow, SVM classification detection is used to quickly screen patients and healthy individuals, classifying them based on relatively intuitive data such as audio features and demographic information. This step aims to efficiently identify potential AD patients and healthy control samples, providing a target group for subsequent in-depth analysis. After SVM classification detection, more in-depth semantic analysis is performed on the data of suspected AD patients. The large language model can comprehensively consider language impairment characteristics, such as reduced information content and lexical semantic defects, to conduct more complex cognitive function assessments. This step not only enhances the depth of diagnosis but also provides interpretability of the diagnostic results.

[0112] Figure 5 This paper demonstrates how Prompt Engineering can be used to optimize the input of large pre-trained language models, where zero-shot thought chains are used as a specific implementation to handle complex problems without training data by simulating stepwise human reasoning. This technique guides the model through detailed diagnostic analysis using specific task descriptions and input formats to determine whether a participant is healthy or has Alzheimer's disease (AD). In this embodiment, the use of zero-shot thought chains not only improves the diagnostic accuracy of AD but also enables the model to operate effectively with limited labeled data and enhances the interpretability of diagnostic results through stepwise reasoning, which is crucial for improving the accuracy and reliability of medical diagnoses.

[0113] S52, Language Barrier Recognition

[0114] Based on Louise et al.'s in-depth study of the differences in language descriptions between hundreds of normal individuals and AD patients in *Describing the cookie theft picture: Sources of breakdown in alzheimer's dementia*, this embodiment summarizes the core language impairments in AD patients. These impairments, ranked by severity, mainly include reduced information content, lexical semantic deficits, poor referential cohesion, and lack of mental state language. To more accurately analyze these impairments, this embodiment limits the model analysis to the following four aspects and analyzes each language impairment individually, ultimately providing a final diagnostic result.

[0115] (1) Reduced information content

[0116] Definition: Reduced information content refers to a decrease in both the quantity and quality of information conveyed in verbal expression. Patients are unable to provide sufficient detail when describing events, objects, or scenes, resulting in a lack of specificity and completeness in their narratives. This is often due to cognitive decline leading to weakened memory and poor concentration.

[0117] Analysis Methods: The information richness of patients’ descriptions of specific pictures or scenes (such as the frequency and level of detail of mention of key objects, actions and attributes) was analyzed and quantified. Natural language processing techniques were used to compare the differences between the description text and the picture being described (such as missing information or omission of important details) to assess whether there was a reduction in the amount of information.

[0118] (2) Lexical semantic defects

[0119] Definition: Lexical semantic deficits refer to difficulties in acquiring and using vocabulary. Patients experience challenges in finding appropriate words to express their thoughts and may use incorrect or inaccurate words to describe specific objects or situations. This often manifests as using overly general or irrelevant words instead of more specific or appropriate ones, reflecting a decline in semantic memory.

[0120] Analysis methods: Semantic analysis and lexical network analysis are used to describe the text (especially to analyze the usage of specific nouns and verbs), detect unconventional or incorrect word usage (such as replacing specific nouns with overly generic or irrelevant words), and assess whether there are any semantic defects in the text.

[0121] (3) Poor coherence of reference

[0122] Definition: Poor referential coherence refers to problems with the coherence and consistency of the use of pronouns and other referents (such as noun phrases). Patients may have difficulty using pronouns correctly to refer to previously mentioned content, or may use pronouns prematurely in inappropriate contexts, making the conversation or narrative difficult to understand. Poor referential coherence reflects a patient's difficulty in maintaining the structure and logical flow of a conversation.

[0123] Analysis Method: Analyze the referential relationships between antecedents and corresponding pronouns in the descriptive text, especially whether the pronouns appropriately refer to the entities mentioned by the antecedents. Then, assess the coherence and correctness of pronoun usage. When there is inappropriate pronoun usage or a lack of the stated antecedent, it is assessed as poor referential coherence.

[0124] (4) Lack of psychological state language

[0125] Definition: Lack of mental state language refers to the absence of mental state language in descriptive texts. Mental state language includes language used to describe a person's beliefs, desires, intentions, and feelings. Individuals with Alzheimer's disease (AD) may have difficulty understanding or expressing their own or others' mental states, which is often associated with a weakened "Theory of Mind" (ToM) capacity—an impaired ability to infer the thoughts and feelings of others.

[0126] Analysis Method: Analyze descriptive texts about interpersonal interactions and personal feelings, analyze the richness and appropriateness of psychological state language in the descriptive texts, and then conduct sentiment analysis on the descriptive texts; when the descriptive texts lack language expressing emotional understanding or psychological motivation, they are assessed as lacking psychological state language.

[0127] S53, Comprehensive Analysis of Large Language Models

[0128] The descriptive text, demographic information, audio-based predictions, and corresponding confidence scores are input into the prompting engineering of a large language model. Based on the given prompts, the large language model employs a zero-shot thought process to conduct in-depth semantic analysis and logical reasoning across n dimensions of the prompting engineering. Subsequently, the model outputs a detailed analysis report, including the identification and analysis process of the language disorder and a preliminary diagnosis of whether the user has Alzheimer's disease (AD). The prompting engineering formula is as follows:

[0129] Model(f(x, a1, ..., a) n ))=p

[0130] Where Model represents the large language model;

[0131] f: indicates prompting engineering, which guides the model on how to process input data, perform reasoning, and produce expected output. Through well-designed prompts, it ensures that the large language model can understand and process information in a predetermined way.

[0132] x: The participant's descriptive text, which forms the basis for Model generation analysis;

[0133] a1, ..., a n : Represents n dimensions of consideration for the prompting project, including medical symptom evidence and audio characteristics related to AD;

[0134] p: The Model outputs the results after comprehensive analysis of the input descriptive text, audio features, and demographic information. The output is not just a simple label, but includes detailed analysis and possible diagnostic suggestions.

[0135] Figure 6 This study demonstrates the application of large language models in early Alzheimer's disease (AD) screening using zero-shot thought chains, specifically illustrating how the model analyzes participants' descriptions of scenarios to identify language disorders. In this process, the model assesses whether participants' descriptions exhibit four main language disorders: reduced information content, lexical semantic deficits, poor referential coherence, and lack of mental state language. Through the identification and semantic analysis of these language disorders, the model can determine whether participants are exhibiting early symptoms of AD. Figure 6 The diagram of the 202 Alzheimer's disease early screening report effectively demonstrates how a large language model can assess disease risk based on language performance, thereby providing a preliminary diagnostic basis for potential AD patients.

[0136] The prompt indicates that the project consists of three parts, namely<Task Description> (Task Description)<User Input> (User input) and<Model Ouput> (Model output).

[0137] The task description in the project is as follows:

[0138] <Task Description>Combining relevant data from the dataset with pathological evidence associated with Alzheimer's disease (AD), this study focuses on analyzing four main language impairments in descriptive texts: reduced information content, lexical semantic deficits, poor referential coherence, and lack of mental state language, to determine whether participants have AD. Furthermore, to mitigate the influence of differences in education level and subjective awareness,<Task Description> The study also incorporated participants' demographic information, such as age, gender, and education level, aiming to improve the objectivity and accuracy of the diagnosis. In this way, large language models can take into account social and psychological factors that may influence language use during the analysis process, thereby providing more accurate and personalized diagnostic results.

[0139] Research has found that language use is influenced by a variety of factors, including gender, age, education level, and an individual's subjective consciousness and emotional state. Positive emotions generally encourage the use of a more diverse vocabulary and more complex sentence structures, while negative emotions may lead to simpler, more repetitive language use. Individuals with higher levels of education typically use more precise and richer language with clearer logic; conversely, their language may be more simplistic and illogical. These psychosocial factors pose additional challenges to the diagnosis of Alzheimer's disease (AD) because relying solely on language characteristics is insufficient for a completely accurate diagnosis and carries a certain risk of bias.

[0140] To accurately consider specific factors during the diagnostic process, especially the participant's demographic information and emotional state, this embodiment...<Task Description> Specific demographic information of the test samples was incorporated to reduce the subjective influence of individual differences on diagnostic results. This embodiment established an optimized task description by explicitly focusing the task description on diagnosing whether participants have dementia or are in good health. The optimized task description clearly states that the model needs to perform step-by-step analysis and judgment based on four language impairments. This structured evaluation method greatly improves the model's systematicity and diagnostic transparency.

[0141] like Figure 7 As shown, the optimized prompt project<Task Description> The instructions are as follows: "Based on the examples provided below, and considering the participant's demographic factors (age, sex, and education level) and audio detection results, please analyze the description to diagnose whether it is dementia or a healthy state. Focus on the following aspects: reduced information content, semantic deficits in vocabulary, lack of mental state language, and poor referential coherence. Provide a detailed step-by-step analysis and judgment based on these four criteria. For healthy individuals, please provide 'control'; for AD patients, please provide 'dementia'."

[0142] By incorporating consideration of emotional states, this optimization ensures<Task Description> It can provide more comprehensive guidance for large language models in the early screening and diagnosis of AD, and enhances the objectivity and accuracy of diagnosis by introducing demographic information and a structured analysis process.

[0143] Prompt user input in the project:

[0144] <User Input> The input includes participant descriptions, demographic information, and audio-based predictions and corresponding confidence scores.

[0145] Audio Prediction+Confidence provides the model with preliminary analysis results and confidence levels of participants' speech characteristics, obtained based on probability calculations. By integrating these audio analysis results and confidence information into the model input, not only is the understanding of Alzheimer's disease symptoms enhanced, but the accuracy of diagnosis is also improved. Furthermore, this method supports the model in making more accurate diagnostic decisions by weighing the reliability of preliminary cues in complex or ambiguous situations.

[0146] Leveraging the zero-shot thought process, the model can derive diagnostic results directly from user input without providing specific inference samples. Since the training set and test samples describe semantically identical content, differing only in the words and sentences used, and TF-IDF (Term Frequency-Inverse Document Frequency) is a statistical technique used to measure the importance of words in a set of descriptive texts, we use this method to calculate semantic similarity. First, we calculate the TF-IDF matrix for all samples in both the training and test sets. Then, we calculate the cosine similarity between the test sample and each sample in the training set. The closer the cosine similarity is to 1, the more similar the two texts are. Higher text similarity corresponds to more similar content and vocabulary in the descriptive texts, making the sample more reliable.

[0147] For each descriptive text d and character t, calculate the term frequency (TF) and inverse document frequency (IDF).

[0148]

[0149] TF-IDF(t,d)=TF(t,d)×IDF(t)

[0150] Where (t, d) represents the frequency of character t in the description text d, t′ represents traversing each character t in the description text d; N represents the total number of description texts, and |{d∈D:t∈d}| represents the number of description texts containing character t.

[0151] To improve the accuracy of inference, this embodiment specifically introduces negative training samples from the training set that are closest to the current test samples, using the method described above.<Similar Negative Example> and positive training samples<SimilarPositive Example> For reference, these samples carry their respective labels. <label>The accompanying descriptive text of the test samples provides the model with rich contextual information, helping it to understand the situation more accurately and improve the reliability of its reasoning. For example... Figure 8 As shown, the semantic similarity analysis of participant 202 and the positive and negative training samples reveals a strong similarity in their attention to detail, language organization skills, and ability to understand complex situations. This method can effectively identify individuals who may have Alzheimer's disease.

[0152] Prompt for model output in the project:

[0153] <Model Output> The model's specific output is displayed, including detailed step-by-step analysis and comprehensive diagnostic results. This output is based on the model's analysis of four major language disorders through zero-shot thought chains, including reduced information content, lexical semantic deficits, poor referential coherence, and lack of mental state language. The model's predictions... <label>The label outputs, such as "dementia" and "control", where "control" represents healthy individuals and "dementia" represents AD patients.

[0154] Although SVM can quickly distinguish patients from healthy individuals by analyzing audio and other biomarker features, audio features alone cannot fully capture the changes in patients' thinking logic and memory capacity. To make up for this deficiency, the present embodiment combines audio features with semantic features, using audio-based prediction results and corresponding confidence as supplementary input for large language models. This fusion method improves the speed, accuracy of diagnosis, and enhances the interpretability of the results.

[0155] Finally, the <Model Output> not only provides classification labels and probabilities, but also increases the transparency and understandability of the output through detailed text descriptions. The output format of the <Model Output> is also extended from simple diagnostic labels to comprehensive screening reports that include detailed analysis processes, clearly labeled as "control" or "dementia", thereby improving the quality of medical decision-making and the consistency of data processing.

[0156] Overall, the present embodiment is not only suitable for technical developers and data scientists, but also facilitates the use and understanding of clinical physicians and researchers, providing strong technical support for the diagnosis of complex diseases such as Alzheimer's disease.

[0157] Example 3

[0158] Example 3 is a participant 203 sample analysis of a multi-modal Alzheimer's disease early screening algorithm based on a large language model.

[0159] First, the participant 203 describes the scene shown in FIG. 1; then, the participant 203's voice data is collected and subjected to noise suppression, voice enhancement, and speaker separation operations to clearly capture the language content; next, the processed voice data is transcribed into text to form a description text. Figure 9 The eGeMAPS feature set is extracted from the transcribed voice data, and the extracted features are combined with demographic information to form a multi-modal feature matrix containing audio features and demographic information. The multi-modal feature matrix is processed using an attention pooling module to extract key features; SVM support vector machines are used to classify the pooled features to obtain audio-based prediction results and confidence.

[0160] As shown in FIG. 2, the <Model Input> includes the description text and the multi-modal feature matrix.

[0161] Figure 10 ​As shown: Participant 203's demographic information is a 71-year-old male with 8 years of education; his voice data was predicted using audio-based prediction, and the prediction result was "control" with a confidence level of 63.08; Participant 203... Figure 9 The description text for the content shown is: "Okay, the boy is in the cookie jar. He's about to fall off the stool. The mother's bathtub is overflowing on the floor. The child will fall on the floor. Where is the girl? She's reaching for the cookies. The poor old mother has to go help her... That's all, right?"

[0162] The descriptive text of participant 203, demographic information, audio-based prediction results, and corresponding confidence input prompts were engineered. Negative and positive training samples closest to the current test sample were specifically introduced from the training set as references, such as... Figure 11 As shown. Figure 11 The document presents participant 203's descriptive text and demographic information, along with similar positive and negative training samples. These samples and their related information are used to assist the large language model in accurately determining whether participant 203 has Alzheimer's disease through comparison and analysis. This method allows the model to more comprehensively understand participants' language expression and cognitive state, thereby improving the accuracy and reliability of the diagnosis. Next, through a zero-shot thought process, it obtains... Figure 12 The following are early Alzheimer's disease screening reports for 203 participants:

[0163] Reduced information content: The description provides some information about the scene, including the boy's actions, the mother's actions, and their consequences. However, the description is relatively brief, lacks detail, and is sometimes unclear or ambiguous (e.g., "Where is the girl?").

[0164] Lexical semantic defects: The vocabulary used is simple and limited, with no obvious errors in word choice or meaning. The language description is not particularly detailed or nuanced.

[0165] Lack of mental state language: The use of mental state language, such as thoughts, feelings, or intentions, is minimal. Descriptions primarily focus on physical actions and events.

[0166] Poor referential coherence: The descriptions are somewhat fragmented, and the connections between events and objects are unclear. Participants seem to jump between different ideas, lacking coherence and exhibiting some repetition (e.g., "That's all, right?").

[0167] Demographic information: The participant was a 71-year-old male with 8 years of education, which increases the risk of developing Alzheimer's disease.

[0168] Audio-based prediction: The audio-based prediction is "control" with a confidence level of 63.08%, which indicates that the probability of the individual being healthy is relatively moderate.

[0169] Final assessment: Considering the analysis of the participant's descriptions and demographic information, we disagree with the audio-based prediction and instead diagnose this person with "dementia" (Alzheimer's disease). The relatively brief and fragmented descriptions, lack of mental state language, and poor referential coherence suggest a higher likelihood of cognitive impairment.

[0170] Diagnosis: Dementia.

[0171] This embodiment demonstrates a multimodal Alzheimer's disease early screening algorithm based on a large language model, enabling early screening of potential Alzheimer's patients. By utilizing audio, language, and demographic information, this algorithm comprehensively evaluates the patient's condition, providing highly personalized and detailed diagnostic results.

[0172] Example 4

[0173] This embodiment describes a multimodal Alzheimer's disease early screening system based on a large language model. It effectively combines audio and demographic data by integrating speech processing, feature extraction, attention pooling, support vector machine classification, and deep semantic analysis techniques to accurately diagnose and analyze the early symptoms of Alzheimer's disease.

[0174] System components:

[0175] The data collection unit is used to collect participants' voice data and demographic information;

[0176] The audio processing unit is used to perform noise suppression, voice enhancement and speaker separation operations on the speech data to obtain the processed speech data;

[0177] The feature extraction unit is used to extract the eGeMAPS feature set of the processed speech data and to concatenate different time series data in the eGeMAPS feature set in a time sequence to generate a feature matrix with time series information.

[0178] The feature processing unit is used to process the multimodal feature matrix using the attention pooling module to obtain the pooled features;

[0179] The classification unit is used to classify and detect the pooled features using a support vector machine, obtain audio-based prediction results, and distinguish between potential Alzheimer's patients and healthy individuals.

[0180] The large language model analysis unit is used to take the descriptive text, the demographic information, the audio-based prediction results, and the confidence level as input, and use zero-shot thinking chain to analyze the multimodal data based on language barriers.

[0181] The report generation unit is used to generate a screening report based on the analysis to determine whether the participant has Alzheimer's disease.

[0182] Workflow:

[0183] 1. Participants provide voice samples and related demographic information through the data collection unit.

[0184] 2. The speech samples are processed by the audio processing unit and transmitted to the feature extraction unit in a clear and enhanced format.

[0185] 3. The feature extraction unit extracts key acoustic features and generates a feature matrix, which is then optimized by the feature processing unit and passed to the classification unit.

[0186] 4. The classification unit analyzes the features, preliminarily classifies the potential health status, and transmits the results to the large language model analysis unit.

[0187] 5. The large language model analysis unit integrates all input data, uses deep learning technology to analyze and identify any language barriers, and produces diagnostic results.

[0188] 6. The report generation unit compiles the final screening report based on the analysis results. This report details the analysis process and diagnostic results for use by medical professionals.

[0189] This system integrates multiple technologies and units to provide a comprehensive and efficient early screening tool for Alzheimer's disease, aiming to improve the accuracy and timeliness of screening through advanced large language models and machine learning methods.

[0190] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.< / label> < / label>

Claims

1. A multimodal Alzheimer's disease early screening algorithm based on a large language model, characterized in that, Includes the following steps: S1. Collect participants' voice data and demographic information; the demographic information includes gender, age, and education level; convert the voice data into descriptive text; S2. Extract the eGeMAPS feature set of the speech data; concatenate different time series data in the eGeMAPS feature set to generate a feature matrix with time series information, and integrate demographic information into this feature matrix to obtain a multimodal feature matrix containing audio features and demographic information. S3. Process the multimodal feature matrix using the attention pooling module to obtain the pooled features; Step S3 further includes: a. Divide the processed speech data into equal parts. Each segment of speech data is processed, and the corresponding eGeMAPS feature set is extracted; the eGeMAPS feature set contains... Acoustic parameters in several dimensions; the acoustic parameters are represented in float type; b. The eGeMAPS feature sets are spliced ​​together to form matrix The matrix ;in The matrix contains only real numbers. c. Convert the demographic information of the current sample to float type and obtain the demographic information matrix. The matrix ;in This indicates the number of categories of the demographic information; the demographic information includes gender, age, and education level. d. The matrix Fill in according to the time dimension matrix ';The matrix ; e. Matrix With matrix To splice: Where C is the characteristic matrix; S4. Use a support vector machine to classify and detect the pooled features, and obtain the audio-based prediction results and corresponding confidence scores. S5. The descriptive text, demographic information, audio-based prediction results, and confidence scores are used as inputs to the prompting engineering of the large language model, and the output is a report on whether the participant has Alzheimer's disease screening. The semantic similarity of positive and negative training samples is introduced as a reference for the prompting engineering. The prompting engineering uses zero-shot thinking chain to analyze multimodal data based on language barriers. The language barriers are reduced information content, lexical semantic defects, poor referential coherence, and lack of mental state language.

2. The multimodal Alzheimer's disease early screening algorithm based on a large language model according to claim 1, characterized in that, Step S1 further performs noise suppression, voice enhancement, and speaker separation operations on the speech data to obtain processed speech data; and converts the processed speech data into descriptive text.

3. The multimodal Alzheimer's disease early screening algorithm based on a large language model according to claim 2, characterized in that, Step S1 further uses the Koala model of PICOVOICE to suppress noise in the speech data to preserve human voice and enhance human voice; and uses the Falcon model of PICOVOICE to separate the speaker in order to accurately extract the patient's speech. The patient's speech is converted into the descriptive text using automatic speech recognition technology, and the descriptive text is then calibrated.

4. The multimodal Alzheimer's disease early screening algorithm based on a large language model according to claim 2, characterized in that, The processed speech data is sliced ​​to extract the eGeMAPS feature set for each slice; the slices have the same duration to ensure the consistency of the time series information of the eGeMAPS feature set.

5. The multimodal Alzheimer's disease early screening algorithm based on a large language model according to claim 4, characterized in that, The design of the prompting project mentioned in step 5 is as follows: in Representing a large language model, As a hint, To indicate the output of the project, To describe the text, express The thought dimensions of the prompting project include the gender, age, education level, audio-based prediction results, and confidence level.

6. The multimodal Alzheimer's disease early screening algorithm based on a large language model according to claim 5, characterized in that, Step 5 further includes calculating the semantic similarity between positive and negative samples using the TF-IDF method. in, Indicate word frequency, Indicates inverse document frequency. Character In the description text Frequency of occurrence in Indicates traversal of the description text Each character in ; This indicates the total number of descriptive texts. Indicates the presence of characters The amount of descriptive text.

7. The multimodal Alzheimer's disease early screening algorithm based on a large language model according to claim 5, characterized in that, The language barrier mentioned in step S5 is defined as follows: Reduced information content refers to a decrease in both the quantity and quality of information conveyed in language expression; Lexical semantic defects refer to difficulties in acquiring and using vocabulary; Poor referential coherence refers to problems with coherence and consistency in the use of pronouns and other referential expressions. The lack of psychological state language refers to the absence of language in the descriptive text that describes a person's beliefs, desires, intentions, and emotions.

8. The multimodal Alzheimer's disease early screening algorithm based on a large language model according to claim 5, characterized in that, Further analysis of language barriers includes: The information richness and relevance of the descriptive text are analyzed and quantified. Natural language processing techniques are used to compare the differences between the descriptive text and the image being described to assess whether there is a reduction in information content. The presence of semantic defects is assessed by detecting unconventional or erroneous word usage in the descriptive text through semantic analysis and lexical networks; the lexical terms include nouns and verbs. Analyze the referential relationship between antecedents and corresponding pronouns in the descriptive text. When there is inappropriate use of pronouns or a lack of the antecedents, the reference coherence is assessed as poor. Identify and analyze the psychological state language in the descriptive text. When there is a lack of language expressing emotional understanding or psychological motivation, it is assessed as a lack of psychological state language. The psychological state language includes language describing a person's beliefs, desires, intentions, and emotions.

9. A multimodal Alzheimer's disease early screening system based on a large language model, characterized in that, Includes the following units: The data collection unit is used to collect participants' voice data and demographic information; The audio processing unit is used to perform noise suppression, voice enhancement and speaker separation operations on the speech data to obtain the processed speech data; The feature extraction unit is used to extract the eGeMAPS feature set of the processed speech data and to concatenate different time series data in the eGeMAPS feature set in a time sequence to generate a feature matrix with time series information. The feature processing unit is used to process the multimodal feature matrix using the attention pooling module to obtain the pooled features; where, a. Divide the processed speech data into equal parts. Each segment of speech data is processed, and the corresponding eGeMAPS feature set is extracted; the eGeMAPS feature set contains... Acoustic parameters in several dimensions; the acoustic parameters are represented in float type; b. The eGeMAPS feature sets are spliced ​​together to form matrix The matrix ;in The matrix contains only real numbers. c. Convert the demographic information of the current sample to float type and obtain the demographic information matrix. The matrix ;in This indicates the number of categories of the demographic information; the demographic information includes gender, age, and education level. d. The matrix Fill in according to the time dimension matrix ';The matrix ; e. Matrix With matrix To splice: Where C is the characteristic matrix; The classification unit is used to classify and detect the pooled features using a support vector machine, and obtain audio-based prediction results and corresponding confidence scores to distinguish potential Alzheimer's patients from healthy individuals. The large language model analysis unit is used to analyze multimodal data based on language barriers by taking the descriptive text, the demographic information, the audio-based prediction results and the confidence level as inputs; The report generation unit is used to generate a screening report based on the analysis to determine whether the participant has Alzheimer's disease.

Images