Multi-modal Alzheimer's disease early screening algorithm based on large language model

Abstract

The invention relates to the technical field of medical information, and particularly provides a multi-modal Alzheimer's disease early screening algorithm based on a large language model. The algorithm first collects voice data and demographic information of participants. Voice data is processed through noise suppression, voice enhancement and speaker separation and is transcribed into a description text. An eGeMAPS feature set is extracted from the voice data and is subjected to time sequence splicing, and a multi-modal feature matrix is formed in combination with demographic information; the attention pooling module is used for obtaining key features, a support vector machine is used for feature classification, and potential Alzheimer's disease patients and healthy individuals are distinguished. And inputting the classification result, the description text and the demographic information into a large language model, deeply analyzing four language barriers by adopting a zero sample thinking chain: information content reduction, vocabulary semantic defects, poor reference continuity and psychological state language deficiency, and finally generating a comprehensive screening report. According to the algorithm, the accuracy of early diagnosis is improved by combining audio and semantic features.

Description

Technical Field

[0001] The present invention relates to the field of medical information technology, and in particular to a multimodal Alzheimer's disease early screening algorithm based on a large prediction model. Background Art

[0002] Alzheimer's disease (AD) is one of the most common types of dementia and is actually the most common cause of dementia. Dementia is a broad term used to describe a group of symptoms that involve memory loss and the deterioration of thinking and social skills to a degree that interferes with daily life. Alzheimer's disease is the specific disease that causes these symptoms, while dementia itself is not a disease, but an umbrella term for a group of symptoms caused by a variety of different diseases. In short, Alzheimer's disease is a form of dementia and is one of the main causes of dementia symptoms. As AD progresses, the patient's cognitive function gradually declines, eventually manifesting as typical dementia symptoms.

[0003] Compared with normal aging people, AD patients show significant differences in semantic understanding, speech production, and memory function. In order to diagnose and evaluate the symptoms of AD patients, existing technologies mainly rely on analyzing the voice performance of patients describing pictures to explore subtle changes in their voice and semantics. Based on the type of features, these technologies can be roughly divided into three categories: audio-based feature analysis, semantic feature analysis based on transcribed text, and comprehensive analysis combining text and semantic features.

[0004] AD patients often show characteristics of speech disfluency, such as slow speech, prolonged pauses, and difficulty in vocabulary search. In response to these characteristics, Longbin Jin et al. innovatively combined multiple audio features, including VGGish, wav2vec, and i-vector, as well as 18 disfluency features extracted from the original audio, such as the frequency and number of pauses in the paused audio, in the CONSEN system proposed in 2023. This comprehensive analysis method can effectively identify AD patients of different languages.

[0005] In addition, Utkarsh Sarawgi et al. further explored the method of combining the disfluency feature set and the emotion detection audio feature set (ComparE2013), and introduced the sequence of speaker identity changes in the conversation as a simple semantic feature reflecting the patient's logical thinking ability. They provided a more comprehensive basis for the diagnosis of AD patients by comprehensively judging the results of these three feature models.

[0006] Zih-Jyun Lin et al. proposed a new idea of ​​using the existing mature automatic speech recognition (ASR) model for AD detection. Through transfer learning, they enabled the ASR model to extract specific audio features for AD patients and classify them based on this.

[0007] However, it should be pointed out that current methods mainly focus on the analysis of audio features. Although they have shown some effectiveness in detecting AD patients, they ignore the semantic characteristics of patients and therefore have difficulty in deeply capturing the changes in thinking behind the patient's language. Considering that early AD patients tend to show more significant memory deterioration and semantic comprehension problems, relying solely on audio may not be enough to accurately identify patients at an early stage.

[0008] In addition, these technologies face another challenge, that is, they usually rely on a large amount of labeled data for training. However, the currently available AD datasets are mainly focused on English, and English-speaking AD patients worldwide account for less than 50% of the total patient population. For other languages, there is a lack of corresponding datasets for model training, which limits the practical application of the above methods.

[0009] AD is a degenerative neurological disease that significantly affects patients' language comprehension, speech production, and memory, especially in the early stages, when they show symptoms of confused speech logic and difficulty finding words. In order to effectively detect AD patients, researchers tried to obtain information from the semantic features of the pictures described by the patients.

[0010] In existing studies, Sarawgi et al. used the GloVe model to extract lexical features from the image description text and processed them uniformly through LSTM. Ben Ammar et al. proposed an AD detection model based on the language features of the patient's speech transcription, using the KNN algorithm for feature selection, and then training machine learning classifiers such as support vector machines (SVM) for diagnosis. Yamanki et al. further introduced a contrastive learning model based on Siamese BERT, aiming to extract discriminative features from the text of the patient's speech transcription and other features (such as demographic, vocabulary, and semantic information) to support the diagnosis of AD.

[0011] Recently, Yi Wang et al. proposed a Prompt-based pre-trained language model fine-tuning method for AD detection in 2023. This method uses pre-trained language models such as Roberta and Bert to extract semantic features from speech-recorded text, and transforms the classification problem into a label generation problem through prompt engineering technology. However, these methods face several challenges in practical applications.

[0012] First, they rely on a large amount of labeled data for training, which is often limited to the content described in a specific dataset and fails to deeply explore the changes in patients' specific logical thinking and memory abilities. Therefore, these methods have limited generalization capabilities, especially when faced with patients with different pictures or different life backgrounds.

[0013] Secondly, most of the currently available datasets focus on the "cookie theft" image, which makes these models only applicable to people with similar life experiences. However, this group of people accounts for a relatively small proportion of the total AD patient population, limiting the applicability of the model in actual clinical settings.

[0014] Finally, these semantic feature-based models lack interpretability when outputting results, making it difficult for clinicians to understand and trust the model’s diagnostic results. This further hinders the widespread application of these models in clinical practice.

[0015] Since only the participants' voice data can be obtained initially, it needs to be converted into text information through the automatic speech recognition (ASR) model. Therefore, in practical applications, the fusion of audio and semantic features mostly occurs at the decision-making level.

[0016] In recent years, researchers have proposed a series of multimodal AD detection methods. Among them, Xuchu Chen et al. proposed an algorithm that used the paralinguistic feature set IS10 and XLSR-50 in terms of audio features, and used the RoBerta model for feature extraction and classification detection in terms of semantic features. Finally, they realized the decision-level fusion of audio and semantic features through multi-layer perceptron (MLP) and SVM, thereby achieving multimodal comprehensive detection.

[0017] Similarly, Kangdi Mei et al. also proposed a multimodal AD detection algorithm, which is similar in structure to the method of XuchuChen et al., but they tried more sound feature sets, such as ComParE and eGeMAPS, in terms of audio features, and supplemented the disfluency feature set. This combination of multimodal features further improved the accuracy of diagnosis.

[0018] Although the above methods utilize both speech features and semantic features and achieve good detection results, they still face some challenges. First, these methods require a large amount of annotated data sets for model training, which may be limited in practical applications. Second, although audio features are not affected by language content to a certain extent, the semantic feature extraction method used is still limited by the content of the data set, so its generalization ability needs to be improved.

[0019] In 2023, Wei et al. proposed the "Chain-of-Thought" technology, which significantly improves the performance of large language models in solving complex reasoning problems by providing them with a multi-step reasoning process. However, this reasoning process generally needs to be customized manually, which is time-consuming and labor-intensive, and the performance improvement of the model depends on the quality of the manually customized reasoning process. In order to solve the problem of manual labeling of the reasoning process, Kojima et al. proposed the Zero-Shot Chain-of-Thought technology, which induces the instruction "Let's think step by step" to enable the model to automatically generate reasoning steps without providing reasoning steps, with excellent final effect.

[0020] Although existing AD diagnostic technologies have shown advantages in some aspects, they still face many challenges: reliance on a large amount of annotated data, lack of language and cultural diversity, insufficient model generalization, immature multimodal data fusion, lack of interpretability of diagnostic models, and privacy and ethical issues. These issues limit the widespread application and accuracy of the technology, and urgently need to be improved through technological innovation and data diversification. Summary of the invention

[0021] The purpose of the present invention is to overcome several major shortcomings of the prior art: first, the traditional method relies on a large amount of labeled data for model training, which limits the versatility and practicality of the algorithm when data is scarce; second, screening that relies solely on semantic features is easily affected by the patient's subjective consciousness and lacks comprehensiveness; third, the lack of interpretability of traditional screening makes it difficult for doctors to understand and trust the diagnostic results of the model. To solve these problems, the present invention proposes a multimodal AD early screening method based on a large language model. This method combines audio features and semantic features, uses a general large language model to perform effective detection in zero-sample and small-sample situations, and provides interpretable diagnostic results. By using the thinking method of the zero-sample thinking chain, the present invention can achieve complex reasoning and higher diagnostic accuracy without the need for a large amount of labeled data. In addition, the fusion of audio and semantic features enhances the comprehensiveness of detection, making early screening more accurate and reliable.

[0022] In view of the above situation, the present invention provides a multimodal early screening algorithm for Alzheimer's disease based on a large language model, which specifically includes the following steps:

[0023] S1. Collecting voice data and demographic information of participants; the demographic information includes gender, age and education level; converting the voice data into descriptive text;

[0024] S2, extracting the eGeMAPS feature set of the speech data; performing time-series splicing on different time series data in the eGeMAPS feature set to generate a feature matrix with time series information, and integrating demographic information into the feature matrix to obtain a multimodal feature matrix containing audio features and demographic information;

[0025] S3, using an attention pooling module to process the multimodal feature matrix to obtain pooled features;

[0026] S4. Using a support vector machine to perform classification detection on the pooled features to obtain audio-based prediction results and corresponding confidence levels;

[0027] S5. The description text, the demographic information, the audio-based prediction results and the confidence level are used as inputs of a prompting project of a large language model to output a screening report on whether the participant suffers from Alzheimer's disease; the semantic similarity of positive and negative training samples is introduced as a reference for the prompting project; the prompting project uses a zero-sample thinking chain to analyze the multimodal data based on language barriers; the language barriers are reduced information content, lexical and semantic defects, poor referential coherence and lack of mental state language.

[0028] Preferably, the step S1 further performs noise suppression, voice enhancement and speaker separation operations on the voice data to obtain processed voice data; and converts the processed voice data into description text.

[0029] Preferably, the step S1 further uses the Koala model of PICOVOICE to suppress noise on the voice data to retain the human voice and enhance the human voice; uses the Falcon model of PICOVOICE to perform speaker separation to accurately extract the patient's voice; uses automatic speech recognition technology to convert the patient's voice into the description text, and calibrates the description text.

[0030] Preferably, a slice operation is performed on the processed speech data to extract the eGeMAPS feature set of each slice; the slices have the same duration to ensure consistency of time series information of the eGeMAPS feature set.

[0031] Preferably, step S3 further comprises:

[0032] a. Dividing the processed speech data into x segments on average, extracting the eGeMAPS feature set corresponding to each segment of speech data; the eGeMAPS feature set comprises acoustic parameters of y dimensions; the acoustic parameters are represented by float type;

[0033] b. Concatenate the x segments of eGeMAPS feature sets to form a matrix A of [x, y]; the matrix A∈R x×y ; Where R is the matrix whose elements are all real numbers;

[0034] c. Convert the demographic information of the current sample into float type to obtain the demographic information matrix B; the matrix B∈R n ; Wherein n represents the number of types of the demographic information; the demographic information includes gender, age and education level, etc.;

[0035] d. Fill the matrix B into a [x+n] matrix B' according to the time dimension; the matrix B'∈R x×n ;

[0036] e. Concatenate matrix A and matrix B′:

[0037] C=[A|B′]∈R x×(y+n) ,

[0038] Among them, C is the feature matrix.

[0039] Preferably, step 5 further includes calculating the semantic similarity of positive and negative samples using the TF-IDF method.

[0040]

[0041] TF-IDF(t,d)=TF(t,d)×IDF(t)

[0042] Among them, TF represents term frequency, IDF represents inverse document frequency, (t, d) represents the frequency of character t appearing in description text d, t′ represents traversing every character t in description text d; N represents the total number of description texts, |{d∈D:t∈d}| represents the number of description texts containing character t.

[0043] Preferably, the design of the prompting project in step 5 is as follows:

[0044] Model(f(x,a 1 , …, a n ))=p

[0045] Model represents the large language model, f is the prompt, p is the output of the prompt project, x is the description text, and a 1 , ..., a n Representing the thinking dimensions of n kinds of prompt engineering; the thinking dimensions include the gender, the age, the education level, the audio-based prediction results and the confidence.

[0046] Preferably, the language barrier in step S5 is defined as:

[0047] Information content reduction refers to the reduction in the amount and quality of information conveyed in language expression;

[0048] Lexical-semantic deficits refer to difficulties in vocabulary acquisition and use;

[0049] Poor referential coherence refers to problems with coherence and consistency in the use of pronouns and other referential expressions;

[0050] The lack of mental state language refers to the lack of language in the descriptive text that describes a person's beliefs, desires, intentions, and emotions.

[0051] Preferably, analyzing the language barrier further comprises:

[0052] Analyze and quantify the information richness and relevance of the description text, and use natural language processing technology to compare the difference between the description text and the described image to evaluate whether there is a reduction in the information content;

[0053] By detecting the unconventional or incorrect use of words in the description text through semantic analysis and vocabulary network, the existence of the lexical semantic defect is evaluated; preferably, the vocabulary includes nouns and verbs;

[0054] Analyze the reference relationship between the antecedent and the corresponding pronoun in the description text, and evaluate the reference coherence as poor when there is improper use of pronouns or the antecedent is missing;

[0055] The psychological state language in the descriptive text is identified and analyzed, and when there is a lack of language expressing emotional understanding or psychological motivation, it is assessed as a lack of psychological state language; the psychological state language includes language that describes a person's beliefs, desires, intentions, and emotions.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] 1. Reduce the reliance on labeled data: Existing technologies usually rely on a large amount of labeled data for model training, which is limited in the AD field due to data scarcity. The present invention uses a universal large language model that can work effectively even in the case of zero or very few samples, significantly improving its practicality and applicability.

[0058] 2. Comprehensive detection through multimodal features: Existing technologies often fail to capture subtle changes in patients’ cognitive and memory abilities through audio features alone. This invention integrates audio and semantic features to ensure a more comprehensive analysis and improve the accuracy and reliability of early detection of AD.

[0059] 3. Improved explainability and trust: Traditional diagnostic models often lack explainability, which makes it difficult for medical professionals to trust and understand these models. This invention solves this problem by providing explainable diagnostic results, using the zero-sample thinking chain thinking method to simulate the human reasoning process, enhancing the transparency and credibility of the diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a diagram of the large language model analysis framework described in Example 2 of the present invention;

[0061] Figure 2 This is a flow chart of voice data preprocessing according to Embodiment 2 of the present invention;

[0062] Figure 3 This is a multimodal data fusion strategy diagram described in Example 2 of the present invention;

[0063] Figure 4 This is a schematic diagram of the structure of the feature splicing and attention pooling model described in Example 2 of the present invention;

[0064] Figure 5 This is a schematic diagram of the prompting project and zero-sample thinking chain described in Example 2 of the present invention;

[0065] Figure 6 This is a schematic diagram of an early screening report for Alzheimer's disease of the participant 202 described in Example 2 of the present invention;

[0066] Figure 7 This is a schematic diagram of the optimized prompting engineering and zero-sample thinking chain described in Example 2 of the present invention;

[0067] Figure 8 The semantic similarity analysis between the participant 202 and the positive and negative training samples described in Embodiment 2 of the present invention;

[0068] Fig. 9 The description pictures described in Embodiment 2 and Embodiment 3 of the present invention;

[0069] Fig.10 The input information of the participant 203 in the large language model described in Embodiment 3 of the present invention;

[0070] Fig.11 The semantic similarity analysis between the participant 203 and the positive and negative training samples described in Example 3 of the present invention;

[0071] Fig.12 This is a schematic diagram of an early screening report for Alzheimer's disease of participant 203 described in Example 3 of the present invention. DETAILED DESCRIPTION

[0072] The present invention is further described in detail below in conjunction with test examples and specific implementation methods. However, this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the content of the present invention belong to the scope of the present invention.

[0073] This embodiment provides a multimodal Alzheimer's disease early screening algorithm based on a large language model, and the specific steps are as follows:

[0074] S1. Collect the participants’ voice data and demographic information. The demographic information includes gender, age, and education level, which is crucial for subsequent analysis and helps improve the diagnostic accuracy of the model. Preprocess the collected voice data and convert it into descriptive text.

[0075] S2. Extracting the eGeMAPS feature set of the speech data; performing time-series splicing on different time series data in the eGeMAPS feature set to generate a feature matrix with time series information, and integrating demographic information into the feature matrix to obtain a multimodal feature matrix containing audio features and demographic information.

[0076] S3. Use an attention pooling module to process the multimodal feature matrix to obtain pooled features.

[0077] S4. Use a support vector machine to classify and detect the pooled features to obtain audio-based prediction results and corresponding confidence levels to distinguish potential Alzheimer's patients from healthy individuals.

[0078] S5. Using the descriptive text, the demographic information, the audio-based prediction results and the confidence as inputs of a prompting project of a large language model, and outputting a screening report on whether the participant suffers from Alzheimer's disease; the prompting project adopts a zero-sample thinking chain to analyze the multimodal data based on language barriers; the language barriers are reduced information content, lexical and semantic defects, poor referential coherence and lack of mental state language.

[0079] The screening reports generated will provide clinicians with key diagnostic information to aid in early identification and intervention of Alzheimer's disease.

[0080] Through this series of steps, this embodiment not only improves the accuracy and efficiency of early screening for Alzheimer's disease, but also increases the interpretability of the diagnostic process, providing powerful decision-making support for medical professionals and patients' families.

[0081] Example 2

[0082] This embodiment proposes a multimodal Alzheimer's disease (AD) early screening algorithm based on a large language model. This method not only utilizes the excellent task reasoning and general task solving capabilities of the large language model, but also integrates audio and semantic features to improve the accuracy of diagnosis.

[0083] Alzheimer's disease is the most common form of dementia, which is mainly manifested by symptoms such as memory loss, mental degeneration and decreased social skills, which seriously affect daily life. In this embodiment, the two terms "Alzheimer's disease" and "dementia" are essentially used to describe the symptoms of dementia caused by AD, emphasizing that AD is the main disease causing dementia.

[0084] A significant advantage of large language models is that they can learn effectively in the absence of task-specific data, which is particularly critical in early AD screening. AD patients will show obvious language impairments in the early stages, including in semantics, pragmatics, and grammar. Large language models can effectively capture these language features by leveraging their powerful contextual learning and semantic understanding capabilities.

[0085] However, since semantic content is easily affected by subjective factors, relying solely on semantic analysis may not be sufficient to ensure accuracy. Therefore, this embodiment uses a multimodal detection strategy that combines audio and semantic features to comprehensively evaluate the patient's language performance, thereby improving the accuracy of detection. Figure 1 As shown, this multimodal data processing flow is described.

[0086] S1. Preprocessing of speech data

[0087] like Figure 2 As shown, this embodiment performs three preprocessing operations on the acquired voice data, namely noise suppression, speaker separation and voice transcription. The Koala model of PICOVOICE is used to suppress the noise of the voice data to retain the human voice and enhance the human voice; the Falcon model of PICOVOICE is used to separate the speaker to accurately extract the patient's voice; then, the ASR model is used to transcribe the patient's voice and calibrate the transcribed text information.

[0088] S2. Feature extraction and splicing

[0089] In the AD detection method for audio features, this embodiment uses the eGeMAPS feature set (extended Geneva Minimalistic Acoustic Parameter Set) to classify speech, which is an efficient acoustic parameter set that includes the time-sequential splicing of multiple audio features. This splicing method not only covers the basic features of the sound, but also includes some more complex acoustic measurements, such as fundamental frequency changes, formant characteristics, and multi-dimensional acoustic parameters such as sound quality.

[0090] First, the speech data is sliced ​​accurately to ensure that each slice has the same duration, thereby maintaining the consistency of the time series information. Then, for each slice segment, the corresponding eGeMAPS feature set is extracted. The eGeMAPS feature sets of different time series are spliced ​​to obtain a matrix sequence with time series information. These audio features are combined into a feature vector through the time series splicing method, and the feature vector will be used as input data for subsequent model training and analysis. This time series splicing method not only retains the time series information in the audio, but also enhances the correlation between features. The feature vector contains multi-dimensional acoustic parameters, each of which is represented by float type to ensure the consistency of data processing. Next, the patient's demographic information is filled along the time series dimension, and the demographic information is spliced ​​into the matrix sequence with time series information to obtain a multimodal feature matrix containing audio features and demographic information. The specific steps are as follows:

[0091] S21, divide a segment of audio into x segments evenly, and extract the eGeMAPS feature set corresponding to each segment of audio, wherein the eGeMAPS feature set includes acoustic parameters of y dimensions, and all acoustic parameters are represented by float type.

[0092] S22. Concatenate the x segments of eGeMAPS feature sets to form a two-dimensional matrix A of [x, y]:

[0093] A∈R x× y

[0094] Among them, the elements in the matrix R are all real numbers.

[0095] S23. Convert the demographic information of the current sample into float type and obtain the demographic information matrix B:

[0096] B∈R n

[0097] Where n represents the number of types of demographic information. When the demographic information consists of three types of information: gender, age, and education level, n = 3. The higher the correlation between demographic information and Alzheimer's disease and the richer the types, the more accurate the prediction results.

[0098] S24, fill the matrix B into the [x+n] matrix B' according to the time dimension:

[0099] B′∈R x×n

[0100] S25. Concatenate matrix A and matrix B′ to obtain a multimodal feature matrix C:

[0101] C=[A|B′]∈R x×(y+n)

[0102] S3. Attention pooling processing

[0103] Use the attention pooling module to process the multimodal feature matrix. Figure 4 The following is a schematic diagram of the structure of the feature concatenation and attention pooling model. The specific steps for implementing attention pooling on the multimodal feature matrix are as follows:

[0104] (1) Calculation of attention weights: Using the attention pooling module, the model calculates an attention weight for each feature in the multimodal feature matrix. These weights reflect the importance of each feature in a specific task. Features with higher weights will be given greater influence in the final output.

[0105] (2) Weighted pooling: Based on the calculated attention weights, the model will perform a weighted pooling operation, that is, use the weights to perform weighted averaging or weighted summation on the multimodal feature matrix. In this way, the feature representation finally generated will focus on the most critical features.

[0106] S4, SVM classification detection

[0107] Next, SVM is used to classify and detect the features processed by attention pooling, and the audio-based prediction results and corresponding confidence levels are obtained to preliminarily distinguish between AD patients and healthy individuals. In the prediction results, "dementia" represents AD patients, and "control" represents healthy individuals.

[0108] In order to further improve the accuracy and robustness of classification, this embodiment uses 10 different random seeds to randomly initialize the parameters of the model, then trains them separately, and then uses a small sample to perform k-fold fine-tuning. Finally, the parameters of the 10 fine-tuned models are simply summed and averaged.

[0109] S5. Multimodal early screening of Alzheimer's disease based on large language model

[0110] S51. Input data preparation

[0111] In this embodiment, the application of SVM classification detection and large language model constitutes a multi-layer diagnostic strategy, the purpose of which is to take advantage of the advantages of different methods to improve the accuracy and reliability of AD diagnosis. In the preliminary stage of the processing flow, SVM classification detection is used to quickly screen patients and healthy individuals, and classify them based on more intuitive data such as audio features and demographic information. This step aims to efficiently identify potential AD patients and healthy control samples, and provide target groups for subsequent in-depth analysis. After SVM classification detection, a more in-depth semantic analysis is performed on the data of suspected AD patients. The large language model can comprehensively consider the characteristics of language disorders, such as reduced information content, lexical and semantic defects, and perform more complex cognitive function assessments. This step not only enhances the depth of diagnosis, but also provides the interpretability of diagnostic results.

[0112] Figure 5 It demonstrates how to use prompt engineering to optimize the input of a large pre-trained language model, where zero-shot thought chaining is used as a specific implementation to handle complex problems without training data by simulating human step-by-step reasoning. This technology uses specific task descriptions and input formats to guide the model to perform detailed diagnostic analysis to determine whether the participant is healthy or has AD. In this embodiment, the use of zero-shot thought chaining not only improves the diagnostic accuracy of AD, but also enables the model to run effectively with a small amount of labeled data, and enhances the interpretability of the diagnostic results through step-by-step reasoning, which is crucial to improving the accuracy and trust of medical diagnosis.

[0113] S52, Language Barrier Identification

[0114] Based on Louise et al.'s in-depth study of the differences in language descriptions between hundreds of normal people and AD patients in Describing the cookie theft picture: Sources of breakdown in alzheimer's dementia, this example summarizes the core language barriers of AD patients, which mainly include reduced information content, lexical semantic deficients, poor referential cohesion, lack of mental state language, etc. In order to analyze these barriers more accurately, this example limits the model analysis angle to the following four aspects, analyzes each language barrier one by one, and then gives the final diagnosis result.

[0115] (1) Reduction of information content

[0116] Definition: Decreased information content refers to a decrease in the amount and quality of information conveyed in verbal communication. Patients are unable to provide sufficient detail when describing events, objects, or scenes, making their narrative lack specificity and completeness. This is usually due to cognitive deterioration leading to decreased memory and poor concentration.

[0117] Analysis methods: Analyze and quantify the information richness of patients' descriptions of specific pictures or scenes (such as the frequency and level of detail of key objects, actions, and attributes), use natural language processing technology to compare the differences between the description text and the described pictures (such as missing information, omission of important details), and evaluate whether there is a reduction in the information content.

[0118] (2) Lexical and semantic defects

[0119] Definition: Lexical-semantic deficits refer to difficulties in vocabulary acquisition and use. Patients have challenges finding the right words to express their thoughts and may use incorrect or inaccurate words to describe specific objects or situations. This often manifests as the use of overly general or irrelevant words in place of more specific or appropriate words, reflecting a deterioration in semantic memory.

[0120] Analysis methods: Use semantic analysis and lexical network analysis to describe the text (especially the use of specific nouns and verbs), detect unusual or incorrect vocabulary usage (such as replacing specific nouns with overly general or irrelevant words), and evaluate whether there are lexical and semantic defects.

[0121] (3) Poor referential coherence

[0122] Definition: Referential incoherence refers to problems with consistency and agreement in the use of pronouns and other referents (such as noun phrases) in speech. The patient may have difficulty using pronouns correctly to refer to previously mentioned items in speech or may use pronouns prematurely in inappropriate contexts, making the conversation or narrative difficult to understand. Referential incoherence reflects the patient's difficulty maintaining the structure and logical flow of the conversation.

[0123] Analysis method: Analyze the referential relationship between the antecedent and the corresponding pronoun in the descriptive text, especially whether the pronoun properly refers to the entity mentioned by the antecedent, and then judge the coherence and correctness of the use of pronouns. When there is improper use of pronouns or the absence of the antecedent, it is evaluated as poor referential coherence.

[0124] (4) Lack of mental state language

[0125] Definition: Deficient mental state language involves the absence of mental state language in descriptive text. Mental state language includes language that describes a person’s beliefs, desires, intentions, and emotions. People with AD may have difficulty understanding or expressing their own or others’ mental states, which is often associated with a reduction in “Theory of Mind” (ToM), the ability to reason about other people’s thoughts and feelings.

[0126] Analysis method: Analyze descriptive texts about interpersonal interactions and personal feelings, analyze the richness and appropriateness of psychological state language in the descriptive texts, and then conduct sentiment analysis on the descriptive texts; when the descriptive text lacks language that expresses emotional understanding or psychological motivation, it is assessed as lacking psychological state language.

[0127] S53. Comprehensive Analysis of Large Language Models

[0128] The description text, demographic information, audio-based prediction results and corresponding confidence levels are input into the prompt engineering of the large language model. The large language model uses the zero-sample thinking chain thinking method based on the set prompts to conduct in-depth semantic analysis and logical reasoning on the thinking dimensions of n prompt engineering. Subsequently, the model will output a detailed analysis report, including the identification and analysis process of language disorders and the preliminary diagnosis results of whether AD is present. The formula for prompt engineering is as follows:

[0129] Model(f(x,a 1 , …, a n ))=p

[0130] Among them, Model: represents the large language model;

[0131] f: indicates prompt engineering, which is used to guide the model on how to process input data, make inferences, and generate expected outputs. Through carefully designed prompts, it ensures that the large language model can understand and process information in a predetermined manner.

[0132] x: The description text of the participant, which is the basis for model generation analysis;

[0133] a 1 , ..., a n : represents the thinking dimensions of n kinds of prompt engineering, including the medical symptom basis and audio characteristics of AD;

[0134] p: The output of the model after comprehensive analysis based on the input description text, audio features and demographic information. The output is not just a simple label, but includes detailed analysis and possible diagnostic suggestions.

[0135] Figure 6 The application of the large language model using zero-shot thought chaining in early AD screening was demonstrated, specifically describing how the model can be used to analyze participants' descriptions of scenes to identify language disorders. In this process, the model evaluated whether the participants' descriptions showed the following four major language disorders: reduced information content, lexical semantic defects, poor referential coherence, and lack of mental state language. Through the identification and semantic analysis of these language disorders, the model was able to determine whether the participants showed early symptoms of AD. Figure 6 As a schematic diagram of the early screening report of Alzheimer's disease for participant 202, it effectively demonstrates how the large language model can perform disease risk assessment based on language performance, thereby providing a preliminary diagnostic basis for possible AD patients.

[0136] The prompt project consists of three parts:<Task Description> (Task Description),<User Input> (user input) and<Model Ouput> (Model output).

[0137] Description of tasks in the prompt project:

[0138] <Task Description>Combining the relevant information of the dataset and the pathological evidence related to AD, we focused on analyzing the four main language disorders: reduced information content in the descriptive text, lexical semantic defects, poor referential coherence, and lack of mental state language, to determine whether the participants had AD. In addition, in order to reduce the impact caused by differences in education level and subjective consciousness,<Task Description> The participants' demographic information, such as age, gender, and education level, is also included to improve the objectivity and accuracy of the diagnosis. In this way, the large language model is able to take into account social and psychological factors that may affect language use during the analysis process, thereby providing more precise and personalized diagnostic results.

[0139] Studies have found that language use is affected by many factors, including gender, age, education level, and the individual's subjective consciousness and emotional state. Positive emotions generally prompt the use of more diverse vocabulary and more complex sentence structures, while negative emotions may lead to simpler, repetitive language use. Individuals with higher levels of education generally use more accurate and rich language with clearer logic; conversely, the language may be more monotonous and logically confusing. These social and psychological factors pose additional challenges to the diagnosis of AD, because it is difficult to determine AD completely and accurately based on language characteristics alone, and there is a certain risk of bias.

[0140] In order to accurately consider certain factors in the diagnostic process, especially the demographic information and emotional state of the participants, this embodiment<Task Description> Specific test sample demographic information is added to reduce the subjective impact of individual differences on the diagnostic results. This example establishes an optimized task description by explicitly focusing the task description on diagnosing whether the participant has dementia or health. The optimized task description clearly states that the model needs to perform step-by-step analysis and judgment based on the four language barriers. The structured evaluation method greatly improves the systematicness of the model and the transparency of the diagnosis.

[0141] like Figure 7 As shown, the optimized prompt engineering<Task Description> As follows: "Based on the examples provided below, and taking into account the demographic factors of the participants (age, gender, and education level) and the audio test results, please analyze the descriptions to diagnose dementia or health. Focus on the following aspects: reduced information content, lexical semantic deficits, lack of mental state language, and poor referential coherence. Provide a detailed step-by-step analysis and judgment based on these four criteria. For healthy individuals, please provide "control"; for AD patients, please provide "dementia".

[0142] By taking emotional states into account, this optimization ensures<Task Description> It can more comprehensively guide large language models to conduct early screening and diagnosis of AD, and at the same time enhance the objectivity and accuracy of diagnosis by introducing demographic information and structured analysis process.

[0143] Prompt user input in the project:

[0144] <User Input> The input content includes the participant's description text (Participant Description), demographic information (Demographic Information), and the prediction results based on audio features and the corresponding confidence (Audio Prediction+Confidence).

[0145] Audio Prediction+Confidence provides the model with preliminary analysis results and confidence levels of the participants’ voice characteristics, which are obtained based on probability calculations. By integrating these audio analysis results with confidence information into the model input, it not only enhances the understanding of Alzheimer’s disease symptoms, but also improves the accuracy of diagnosis. In addition, this approach supports the model to make more precise diagnostic decisions by weighing the reliability of preliminary clues in complex or ambiguous situations.

[0146] Using the zero-sample thinking chain thinking method, the model can derive diagnostic results directly from user input without providing specific reasoning samples. Because the content described in the training set and the test object is semantically consistent, the difference lies in the words and sentences used, and TF-IDF (Term Frequency-Inverse Document Frequency) is a statistical technique used to measure the importance of words in a descriptive text set. We use this method to calculate semantic similarity. First, we calculate the TF-IDF matrix of all samples in the training set and test samples, and then calculate the cosine similarity of the test sample and each sample in the training set. The closer the cosine similarity is to 1, the more similar the two texts are. The higher the text similarity, the closer the content and wording of the corresponding description text are, and the stronger the reference value of the sample.

[0147] For each description text d and character t, calculate the term frequency (TF) and inverse document frequency (IDF)

[0148]

[0149] TF-IDF(t,d)=TF(t,d)×IDF(t)

[0150] Among them, (t, d) represents the frequency of occurrence of character t in description text d, t′ represents traversing each character t in description text d; N represents the total number of description texts, and |{d∈D:t∈d}| represents the number of description texts containing character t.

[0151] In order to improve the accuracy of reasoning, this embodiment specifically introduces the negative training sample closest to the current test sample from the training set through the above method.<Similar Negative Example> and positive training samples<SimilarPositive Example> As a reference, these samples carry their own labels <label>The description text of the corresponding test sample provides the model with rich contextual information, which helps the model understand the situation more accurately and improves the reliability of reasoning. Figure 8 As shown, the semantic similarity analysis between the participant 202 and the positive and negative training samples is shown. From the semantic analysis between the patient 202 and the positive and negative training samples, it can be seen that their attention to details, language organization ability and ability to understand complex situations have strong similarities. In this way, individuals who may have Alzheimer's disease can be effectively identified.

[0152] Model output in prompt project:

[0153] <Model Output> The detailed output of the model is shown, including a detailed step-by-step analysis and a comprehensive diagnosis. This output is based on the model's analysis of four major language disorders through zero-shot thought chains, including reduced information content, lexical semantic deficits, poor referential coherence, and lack of mental state language. <label>Label output, such as "dementia" and "control", where "control" represents healthy individuals and "dementia" represents AD patients.

[0154] Although SVM can quickly distinguish patients from healthy individuals by analyzing biomarker features such as audio, audio features alone cannot fully capture the changes in patients' thinking logic and memory ability. To make up for this shortcoming, this embodiment combines audio features with semantic features, and uses audio-based prediction results and corresponding confidence levels as supplementary inputs to the large language model. This fusion method improves the speed and accuracy of diagnosis and enhances the interpretability of the results.

[0155] at last,<Model Output> Not only does it provide classification labels and probabilities, it also increases the transparency and understandability of the output through detailed text descriptions.<Model Output> The output format of the system has also been expanded from a simple diagnosis label to a comprehensive screening report that includes a detailed analysis process and is clearly marked as "control" or "dementia", thereby improving the quality of medical decision-making and the consistency of data processing.

[0156] In general, this embodiment is not only suitable for technology developers and data scientists, but also easy for clinicians and researchers to use and understand, and provides strong technical support for the diagnosis of complex diseases such as Alzheimer's disease.

[0157] Example 3

[0158] Example 3 is a sample analysis of participant 203 of a multimodal Alzheimer's disease early screening algorithm based on a large language model.

[0159] First, participants 203 Fig. 9 The scene shown is described; then, the voice data of participant 203 is collected and noise suppression, voice enhancement and speaker separation operations are performed to clearly capture the language content; then, the processed voice data is transcribed into text to form a description text.

[0160] The eGeMAPS feature set is extracted from the transcribed speech data and spliced ​​in time series. The extracted features are combined with demographic information to form a multimodal feature matrix containing audio features and demographic information. The attention pooling module is used to process the multimodal feature matrix to extract key features; the SVM support vector machine is used to classify the pooled features to obtain audio-based prediction results and confidence.

[0161] like Fig.10 As shown: The demographic information of participant 203 is 71 years old male, 8 years of education; his voice data is predicted based on audio, and the prediction result is "control" with a confidence level of 63.08; the participant 203 Fig. 9 The description text for the content shown reads: "Okay, the boy is in the cookie jar. He's about to fall off the stool. The mother's bathtub is overflowing on the floor. The child is going to fall on the floor. Where is the girl? She's reaching for a cookie. Poor old mom has to help her...that's all, right?"

[0162] The description text, demographic information, audio-based prediction results and corresponding confidence of the participant 203 are input into the prompt engineering. The negative training samples and positive training samples closest to the current test sample are specially introduced from the training set as references, such as Fig.11 shown. Fig.11 The description text and demographic information of participant 203 are shown in Figure 2, as well as positive and negative training samples similar to him. These samples and related information are used to assist the large language model to accurately determine whether participant 203 has Alzheimer's disease through comparison and analysis. In this way, the model can more comprehensively understand the language expression and cognitive state of the participant, thereby improving the accuracy and reliability of the diagnosis. Then, through the thinking method of zero-sample thinking chain, we get the following Fig.12 The Alzheimer's disease early screening report of the participant 203 sample is shown:

[0163] Reduced Information Content: The description provides some information about the scene, including the boy's actions, the mother's actions, and their consequences. However, the description is relatively brief, lacks detail, and is somewhat unclear or ambiguous (e.g., "Where is the girl?").

[0164] Lexical and semantic deficits: Use of a simple and limited vocabulary with no obvious errors in word choice or meaning. Language is not particularly detailed or nuanced.

[0165] Lack of mental state language: There is minimal use of mental state language, such as thoughts, feelings, or intentions. Descriptions focus primarily on physical actions and events.

[0166] Poor referential coherence: Descriptions are somewhat fragmented, and connections between events and objects are unclear. Participants seem to jump between different ideas, lack coherence, and are somewhat repetitive (e.g., "That's all, isn't it?")

[0167] Demographic Information: The participant was a 71-year-old male with 8 years of education, which increases the risk of developing Alzheimer's disease.

[0168] Audio-based prediction: The audio-based prediction is "control" with a confidence of 63.08%, which indicates a relatively moderate likelihood of being a healthy individual.

[0169] Final judgment: Considering the analysis of the participant's description and demographic information, we disagree with the prediction based on the audio and instead diagnose this person as "demented" (Alzheimer's disease). The relatively brief and fragmented description, lack of mental state language, and poor referential coherence suggest a higher likelihood of cognitive impairment.

[0170] Diagnosis: Dementia.

[0171] This example demonstrates a multimodal Alzheimer's disease early screening algorithm based on a large language model to perform early screening on potential Alzheimer's patients. Through audio, language, and demographic information, this algorithm can comprehensively evaluate the patient's condition and provide highly personalized, detailed, and accurate diagnostic results.

[0172] Example 4

[0173] This embodiment describes a multimodal Alzheimer's disease early screening system based on a large language model, which effectively combines audio and demographic data by integrating speech processing, feature extraction, attention pooling, support vector machine classification, and deep semantic analysis techniques to accurately diagnose and analyze the early symptoms of Alzheimer's disease.

[0174] System composition:

[0175] a data collection unit to collect voice data and demographic information of participants;

[0176] An audio processing unit, used to perform noise suppression, voice enhancement and speaker separation operations on the voice data to obtain processed voice data;

[0177] A feature extraction unit, used to extract an eGeMAPS feature set of the processed speech data, and perform time-series splicing on different time series data in the eGeMAPS feature set to generate a feature matrix with time series information;

[0178] A feature processing unit, used to process the multimodal feature matrix using an attention pooling module to obtain pooled features;

[0179] A classification unit, configured to perform classification detection on the pooled features using a support vector machine, obtain audio-based prediction results, and distinguish potential Alzheimer's disease patients from healthy individuals;

[0180] A large language model analysis unit, configured to take the description text, the demographic information, the audio-based prediction result, and the confidence as inputs, and analyze the multimodal data based on language barriers using a zero-sample thinking chain;

[0181] A report generating unit is used to generate a screening report based on the analysis to determine whether the participant suffers from Alzheimer's disease.

[0182] Workflow:

[0183] 1. Participants provide voice samples and relevant demographic information through the data collection unit.

[0184] 2. The speech samples are processed by the audio processing unit and transmitted to the feature extraction unit in a clear and enhanced format.

[0185] 3. The feature extraction unit extracts key acoustic features and generates a feature matrix, which is then optimized by the feature processing unit and passed to the classification unit.

[0186] 4. The classification unit analyzes the features, preliminarily classifies the potential health status, and transmits the results to the large language model analysis unit.

[0187] 5. The large language model analysis unit integrates all input data, uses deep learning technology to analyze and identify any language barriers, and produces diagnostic results.

[0188] 6. The report generation unit prepares a final screening report based on the analysis results, which explains the analysis process and diagnostic results in detail for use by medical professionals.

[0189] This system provides a comprehensive and efficient early screening tool for Alzheimer's disease by integrating multiple technologies and units, aiming to improve the accuracy and timeliness of screening through advanced large language models and machine learning methods.

[0190] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.< / label> < / label>

Claims

1. A multimodal early screening algorithm for Alzheimer's disease based on a large language model, characterized in that: The following steps are involved: S1. Collecting voice data and demographic information of participants; the demographic information includes gender, age and education level; converting the voice data into descriptive text; S2, extracting the eGeMAPS feature set of the speech data; performing time-series splicing on different time series data in the eGeMAPS feature set to generate a feature matrix with time series information, and integrating demographic information into the feature matrix to obtain a multimodal feature matrix containing audio features and demographic information; S3, using an attention pooling module to process the multimodal feature matrix to obtain pooled features; S4. Using a support vector machine to perform classification detection on the pooled features to obtain audio-based prediction results and corresponding confidence levels; S5. The description text, the demographic information, the audio-based prediction results and the confidence level are used as inputs of a prompting project of a large language model to output a screening report on whether the participant suffers from Alzheimer's disease; the semantic similarity of positive and negative training samples is introduced as a reference for the prompting project; the prompting project uses a zero-sample thinking chain to analyze the multimodal data based on language barriers; the language barriers are reduced information content, lexical and semantic defects, poor referential coherence and lack of mental state language.

2. According to claim 1, a multimodal Alzheimer's disease early screening algorithm based on a large language model is characterized in that: The step S1 further performs noise suppression, voice enhancement and speaker separation operations on the voice data to obtain processed voice data; and converts the processed voice data into description text.

3. The multimodal early screening algorithm for Alzheimer's disease based on a large language model according to claim 2, characterized in that: The step S1 further uses the Koala model of PICOVOICE to suppress noise on the speech data to retain the human voice and enhance the human voice; uses the Falcon model of PICOVOICE to perform speaker separation to accurately extract the patient's voice; The patient's speech is converted into the description text using automatic speech recognition technology, and the description text is calibrated.

4. The multimodal Alzheimer's disease early screening algorithm based on a large language model according to claim 2, characterized in that: The processed speech data is sliced ​​to extract the eGeMAPS feature set of each slice; the slices have the same duration to ensure the consistency of the time series information of the eGeMAPS feature set.

5. The multimodal early screening algorithm for Alzheimer's disease based on a large language model according to claim 4, characterized in that: Step S3 further comprises: a. Dividing the processed speech data into x segments on average, extracting the eGeMAPS feature set corresponding to each segment of speech data; the eGeMAPS feature set comprises acoustic parameters of y dimensions; the acoustic parameters are represented by float type; b. Concatenate the x segments of eGeMAPS feature sets to form a matrix A of [x, y]; the matrix A∈R x×y ; Where R is the matrix whose elements are all real numbers; c. Convert the demographic information of the current sample into float type to obtain the demographic information matrix B; the matrix B∈R n ; Wherein n represents the number of types of the demographic information; the demographic information includes gender, age and education level, etc.; d. Fill the matrix B into a [x+n] matrix B' according to the time dimension; the matrix B'∈R x×n ; e. Concatenate matrix A and matrix B′: C=[A|B′]∈R x×(y+n) , Among them, C is the feature matrix.

6. The multimodal early screening algorithm for Alzheimer's disease based on a large language model according to claim 4, characterized in that: The design of the prompt project described in step 5 is as follows: Model(f(x,a1,…,a n ))=p Where Model represents the large language model, f is the prompt, p is the output of the prompt project, x is the description text, a1,...,a n Representing the thinking dimensions of n kinds of prompt engineering; the thinking dimensions include the gender, the age, the education level, the audio-based prediction results and the confidence.

7. The multimodal early screening algorithm for Alzheimer's disease based on a large language model according to claim 6, characterized in that: Step 5 further includes calculating the semantic similarity of positive and negative samples using the TF-IDF method. TF-IDF(t,d)=TF(t,d)×IDF(t) Among them, TF represents term frequency, IDF represents inverse document frequency, (t, d) represents the frequency of character t appearing in description text d, t′ represents traversing every character t in description text d; N represents the total number of description texts, and |{d∈D:t∈d}| represents the number of description texts containing character t.

8. The multimodal early screening algorithm for Alzheimer's disease based on a large language model according to claim 6, characterized in that: The language barrier in step S5 is defined as: Information content reduction refers to the reduction in the amount and quality of information conveyed in language expression; Lexical-semantic deficits refer to difficulties in vocabulary acquisition and use; Poor referential coherence refers to problems with coherence and consistency in the use of pronouns and other referential expressions; The lack of mental state language refers to the lack of language in the descriptive text that describes a person's beliefs, desires, intentions, and emotions.

9. The multimodal early screening algorithm for Alzheimer's disease based on a large language model according to claim 6, characterized in that: The analysis of language barriers further includes: Analyze and quantify the information richness and relevance of the description text, and use natural language processing technology to compare the difference between the description text and the described image to evaluate whether there is a reduction in the information content; By detecting the unconventional or incorrect use of words in the description text through semantic analysis and vocabulary network, the existence of the lexical semantic defect is evaluated; preferably, the vocabulary includes nouns and verbs; Analyze the reference relationship between the antecedent and the corresponding pronoun in the description text, and evaluate the reference coherence as poor when there is improper use of pronouns or the antecedent is missing; The psychological state language in the descriptive text is identified and analyzed, and when there is a lack of language expressing emotional understanding or psychological motivation, it is assessed as a lack of psychological state language; the psychological state language includes language that describes a person's beliefs, desires, intentions, and emotions.

10. A multimodal Alzheimer's disease early screening system based on a large language model, characterized in that: The following units are included: a data collection unit to collect voice data and demographic information of participants; An audio processing unit, used to perform noise suppression, voice enhancement and speaker separation operations on the voice data to obtain processed voice data; A feature extraction unit, used to extract an eGeMAPS feature set of the processed speech data, and perform time-series splicing on different time series data in the eGeMAPS feature set to generate a feature matrix with time series information; A feature processing unit, used to process the multimodal feature matrix using an attention pooling module to obtain pooled features; A classification unit, configured to perform classification detection on the pooled features using a support vector machine, and obtain audio-based prediction results and corresponding confidence levels to distinguish potential Alzheimer's disease patients from healthy individuals; A large language model analysis unit, configured to take the description text, the demographic information, the audio-based prediction result, and the confidence as inputs, and analyze the multimodal data based on language barriers using a zero-sample thinking chain; A report generating unit is used to generate a screening report based on the analysis to determine whether the participant suffers from Alzheimer's disease.

Images