Alzheimer's disease early recognition system based on attention mechanism

Abstract

The invention discloses an attention mechanism-based Alzheimer's disease early recognition system, which comprises a data preprocessing module used for preprocessing long audio clips; the feature extraction module is used for extracting acoustic features A from the audio clips and extracting traditional language features T and deep language features L from the text data; the first splicing module is used for splicing the A, T and L features to obtain a spliced feature Y; the self-attention module is used for performing feature extraction on the Y by using a self-attention mechanism to obtain a fusion feature Z; the cross-modal attention module is used for enabling the A, T and L features to absorb the features in the Z to obtain three enhanced features A +, T + and L +; the second splicing module is used for splicing A +, T + and L + to obtain a spliced feature Y +; and the identification module is used for carrying out classification identification on the feature Y + by adopting a linear layer and a cross entropy loss function to obtain an identification result.

Description

Technical Field

[0001] This application relates to the field of early Alzheimer's disease identification technology, and more particularly to an early Alzheimer's disease identification system based on attention mechanisms. Background Technology

[0002] Alzheimer's disease (AD), a major type of dementia in the elderly, is currently incurable. However, early diagnosis can effectively slow its progression, making early detection crucial. The disease primarily affects people aged 65 to 90, and its main characteristic is gradual memory loss. As the disease progresses, patients not only experience forgetfulness but also decline in language skills, eye movement abnormalities, cognitive decline, weakened calculation abilities, reduced scene semantic perception, and abnormal changes in brain structure, accompanied by significant changes in personality and behavior. These neurological abnormalities severely impact patients' daily lives.

[0003] Alzheimer's disease is often accompanied by language problems, which are the most direct and unavoidable consequence of cognitive impairment. Previous studies have shown that the language centers in AD patients are damaged, leading to problems in thinking, expressing themselves, understanding others' speech, and memorizing vocabulary. Regarding language expression, the main problems include: First, AD patients often experience a reduced vocabulary and difficulty memorizing words. This means they may forget some everyday phrases or names, or be unable to use words correctly, causing communication difficulties. Second, AD patients also exhibit cognitive impairment in understanding others' speech. They have difficulty understanding or correctly comprehending the meaning of sentences spoken by others, thus struggling to respond appropriately. Third, AD patients experience difficulties in sentence construction, including grammatical, lexical, and semantic problems, resulting in disorganized and incoherent expression. These language-related cognitive impairments usually become apparent in the early clinical stages. To address these issues, Qin et al. used Wav2Vec2.0 for speech feature extraction, achieving good results in AD diagnostic tasks. Liu et al. created a new speech dataset including AD and healthy controls, and used a logistic regression cross-validation model (LogRegCV) for classification, demonstrating the effectiveness of extracting spectrogram features from speech data. Calzà L et al. conducted acoustic, rhythmic, morphological, syntactic, and lexical analyses of speech samples, and trained automatic classifiers using algorithms such as Support Vector Machine (SVM) and Random Forest (RF) to distinguish between healthy controls and mild cognitive impairment groups. Wang et al. analyzed the performance of Percentage of Silence Duration (PSD) in AD, amnestic mild cognitive impairment (aMCI), and normal control (NC) patients, finding significantly increased PSD in aMCI and AD patients, confirming PSD as a reliable diagnostic biomarker for early AD and aMCI. Agbavor et al. used a pre-trained audio model to extract features from patient audio and evaluated these features on the ADReSSO dataset.

[0004] Some scholars focus on using textual information to study AD features for classification. For example, Adhikari S et al. used the resource-scarce Nepali language as their subject, employing techniques such as TF-IDF, CountVectorizer (CV), Word2Vec, and FastText to convert text documents into vector representations. Regarding machine learning models, the Naive Bayes classifier performed best when using CV and TF-IDF vectorization techniques. In terms of deep learning models, a CNN model with Word2Vec achieved the best results when applying attention mechanisms, highlighting their potential in early Alzheimer's diagnosis, especially in the early detection of speech degeneration. Khan YF et al. used three different approaches: traditional machine learning models, sequential deep learning models, and XLNet and BERT models. These models showed varying accuracies in automatically detecting language indicators of cognitive memory loss in AD patients. Ilias et al. used a multi-task learning approach, utilizing a transformer-based model for AD diagnosis, demonstrating high accuracy in both single-task and multi-task learning, and revealing language differences between AD and non-AD patients. YF Khan et al. used the bag-of-words method in the feature extraction stage to represent text data and developed a stacked deep dense neural network (SDDNN) model for text classification. They also used audio recording data to diagnose Alzheimer's disease.

[0005] The aforementioned methods do not fully consider some problems inherent in single-modality models, such as confusion with other diseases and difficulty in effective early diagnosis. Y Huang et al. also mathematically verified that multimodal models are superior to any single-modality model. Therefore, to address the problems caused by single-modality models, Ying et al. used multimodal features, combining acoustic and linguistic features, to identify AD. They used the IS10 toolkit to extract traditional acoustic features, a fine-tuned Wav2Vec2.0 model to extract deep acoustic features, and a fine-tuned BERT model to extract linguistic features. The features were concatenated and classified using an SVM classifier. This method achieved 89.1% accuracy in long audio and 84.0% accuracy in short audio. However, this method simply concatenates features without fully considering the correlation between modalities. Mittal A et al. also proposed a multimodal deep learning method that simultaneously detects AD using speech and corresponding transcripts. This method uses transfer learning to address the problem of insufficient data and tested the feasibility of using text generated by an Automatic Speech Recognition (ASR) system instead of manual transcription. The model achieved an accuracy of 85.3% on the DementiaBank dataset, and age and gender biases were analyzed. PanY et al. used an automatic speech recognition system to obtain temporal alignment information and confidence scores for audio segments, improving the robustness of acoustic feature extraction. They combined acoustic and linguistic features, using a bidirectional Long Short-Term Memory (LSTM) network and attention mechanisms to classify the combined features, demonstrating that the method can relatively accurately distinguish audio segments containing information about cognitive impairment, providing strong support for early diagnosis. Li et al. proposed several effective methods for extracting more Alzheimer's disease-related cues from high-level acoustic and linguistic features. The results were validated by comparing the performance of combinations of acoustic, linguistic, and task-related features. Martinc et al. analyzed diagnostic methods for Alzheimer's disease using audio feature engineering. They used the OpenSmile toolkit and GloVe to extract acoustic and linguistic features from speech segments, and then constructed Active Data Representation (ADR) features.

[0006] In summary, the current method has the following drawbacks:

[0007] 1. The monomodal approach is easily confused with other diseases in the early identification of Alzheimer's disease.

[0008] 2. In the extraction of features from speech and text data, it is not possible to fully capture semantic language patterns and language changes specific to AD patients.

[0009] 3. The method of directly splicing features does not take into account the correlation between different modalities and cannot make the most of multimodal information. Summary of the Invention

[0010] In view of this, embodiments of this application provide an early Alzheimer's disease identification system based on an attention mechanism to solve the problems existing in the background art.

[0011] According to an embodiment of this application, an early Alzheimer's disease identification system based on an attention mechanism is provided, comprising:

[0012] A data preprocessing module is used to uniformly divide long audio segments into N audio segments, and then use ASR to transcribe the audio segments into text data. The long audio segments include audio from both healthy individuals and Alzheimer's patients.

[0013] The feature extraction module is used to input the audio segment into the Wav2Vec2.0 model and output acoustic features A, and to process the text data through the TF-IDF algorithm and the Word2Vec model to output traditional language features T and deep language features L.

[0014] The first splicing module is used to splice the features A, T and L to obtain the spliced ​​feature Y;

[0015] The self-attention module is used to extract semantic information features from Y using a self-attention mechanism to obtain fused features Z.

[0016] The cross-modal attention module is used to use the cross-modal attention mechanism to enable the A, T, and L features to extract information from the Z feature, resulting in three enhanced features A+, T+, and L+.

[0017] The second splicing module is used to splice A+, T+ and L+ to obtain the spliced ​​feature Y+;

[0018] The recognition module is used to classify and recognize feature Y+ using a linear layer and cross-entropy loss function to obtain the recognition result.

[0019] Optionally, the Wav2Vec2.0 model is used to encode audio segments using a multi-layer convolutional neural network to generate a latent audio representation H, to perform a masking operation on the latent audio representation H, and to input it into a Transformer architecture to generate a context feature sequence C. The feature sequence C is compressed into a fixed-dimensional acoustic feature vector through an average pooling layer, and then the acoustic feature A is output through an embedding layer.

[0020] Optionally, the TF-IDF algorithm is used to calculate the term frequency (TF) of each word in the document and the inverse document frequency (IDF) of each word, and multiply the TF and IDF to obtain the TF-IDF weights, which will form the traditional language features T.

[0021] Optionally, the Word2Vec model is used to maximize the probability of predicting the center word given contextual word conditions through an objective function, thereby obtaining deep language features L.

[0022] Optionally, the self-attention mechanism is used to divide the feature Y into several small parts, each part as an attention head. A linear transformation is performed on each attention head to obtain query, key, and value vectors. A dot product operation is performed on the query, key, and value vectors to obtain an attention score, which is then normalized using Softmax. Finally, the output of the multi-head attention mechanism is obtained by weighted summation, and a corresponding weight matrix is ​​introduced for each attention head. The outputs of the multi-head attention mechanism are concatenated and multiplied by the weight matrix. Feature fusion is performed through residual connections and layer normalization, and then transformed through a fully connected layer to obtain the fused feature Z.

[0023] Optionally, the cross-modal attention mechanism is used to input features A, T, and L, as well as the fused feature Z. A one-dimensional convolutional layer maps different feature dimensions to a unified dimension and introduces positional embedding. A cross-modal multi-head attention mechanism performs dot product operations on the query, key, and value vectors to obtain attention scores, which are then normalized using Softmax. Finally, a weighted summation is used to obtain the output of the multi-head attention mechanism. The outputs of the multi-head attention mechanism are concatenated and multiplied by a weight matrix to obtain the final output of the multi-head attention mechanism. Residual connections and layer normalization, as well as fully connected layers and residual connections, are used to calculate the output, resulting in enhanced features A+, T+, and L+. The query vector is obtained by linear transformation of A, T, and L, while the key and value vectors are obtained by linear transformation of the fused feature Z.

[0024] The technical solutions provided by the embodiments of this application may include the following beneficial effects:

[0025] As can be seen from the above embodiments, this application combines multimodal data of speech and text for early identification of Alzheimer's disease. After in-depth analysis and comparison of numerous existing feature extraction techniques, three methods with complementary advantages are combined: TF-IDF, Wav2Vec2.0, and Word2Vec. Self-attention and cross-modal attention mechanisms are introduced, and by fusing information from different data sources and strengthening feature representation, better modeling of complex relationships is achieved.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0028] Figure 1 This is a block diagram illustrating an attention-based early detection system for Alzheimer's disease according to an exemplary embodiment.

[0029] Figure 2 This is a schematic diagram of the Wav2vec2.0 model structure according to an exemplary embodiment.

[0030] Figure 3 This is a schematic diagram of a self-attention module structure according to an exemplary embodiment.

[0031] Figure 4 This is a schematic diagram of a cross-modal attention module structure according to an exemplary embodiment. Detailed Implementation

[0032] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. In the following description relating to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements.

[0033] Figure 1 This is a block diagram illustrating an attention-based early detection system for Alzheimer's disease, according to an exemplary embodiment. Figure 1 As shown, the system includes: data preprocessing module 1, feature extraction module 2, first stitching module 3, self-attention module 4, cross-modal attention module 5, second stitching module 6, and recognition module 7. Each module is further described in detail below.

[0034] Data preprocessing module 1 is used to preprocess long audio segments, including audio from both normal individuals and Alzheimer's patients. First, the long audio segments are evenly divided into N shorter audio segments, making the large audio data easier to process and analyze. Second, ASR is used to transcribe the audio segments into text data.

[0035] Feature extraction module 2, after in-depth analysis and comparison of many existing feature extraction technologies, combines three methods with complementary advantages: TF-IDF, Wav2Vec2.0 and Word2Vec, which can capture language features more effectively.

[0036] like Figure 2As shown, Wav2vec 2.0 is used to extract acoustic features. Wav2vec 2.0 focuses more on acoustic feature extraction, primarily on acoustic properties such as waveform and spectrum, rather than directly parsing language content. This helps maintain feature consistency, unaffected by language differences and data complexity, and requires no large-scale labeled data, making it more suitable for acoustic feature extraction tasks in multimodal datasets. First, a multi-layer convolutional neural network is used to encode the audio segment, generating a latent audio representation H. Convolutional neural networks excel at extracting audio features, capturing local features and spectral information in audio and transforming them into meaningful representations, which helps the model understand the structure and features of the audio. Simultaneously, the latent audio representation H is masked and input into a Transformer architecture to generate a contextual feature sequence C. The Transformer architecture is used to capture long-range contextual information between sequences, helping the model understand the contextual relationships of speech, reducing data dimensionality, and extracting key information from the audio. Secondly, the feature sequence C is compressed into a fixed 1024-dimensional acoustic feature vector using an average pooling layer, and then an embedding layer with an activation function is used to obtain a 192-dimensional feature representation A. This step maps the context features to a consistent representation space for subsequent processing. Finally, a fully connected layer linearly transforms the output of the embedding layer to generate the final feature representation for different tasks.

[0037] TF-IDF is used to extract traditional language features. The TF-IDF algorithm helps identify words that are relatively uncommon in the general population but more specific to certain documents or categories. It can also capture unique language patterns, such as repetitive words, grammatical errors, and disorganized expressions, which are more prevalent in the speech of AD patients. The TF-IDF algorithm is used to calculate the term frequency (TF) of each word in the document and the inverse document frequency (IDF) of each word. The TF and IDF are multiplied to obtain TF-IDF weights, which form the traditional language features T for further analysis.

[0038] Deep language features are extracted using Word2Vec. Word2Vec is a powerful and widely used natural language processing tool that efficiently maps words to a continuous vector space, capturing semantic relationships. Word2Vec's vector representation preserves semantic information between words, making similar words closer together in the vector space. Through dimensionality reduction, Word2Vec transforms the original sparse, high-dimensional word vectors into low-dimensional, dense vectors, thereby improving model training and storage efficiency. The objective function maximizes the probability of predicting the center word given contextual word conditions. The training objective function is calculated as follows:

[0039] maximizelog P(w|c) (1)

[0040] Here, `maximize` represents the function to be maximized, and `log P(w|c)` represents the probability of word `w` appearing given context word `c`. Taking the negative of the logarithm of this probability and summing the results yields the conditional entropy, which measures the average amount of information given conditions.

[0041] The first splicing module 3 uses concatenation to splice the features A, T, and L to obtain the spliced ​​feature Y.

[0042] Self-attention module 4 uses a self-attention mechanism to further extract fused features. For example... Figure 3 As shown, firstly, this structure possesses highly parallel computing capabilities, enabling it to simultaneously focus on all positions within a sequence, exhibiting particularly excellent performance when processing long sequences. Secondly, because the self-attention mechanism can model complex dependencies between different positions within a sequence, it can capture key features and patterns in the input data, thereby achieving feature extraction. Furthermore, the multi-head attention mechanism has significant advantages in multimodal information fusion, allowing the model to independently focus on features of different modalities. Each head focuses on learning useful information specific to a particular modality to better learn and express the correlations between features of different modalities. It also endows the model with the ability to selectively focus on key features of specific modalities for specific tasks. Based on the vector dimension d... model Divide Y into several smaller parts, each of which serves as an attention head, for a total of i.

[0043] The calculation method for a specific attention head is as follows:

[0044]

[0045] in, These represent the query vector, key vector, and value vector generated after linear transformation, respectively.

[0046] Secondly, use Concatenate to apply the outputs of all attention heads: head1, head2, ..., head1. h Concatenate the results and combine them with the weight matrix. Multiplying these results yields the final output of the multi-head attention mechanism, calculated as follows:

[0047] F=MHA(Q,K,V)=Concat(head1,head2,...,head h W O (3)

[0048] Subsequently, the multimodal features were fused using residual connections and layer normalization, with fully connected layers and residual connections used for computation, to obtain contextual information fusion features. Specifically as follows:

[0049]

[0050] Where, γ LN1 γ LN2 and β LN1 ,β LN2 These are parameters that control the normalization scaling and offset, respectively, and they affect the normalization of the two layers. W 1FFN W 2FFN It is a weight matrix of linear transformation. The former transforms the output F of multi-head attention to a higher-dimensional intermediate layer, while the latter transforms the output of the intermediate layer to the final output dimension.

[0051] Cross-modal attention module 5 uses a cross-modal attention mechanism to allow information from different modalities to interact and enhance their respective representations. For example... Figure 4 As shown, firstly, input features A, T, and L, along with the fused feature Z, are processed through a one-dimensional convolutional layer to map the different feature dimensions to a unified dimension d. For the acoustic modality (A), after convolution, it is represented as... The fusion feature (Z) is obtained Features from different modalities maintain a unified dimensionality in subsequent processing, facilitating dot product operations in cross-modal attention mechanisms and enabling information interaction and fusion between modalities. Furthermore, to fully integrate temporal information, positional embeddings are introduced and incorporated into the representations of each modality to capture temporal structure. This operation ensures that the sequence carries not only modal features but also time-related positional information.

[0052] Subsequently, in the cross-modal multi-head attention layer, a multi-head attention mechanism is used, where the attention matrix of the i-th head is calculated as follows:

[0053]

[0054] in and These represent acoustic modes A and B, respectively. cross and fusion feature Z cross It is obtained through linear transformation. It is the linear transformation weight matrix of the i-th attention head corresponding to the modality. Where, d cross =d / h2 represents the dimension of the head, and h2 represents the number of heads.

[0055] The cross-modal attention mechanism is a key component of this method, fully embodying the core idea of ​​cross-modal multi-head attention mechanisms. It enables one modality to acquire information from another, thereby promoting information interaction and fusion between modalities. The calculation method for the i-th layer of the cross-modal multi-head attention layer is as follows:

[0056]

[0057] in It is the output of the cross-modal attention module in layer i-1. It is a weight matrix.

[0058] Next, residual connections and layer normalization are used, along with fully connected layers and residual connections, to calculate the output of the current layer, resulting in A+. L+ and T+ are obtained similarly.

[0059] The second concatenation module 6 concatenates the enhanced feature pairs A+, T+, and L+ to obtain the concatenated feature Y+.

[0060] The recognition module 7 uses a linear layer and a cross-entropy loss function to generate classification outputs. The linear layer maps the model's features to score spaces for different categories and internally applies the ReLU (Rectified LinearUnit) activation function to introduce non-linearity. The cross-entropy loss function internally performs a softmax operation, transforming the original output scores into a probability distribution before calculating the loss, enabling the model to learn how to better classify the input data. This combined operation allows the model to effectively perform multi-class classification, mapping input data to corresponding categories and making classification decisions.

[0061] The method described in this application achieved accuracy, precision, recall, and F1 score of 91.6%, 91.3%, 91.5%, and 91.3% for early Alzheimer's disease identification on the NCMMSC2021 long speech dataset, respectively. In short speech, the accuracy, precision, recall, and F1 score were 85.2%, 84.6%, 84.6%, and 84.6%, respectively.

[0062] In long speech, the proposed method achieved an 11.8% improvement over the baseline, an 8.4% improvement over Qin et al.'s method, and a 2.5% improvement over Yang et al.'s method. In short speech, the proposed method achieved an 11.2% improvement over the baseline, a 7.2% improvement over Qin et al.'s method, and a 1.2% improvement over Yang et al.'s method.

[0063] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0064] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. An early Alzheimer's disease identification system based on attention mechanisms, characterized in that, include: The data preprocessing module is used to evenly divide a long audio segment into N audio segments and use ASR to transcribe the audio segments into text data. The long audio segments include audio from normal people and Alzheimer's patients. The feature extraction module is used to input the audio segment into the Wav2Vec2.0 model and output acoustic features A, and to process the text data through the TF-IDF algorithm and the Word2Vec model to output traditional language features T and deep language features L. The first splicing module is used to splice the features A, T and L to obtain the spliced ​​feature Y; The self-attention module is used to extract semantic information features from Y using a self-attention mechanism to obtain fused features Z. The cross-modal attention module is used to use the cross-modal attention mechanism to enable the A, T, and L features to extract information from the Z feature, resulting in three enhanced features A+, T+, and L+. The second splicing module is used to splice A+, T+ and L+ to obtain the spliced ​​feature Y+; The recognition module is used to classify and recognize feature Y+ using a linear layer and cross-entropy loss function to obtain the recognition result.

2. The Alzheimer's disease early identification system based on attention mechanism according to claim 1, characterized in that, The Wav2Vec2.0 model is used to encode audio segments using a multi-layer convolutional neural network to generate a latent audio representation H. The latent audio representation H is masked and input into a Transformer architecture to generate a context feature sequence C. The feature sequence C is compressed into a fixed-dimensional acoustic feature vector through an average pooling layer, and then the acoustic feature A is output through an embedding layer.

3. The Alzheimer's disease early identification system based on attention mechanism according to claim 1, characterized in that, The TF-IDF algorithm is used to calculate the term frequency (TF) of each word in the document and the inverse document frequency (IDF) of each word. The TF and IDF are multiplied to obtain the TF-IDF weights, which will form the traditional language features T.

4. The Alzheimer's disease early identification system based on attention mechanism according to claim 1, characterized in that, The Word2Vec model is used to maximize the probability of predicting the center word given contextual word conditions through an objective function, thereby obtaining deep language features L.

5. The Alzheimer's disease early identification system based on attention mechanism according to claim 1, characterized in that, The self-attention mechanism is used to divide the feature Y into several small parts, each of which is an attention head. A linear transformation is performed on each attention head to obtain the query, key, and value vectors. The query, key, and value vectors are then multiplied by a dot product to obtain an attention score, which is then normalized using Softmax. Finally, the output of the multi-head attention mechanism is obtained by weighted summation, and a corresponding weight matrix is ​​introduced for each attention head. The outputs of the multi-head attention mechanism are concatenated and multiplied by the weight matrix. Feature fusion is then performed through residual connections and layer normalization, and finally transformed by a fully connected layer to obtain the fused feature Z.

6. The Alzheimer's disease early identification system based on attention mechanism according to claim 1, characterized in that, The cross-modal attention mechanism is used to input features A, T, and L, as well as the fused feature Z. A one-dimensional convolutional layer maps different feature dimensions to a unified dimension and introduces positional embedding. A cross-modal multi-head attention mechanism performs dot product operations on the query, key, and value vectors to obtain attention scores, which are then normalized using Softmax. Finally, a weighted summation is used to obtain the output of the multi-head attention mechanism. The outputs of the multi-head attention mechanism are concatenated and multiplied by a weight matrix to obtain the final output. Residual connections and layer normalization, as well as fully connected layers and residual connections, are used to calculate the output, resulting in enhanced features A+, T+, and L+. The query vector is obtained by linear transformation of A, T, and L, while the key and value vectors are obtained by linear transformation of the fused feature Z.

Images