Alzheimer disease screening method based on spontaneous language barrier

Abstract

The invention relates to the technical field of voice signal processing, and discloses an Alzheimer's disease screening method based on spontaneous language barrier, and the method comprises the following steps: S1, training a pre-training and deep fusion model for later use, S2, enabling a detected person to carry out different spoken language output tasks, and collecting the voice of the detected person, S3, carrying out the preprocessing of the collected voice, and S4, carrying out the screening of the Alzheimer's disease based on the pre-training and deep fusion model. S4, extracting voice features in the short voice segments, and screening the possibility of a patient by using a machine learning algorithm, and S5, analyzing the voice features in the test stage by using model parameters learned in the training stage, and predicting to obtain a category label to which the voice data in the test stage belongs. According to the Alzheimer's disease voice signal screening method based on deep learning, AD rapid screening is achieved through deep learning, preliminary judgment is made through short voice, the problem that the data size is small in voice screening is solved, and the method is simple and high in intelligent degree.

Description

Technical Field

[0001] The present invention relates to the technical field of speech signal processing, in particular to a screening method for Alzheimer's disease based on spontaneous language disorder. Background Art

[0002] Alzheimer's disease (AD) is the world's largest neurodegenerative disease. The typical clinical manifestation of patients is language disorders. Currently, the commonly used screening methods include expensive electroencephalograms, magnetic resonance imaging, cerebrospinal fluid, which is difficult to obtain and traumatic, and neuropsychological assessments that take a long time. It is very difficult to use this conventional approach to screen thousands of potential dementia patients.

[0003] Large-scale screening of Alzheimer's patients through language disorders can better solve these problems. However, most of the current research on acoustic features is based on simple linear features, especially research on prosodic features. However, these studies have many shortcomings: first, since these features are usually obtained using specific procedures, they are not very comparable and difficult to further evaluate; second, in the context of deep learning, the data sets used in these studies are relatively small; finally, although auxiliary screening based on linguistic features is very effective in improving the sensitivity of AD screening, its extraction process is highly dependent on the tasks performed and the linguistic and medical expertise of the researchers, which takes too long and costs too much, and also increases the cost of model construction and reduces the generalization ability of model application. Therefore, it is necessary to find a more convenient and effective way to combine linguistic knowledge and artificial intelligence methods based on current research.

[0004] If the current screening method is to be applied to large-scale screening, two problems need to be solved urgently: on the one hand, the current mainstream method is to conduct screening based on 1 minute of voice data, but it is difficult to obtain such a long period of data in real scenarios; on the other hand, in the medical context, how to solve the problem of small data volume. Therefore, through AutoGluon to achieve automatic machine learning, it can automatically process tabular data, text, images and their combinations, automatically adjust restrictions, use existing resources to find the optimal model, and design a fusion model for the voice features and text features of AD patients to achieve rapid screening in a non-invasive manner, providing a low-cost, high-feasibility, simple structure, and intelligent objective measurement method for the clinical diagnosis of AD. Summary of the invention

[0005] The purpose of the present invention is to provide a screening method for Alzheimer's disease based on spontaneous language disorder, aiming to use deep learning and speech processing technology to achieve rapid and low-cost screening for Alzheimer's disease.

[0006] To achieve the above object, the present invention provides the following technical solution: a screening method for Alzheimer's disease based on spontaneous language disorder, the screening method comprising the following steps:

[0007] S1: Training pre-training and deep fusion model backup,In this step, we train the deep learning model, including pre-training and fusion model backup;

[0008] S2: Asking the subject to perform different oral production tasks and collecting the subject's speech;

[0009] S3: pre-processing the collected speech;

[0010] S4: Extract speech features from short speech clips and use machine learning algorithms to screen patients for potential;

[0011] S5: Analyze the speech features of the test phase using the model parameters learned in the training phase, and predict the category label of the speech data in the test phase.

[0012] Preferably, the step S2 specifically includes: measuring the on-site noise, eliminating the noise source, and collecting voice after the noise meets the requirements; during the voice collection process, performing different oral production tasks on the subject and marking and arranging the voice.

[0013] Preferably, the step S2 specifically includes: measuring the noise on site, eliminating the noise source, and collecting voice after the noise meets the requirements; during the voice collection process, the subject is subjected to different oral production tasks, including self-introduction, picture description tasks and fluency test tasks, and the voice is marked and sorted. In traditional speech signal processing technology, for the screening of Alzheimer's disease, voice samples are usually simply collected, but there is a lack of targeted oral production tasks. The improvement of the present invention in this step is that a variety of oral production tasks are performed on the subject to obtain more comprehensive and rich voice data, so as to more accurately identify and analyze potential Alzheimer's patients. Specifically including self-introduction, picture description tasks and fluency test tasks, these tasks cover different aspects of language ability and cognitive function, making the collected voice samples more representative and reliable. The advantage of this improvement is that it can more comprehensively reflect the language expression ability and cognitive state of the subject, and improve the accuracy and credibility of the screening method.

[0014] Preferably, the step S3 specifically includes: in this step, the voice is subjected to a unified sampling rate, quantization bit number and number of channels, and the label and quality of the data are verified. Then, the privacy information of the speaker is removed, and the voice is subjected to denoising and segmentation. In the present invention, a systematic pre-processing is carried out for the collected voice data, including a unified sampling rate, quantization bit number and number of channels, and the label and quality of the data are verified. Further, the privacy information of the speaker is also removed, and the voice is subjected to denoising and segmentation to prepare for subsequent feature extraction and analysis. This improvement makes the collected voice data more standardized and reliable, reduces the influence of noise and other interference factors, and lays a good foundation for subsequent feature extraction and model training.

[0015] Preferably, step S4 specifically includes: In this step, we use machine learning to automatically adjust the constraints and use existing resources to find the optimal model. The extracted speech features include prosody features, spectral features, sound quality features, and the eGeMAPS standard feature set. Then, combined with the machine learning algorithm, the patients are screened. In this step, the present invention uses a deep learning model to extract pathological features related to Alzheimer's disease in short speech segments, and optimizes and trains these features. Machine learning is used to automatically adjust the constraints and find the optimal model, thereby improving the recognition ability of Alzheimer's disease.

[0016] Preferably, step S5 specifically includes: In this step, we use the model parameters learned in the training phase to analyze the speech features of the test phase to obtain the corresponding classification accuracy and confusion matrix. Then, the category label of the speech data in the test phase is predicted and output. In the present invention, the speech features of the test phase are analyzed using the model parameters learned in the training phase, and the category label of the speech data in the test phase is predicted by the deep learning model. This step makes full use of the knowledge learned in the training phase to quickly and accurately classify and identify new speech data.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] The present invention realizes rapid screening for Alzheimer's disease through deep learning and speech processing technology, solves the problems of high cost and long time of traditional screening methods, and provides a low-cost and high-efficiency method for the clinical diagnosis of Alzheimer's disease. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic flow chart of the method of the present invention;

[0020] Figure 2 It is a schematic diagram of the speech preprocessing process;

[0021] Figure 3 This is a schematic diagram of the AD short speech screening system based on acoustic features;

[0022] Figure 4 This is a schematic diagram of the pre-trained model based on the wav2vec 2.0 framework;

[0023] Figure 5 This is a schematic diagram of the pre-trained model based on the XLSR-Wav2vec 2.0 framework;

[0024] Figure 6 It is a schematic diagram of three fusion strategies;

[0025] Figure 7 This is a schematic diagram of a fusion system based on enhanced features and pre-trained features;

[0026] Figure 8 This is the structure diagram of Transformer. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] Figure 1 : is a flow chart of a method for screening Alzheimer's disease based on spontaneous language disorder of the present invention, wherein the screening method comprises the following steps:

[0029] S1: training pre-training and deep fusion models for standby, collecting and organizing the voice of the subject when performing the picture description task;

[0030] S2: Let the subject perform different oral production tasks and collect the subject's speech, and pre-process the speech and data;

[0031] S3: Preprocess the collected speech, extract the acoustic features of the speech and input them into the deep fusion model, and classify the extracted features using classical machine learning methods and automatic machine learning methods;

[0032] S4: Extract speech features from short speech clips and use machine learning algorithms to screen the possibility of patients. Directly process AD short speech samples through transfer learning technology to learn the pathological information of patients' speech.

[0033] S5: Analyze the speech features of the test phase using the model parameters learned in the training phase, and predict the category labels of the speech data in the test phase, perform decision classification and output labels respectively.

[0034] In this embodiment: Step S2 specifically includes: measuring the noise on site, eliminating the noise source, and collecting voice after the noise meets the requirements; during the voice collection process, performing different oral production tasks on the subject and marking and arranging the voice.

[0035] In this embodiment: Step S2 specifically includes: measuring the on-site noise, eliminating the noise source, and performing voice collection after the noise meets the requirements; during the voice collection process, performing different oral production tasks on the subject, the oral production tasks include self-introduction, picture description tasks and fluency test tasks, and marking and organizing the voice.

[0036] In this embodiment: Step S3 specifically includes: unifying the sampling rate, the number of quantization bits and the number of channels, verifying the label and quality of the data, removing the privacy information of the speaker, and denoising and segmenting the speech.

[0037] In this embodiment: Step S3 specifically includes: setting the sampling rate to 16kHz, the quantization bit to 16bit, the number of channels to single channel, verifying the label and quality of the data, removing the privacy information of the speaker, denoising the speech, dividing the pre-processed speech into short speech segments of 4s and 6s, and using the wav2vec 2.0 model to convert the frame-level context representation into a sentence-level representation. Figure 2 This is a schematic diagram of the preprocessing process. The training data and test data are sampled at 16Hz, privacy information is removed, irrelevant information is removed, speech is segmented, and weighted. The function of this part is to process the original data and use it as a training speech file that is easy to input for the subsequent algorithm.

[0038] In this embodiment: Step S3 specifically includes: Step S4 specifically includes: Using machine learning to automatically adjust the restriction conditions, using existing resources to find the optimal model, extracting speech features, including prosody features, spectral features, sound quality features and eGeMAPS standard feature set, and combining machine learning algorithms to screen the possibility of dementia patients, extracting two types of features, the standard feature set eGeMAPS and the acoustic feature set acoustic, after processing the collected speech signal, to examine the impact of different types of acoustic features on AD screening. The standard feature set eGeMAPS can be publicly obtained through the openSMILE toolkit. The eGeMAPS standard set contains a total of 88 dimensional features, and 88 HSF features are calculated based on 25 LLD features (including 8 frequency-related features, 3 energy / amplitude-related features and 14 spectral features). This experiment customizes the feature set acoustic: In order to fully grasp the impact of acoustic features on the AD screening system, this study customizes a 523-dimensional acoustic feature set, including 120-dimensional prosody features, 273-dimensional spectral features and 130-dimensional sound quality features. Some acoustic characteristics are shown in Table 4-1.

[0039] Table 4-1 Introduction to three types of acoustic characteristics

[0040] Table 4-1 Three types of acoustic features

[0041]

[0042] In this embodiment: Step S4 specifically includes: using AutoGluon to implement automatic machine learning, which can automatically process tabular data, text, images and their combinations, automatically adjust restrictions, and use existing resources to find the optimal model. Use the fusion model to process and extract the top 10 results of the importance of MCI vs HCs 4s features as the enhanced features of the acoustic features, and normalize the enhanced features and pre-trained features and splice them using the concat method.

[0043] In this embodiment: Step S5 specifically includes: using the model parameters learned in the training phase to analyze the speech features in the test phase, obtain the corresponding classification accuracy and confusion matrix, output the corresponding feature importance, record the features with higher importance, and then predict and output the category label of the speech data in the test phase. Taking the spectrogram feature as an example, the input is first initialized. Including: input audio file; the sampling rate of the input audio file, the default is None; the frame length defaults to 512 sampling points (32ms, 16kHz), the same as the window length; the length of the FFT window, the default is the same as the window length; the window shift, the default is 2 / 3, 512*2 / 3=341 sampling points (21ms, 16kHz); the window type, the default is Hamming window;

[0044] Next, the acquired audio data is preprocessed. The preprocessing step includes pre-emphasis processing, the purpose of which is to reduce high-frequency noise in the audio data. In this solution, we use pre-emphasis processing with a pre-emphasis coefficient of 0.97.

[0045] After preprocessing, we calculate the spectrogram features of the audio data. Specifically, we use fast Fourier transform (FFT) and Hamming window technology to calculate different types of spectrograms such as amplitude spectrum, power spectrum, log-scale power spectrum, and Mel spectrum according to the parameters provided by the user, including frame length, FFT window length, window shift, and window type.

[0046] The following is an explanation using MFCC features as an example.

[0047] When extracting the MFCC features of each speech frame, the frequency domain signal is first obtained through Fourier transform and modulus, and the output in the Mel domain is obtained through a triangular filter function. The logarithm is taken and the discrete cosine transform is used for decorrelation processing to obtain the 13th-order MFCC parameters, and then the first-order difference and second-order difference are extracted to form a 39-dimensional MFCC feature.

[0048] The feature extraction method includes using openSMILE to calculate fundamental frequency, fundamental frequency jitter, amplitude shimmer, MFCC, and LPC features; using the custom acoustic feature set of this experiment to extract features including unvoiced and voiced sounds.

[0049] This experiment uses four common classifiers for baseline classification experiments for the classic machine learning method part. Specifically, they are logistic regression (using L2 regularization), random forest (the maximum number of decision trees is 500), support vector machine (using Gaussian kernel function and probability estimation) and K nearest neighbor (the number of nearest neighbors is set to 7). The classification method, evaluation and normalization are mainly implemented based on the Sklearn toolkit.

[0050] The automatic machine learning method uses the AutoGluon framework, which trains each model in a specified order. It first trains models with more reliable performance (such as random forest models), and then gradually trains models with more computational complexity but lower reliability (such as k-nearest neighbor models). This method can help us effectively control the execution time of the fit() function to ensure that the best model is trained within a specific time period. The trained models include random forests, extreme random trees, k-nearest neighbors, LightGBM boosted trees, CatBoost boosted trees, AutoGluon-Tabular deep neural networks, and XGBoost boosted tree models, which are then assembled and stacked to form high-accuracy models.

[0051] The processing results show that the present invention realizes rapid screening of Alzheimer's disease through deep learning and speech processing technology, solves the problems of high cost and long time of traditional screening methods, and provides a low-cost and high-efficiency method for the clinical diagnosis of Alzheimer's disease.

[0052] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0053] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for screening Alzheimer's disease based on spontaneous language disorder, characterized in that: The screening method includes the following steps: S1: training pre-training and deep fusion model standby; S2: Asking the subject to perform different oral production tasks and collecting the subject's speech; S3: pre-processing the collected speech; S4: Extract speech features from short speech clips and use machine learning algorithms to screen patients for potential; S5: Analyze the speech features of the test phase using the model parameters learned in the training phase, and predict the category label of the speech data in the test phase.

2. The method for screening Alzheimer's disease based on spontaneous language disorder according to claim 1, characterized in that: The step S2 specifically includes: measuring the noise on site, eliminating the noise source, and collecting voice after the noise meets the requirements; during the voice collection process, performing different oral production tasks on the subject and marking and arranging the voice.

3. The method for screening Alzheimer's disease based on spontaneous language disorder according to claim 2, characterized in that: The step S2 specifically includes: measuring the noise on site, eliminating the noise source, and collecting voice after the noise meets the requirements; during the voice collection process, performing different oral production tasks on the subject, the oral production tasks include self-introduction, picture description tasks and fluency test tasks, and marking and organizing the voice.

4. The method for screening Alzheimer's disease based on spontaneous language disorder according to claim 1, characterized in that: The step S3 specifically includes: unifying the sampling rate, the number of quantization bits and the number of channels, verifying the label and quality of the data, removing the privacy information of the speaker, and performing denoising and segmentation on the speech.

5. The method for screening Alzheimer's disease based on spontaneous language disorder according to claim 4, characterized in that: The step S3 specifically includes: setting the sampling rate to 16kHz, the quantization bit number to 16bit, the number of channels to single channel, verifying the label and quality of the data, removing the speaker's privacy information, denoising the speech, dividing the preprocessed speech into short speech segments of 4s and 6s, and using the wav2vec 2.0 model to convert the frame-level context representation into a sentence-level representation.

6. The method for screening Alzheimer's disease based on spontaneous language disorder according to claim 1, characterized in that: The step S4 specifically includes: using machine learning to automatically adjust restrictions, using existing resources to find the optimal model, extracting speech features, including prosodic features, spectral features, and sound quality features as well as the eGeMAPS standard feature set, and combining machine learning algorithms to screen the possibility of dementia patients.

7. The method for screening Alzheimer's disease based on spontaneous language disorder according to claim 1, characterized in that: The step S4 specifically includes: using AutoGluon to implement automatic machine learning, which can automatically process tabular data, text, images and their combinations, automatically adjust restrictions, and use existing resources to find the optimal model. The fusion model is used to process and extract the top 10 results of the MCI vs HCs 4s feature importance as the enhanced features of the acoustic features, and the enhanced features and pre-trained features are normalized and spliced ​​using the concat method.

8. The method for screening Alzheimer's disease based on spontaneous language disorder according to claim 1, characterized in that: The step S5 specifically includes: using the model parameters learned in the training phase to analyze the speech features in the test phase, obtaining the corresponding classification accuracy and confusion matrix, outputting the corresponding feature importance, recording the features with higher importance, and then predicting and outputting the category label of the speech data in the test phase.

Images