An audio-based mild cognitive impairment auxiliary diagnosis method and device

Abstract

The application provides a mild cognitive impairment auxiliary diagnosis method and device based on audio, and relates to the technical field of computer auxiliary diagnosis.The method comprises the following steps: obtaining original dialogue audio; performing audio and acoustic parameter feature extraction according to the original dialogue audio to obtain multi-resolution spectrum graph features and paralanguage features; using a multi-scale double attention mechanism based on a hierarchical attention weight generation strategy to perform dynamic feature fusion according to the multi-resolution spectrum graph features to obtain optimized spectrum graph features; using the optimized spectrum graph features and the paralanguage features to optimize and fine-tune a pre-trained auxiliary diagnosis model to obtain an optimized auxiliary diagnosis model; obtaining actual dialogue audio, using the optimized auxiliary diagnosis model to perform cognitive impairment detection, and obtaining an auxiliary diagnosis result.The application is an efficient and accurate mild cognitive impairment auxiliary diagnosis method based on audio.

Description

Technical Field

[0001] This invention relates to the field of computer-aided diagnostic technology, and in particular to an audio-based method and apparatus for assisting in the diagnosis of mild cognitive impairment. Background Technology

[0002] Mild cognitive impairment (MCI) is an early stage of neurodegenerative diseases such as Alzheimer's disease (AD). Early diagnosis is crucial for slowing the progression of AD, but traditional methods rely on medical imaging, such as magnetic resonance imaging (MRI) and positron emission tomography (PET), which are costly, invasive, and poorly accessible. Speech, as a non-invasive and readily available biomarker, can reflect language impairments in MCI patients (such as difficulty in vocabulary retrieval and semantic comprehension breakdown), providing a new low-cost screening approach.

[0003] Existing speech diagnostic technologies mainly fall into three research paths: Traditional machine learning methods typically employ algorithms such as Support Vector Machines (SVM) or Random Forests, classifying based on manually extracted Mel-Frequency Cepstral Coefficients (MFCC) features or a small number of paralinguistic features, maintaining an accuracy range of 60%-87%. However, they suffer from inherent drawbacks such as strong subjectivity in feature selection and difficulty in capturing complex acoustic patterns. Deep learning methods achieve automatic feature extraction through convolutional neural networks or long short-term memory networks. For example, after converting speech signals into Mel spectrograms, a densely connected convolutional network with 121 layers (DenseNet121) model achieves an accuracy of 90%, but the requirement for a large amount of labeled data limits its clinical application. Pre-trained model fine-tuning techniques, although using hidden unit autoregressive Transformers (Hidden Unit Autoregressive Transformers), can still achieve similar results. Models such as UnitBERT and HuBERT perform well in speech recognition tasks, but their adaptability to MCI classification tasks is still insufficient. For example, an accuracy of 83.2% can be achieved by fine-tuning the Automatic Speech Recognition (ASR) model, but no specific optimization is made for the MCI binary classification task.

[0004] Current technical approaches suffer from several key shortcomings: In terms of feature engineering, existing research does not fully utilize paralinguistic features, mostly focusing only on a few features such as pitch and speech rate, failing to systematically verify the diagnostic value of large feature sets; in the data preprocessing stage, commonly used datasets such as ADReSS and DementiaBank are mostly long audio recordings containing dialogues between medical staff, requiring complex and tedious speech segmentation and noise reduction processing. These technical bottlenecks severely restrict the clinical application and promotion of speech-based MCI diagnostic methods.

[0005] Currently, there is a lack of an efficient and accurate audio-based method for assisting in the diagnosis of mild cognitive impairment. Summary of the Invention

[0006] To address the shortcomings of existing technologies, such as the high subjectivity of feature selection, difficulty in comprehensively capturing complex acoustic patterns in the speech of MCI patients, large requirements for labeled data, and poor model interpretability, this invention provides an audio-based auxiliary diagnostic method and device for mild cognitive impairment. The technical solution is as follows:

[0007] On the one hand, an audio-based auxiliary diagnostic method for mild cognitive impairment is provided, which is implemented by a mild cognitive impairment auxiliary diagnostic device, and includes:

[0008] Acquire the original dialogue audio; extract audio and acoustic parameter features from the original dialogue audio to obtain multi-resolution spectrogram features and paralinguistic features;

[0009] A multi-scale dual attention mechanism based on a hierarchical attention weight generation strategy is used to dynamically fuse features according to multi-resolution spectrogram features to obtain optimized spectrogram features.

[0010] The pre-trained auxiliary diagnostic model is optimized and fine-tuned using optimized spectrogram features and sub-language features to obtain an optimized auxiliary diagnostic model.

[0011] Acquire actual dialogue audio; extract audio and acoustic parameter features based on the actual dialogue audio to obtain actual multi-resolution spectrogram features and actual paralinguistic features; based on a multi-scale dual attention mechanism with a hierarchical attention weight generation strategy, use an optimized auxiliary diagnostic model to detect cognitive impairment based on the actual multi-resolution spectrogram features and actual paralinguistic features, and obtain auxiliary diagnostic results.

[0012] On the other hand, an audio-based auxiliary diagnostic device for mild cognitive impairment is provided, which is applied to an audio-based auxiliary diagnostic method for mild cognitive impairment. The device includes:

[0013] The feature extraction module is used to acquire the original dialogue audio; based on the original dialogue audio, audio and acoustic parameter features are extracted to obtain multi-resolution spectrogram features and paralinguistic features;

[0014] The spectrogram feature optimization module is used for a multi-scale dual attention mechanism based on a hierarchical attention weight generation strategy. It performs dynamic feature fusion based on multi-resolution spectrogram features to obtain optimized spectrogram features.

[0015] The model fine-tuning module is used to optimize and fine-tune the pre-trained auxiliary diagnostic model using optimized spectrogram features and sub-language features to obtain an optimized auxiliary diagnostic model.

[0016] The auxiliary diagnostic result prediction module is used to acquire actual dialogue audio; based on the actual dialogue audio, audio and acoustic parameter features are extracted to obtain actual multi-resolution spectrogram features and actual paralinguistic features; based on a multi-scale dual attention mechanism with a hierarchical attention weight generation strategy, an optimized auxiliary diagnostic model is used to detect cognitive impairment and obtain auxiliary diagnostic results.

[0017] On the other hand, a mild cognitive impairment auxiliary diagnostic device is provided, the mild cognitive impairment auxiliary diagnostic device comprising: a processor; a memory, the memory storing computer-readable instructions, which, when executed by the processor, implement any of the methods in the above-described audio-based mild cognitive impairment auxiliary diagnostic methods.

[0018] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement any of the above-described audio-based methods for assisting diagnosis of mild cognitive impairment.

[0019] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0020] This invention proposes an audio-based method for assisting the diagnosis of mild cognitive impairment. It employs a multi-scale dual attention mechanism based on a hierarchical attention weight generation strategy to capture key information at different time scales. This mechanism effectively enhances the model's ability to perceive subtle changes in speech signals, thereby improving classification performance and significantly increasing the model's accuracy and robustness in MCI classification tasks. Combining spectrogram features with traditional paralinguistic features produces a complementary effect, further enhancing the model's classification performance. Through the effective fusion of these two different types of features, the model can not only extract information from the time-frequency representation of audio signals but also utilize paralinguistic features to capture important clues such as the speaker's emotional state and health status, further improving classification accuracy. This invention provides an efficient and accurate audio-based method for assisting the diagnosis of mild cognitive impairment. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of an audio-based auxiliary diagnostic method for mild cognitive impairment provided by an embodiment of the present invention;

[0023] Figure 2 This is a block diagram of an audio-based auxiliary diagnostic device for mild cognitive impairment provided in an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of the structure of an auxiliary diagnostic device for mild cognitive impairment provided in an embodiment of the present invention. Detailed Implementation

[0025] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0026] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0027] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0028] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0029] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0030] This invention provides an audio-based method for assisting in the diagnosis of mild cognitive impairment. This method can be implemented using a diagnostic device for mild cognitive impairment, which can be a terminal or a server. Figure 1 The flowchart shown is for an audio-based method for assisting in the diagnosis of mild cognitive impairment. The processing flow of this method may include the following steps:

[0031] S1. Obtain the original dialogue audio; extract audio and acoustic parameter features based on the original dialogue audio to obtain multi-resolution spectrogram features and paralinguistic features.

[0032] Optionally, audio and acoustic parameter features are extracted from the original dialogue audio to obtain multi-resolution spectrogram features and paralinguistic features, including:

[0033] The original dialogue audio is preprocessed to obtain the processed dialogue audio;

[0034] Based on the Fast Fourier Transform method, Mel spectrogram features are extracted from the processed dialogue audio to obtain multi-resolution spectrogram features.

[0035] The OpenSmile tool was used to perform multi-dimensional acoustic parameter feature analysis on the processed dialogue audio to obtain paralinguistic features.

[0036] In one feasible implementation, the present invention uses the original dialogue audio from the DementoaBank English Protocol Delaware Corpus dataset, which is used to study changes in language ability in Alzheimer's disease-related dementia and mild cognitive impairment. To address the issues of interference from healthcare professionals and content redundancy in the original dialogue audio, the participants' speech is extracted, and long audio clips are segmented into standard 30-60 second segments to form processed dialogue audio suitable for fine-tuning of pre-trained models. The preprocessing preserves natural pauses and prosodic features in the speech, ensuring the integrity of acoustic information.

[0037] In audio signal processing, spectrogram and paralinguistic feature techniques address the representation needs of signal frequency domain structure and non-linguistic information, respectively.

[0038] The spectrogram-related technology is based on the Mel spectrogram. After pre-emphasizing and enhancing high-frequency components, the audio is segmented into short-time windows and Hamming windows are applied to reduce spectral leakage. The power spectral density of each frame is calculated using fast Fourier transform. Then, the frequency axis is nonlinearly mapped by a triangular filter bank under the Mel scale. The time-frequency energy distribution that conforms to the characteristics of human ear perception is generated by weighted averaging.

[0039] The extraction of secondary language features relies on the Opensmile tool to achieve multi-dimensional acoustic parameter analysis. The technical solution covers two stages: low frequency descriptor (LLD) extraction and statistical function aggregation.

[0040] In this process, 88 LLDs were extracted based on the eGeMAPSv02 feature set of the OpenSmile tool. The parameters include fundamental frequency, intensity, and spectral tilt. A 23-dimensional high-level feature set is generated using statistical functions such as mean and standard deviation, focusing on capturing changes in speech rhythm and tone quality. The emobase feature set expands to 1582 dimensions, integrating emotion-related acoustic indicators such as jitter and tremor, and using multi-scale statistics to describe non-periodic fluctuations in sound. The large-scale emotion feature set further integrates time-frequency dynamic features and cross-channel correlation analysis, constructing a 3840-dimensional feature vector to analyze subtle differences in emotional states. The INTERSPEECH series feature sets adopt a differentiated design strategy: IS10 analyzes emotional characteristics using 6500+ LLD parameters (covering prosody and spectral quality parameters); IS11 focuses on assessing the speaker's physiological state, such as stress and fatigue; and IS12 extracts long-term stable acoustic patterns for personality traits. All three generate high-level features through statistical aggregation (extreme values, quantiles, regression coefficients). The core technology lies in the refined selection of LLD parameters (such as fundamental frequency trajectory, formant bandwidth, and harmonic-to-noise ratio) and the combined optimization of statistical modeling methods to achieve sensitive capture of subtle acoustic abnormalities in the speech of MCI patients.

[0041] S2. A multi-scale dual attention mechanism based on a hierarchical attention weight generation strategy is used to dynamically fuse features according to multi-resolution spectrogram features to obtain optimized spectrogram features.

[0042] Optionally, a multi-scale dual attention mechanism based on a hierarchical attention weight generation strategy is used to dynamically fuse features according to multi-resolution spectrogram features to obtain optimized spectrogram features, including:

[0043] Based on a multi-scale filter bank, feature analysis is performed according to multi-resolution spectral characteristics to obtain multi-resolution FBANK features.

[0044] Data reconstruction is performed based on multi-resolution FBANK features to obtain a set of FBANK feature blocks;

[0045] Based on the dual attention mechanism, a graph attention network is used to dynamically aggregate features according to the FBANK feature block set to obtain the final attention features;

[0046] The final attention features are dimensionality reduced to obtain optimized spectrogram features.

[0047] In one feasible implementation, the present invention employs filterbank energy (Fbank) technology to discard complex phase information and retain only the log-Mel filterbank energy value, thereby achieving data dimensionality reduction and preservation of key phoneme features. Based on the Fbank, a discrete cosine transform (DCT) is performed to extract the first 12-13 dimensions of coefficients to compress the feature space and effectively characterize the formant characteristics of speech.

[0048] Multi-resolution FBANK features are extracted by modifying the overlap or spacing parameter (frame_shift) between adjacent frames. After cropping, padding, and merging, the resulting data dimension is (W, target_length, 128), where W represents the number of different resolutions, target_length is the number of frames set uniformly, and 128 is the number of filter banks. The extracted FBANK features are then patched, that is, the audio feature sequence is divided into multiple small blocks to better capture local features. After patching, the data dimension becomes (W, patch_num, 16, 16), where (16, 16) represents the size of each patch, and patch_num is the number of patches after partitioning.

[0049] This invention proposes a multi-scale dual attention (MSDA) mechanism based on a hierarchical attention weight generation strategy. MSDA aims to enhance the model's ability to model local details and global temporal dependencies in audio data through the synergistic effect of local enhancement layers and global association layers. The model uses a graph attention network (GAT) to model global topological relationships and combines a gated attention (GA) mechanism to dynamically fuse local and global features, ultimately generating discriminative attention weights.

[0050] The feature data obtained through the dual attention mechanism will be 16 at the same resolution. The 16-patch alignment and concatenation results in 128 data points. The attention across the W dimensions (different resolutions) is averaged, resulting in a data dimension of (target_length, 128). This method fuses information from multiple resolution features, reducing the vector to two dimensions. This step not only preserves the advantages of multi-resolution features but also simplifies the data structure, facilitating subsequent model processing.

[0051] Optionally, based on the dual attention mechanism, a graph attention network is used to dynamically aggregate features according to the FBANK feature block set to obtain the final attention features, including:

[0052] Based on the local attention mechanism, local attention features are obtained by weighting the features within the feature blocks according to the FBANK feature block set.

[0053] Based on the FBANK feature block set, a graph attention network is used to model the relationships between feature blocks to obtain global attention features.

[0054] Based on the gating attention mechanism, the final attention features are obtained by dynamically fusing local attention features and global attention features.

[0055] In one feasible implementation, this invention proposes a dual attention mechanism to model the temporal dependencies within and between feature blocks. This mechanism not only considers local contextual information but also introduces a graph attention network to model the topological relationships between blocks, significantly improving the model's ability to express complex temporal patterns.

[0056] Within each patch, a Local Attention mechanism is employed to enhance the modeling capability of adjacent time frames, capturing local details in the audio signal (such as short-time frequency band variations in speech). A sliding window strategy is introduced into the self-attention mechanism within each patch to limit the attention computation to adjacent frames. Time frames (e.g.) This avoids redundant computations in global attention. Let the input features be... (in For the number of patches, (where the feature dimension is used), then the local attention weights Calculate as follows (1):

[0057] (1);

[0058] in, , representing the unnormalized local attention coefficient; This represents the unnormalized local attention coefficients after introducing a sliding window strategy to limit the scope of attention calculation. Represents a node The local neighborhood (defined by the sliding window) is represented by LeakyReLU, which denotes the activation function.

[0059] Between patches, a Graph Attention Network (GAT) is used to model the topological relationships between patches. Each patch can be viewed as a node in a graph, with the node features being the feature vectors within the patch. Then construct the adjacency matrix. The correlation between patches is calculated using similarity (such as cosine similarity); finally, a graph attention mechanism is used to calculate the global attention weights. The calculation is as follows (2):

[0060] (2);

[0061] in, , representing the unnormalized global attention coefficient; Represents the unnormalized global attention coefficient of the m-th node; Represents a node The global neighborhood (composed of the adjacency matrix) definition).

[0062] The multi-head graph attention mechanism (Multi-headGAT) is used to run multiple independent attention heads in parallel to improve the robustness of the model. The process is as follows (3):

[0063] (3);

[0064] in, The number of attention heads.

[0065] In obtaining local attention weights and global attention weights Then, the corresponding features are calculated based on the FBANK feature block set. A gated attention mechanism is introduced to dynamically fuse the two, generating the final attention weights. This mechanism controls the fusion ratio of local and global information through gating units.

[0066] Local attention features With global attention features The splicing process is as follows (4):

[0067] (4);

[0068] Gated weights are generated using a multilayer perceptron (MLP) with learnable parameters. The process is as follows (5):

[0069] (5);

[0070] in, This represents the Sigmoid activation function; These are the learnable weights of the multilayer perceptron; This refers to the bias in a multilayer perceptron.

[0071] The local and global features are weighted and fused according to the gating weights, as shown in equation (6): (6);

[0072] This fusion approach enables the model to dynamically balance the influence of local details and global correlations based on the importance of input features, adapting to the needs of different audio processing tasks (such as speech recognition requiring more local details, and sentiment analysis requiring more global semantic information).

[0073] S3. Using optimized spectrogram features and sub-language features, the pre-trained auxiliary diagnostic model is fine-tuned to obtain an optimized auxiliary diagnostic model.

[0074] The auxiliary diagnostic model is the SSAST-Base-Frame-400 model; the auxiliary diagnostic model is pre-trained using the AudioSet dataset and the LibriSpeech dataset.

[0075] In one feasible implementation, the present invention uses a self-supervised audio spectrogram transformer (SSAST) model for fine-tuning classification. By utilizing a large amount of unlabeled audio data for self-supervised learning, it effectively alleviates the need for large-scale labeled data in traditional supervised methods, reduces data acquisition costs, and improves the generalization ability of the model.

[0076] The SSAST model is built on the standard Transformer architecture, consisting of an encoder and a decoder. The encoder comprises multiple stacked Transformer layers, each containing a multi-head self-attention mechanism and a feedforward neural network. SSAST introduces the Masked Spectrogram Patch Modeling (MSPM) task, which predicts randomly masked portions of the spectrogram. This forces the model to learn the time-frequency structure of audio segments, focusing not only on time-series prediction but also on frequency-dimensional dependencies, thus better capturing the essential features of the audio.

[0077] This invention examined the scores of different models on various speech tasks and ultimately selected the SSAST-Base-Frame-400 model trained on the AudioSet and LibriSpeech datasets. In the audio event classification task, masking 400 patches outperformed other settings, indicating that frame-based models are more suitable for speech tasks. "Base" represents the size of the model parameters; the Base model's Transformer encoder has 12 layers, each with 12 attention heads, an embedding dimension of 768, and 89 million parameters. It outperformed the small and tiny models in its respective tasks.

[0078] Fine-tuning employs a hierarchical learning rate approach, setting larger learning rates for the feature encoding layer and fully connected layers to quickly adjust their weights and better adapt to the needs of new tasks. The model itself is given a smaller learning rate to prevent the destruction of already learned useful features; these are typically general features with high transferability across multiple tasks. A learning rate warm-up adjustment method is used, employing a small learning rate in the first few epochs of training and gradually increasing it to the predetermined rate. This approach helps the model converge more stably in the early stages of training, is less prone to gradient explosion, helps the model find a good local optimum faster, and maintains high stability throughout training.

[0079] S4. Obtain the actual dialogue audio; extract audio and acoustic parameter features based on the actual dialogue audio to obtain actual multi-resolution spectrogram features and actual paralinguistic features; based on the multi-scale dual attention mechanism of the hierarchical attention weight generation strategy, use the optimized auxiliary diagnostic model to detect cognitive impairment based on the actual multi-resolution spectrogram features and actual paralinguistic features, and obtain auxiliary diagnostic results.

[0080] Optionally, based on a multi-scale dual attention mechanism using a hierarchical attention weight generation strategy, and according to actual multi-resolution spectrogram features and actual paralinguistic features, an optimized auxiliary diagnostic model is used to detect cognitive impairment and obtain auxiliary diagnostic results, including:

[0081] Based on a hierarchical attention weight generation strategy, a multi-scale dual attention mechanism is used to dynamically fuse features according to the actual multi-resolution spectrogram features to obtain optimized actual spectrogram features.

[0082] Based on the optimized actual spectrogram features and actual secondary language features, feature splicing is performed to obtain spliced ​​features;

[0083] Based on the splicing features, dimensionality reduction features are obtained by optimizing the linear projection layer of the auxiliary diagnostic model for dimensionality reduction mapping.

[0084] Based on the dimensionality reduction features, the position addition process is performed by optimizing the position embedding layer of the auxiliary diagnostic model to obtain the processed features;

[0085] Based on the processed features, semantic encoding is performed by optimizing the feature encoding layer of the auxiliary diagnostic model to obtain encoded features.

[0086] Based on the encoded features, cognitive impairment classification prediction is performed by optimizing the fully connected layer of the auxiliary diagnostic model to obtain auxiliary diagnostic results.

[0087] In one feasible implementation, the data processed by MSDA is input into a pre-trained model. A linear transformation is performed through a linear projection layer, followed by a positional embedding layer to add positional information, and then the data is fed into a feature encoding layer for encoding. In this way, the pre-trained model can better understand and model the time-series features in the audio data, improving the performance of audio processing tasks.

[0088] Through MSDA, the proposed model can more comprehensively capture the temporal dependencies and local details in audio data, providing richer feature representations for audio processing tasks.

[0089] This invention proposes an audio-based method for assisting the diagnosis of mild cognitive impairment. It employs a multi-scale dual attention mechanism based on a hierarchical attention weight generation strategy to capture key information at different time scales. This mechanism effectively enhances the model's ability to perceive subtle changes in speech signals, thereby improving classification performance and significantly increasing the model's accuracy and robustness in MCI classification tasks. Combining spectrogram features with traditional paralinguistic features produces a complementary effect, further enhancing the model's classification performance. Through the effective fusion of these two different types of features, the model can not only extract information from the time-frequency representation of audio signals but also utilize paralinguistic features to capture important clues such as the speaker's emotional state and health status, further improving classification accuracy. This invention provides an efficient and accurate audio-based method for assisting the diagnosis of mild cognitive impairment.

[0090] Figure 2 This is a block diagram illustrating an audio-based auxiliary diagnostic device for mild cognitive impairment according to an exemplary embodiment. The device is used in an audio-based auxiliary diagnostic method for mild cognitive impairment. (Refer to...) Figure 2 The device includes a feature extraction module 210, a spectrogram feature optimization module 220, a model fine-tuning module 230, and an auxiliary diagnostic result prediction module 240. Among them:

[0091] The feature extraction module 210 is used to acquire the original dialogue audio; and to extract audio and acoustic parameter features based on the original dialogue audio to obtain multi-resolution spectrogram features and paralinguistic features.

[0092] The spectrogram feature optimization module 220 is used for a multi-scale dual attention mechanism based on a hierarchical attention weight generation strategy to perform dynamic feature fusion based on multi-resolution spectrogram features and obtain optimized spectrogram features.

[0093] The model fine-tuning module 230 is used to optimize and fine-tune the pre-trained auxiliary diagnostic model using optimized spectrogram features and sub-language features to obtain an optimized auxiliary diagnostic model.

[0094] The auxiliary diagnostic result prediction module 240 is used to acquire actual dialogue audio; extract audio and acoustic parameter features based on the actual dialogue audio to obtain actual multi-resolution spectrogram features and actual paralinguistic features; and use an optimized auxiliary diagnostic model to detect cognitive impairment based on the actual multi-resolution spectrogram features and actual paralinguistic features, based on a multi-scale dual attention mechanism with a hierarchical attention weight generation strategy, to obtain auxiliary diagnostic results.

[0095] Optionally, the feature extraction module 210 is further used for:

[0096] The original dialogue audio is preprocessed to obtain the processed dialogue audio;

[0097] Based on the Fast Fourier Transform method, Mel spectrogram features are extracted from the processed dialogue audio to obtain multi-resolution spectrogram features.

[0098] The OpenSmile tool was used to perform multi-dimensional acoustic parameter feature analysis on the processed dialogue audio to obtain paralinguistic features.

[0099] Optionally, the spectrogram feature optimization module 220 is further used for:

[0100] Based on a multi-scale filter bank, feature analysis is performed according to multi-resolution spectral characteristics to obtain multi-resolution FBANK features.

[0101] Data reconstruction is performed based on multi-resolution FBANK features to obtain a set of FBANK feature blocks;

[0102] Based on the dual attention mechanism, a graph attention network is used to dynamically aggregate features according to the FBANK feature block set to obtain the final attention features;

[0103] The final attention features are dimensionality reduced to obtain optimized spectrogram features.

[0104] Optionally, the spectrogram feature optimization module 220 is further used for:

[0105] Based on the local attention mechanism, local attention features are obtained by weighting the features within the feature blocks according to the FBANK feature block set.

[0106] Based on the FBANK feature block set, a graph attention network is used to model the relationships between feature blocks to obtain global attention features.

[0107] Based on the gating attention mechanism, the final attention features are obtained by dynamically fusing local attention features and global attention features.

[0108] The auxiliary diagnostic model is the SSAST-Base-Frame-400 model; the auxiliary diagnostic model is pre-trained using the AudioSet dataset and the LibriSpeech dataset.

[0109] Optionally, the auxiliary diagnostic result prediction module 240 is further used for:

[0110] Based on a hierarchical attention weight generation strategy, a multi-scale dual attention mechanism is used to dynamically fuse features according to the actual multi-resolution spectrogram features to obtain optimized actual spectrogram features.

[0111] Based on the optimized actual spectrogram features and actual secondary language features, feature splicing is performed to obtain spliced ​​features;

[0112] Based on the splicing features, dimensionality reduction features are obtained by optimizing the linear projection layer of the auxiliary diagnostic model for dimensionality reduction mapping.

[0113] Based on the dimensionality reduction features, the position addition process is performed by optimizing the position embedding layer of the auxiliary diagnostic model to obtain the processed features;

[0114] Based on the processed features, semantic encoding is performed by optimizing the feature encoding layer of the auxiliary diagnostic model to obtain encoded features.

[0115] Based on the encoded features, cognitive impairment classification prediction is performed by optimizing the fully connected layer of the auxiliary diagnostic model to obtain auxiliary diagnostic results.

[0116] This invention proposes an audio-based method for assisting the diagnosis of mild cognitive impairment. It employs a multi-scale dual attention mechanism based on a hierarchical attention weight generation strategy to capture key information at different time scales. This mechanism effectively enhances the model's ability to perceive subtle changes in speech signals, thereby improving classification performance and significantly increasing the model's accuracy and robustness in MCI classification tasks. Combining spectrogram features with traditional paralinguistic features produces a complementary effect, further enhancing the model's classification performance. Through the effective fusion of these two different types of features, the model can not only extract information from the time-frequency representation of audio signals but also utilize paralinguistic features to capture important clues such as the speaker's emotional state and health status, further improving classification accuracy. This invention provides an efficient and accurate audio-based method for assisting the diagnosis of mild cognitive impairment.

[0117] Figure 3 This is a schematic diagram of the structure of an auxiliary diagnostic device for mild cognitive impairment provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the auxiliary diagnostic device for mild cognitive impairment may include the above-mentioned Figure 2 The illustrated audio-based diagnostic aid for mild cognitive impairment is shown. Optionally, the diagnostic aid for mild cognitive impairment 310 may include a first processor 2001.

[0118] Optionally, the mild cognitive impairment assistive diagnostic device 310 may also include a memory 2002 and a transceiver 2003.

[0119] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0120] The following is combined with Figure 3 A detailed description of each component of the mild cognitive impairment auxiliary diagnostic device 310 is provided below:

[0121] The first processor 2001 is the control center of the mild cognitive impairment auxiliary diagnostic device 310. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0122] Optionally, the first processor 2001 can perform various functions of the mild cognitive impairment auxiliary diagnostic device 310 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0123] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 3 CPU0 and CPU1 are shown in the diagram.

[0124] In a specific implementation, as one example, the mild cognitive impairment assistive diagnostic device 310 may also include multiple processors, such as... Figure 3 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0125] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0126] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the mild cognitive impairment assistive diagnostic device 310. Figure 3 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0127] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0128] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 3 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0129] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or it can exist independently and be connected to the interface circuit of the mild cognitive impairment auxiliary diagnostic device 310. Figure 3 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0130] It should be noted that, Figure 3 The structure of the mild cognitive impairment auxiliary diagnostic device 310 shown in the figure does not constitute a limitation on the router. Actual knowledge structure recognition devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0131] Furthermore, the technical effects of the mild cognitive impairment auxiliary diagnostic device 310 can be referred to the technical effects of the audio-based mild cognitive impairment auxiliary diagnostic method described in the above method embodiments, and will not be repeated here.

[0132] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0133] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0134] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0135] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0136] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0137] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0138] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0139] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0140] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0141] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0142] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0143] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0144] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An audio-based auxiliary diagnostic method for mild cognitive impairment, characterized in that, The method includes: Acquire the original dialogue audio; extract audio features and acoustic parameter features from the original dialogue audio to obtain multi-resolution spectrogram features and paralinguistic features; A multi-scale dual attention mechanism based on a hierarchical attention weight generation strategy dynamically fuses features from multi-resolution spectrogram features to obtain optimized spectrogram features, including: Based on the multi-scale filter bank, feature analysis is performed according to the multi-resolution spectrogram characteristics to obtain multi-resolution FBANK features. Data reconstruction is performed based on multi-resolution FBANK features to obtain a set of FBANK feature blocks; Based on the dual attention mechanism, a graph attention network is used to dynamically aggregate features according to the FBANK feature block set to obtain the final attention features; The final attention features are dimensionality-reduced to obtain optimized spectrogram features; The method based on the dual attention mechanism, which uses a graph attention network to dynamically aggregate features according to the FBANK feature block set to obtain the final attention features, includes: Based on the local attention mechanism, local attention features are obtained by weighting the features within the feature blocks according to the FBANK feature block set. Based on the FBANK feature block set, a graph attention network is used to model the relationships between feature blocks to obtain global attention features. Based on the gating attention mechanism, the final attention features are obtained by dynamically fusing local attention features and global attention features. The pre-trained auxiliary diagnostic model is optimized and fine-tuned using optimized spectrogram features and sub-language features to obtain an optimized auxiliary diagnostic model. Acquire actual dialogue audio; extract audio features and acoustic parameter features based on the actual dialogue audio to obtain actual multi-resolution spectrogram features and actual paralinguistic features; based on a multi-scale dual attention mechanism with a hierarchical attention weight generation strategy, use an optimized auxiliary diagnostic model to detect cognitive impairment based on the actual multi-resolution spectrogram features and actual paralinguistic features to obtain auxiliary diagnostic results.

2. The audio-based auxiliary diagnostic method for mild cognitive impairment according to claim 1, characterized in that, The step of extracting audio features and acoustic parameter features from the original dialogue audio to obtain multi-resolution spectrogram features and paralinguistic features includes: The original dialogue audio is preprocessed to obtain the processed dialogue audio; Based on the Fast Fourier Transform method, Mel spectrogram features are extracted from the processed dialogue audio to obtain multi-resolution spectrogram features. The OpenSmile tool was used to perform multi-dimensional acoustic parameter feature analysis on the processed dialogue audio to obtain paralinguistic features.

3. The audio-based auxiliary diagnostic method for mild cognitive impairment according to claim 1, characterized in that, The auxiliary diagnostic model is the SSAST-Base-Frame-400 model; the auxiliary diagnostic model is pre-trained using the AudioSet dataset and the LibriSpeech dataset.

4. The audio-based auxiliary diagnostic method for mild cognitive impairment according to claim 1, characterized in that, The multi-scale dual attention mechanism based on a hierarchical attention weight generation strategy uses an optimized auxiliary diagnostic model to detect cognitive impairment based on actual multi-resolution spectrogram features and actual paralinguistic features, obtaining auxiliary diagnostic results, including: Based on a hierarchical attention weight generation strategy, a multi-scale dual attention mechanism is used to dynamically fuse features according to the actual multi-resolution spectrogram features to obtain optimized actual spectrogram features. Based on the optimized actual spectrogram features and actual secondary language features, feature splicing is performed to obtain spliced ​​features; Based on the splicing features, dimensionality reduction features are obtained by optimizing the linear projection layer of the auxiliary diagnostic model for dimensionality reduction mapping. Based on the dimensionality reduction features, the position addition process is performed by optimizing the position embedding layer of the auxiliary diagnostic model to obtain the processed features; Based on the processed features, semantic encoding is performed by optimizing the feature encoding layer of the auxiliary diagnostic model to obtain encoded features. Based on the encoded features, cognitive impairment classification prediction is performed by optimizing the fully connected layer of the auxiliary diagnostic model to obtain auxiliary diagnostic results.

5. An audio-based auxiliary diagnostic device for mild cognitive impairment, used to implement the audio-based auxiliary diagnostic method for mild cognitive impairment as described in any one of claims 1-4, characterized in that, The device includes: The feature extraction module is used to acquire the original dialogue audio; based on the original dialogue audio, audio features and acoustic parameter features are extracted to obtain multi-resolution spectrogram features and paralinguistic features; The spectrogram feature optimization module is used for a multi-scale dual attention mechanism based on a hierarchical attention weight generation strategy. It performs dynamic feature fusion based on multi-resolution spectrogram features to obtain optimized spectrogram features. The model fine-tuning module is used to optimize and fine-tune the pre-trained auxiliary diagnostic model using optimized spectrogram features and sub-language features to obtain an optimized auxiliary diagnostic model. The auxiliary diagnostic result prediction module is used to acquire actual dialogue audio; extract audio features and acoustic parameter features based on the actual dialogue audio to obtain actual multi-resolution spectrogram features and actual paralinguistic features; and use an optimized auxiliary diagnostic model to detect cognitive impairment based on the actual multi-resolution spectrogram features and actual paralinguistic features, based on a multi-scale dual attention mechanism with a hierarchical attention weight generation strategy, to obtain auxiliary diagnostic results.

6. A diagnostic aid for mild cognitive impairment, characterized in that, The mild cognitive impairment auxiliary diagnostic device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 4.

Images