Snn-based post-stroke dysarthria pathological speech recognition method and device

Abstract

The application discloses a post-stroke dysarthria pathological speech recognition method and device based on SNN, and aims to solve the technical problem that the existing dysarthria pathological speech recognition method based on a deep learning algorithm causes excessive power consumption in the model operation process. The method comprises the following steps: obtaining a plurality of initial audio data of a patient to be detected, and respectively pre-processing each initial audio data to output a mel-frequency cepstral coefficient feature corresponding to each initial audio data; inputting each mel-frequency cepstral coefficient feature into a pulse neural network that fuses word attention and channel attention for recognition, and outputting a plurality of syllable category disease probability; and generating a post-stroke dysarthria pathological speech recognition result corresponding to the patient to be detected according to the plurality of syllable category disease probability.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and device for recognizing pathological speech of post-stroke dysarthria based on SNN. Background Art

[0002] In modern society, the incidence of stroke is increasing year by year. Post-Stroke Dysarthria (PSD) is a common sequelae of stroke that can lead to a decrease in voice quality and even inability to communicate normally, seriously affecting patients' quality of life. Traditionally, the diagnosis of dysarthria is primarily based on a series of procedures, including patient interviews, invasive surgical examinations using instruments such as laryngoscopy and endoscopy, acoustic analysis, and subjective speech assessment by doctors. These methods are time-consuming and cumbersome.

[0003] In recent years, the development of deep learning algorithms has led to increasingly superior performance in image and speech classification algorithms. Consequently, some have proposed using these algorithms to assist physicians in identifying speech pathologies such as dysarthria. This approach could enable non-invasive, cost-effective, and unbiased testing, and rapidly assess speech pathology.

[0004] Existing methods for recognizing dysarthric pathological speech based on deep learning algorithms mostly use deep CNN models or large Transformer models to complete the recognition of dysarthric pathological speech. However, these models all belong to artificial neural networks. Whether in the training stage or the inference stage, artificial neural networks require a large amount of computing resources to support their complex neuronal connections and calculation operations, resulting in excessive power consumption in the model calculation process. Summary of the Invention

[0005] The present invention provides a method and device for post-stroke dysarthria pathological speech recognition based on SNN, which is used to solve the technical problem that the existing dysarthria pathological speech recognition method based on deep learning algorithm causes excessive power consumption in the model calculation process.

[0006] The first aspect of the present invention provides a method for recognizing pathological speech of post-stroke dysarthria based on SNN, comprising:

[0007] Acquire multiple initial audio data of the patient to be tested, preprocess each of the initial audio data, and output Mel-frequency cepstral coefficient features corresponding to each of the initial audio data;

[0008] Inputting each of the Mel-frequency cepstral coefficient features into a spiking neural network that integrates word attention and channel attention for recognition, and outputting disease probabilities for multiple syllable categories;

[0009] According to the disease probabilities of the multiple syllable categories, a pathological speech recognition result of post-stroke dysarthria corresponding to the patient to be detected is generated.

[0010] Optionally, preprocessing each of the initial audio data and outputting a Mel-frequency cepstral coefficient feature corresponding to each of the initial audio data includes:

[0011] Pre-emphasize each of the initial audio data, and output the pre-emphasized audio data corresponding to each of the initial audio data;

[0012] respectively dividing the pre-emphasized audio data into frames to generate a plurality of audio frame data;

[0013] Performing windowing on each of the audio frame data, and outputting window audio data corresponding to each of the audio frame data;

[0014] Performing a fast Fourier transform on the audio data in each window, and outputting a target spectrum corresponding to the audio data in each window;

[0015] Performing filtering operations on each of the target spectra to generate a Mel spectrum corresponding to each of the target spectra;

[0016] Perform logarithmic operation and discrete cosine transform on each of the mel spectrums, and output mel cepstral coefficient features corresponding to each of the mel spectrums.

[0017] Optionally, the spiking neural network integrating word attention and channel attention includes an embedding module, a spiking self-attention module, and a linear output layer; the inputting of each of the Mel-frequency cepstral coefficient features into the spiking neural network integrating word attention and channel attention for recognition, and outputting disease probabilities of multiple syllable categories, includes:

[0018] Using an embedding module to perform enhanced feature extraction on each of the Mel-frequency cepstral coefficient features, and generating a target pulse nonlinear feature corresponding to each of the Mel-frequency cepstral coefficient features;

[0019] Using each of the target pulse nonlinear features as the input of the pulse self-attention module, and outputting the target attention matrix corresponding to each of the target pulse nonlinear features;

[0020] Each of the target attention matrices is used as the input of the linear output layer, and the disease probability of the syllable category corresponding to each of the target attention matrices is output.

[0021] Optionally, the embedding module comprises a first convolutional layer, a first batch normalization layer, a first leaky pulse neuron, a second convolutional layer, a second batch normalization layer, and a second leaky pulse neuron; the embedding module is used to perform strong feature extraction on each of the mel-frequency cepstral coefficient features to generate target pulse nonlinear features corresponding to each of the mel-frequency cepstral coefficient features, comprising:

[0022] The first convolutional layer is used to perform convolutional operation on each of the mel-frequency cepstral coefficient features to output first convolutional features corresponding to each of the mel-frequency cepstral coefficient features;

[0023] The first batch normalization layer is used to normalize each of the first convolutional features to generate first normalized features corresponding to each of the first convolutional features;

[0024] Each of the first normalized features is taken as an input of the second leaky pulse neuron to output initial pulse nonlinear features corresponding to each of the first normalized features;

[0025] The second convolutional layer is used to perform convolutional operation on each of the initial pulse nonlinear features to output second convolutional features corresponding to each of the initial pulse nonlinear features;

[0026] The second batch normalization layer is used to normalize each of the second convolutional features to generate second normalized features corresponding to each of the second convolutional features;

[0027] Each of the second normalized features is taken as an input of the second leaky pulse neuron to output target pulse nonlinear features corresponding to each of the second normalized features.

[0028] Optionally, the pulse self-attention module comprises a word attention layer, a channel attention layer, and a multi-layer perception layer; the pulse self-attention module is used to take each of the target pulse nonlinear features as an input to output a target attention matrix corresponding to each of the target pulse nonlinear features, comprising:

[0029] Each of the target pulse nonlinear features is taken as an input of the word attention layer to output a word attention matrix corresponding to each of the pulse nonlinear features;

[0030] Each of the target pulse nonlinear features is taken as an input of the channel attention layer to output a channel attention matrix corresponding to each of the pulse nonlinear features;

[0031] Each of the target pulse nonlinear features, the word attention matrix corresponding to each of the target pulse nonlinear features, and the channel attention matrix corresponding to each of the target pulse nonlinear features is summed and superimposed to output an attention residual matrix corresponding to each of the target pulse nonlinear features;

[0032] The attention residual matrices are respectively taken as inputs of multilayer perceptron layers, and the attention residual matrices correspondingly output perceptual pulse nonlinear features;

[0033] The attention residual matrices and the perceptual pulse nonlinear features are respectively summed and superimposed, and target attention matrices corresponding to the target pulse nonlinear features are output.

[0034] Optionally, the generating of the post-stroke dysarthria pathological speech recognition result corresponding to the to-be-detected patient according to the plurality of syllable category disease probabilities comprises:

[0035] The plurality of syllable category disease probabilities are subjected to mean operation, and a disease probability mean value is output.

[0036] Based on a comparison result of the disease probability mean value and a preset probability threshold, the post-stroke dysarthria pathological speech recognition result corresponding to the to-be-detected patient is generated.

[0037] Optionally, the model training process of the pulse neural network fusing word attention and channel attention comprises:

[0038] An audio data training set is acquired, and the audio data training set is preprocessed, and a plurality of to-be-trained mel-frequency cepstrum coefficient features are output.

[0039] Each to-be-trained mel-frequency cepstrum coefficient feature is input to an initial pulse neural network fusing word attention and channel attention for recognition, and a plurality of to-be-trained syllable category disease probabilities are output.

[0040] The plurality of to-be-trained syllable category disease probabilities are subjected to mean calculation to determine a to-be-trained disease probability mean value.

[0041] A target loss value is calculated according to the to-be-trained disease probability mean value.

[0042] Based on the target loss value, model parameters of the initial pulse neural network fusing word attention and channel attention are updated to determine an intermediate pulse neural network fusing word attention and channel attention, and an iteration number is counted in real time.

[0043] It is judged whether the iteration number reaches a preset iteration threshold.

[0044] If yes, the intermediate pulse neural network fusing word attention and channel attention is taken as the trained pulse neural network fusing word attention and channel attention.

[0045] The second aspect of the present application provides a post-stroke dysarthria pathological speech recognition device based on SNN, comprising:

[0046] An acquisition module is used to acquire multiple initial audio data of the patient to be tested, pre-process each of the initial audio data, and output Mel-frequency cepstral coefficient features corresponding to each of the initial audio data;

[0047] A recognition module is used to input each of the Mel-frequency cepstral coefficient features into a spiking neural network that integrates word attention and channel attention for recognition, and output disease probabilities for multiple syllable categories;

[0048] The prediction module is used to generate a pathological speech recognition result of post-stroke dysarthria corresponding to the patient to be detected based on the disease probabilities of the multiple syllable categories.

[0049] The third aspect of the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the SNN-based post-stroke dysarthria pathological speech recognition method as described in any one of the above items.

[0050] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the steps of the SNN-based post-stroke dysarthria pathological speech recognition method as described in any one of the above.

[0051] It can be seen from the above technical solutions that the present invention has the following advantages:

[0052] The above technical solution of the present invention provides a pathological speech recognition method for post-stroke dysarthria based on SNN. First, multiple initial audio data of the patient to be tested are obtained, and each initial audio data is preprocessed respectively, and the Mel-frequency cepstral coefficient feature corresponding to each initial audio data is output; then, each Mel-frequency cepstral coefficient feature is input into a pulse neural network that integrates word attention and channel attention for recognition, and the disease probabilities of multiple syllable categories are output; finally, according to the disease probabilities of multiple syllable categories, the pathological speech recognition result of post-stroke dysarthria corresponding to the patient to be tested is generated; based on the above solution, after preprocessing the multiple initial audio data obtained, the pulse neural network that integrates word attention and channel attention is used to recognize the multiple Mel-frequency cepstral coefficient features obtained, and the pathological speech recognition result of post-stroke dysarthria is output. The present invention combines word attention and channel attention on the basis of the pulse neural network, which can reduce the computational complexity and parameters of the model, thereby reducing the power consumption of the model calculation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 A flowchart of a method for recognizing pathological speech of post-stroke dysarthria based on SNN provided in Example 1 of the present invention;

[0055] Figure 2 A data diagram of syllables corresponding to the audio data provided in the first embodiment of the present invention;

[0056] Figure 3 This is a diagram showing the framework of the SNN-based TCAFormer network provided in Example 1 of the present invention;

[0057] Figure 4 A flowchart of the steps for model training of a spiking neural network that integrates word attention and channel attention, provided in Example 2 of the present invention;

[0058] Figure 5 A diagram showing the framework structure of a pulse neural network model training that integrates word attention and channel attention, as provided in the second embodiment of the present invention;

[0059] Figure 6 The present invention provides a structural block diagram of a post-stroke dysarthria pathological speech recognition device based on SNN. DETAILED DESCRIPTION

[0060] The embodiments of the present invention provide a method and device for post-stroke dysarthria pathological speech recognition based on SNN, which is used to solve the technical problem that the existing dysarthria pathological speech recognition method based on deep learning algorithm causes excessive power consumption in the model calculation process.

[0061] In order to make the purpose, features, and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0062] Explanation of terms:

[0063] PSD: post-stroke dysarthria.

[0064] SNN: Spiking Neural Network.

[0065] LIF: Leaky Integrate-and-Fire.

[0066] MFCC: Mel-Frequency Cepstral Coefficients.

[0067] See also Figure 1 , Figure 1 This is a flowchart of the steps of a SNN-based post-stroke dysarthria pathological speech recognition method provided in Example 1 of the present invention.

[0068] The present invention provides a SNN-based post-stroke dysarthria pathological speech recognition method, comprising:

[0069] Step 101: Acquire multiple initial audio data of a patient to be tested, pre-process each initial audio data, and output Mel-frequency cepstral coefficient features corresponding to each initial audio data.

[0070] Please note that Figure 2 The patient to be tested has multiple syllables, and each syllable corresponds to an initial audio data. The present invention obtains multiple initial audio data of the patient to be tested from the Chinese Syllable PSD pathological speech database collected by the First Affiliated Hospital of Jinan University. That is, the pathological speech data (audio data) were collected by the Speech Therapy Center of the Affiliated Hospital of Jinan University. All participants were Mandarin-speaking adults, including 17 ordinary males, 23 ordinary females, 38 male patients, and 22 female patients, as shown in Table 1. The age range of the people participating in the collection was between 25 and 83 years old [mean ± standard deviation: 56.74 ± 16.40 years old]. They all underwent physical examination, Frenche dysarthria assessment, and other auxiliary examinations (such as brain CT and MRI).

[0071] Before developing dysarthria, all patients were able to communicate fluently in Mandarin, without visual or severe auditory comprehension impairment. All audio data were collected at a sampling rate of 16K and saved in .wav format. The collected pronunciation syllables covered all Mandarin vowel syllables, consonant syllables, and intonation changes. Figure 2As shown, some test subjects have missing syllables. Table 1 shows the proportion of participants in the dataset. Taking into account the gender of the patients, the dataset is divided into a training set and a test set at a ratio of 4:1. The test set contains all the syllables of 20 people for the purpose of subsequent experiments. The training set (audio data training set) is used to train the pulse neural network that integrates word attention and channel attention proposed in the present invention, and then the audio data in the test set (a plurality of initial audio data of the patient to be tested) is input into the trained pulse neural network that integrates word attention and channel attention to obtain the probability that all syllable categories of the patient to be tested belong to pathological speech. The probability that the speaker is sick (mean probability of illness) is finally obtained by calculating the average of all syllable probabilities.

[0072] Table 1: Information of participants

[0073]

[0074] Furthermore, the process of pre-processing each of the initial audio data and outputting the Mel-frequency cepstral coefficient features corresponding to each of the initial audio data can be achieved by executing the following sub-steps S11 to S16:

[0075] Step S11, pre-emphasize each initial audio data, and output the pre-emphasized audio data corresponding to each initial audio data;

[0076] Step S12: framing each pre-emphasized audio data to generate a plurality of audio frame data;

[0077] Step S13: windowing each audio frame data, and outputting the window audio data corresponding to each audio frame data;

[0078] Step S14: performing fast Fourier transform on the audio data in each window, and outputting the target spectrum corresponding to the audio data in each window;

[0079] Step S15: Perform filtering operations on each target spectrum to generate a Mel spectrum corresponding to each target spectrum;

[0080] Step S16: perform logarithmic operation and discrete cosine transform on each Mel spectrum, and output the Mel cepstral coefficient feature corresponding to each Mel spectrum.

[0081] It should be noted that the present invention obtains multiple windows of audio data by sequentially pre-emphasizing, framing, and windowing the speech signal (initial audio data). Then, 24-dimensional MFCCs (Mel-frequency Cepstral Coefficients) features are extracted from the obtained windowed audio data. The present invention sets the window length to 400 and the frame shift to 160. To ensure consistent time duration for each audio signal, the present invention sets the number of time windows to 120. Time windows shorter than 120 are padded with zeros, and those longer than 120 are truncated.

[0082] Furthermore, the present invention performs a fast Fourier transform on each time window (windowed audio data) to extract a spectrum (target spectrum). This spectrum is further extracted using a Mel filter array to obtain a Mel spectrum. The Mel spectrum is then subjected to logarithmic operations and discrete cosine transforms to ultimately produce Mel-frequency cepstral coefficient features (MFCCs). The dimension of the Mel-frequency cepstral coefficients is the number of Mel filters. Mel-frequency is a nonlinear frequency scale determined by the human ear's sensory perception of equally spaced pitch changes and is highly effective for pathological speech classification.

[0083] In this embodiment, a plurality of initial audio data of a patient to be tested is obtained, and each initial audio data is preprocessed respectively, and a Mel-frequency cepstral coefficient feature corresponding to each initial audio data is output.

[0084] Step 102: Input each Mel-frequency cepstral coefficient feature into a pulse neural network that integrates word attention and channel attention for recognition, and output disease probabilities of multiple syllable categories.

[0085] The spiking neural network that integrates word attention and channel attention includes an embedding module, a spiking self-attention module, and a linear output layer.

[0086] Please note that Figure 3The spiking neural network that integrates word attention and channel attention is a TCAFormer (Transformer-based Cross-Attention Former) network based on SNN, which can complete the task of pathological speech recognition. Since SNN is an event-driven neural network, discrete binarization of floating-point data will lead to the loss of effective features. Therefore, for the audio dataset used in the present invention, it is necessary to repeat the feature map for T time steps to ensure the acquisition of model information. The present invention sets T to 4, that is, repeating the input feature map 4 times; wherein, the spiking neural network that integrates word attention and channel attention consists of an embedding module, a spiking self-attention module and a linear output layer; the embedding module consists of two convolutional submodules (convolutional submodule 1 and convolutional submodule 2), and the convolutional submodule consists of a convolutional layer, a batch normalization layer (BN layer, Batch Normalization layer), and a LIF neuron; the spiking self-attention module consists of a word attention layer, a channel attention layer and a multilayer perceptron layer (MLP layer, Multilayer Perceptron).

[0087] Specifically, step 102 may include the following sub-steps S21-S23:

[0088] Step S21: using an embedding module to perform enhanced feature extraction on each Mel-frequency cepstral coefficient feature to generate a target pulse nonlinear feature corresponding to each Mel-frequency cepstral coefficient feature;

[0089] The embedding module includes a first convolutional layer, a first batch of normalization layers, a first leakage spike neuron, a second convolutional layer, a second batch of normalization layers, and a second leakage spike neuron.

[0090] Furthermore, step S31 may include the following sub-steps S211-S216:

[0091] Step S211: Use the first convolution layer to perform convolution operations on each Mel-frequency cepstral coefficient feature, and output the first convolution feature corresponding to each Mel-frequency cepstral coefficient feature;

[0092] Step S212: normalize each first convolution feature through the first batch of normalization layers to generate a first normalized feature corresponding to each first convolution feature;

[0093] Step S213: using each first normalized feature as an input to a second leaky spike neuron, and outputting an initial spike nonlinear feature corresponding to each first normalized feature;

[0094] Step S214: using the second convolution layer to perform convolution operations on each initial pulse nonlinear feature, and outputting a second convolution feature corresponding to each initial pulse nonlinear feature;

[0095] Step S215: normalize each second convolutional feature through a second batch of normalization layers to generate a second normalized feature corresponding to each second convolutional feature;

[0096] Step S216: Use each second normalized feature as an input of a second leakage pulse neuron, and output a target pulse nonlinear feature corresponding to each second normalized feature.

[0097] It should be noted that for the embedding module, since the data of the present invention is speech data with temporal sequence, in the convolution submodule 1, the present invention modifies the size of the input convolution kernel to (24, 3), where the width is the number of MFCC dimensions. While keeping the number of time windows unchanged, the features of each time window are upgraded to extract features, and then the features are discretized and encoded through LIF neurons. In the convolution submodule 2, the convolution kernel is modified to one dimension to adapt to the shape of the data and further extract local features. The data processing process of the two convolution submodules of the embedding module can be expressed as:

[0098] ;

[0099] Among them, P is the output of the convolution submodule; X is the input of the convolution submodule; LIF stands for LIF neuron, BN stands for batch normalization, and Conv stands for convolutional layer.

[0100] Step S22: using the nonlinear features of each target pulse as input to the pulse self-attention module, and outputting the target attention matrix corresponding to each target pulse nonlinear feature;

[0101] The pulse self-attention module includes a word attention layer, a channel attention layer, and a multi-layer perceptron layer.

[0102] It should be noted that for the pulse self-attention module, the present invention designs a word attention layer and a channel attention layer to better grasp the global features, and inputs them into the MLP layer through residual connection to fully extract global and local features.

[0103] It is worth mentioning that compared with the self-attention layer of the existing Transformer model, whose computational complexity is proportional to the square of the input length, the computational complexity of the pulse self-attention module proposed in this invention is proportional to the length of the input and the embedding dimension, which greatly reduces the number of parameters, helps to speed up the calculation speed and reduce the hardware load.

[0104] Specifically, step S22 may include the following sub-steps S221-S22:

[0105] Step S221: Use each target pulse nonlinear feature as the input of the word attention layer, and output the word attention matrix corresponding to each pulse nonlinear feature;

[0106] Specifically, in the word attention layer, the linear matrix learned by the network is first used to generate two matrices, Q and K. That is, in the word attention layer, the query matrix Q and key matrix K corresponding to each target pulse nonlinear feature are generated according to the nonlinear features of each target pulse. The sizes of the two matrices Q and K are consistent with the input P. The processing process of the query matrix and key matrix can be expressed as:

[0107] ;

[0108] Where Q is the query matrix; P is the output of the convolution submodule; K is the key matrix; is the weight corresponding to the nonlinear feature of the target pulse in the query matrix; is the weight corresponding to the nonlinear feature of the target pulse in the key matrix; is the LIF neuron corresponding to the query matrix; is the LIF neuron corresponding to the key matrix.

[0109] Furthermore, in the word attention layer, the present invention adds the matrix Q row by row to form a row of features, and after processing by the LIF neuron, multiplies it with the matrix K to obtain the word attention matrix TA. The processing process can be expressed as:

[0110] ;

[0111] Among them, TA is the word attention matrix; D is the number of embedding dimensions; N is the number of words; is the value of row i and column j in the query matrix; K is the key matrix; is matrix multiplication.

[0112] Step S222: Use the nonlinear features of each target pulse as the input of the channel attention layer, and output the channel attention matrix corresponding to each pulse nonlinear feature;

[0113] It should be noted that since word attention and channel attention are similar, both generate two matrices Q and K through linear matrices learned by the network, that is, in the channel attention layer, according to the nonlinear features of each target pulse, the query matrix Q and key matrix K corresponding to each target pulse nonlinear feature are generated. Among them, the step principle of generating the query matrix Q and key matrix K corresponding to each target pulse nonlinear feature in the channel attention layer is consistent with the step principle of generating the query matrix Q and key matrix K corresponding to each target pulse nonlinear feature in the word attention layer, and the present invention will not go into details.

[0114] It is worth mentioning that the query matrix Q and key matrix K generated in the channel attention layer are different from those generated in the word attention layer.

[0115] Furthermore, in the channel attention, the present invention adds the matrix Q column by column to form a column of features, and multiplies it with the matrix K after LIF neuron processing to obtain the channel attention matrix CA; wherein, the processing process of the word attention matrix and the channel attention matrix can be expressed as:

[0116] ;

[0117] Among them, TA is the word attention matrix; D is the number of embedding dimensions; N is the number of words; is the value of row i and column j in the query matrix; K is the key matrix; is matrix multiplication.

[0118] Step S223: summing and superimposing each target pulse nonlinear feature and the word attention matrix and channel attention matrix corresponding to each target pulse nonlinear feature, and outputting the attention residual matrix corresponding to each target pulse nonlinear feature;

[0119] Step S224: using each attention residual matrix as the input of a multi-layer perceptron layer, and outputting the perceptual impulse nonlinear features corresponding to each attention residual matrix;

[0120] Step S225: sum and superimpose each attention residual matrix and the perception pulse nonlinear features corresponding to each attention residual matrix, and output the target attention matrix corresponding to each target pulse nonlinear feature.

[0121] In this embodiment, each Mel-frequency cepstral coefficient feature is input into a spiking neural network that integrates word attention and channel attention for recognition, and the disease probabilities of multiple syllable categories are output.

[0122] Step 103: Generate a post-stroke dysarthria pathological speech recognition result corresponding to the patient to be tested based on the disease probabilities of the multiple syllable categories.

[0123] Specifically, step 103 may include the following sub-steps S31-S32:

[0124] S31, performing mean calculation on the disease probabilities of multiple syllable categories and outputting the mean disease probability;

[0125] S32. Based on the comparison result of the disease probability mean and the preset probability threshold, generate the post-stroke dysarthria pathological speech recognition result corresponding to the patient to be detected.

[0126] It should be noted that the present invention averages the time step T, and operates the attention matrix through the linear output layer to obtain the output of the model, and uses it as the probability that the syllable is uttered by the patient. By performing a mean operation on the disease probabilities of all syllable categories, the mean disease probability is output, and the mean disease probability is compared with the preset probability threshold. If the mean disease probability is greater than or equal to the preset probability threshold, the pathological speech recognition result of post-stroke dysarthria is that the patient to be detected is ill. If the mean disease probability is less than the preset probability threshold, the pathological speech recognition result of post-stroke dysarthria is that the patient to be detected is not ill. The preset probability threshold is set as needed, and the present invention is not limited to this.

[0127] In this embodiment, a pathological speech recognition result of post-stroke dysarthria corresponding to the patient to be detected is generated based on the disease probabilities of multiple syllable categories.

[0128] For comparison purposes, existing technologies can be used as a reference. Speech signals are one-dimensional time series signals that contain a wealth of information (including both highly influential and redundant interference). Direct classification is therefore inefficient. Preprocessing the speech signal to extract information is generally performed before classification. Preprocessing often extracts features from the frequency domain, with Mel-Frequency Cepstral Coefficients (MFCCs) and spectrograms being the most commonly used. Other researchers have also employed nonlinear features, multimedia audio description (MPEG-7) features, and MFCC features combined with fundamental frequency, spectral centroid, and prosodic features. With technological advancements, several large end-to-end speech model algorithms have emerged. However, these algorithms require massive datasets and extremely large model parameters. Therefore, particularly in the field of pathological speech, datasets are typically small due to the limited number of experimental subjects and privacy concerns. Therefore, most current research focuses on preprocessing the speech signal to extract features useful for speech classification systems. These features are then processed using deep learning algorithms.

[0129] With the advancement of computer technology, faster computing speeds, and the increase in dedicated high-speed computing memory, neural network models have continuously evolved toward deeper layers and more complex structures. From the initial RNN and CNN models to deep CNN models like ResNet, and finally to the large Transformer model proposed in 2017, their applications have expanded from natural language processing to various fields such as image processing. These well-known algorithms are artificial neural networks. However, the calculation of model parameters in artificial neural networks is essentially floating-point arithmetic, which results in significant power consumption for both training and inference. Spiking neural networks (SNNs), considered the third generation of neural networks and inspired by the brain, are a potential competitor to ANNs. Their event-driven nature makes them highly biologically plausible and energy-efficient. SNNs, built on the characteristics of SNNs and combined with advanced ANN architectures, offer tremendous potential for developing high-performance, low-energy algorithms.

[0130] At the core of an SNN is a spiking neuron that receives input signals and accumulates membrane potential at each time step. When the membrane potential reaches the firing threshold, the neuron emits a spike signal. After emitting the spike signal, the membrane potential returns to its resting potential. If the membrane potential does not reach the threshold, the neuron will not emit a spike signal but will continue to accumulate, and the membrane potential will leak over time. Therefore, the output of a spiking neuron can be considered a binary output. Compared to the floating-point multiplication operations in an ANN network, the multiplication operation performed by a spiking neuron is equivalent to a logical AND operation, which is much more efficient.

[0131] At present, the main research subjects of international research on articulation disorders are people who speak English, French, German, Swedish and other languages. In comparison, there are relatively few studies on articulation disorders in Mandarin Chinese.

[0132] Based on this foundation, most existing technologies still use shallow CNN models, without further research into deeper models and the role of different network topologies. These models cannot effectively handle the inherent dependencies between features at different time steps, resulting in poor practical application results. Furthermore, most existing technologies still rely on ANNs, which consume a lot of power and are unsuitable for deployment on edge devices for inference.

[0133] In response to the above problems, the present invention proposes a pathological speech recognition method for post-stroke dysarthria based on SNN. Specifically, first, by converting the two-dimensional feature map into a one-dimensional feature map in the embedding layer, it helps to reduce the number of model parameters, and makes the shapes of most subsequent convolution kernels consistent, which is conducive to hardware-level implementation. Secondly, the combination of word attention and channel attention in the network replaces the self-attention in the Transformer model to reduce the computational complexity and parameters of the model. In addition, multiple attention mechanisms are used to capture and fuse global and local features, effectively improving the accuracy of the model. The present invention builds a TCAFormer SNN model (a pulse neural network that integrates word attention and channel attention). Thanks to the low power consumption characteristics of the SNN model framework, the network helps to deploy applications on edge devices, thereby expanding application scenarios. In addition, the pulse neural network that integrates word attention and channel attention proposed in the present invention is a deep network that uses multiple self-attentions for fusion to fully extract features of different time periods on the speech signal, thereby better improving the performance of pathological speech classification tasks.

[0134] Compared with traditional methods, the present invention not only has higher accuracy but also greatly reduces power consumption. This is mainly due to the fusion of multiple attention mechanisms of the TCAFormer network, which can comprehensively capture the characteristics of speech. In addition, thanks to the SNN network framework, the power consumption of this network is greatly reduced compared to traditional ANN networks, and it has a wider range of application scenarios. Moreover, experiments have found that this method also has very good effects when applied to the classification of spoken commands with certain background noise in datasets such as SpeechCommands. This method can be applied to more tasks related to speech classification, and can be deployed on edge devices for application in a wider range of scenarios.

[0135] In summary, the pulse neural network that integrates word attention and channel attention proposed in the present invention can be extended to different speech classification tasks. In addition, the low power consumption characteristics of the pulse neural network help to deploy the model to embedded devices and apply it to a wider range of fields; at the same time, the present invention starts from the 24-dimensional MFCCs time-frequency characteristics of the speech signal and captures the dependency of the features of the speech signal in different time windows by integrating the pulse neural network that integrates word attention and channel attention. The core of the pulse neural network that integrates word attention and channel attention is to introduce word attention and channel attention at the same time to jointly extract features. Experiments have shown that the model can fully extract the features of the audio and achieve a high accuracy rate in classification tasks.

[0136] In an embodiment of the present invention, the present invention provides a post-stroke dysarthria pathological speech recognition method based on SNN. First, multiple initial audio data of a patient to be tested are obtained, and each initial audio data is preprocessed separately, and the Mel-frequency cepstral coefficient feature corresponding to each initial audio data is output; then, each Mel-frequency cepstral coefficient feature is input into a pulse neural network that integrates word attention and channel attention for recognition, and the disease probabilities of multiple syllable categories are output; finally, according to the disease probabilities of multiple syllable categories, the post-stroke dysarthria pathological speech recognition result corresponding to the patient to be tested is generated; based on the above scheme, after preprocessing the multiple initial audio data obtained, the pulse neural network that integrates word attention and channel attention is used to recognize the multiple Mel-frequency cepstral coefficient features obtained, and the post-stroke dysarthria pathological speech recognition result is output. The present invention combines word attention and channel attention on the basis of the pulse neural network, which can reduce the computational complexity and parameters of the model, thereby reducing the power consumption of the model calculation process.

[0137] For better explanation, refer to Figure 4 , shows a flowchart of the steps of model training of a spiking neural network that integrates word attention and channel attention provided by the second embodiment of the present invention. The process may include the following steps:

[0138] Step 401: Acquire an audio data training set, pre-process the audio data training set, and output a plurality of Mel-frequency cepstral coefficient features to be trained;

[0139] Step 402: Input each Mel-frequency cepstral coefficient feature to be trained into an initial spiking neural network that integrates word attention and channel attention for recognition, and output disease probabilities of multiple syllable categories to be trained;

[0140] Step 403: Calculate the mean of disease probabilities of multiple syllable categories to be trained to determine the mean of disease probabilities to be trained;

[0141] Step 404: Calculate the target loss value based on the mean disease probability to be trained;

[0142] Step 405: Update the model parameters of the initial spiking neural network that integrates word attention and channel attention based on the target loss value, determine the intermediate spiking neural network that integrates word attention and channel attention, and count the number of iterations in real time;

[0143] Step 406: Determine whether the number of iterations reaches a preset iteration threshold;

[0144] Step 407: If yes, the intermediate pulse neural network that integrates word attention and channel attention is used as the trained pulse neural network that integrates word attention and channel attention.

[0145] It should be noted that the present invention extracts 24-dimensional MFCCs features from the collected audio data. This is based on the Chinese Syllable PSD pathological speech database collected by the First Affiliated Hospital of Jinan University, which is partitioned to obtain an audio data training set. Multiple MFCCs features for model training are extracted based on the audio data training set, and the number of time windows is unified. The constructed SNN-based TCAFormer model (an initial spiking neural network based on word attention and channel attention) is then trained for classification.

[0146] Furthermore, if the number of iterations does not reach the preset iteration threshold, the intermediate pulse neural network of the fusion of word attention and channel attention is used as the new initial pulse neural network of the fusion of word attention and channel attention, and the process jumps to step 402 until the number of iterations reaches the preset iteration threshold, and the intermediate pulse neural network of the fusion of word attention and channel attention determined when the number of iterations reaches the preset iteration threshold is used as the trained pulse neural network of the fusion of word attention and channel attention; wherein, the preset iteration threshold can be set as needed, and the present invention is not limited to this.

[0147] In this embodiment of the present invention, 24-dimensional MFCCs features are extracted from the collected audio data, with a unified number of time windows. The MFCC features extracted from the syllable audio data are then trained using a SNN-based TCAFormer model to determine syllable-level accuracy. Finally, the audio data from the test set is fed into the trained model to determine the probability that all syllable categories of a particular speaker are pathological. The probability that the speaker is ill is ultimately determined by averaging all syllable probabilities.

[0148] See also Figure 6 , Figure 6 The present invention provides a structural block diagram of a post-stroke dysarthria pathological speech recognition device based on SNN.

[0149] The present invention provides a post-stroke dysarthria pathological speech recognition device based on SNN, comprising:

[0150] The acquisition module 601 is used to acquire multiple initial audio data of the patient to be tested, pre-process each initial audio data, and output the Mel-frequency cepstral coefficient feature corresponding to each initial audio data;

[0151] Recognition module 602, configured to input each Mel-frequency cepstral coefficient feature into a spiking neural network that integrates word attention and channel attention for recognition, and output disease probabilities for multiple syllable categories;

[0152] The prediction module 603 is used to generate a pathological speech recognition result of post-stroke dysarthria corresponding to the patient to be tested based on the disease probabilities of multiple syllable categories.

[0153] Furthermore, the acquisition module 601 is specifically configured to:

[0154] Pre-emphasize each initial audio data and output the pre-emphasized audio data corresponding to each initial audio data;

[0155] Frame each pre-emphasized audio data to generate multiple audio frame data;

[0156] Windowing each audio frame data, and outputting the window audio data corresponding to each audio frame data;

[0157] Perform fast Fourier transform on the audio data of each window respectively, and output the target spectrum corresponding to the audio data of each window;

[0158] Perform filtering operations on each target spectrum to generate the Mel spectrum corresponding to each target spectrum;

[0159] Perform logarithmic operation and discrete cosine transform on each Mel spectrum respectively, and output the Mel cepstral coefficient features corresponding to each Mel spectrum.

[0160] Furthermore, the spiking neural network integrating word attention and channel attention includes an embedding module, a spiking self-attention module, and a linear output layer; the recognition module 602 includes:

[0161] The first submodule is used to use the embedding module to perform enhanced feature extraction on each Mel-frequency cepstral coefficient feature, and generate the target pulse nonlinear feature corresponding to each Mel-frequency cepstral coefficient feature;

[0162] The second submodule is used to take the nonlinear features of each target pulse as the input of the pulse self-attention module and output the target attention matrix corresponding to each target pulse nonlinear feature;

[0163] The third submodule is used to use each target attention matrix as the input of the linear output layer and output the disease probability of the syllable category corresponding to each target attention matrix.

[0164] Furthermore, the embedding module includes a first convolutional layer, a first batch of normalization layers, a first leaky spike neuron, a second convolutional layer, a second batch of normalization layers, and a second leaky spike neuron; and a first submodule, specifically configured to:

[0165] The first convolution layer is used to perform convolution operations on each Mel-frequency cepstral coefficient feature, and the first convolution feature corresponding to each Mel-frequency cepstral coefficient feature is output;

[0166] Normalize each first convolution feature through the first batch of normalization layers to generate a first normalized feature corresponding to each first convolution feature;

[0167] Using each first normalized feature as the input of the second leaky spike neuron, and outputting the initial spike nonlinear feature corresponding to each first normalized feature;

[0168] The second convolution layer is used to perform convolution operations on each initial pulse nonlinear feature, and the second convolution features corresponding to each initial pulse nonlinear feature are output;

[0169] Normalizing each second convolution feature through a second batch of normalization layers to generate a second normalized feature corresponding to each second convolution feature;

[0170] Each second normalized feature is used as the input of the second leakage pulse neuron, and the target pulse nonlinear feature corresponding to each second normalized feature is output.

[0171] Furthermore, the pulse self-attention module includes a word attention layer, a channel attention layer, and a multi-layer perceptron layer; the second submodule is specifically used to:

[0172] The nonlinear features of each target pulse are used as the input of the word attention layer, and the word attention matrix corresponding to each pulse nonlinear feature is output;

[0173] The nonlinear features of each target pulse are used as the input of the channel attention layer, and the channel attention matrix corresponding to each pulse nonlinear feature is output;

[0174] The word attention matrix and channel attention matrix corresponding to each target pulse nonlinear feature are summed and superimposed respectively, and the attention residual matrix corresponding to each target pulse nonlinear feature is output;

[0175] Each attention residual matrix is ​​used as the input of the multi-layer perceptron layer, and the nonlinear characteristics of the perceptual impulse corresponding to each attention residual matrix are output;

[0176] Each attention residual matrix and the perceptual pulse nonlinear features corresponding to each attention residual matrix are summed and superimposed respectively, and the target attention matrix corresponding to each target pulse nonlinear feature is output.

[0177] Furthermore, the prediction module 603 is specifically configured to:

[0178] Perform mean calculation on the disease probabilities of multiple syllable categories and output the mean disease probability;

[0179] Based on the comparison result of the mean probability of illness and the preset probability threshold, the pathological speech recognition result of post-stroke dysarthria corresponding to the patient to be detected is generated.

[0180] In an optional embodiment of the device, the device further comprises:

[0181] The first module is used to obtain an audio data training set, preprocess the audio data training set, and output a plurality of Mel-frequency cepstral coefficient features to be trained;

[0182] The second module is used to input the Mel-frequency cepstral coefficient features of each to-be-trained syllable into the initial spiking neural network that integrates word attention and channel attention for recognition, and output the disease probabilities of multiple to-be-trained syllable categories;

[0183] The third module is used to calculate the mean of the disease probabilities of multiple syllable categories to be trained, and determine the mean of the disease probabilities to be trained;

[0184] The fourth module is used to calculate the target loss value based on the mean disease probability to be trained;

[0185] The fifth module is used to update the model parameters of the initial spiking neural network that integrates word attention and channel attention based on the target loss value, determine the intermediate spiking neural network that integrates word attention and channel attention, and count the number of iterations in real time;

[0186] The sixth module is used to determine whether the number of iterations reaches a preset iteration threshold;

[0187] The seventh module is used to: if yes, use the intermediate pulse neural network that integrates word attention and channel attention as the trained pulse neural network that integrates word attention and channel attention.

[0188] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and sub-modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0189] An embodiment of the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory; when the computer program is executed by the processor, the processor executes the steps of the SNN-based post-stroke dysarthria pathological speech recognition method as described in the first embodiment above.

[0190] An embodiment of the present invention also provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the SNN-based post-stroke dysarthria pathological speech recognition method as in any of the above embodiments are implemented.

[0191] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0192] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0193] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A SNN-based method for post-stroke dysarthria pathological speech recognition, characterized in that: include: Acquire multiple initial audio data of the patient to be tested, preprocess each of the initial audio data, and output Mel-frequency cepstral coefficient features corresponding to each of the initial audio data; Inputting each of the Mel-frequency cepstral coefficient features into a spiking neural network that integrates word attention and channel attention for recognition, and outputting disease probabilities for multiple syllable categories; generating a post-stroke dysarthria pathological speech recognition result corresponding to the patient to be tested according to the disease probabilities of the plurality of syllable categories; The spiking neural network integrating word attention and channel attention includes an embedding module, a spiking self-attention module, and a linear output layer; the Mel-frequency cepstral coefficient features are input into the spiking neural network integrating word attention and channel attention for recognition, and the disease probabilities of multiple syllable categories are output, including: Using an embedding module to perform enhanced feature extraction on each of the Mel-frequency cepstral coefficient features, and generating a target pulse nonlinear feature corresponding to each of the Mel-frequency cepstral coefficient features; Using each of the target pulse nonlinear features as the input of the pulse self-attention module, and outputting the target attention matrix corresponding to each of the target pulse nonlinear features; Taking each of the target attention matrices as input to a linear output layer, and outputting the disease probability of the syllable category corresponding to each of the target attention matrices; The pulse self-attention module includes a word attention layer, a channel attention layer and a multi-layer perceptron layer.

2. The SNN-based post-stroke dysarthria pathological speech recognition method according to claim 1, characterized in that: The preprocessing of each of the initial audio data and outputting the Mel-frequency cepstral coefficient features corresponding to each of the initial audio data includes: Pre-emphasize each of the initial audio data, and output the pre-emphasized audio data corresponding to each of the initial audio data; respectively dividing the pre-emphasized audio data into frames to generate a plurality of audio frame data; Performing windowing on each of the audio frame data, and outputting window audio data corresponding to each of the audio frame data; Performing a fast Fourier transform on the audio data in each window, and outputting a target spectrum corresponding to the audio data in each window; Performing filtering operations on each of the target spectra to generate a Mel spectrum corresponding to each of the target spectra; Perform logarithmic operation and discrete cosine transform on each of the mel spectrums, and output mel cepstral coefficient features corresponding to each of the mel spectrums.

3. The SNN-based post-stroke dysarthria pathological speech recognition method according to claim 1, characterized in that: The embedding module includes a first convolutional layer, a first batch of normalization layers, a first leakage pulse neuron, a second convolutional layer, a second batch of normalization layers, and a second leakage pulse neuron; the embedding module is used to perform enhanced feature extraction on each of the Mel-frequency cepstral coefficient features to generate target pulse nonlinear features corresponding to each of the Mel-frequency cepstral coefficient features, including: Using a first convolutional layer to perform a convolution operation on each of the Mel-frequency cepstral coefficient features, and outputting a first convolution feature corresponding to each of the Mel-frequency cepstral coefficient features; Normalizing each of the first convolutional features through a first batch of normalization layers to generate a first normalized feature corresponding to each of the first convolutional features; using each of the first normalized features as an input to a second leaky spike neuron, and outputting an initial spike nonlinear feature corresponding to each of the first normalized features; Using a second convolutional layer to perform convolution operations on each of the initial pulse nonlinear features, and outputting second convolution features corresponding to each of the initial pulse nonlinear features; Normalizing each of the second convolutional features through a second batch of normalization layers to generate second normalized features corresponding to each of the second convolutional features; Each of the second normalized features is used as an input of a second leakage pulse neuron, and a target pulse nonlinear feature corresponding to each of the second normalized features is output.

4. The SNN-based post-stroke dysarthria pathological speech recognition method according to claim 1, characterized in that: The method of using each of the target pulse nonlinear features as the input of the pulse self-attention module and outputting the target attention matrix corresponding to each of the target pulse nonlinear features includes: Using each of the target pulse nonlinear features as the input of the word attention layer, and outputting the word attention matrix corresponding to each of the pulse nonlinear features; Taking each of the target pulse nonlinear features as the input of the channel attention layer, and outputting the channel attention matrix corresponding to each of the pulse nonlinear features; Respectively summing and superimposing each of the target pulse nonlinear features and the word attention matrix and channel attention matrix corresponding to each of the target pulse nonlinear features, and outputting an attention residual matrix corresponding to each of the target pulse nonlinear features; Using each of the attention residual matrices as the input of a multi-layer perceptron layer, and outputting the perceptual pulse nonlinear features corresponding to each of the attention residual matrices; Each of the attention residual matrices and the perception pulse nonlinear features corresponding to each of the attention residual matrices are summed and superimposed respectively, and a target attention matrix corresponding to each of the target pulse nonlinear features is output.

5. The SNN-based post-stroke dysarthria pathological speech recognition method according to claim 1, characterized in that: Generating a pathological speech recognition result of post-stroke dysarthria corresponding to the patient to be detected based on the disease probabilities of the plurality of syllable categories includes: Performing a mean operation on the disease probabilities of the plurality of syllable categories and outputting the mean disease probability; Based on the comparison result of the disease probability mean and the preset probability threshold, a post-stroke dysarthria pathological speech recognition result corresponding to the patient to be detected is generated.

6. The SNN-based post-stroke dysarthria pathological speech recognition method according to claim 1, characterized in that: The model training process of the pulse neural network integrating word attention and channel attention includes: Acquire an audio data training set, preprocess the audio data training set, and output a plurality of Mel-frequency cepstral coefficient features to be trained; Inputting each of the to-be-trained Mel-frequency cepstral coefficient features into an initial spiking neural network that integrates word attention and channel attention for recognition, and outputting disease probabilities of multiple to-be-trained syllable categories; Calculating the mean of disease probabilities of the plurality of syllable categories to be trained to determine the mean of disease probabilities to be trained; Calculating a target loss value according to the mean disease probability to be trained; Update the model parameters of the initial spiking neural network that integrates word attention and channel attention based on the target loss value, determine the intermediate spiking neural network that integrates word attention and channel attention, and count the number of iterations in real time; Determining whether the number of iterations reaches a preset iteration threshold; If so, the intermediate pulse neural network that integrates word attention and channel attention is used as the trained pulse neural network that integrates word attention and channel attention.

7. A post-stroke dysarthria pathological speech recognition device based on SNN, applying the post-stroke dysarthria pathological speech recognition method based on SNN according to claim 1, characterized in that: include: An acquisition module is used to acquire multiple initial audio data of the patient to be tested, pre-process each of the initial audio data, and output Mel-frequency cepstral coefficient features corresponding to each of the initial audio data; A recognition module is used to input each of the Mel-frequency cepstral coefficient features into a spiking neural network that integrates word attention and channel attention for recognition, and output disease probabilities for multiple syllable categories; The prediction module is used to generate a pathological speech recognition result of post-stroke dysarthria corresponding to the patient to be detected based on the disease probabilities of the multiple syllable categories.

8. A computer device, characterized in that: It includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the SNN-based post-stroke dysarthria pathological speech recognition method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the SNN-based post-stroke dysarthria pathological speech recognition method is implemented as described in any one of claims 1 to 6.

Images