A voice recognition method and apparatus for screening cognitive dysfunction

Abstract

The application relates to a voice recognition method and device for screening cognitive dysfunction, wherein the method comprises the following steps: acquiring an audio signal to be detected; inputting the audio signal into a voice recognition model for recognition to obtain a recognition result, wherein an acoustic model part in the voice recognition model is obtained by training voice data; a language model part comprises a general language model and a special disease language model, the special disease language model is obtained according to corresponding text data collected according to Alzheimer's disease and related cognitive dysfunction, the general language model and the special disease language model are self-learning and fused; a feature extraction part is used for acquiring audio features in the audio signal based on the acoustic model part; and a decoder part is used for recognizing the extracted audio features, and in the recognition, the general language model and the special disease language model are selected dynamically according to decoding parameters. The application can balance the recognition effects of general and medical special disease scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech recognition, in particular to a speech recognition method and device for screening cognitive dysfunction. BACKGROUND

[0002] Alzheimer's disease is the most common neurodegenerative disease and the main cause of dementia, which is mainly manifested as progressive decline in cognitive function, accompanied by mental and behavioral disorders, and seriously affects the patient's daily life ability and quality of life. The course of Alzheimer's disease includes preclinical stage, mild cognitive impairment stage and dementia stage. The existing drug treatment can only alleviate the progression of the disease to a certain extent and cannot cure the disease, which emphasizes the importance of early diagnosis and early intervention. However, in clinical practice, due to insidious onset, the patient's cognitive function has been severely impaired when seeking medical treatment.

[0003] Since the 1960s, the method used in speech recognition has mainly been simple template matching. In the 1980s, the research direction of speech recognition gradually shifted to the technical idea based on statistical models, such as Gaussian mixture distribution-hidden Markov model (GMM-HMM), which realized a breakthrough development from isolated words to large-vocabulary speech recognition, but the accuracy and speed of recognition have not yet reached the practical threshold. In 2006, Hinton proposed to use deep belief networks (Deep Belief Networks, DBN) to initialize neural networks, making it easy to train deep neural networks, thereby triggering a wave of deep learning (Deep Learning, DL) represented by deep neural networks (Deep Neural Network, DNN), recurrent neural networks (Recurrent Neural Networks, RNN) and convolutional neural networks (Convolutional Neural Networks, CNN), and good practical results have been achieved. The speech recognition framework based on neural networks mainly consists of three parts of acoustic model, language model and pronunciation model. The acoustic model and the language model are trained separately, and their correlation is weak. Different pronunciation dictionaries need to be constructed for different languages and dialects, which has high professional threshold, high construction cost, long cycle and poor quality, etc. In addition, the existing speech recognition software is not suitable for Alzheimer's disease patients, such as no speech segment detection function and poor recognition effect in the special disease field. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a speech recognition method and device for screening cognitive dysfunction, which can balance the recognition effect of general and medical special disease scenes.

[0005] The technical scheme adopted by the present application to solve its technical problems is to provide a voice recognition method for cognitive dysfunction screening, comprising the following steps:

[0006] Obtaining an audio signal to be detected;

[0007] Inputting the audio signal into a speech recognition model for recognition to obtain a recognition result, wherein the speech recognition model comprises an acoustic model part, a language model part, a feature extraction part and a decoder part, the acoustic model part is obtained by training the speech data; the language model part comprises a general language model and a special disease language model, the special disease language model is obtained according to the corresponding text data collected from Alzheimer's disease and related cognitive impairment, and the general language model and the special disease language model are self-learning and fused; the feature extraction part is used to obtain the audio features in the audio signal based on the acoustic model part; the decoder part is used to recognize the extracted audio features, and in the recognition, the general language model and the special disease language model are dynamically selected according to the decoding parameters.

[0008] The decoder part comprises a first one-way long short-term memory module, an attention layer module and a second one-way long short-term memory module connected in sequence; the output of the second one-way long short-term memory module is fused with the output of the language model part, specifically: the input of the language model part is the output of the last decoder part, and the two long short-term memory modules form an RNN-LM, the output of the RNN-LM is calculated with the output of the second one-way long short-term memory module to form a gate for selecting the general language model and the special disease language model, and after selection, the output of the second one-way long short-term memory module is classified to obtain the final decoding result.

[0009] The acoustic model part also performs multi-accent adaptive processing, specifically: a general recognition model is used to decode the acoustic part of the training speech data, and a residual vector based on the phoneme level is formed by using the decoding result and the label to represent the difference between the current speaker and the standard speaker, then a threshold is set to screen out the part of the current acoustic model that has larger recognition difference for the speaker as the dialect accent residual vector, and the screened dialect accent is clustered and a fixed number of dialect accent memory vectors are formed by using clustering, and the clustered dialect features are weighted by using the attention mechanism to obtain the accent feature; after learning and refining the accent feature, it is integrated into the current acoustic model, and the dialect accent data is mapped to the space of the standard pronunciation.

[0010] The voice recognition method for cognitive dysfunction screening further comprises validity detection on the audio signal.

[0011] The validity detection on the audio signal comprises:

[0012] performing speech segment detection on the audio signal to obtain at least one speech segment;

[0013] For each speech segment, based on the text content corresponding to the speech segment, the speech segment is detected for effectiveness to determine whether the speech segment is an effective speech segment.

[0014] The technical solution adopted by the present application to solve its technical problems is to provide a speech recognition device for screening cognitive dysfunction, comprising:

[0015] The acquisition module is configured to acquire an audio signal to be detected.

[0016] The recognition module is configured to input the audio signal into a speech recognition model for recognition to obtain a recognition result. The speech recognition model includes an acoustic model part, a language model part, a feature extraction part, and a decoder part. The acoustic model part is obtained by training speech data. The language model part includes a general language model and a disease-specific language model. The disease-specific language model is obtained based on corresponding text data collected from Alzheimer's disease and related cognitive impairment. The general language model and the disease-specific language model are self-learning and fused. The feature extraction part is configured to obtain audio features in the audio signal based on the acoustic model part. The decoder part is configured to recognize the extracted audio features. During recognition, the general language model and the disease-specific language model are dynamically selected based on decoding parameters.

[0017] The decoder part includes a first one-way long short-term memory module, an attention layer module, and a second one-way long short-term memory module connected in sequence. The output of the second one-way long short-term memory module is fused with the output of the language model part. Specifically, the input of the language model part is the output of the last decoder part, and the two long short-term memory modules form an RNN-LM. The output of the RNN-LM is calculated with the output of the second one-way long short-term memory module to select the general language model and the disease-specific language model. After selection, the output of the second one-way long short-term memory module is classified to obtain the final decoding result.

[0018] The acoustic model part also performs multi-accent adaptive processing, specifically: a general recognition model is used to decode the acoustic part of the training speech data, and the decoding result is used to form a residual vector based on the phoneme level with the label to represent the difference between the current speaker and the standard speaker, then a threshold is set to filter out the part of the current acoustic model that has larger recognition difference for the speaker as the dialect accent residual vector, and the filtered dialect accent is clustered and a fixed number of dialect accent memory vectors are formed using clustering, and the attention mechanism is used to weight the clustered dialect features to obtain the accent feature; the accent feature is learned and refined and integrated into the current acoustic model, and the dialect accent data is mapped to the space of the standard pronunciation.

[0019] The voice recognition device for cognitive dysfunction screening also includes an effectiveness detection module for detecting the effectiveness of the audio signal.

[0020] The effectiveness detection module includes a first detection unit for detecting speech segments from the audio signal, and a second detection unit for detecting the effectiveness of each speech segment based on the text content corresponding to the speech segment.

[0021] Advantages

[0022] Compared with the prior art, the present application has the following advantages and positive effects: the present application balances the general and medical special scene recognition effect based on the end-to-end super large scale field language model adaptive method, and the dialect accent variation of patients in different regions is statistically analyzed based on known training data to realize the construction of a memory unit based on dialect accent variation rules, thereby improving the recognition robustness and accuracy. The present application also detects the effective speech segments of the patient, further analyzes the number of voiced segments, duration and unvoiced pause, and meets the demand for speech feature analysis of Alzheimer's disease patients. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is the overall flowchart of the voice recognition method of the present application;

[0024] Figure 2 is a multi-accent adaptive processing diagram in the embodiment of the present application;

[0025] Figure 3 is a processing diagram of the decoder part in the embodiment of the present application. DETAILED DESCRIPTION

[0026] The application will be further described in connection with the following embodiments. It should be understood that these embodiments are only used for illustrating but not for limiting the scope of the application. Furthermore, it should be understood that after reading the content of the present application, those skilled in the art can make various modifications or changes to the present application, and these equivalent forms also fall within the scope defined by the appended claims.

[0027] The first embodiment of the present application relates to a voice recognition method for cognitive dysfunction screening, comprising the following steps:

[0028] Step 1, obtaining an audio signal to be detected;

[0029] Step 2, performing validity detection on the audio signal, specifically: performing speech segment detection on the audio signal to obtain at least one speech segment; for each speech segment, based on the text content corresponding to the speech segment, performing validity detection on the speech segment to determine whether the speech segment is a valid speech segment.

[0030] In the present embodiment, after performing speech segment detection on the audio signal to be detected to obtain at least one speech segment, for each speech segment, at least based on the text content corresponding to the speech segment, validity detection is performed on the speech segment to determine whether the speech segment is a valid speech segment. Since the validity of the speech segment is further detected based on the text content corresponding to the speech segment in the present embodiment, compared with directly taking the detected speech segment as a valid speech segment, the detection accuracy of the valid speech segment is improved.

[0031] Step 3, inputting the audio signal into a voice recognition model for recognition to obtain a recognition result. The voice recognition model comprises an acoustic model part, a language model part, a feature extraction part and a decoder part, wherein the acoustic model part is obtained by training voice data; the language model part comprises a general language model and a disease-specific language model, the disease-specific language model is obtained according to corresponding text data collected from Alzheimer's disease and related cognitive disorders, and the general language model and the disease-specific language model are self-learning and fused; the feature extraction part is used to obtain audio features in the audio signal based on the acoustic model part; the decoder part is used to recognize the extracted audio features, and in the recognition, the general language model and the disease-specific language model are dynamically selected according to decoding parameters.

[0032] In the present embodiment, the acoustic model part considers the dialectal and oral variation of patients in different regions based on the diversity of patient population characteristics and accents of different hospitals, and realizes the construction of a memory unit based on the dialectal and oral variation rules by using known training data to statistically analyze the dialectal and oral variation of patients in different regions. Figure 1As shown, first, the acoustic part of the large-scale training data is decoded by means of a general recognition model, and the decoding result is used to form a residual vector based on a phoneme level with the label to represent the difference between the current speaker and the standard speaker, and then the threshold is set to screen out the part with large recognition difference of the current model for the speaker as the dialect accent residual vector, and the screened dialect accent part is clustered in a clustering manner to form a fixed number of dialect accent memory vectors, so as to realize the dialect accent representation of the pronunciation manner. Considering that the characteristics of dialect accents in different regions are similar, a small amount of data is used to learn and refine the accent features, and the corresponding features are integrated into the acoustic model, so as to map the dialect accent data to the space of standard pronunciation, thereby optimizing the recognition effect of different dialect accents. In addition, since the attention mechanism is used to weight the clustered dialect features, for the region with only a small amount of dialect data, the accent memory vector can also achieve a relatively ideal effect in the corresponding dialect region through the fine-tuning of the accent memory vector in the model training process.

[0033] The embodiment balances the general and medical special disease scene recognition effects, and uses an end-to-end super large field language model self-adaptive method. First, corresponding text data of neurological diseases such as Alzheimer's disease needs to be collected to train a special disease language model; second, the general language model and the special disease model are self-learned and fused, so as to dynamically realize the recognition effect of special disease application fields such as Alzheimer's disease evaluation while ensuring the general effect of daily language. The technology needs to fuse the general main language model and the medical special disease language model through a gating mechanism, the gating parameters are dynamically learned in the model training process, and the decoder dynamically selects the general main language model and the medical special disease language model according to the related decoding parameters, so as to better guarantee the recognition effect, as shown in the following specific embodiment. Figure 3

[0034] Figure 3 The left side in the middle is a general end-to-end (Encoder-Decoder, ED) model structure diagram. The ED decoder is an autoregressive decoding structure, the input is the last decoding result (y t-1 ) and the context vector (c t-1 ) of the last decoding, which is used for autoregressive decoding through a unidirectional long short-term memory module (lstmp). The hidden layer vector (s t d ) of the autoregressive decoding is subjected to Attention operation (Att-ctx-dec) with the ED encoder output (H enc ) to obtain the context vector (c t ​), the context vector of this decoding is further subjected to a one-way long short-term memory module (lstmp) and a classification module to obtain the decoding result of this decoding. The ED model can be well used for sequence modeling, but is limited to training data. Only using speech annotation data to train the ED model, the advantages of large data of the language model cannot be well utilized. The most direct way is to use the language model to simply score the shallow fusion of the decoding result of the ED. Although this shallow fusion method can correct the ED result to a certain extent, the scenario covered by the language model trained by a large amount of data is greatly deviated from the medical special disease corpus trained by the ED, and this deviation will lead to limited fusion effect. Therefore, the ED model is adjusted, and an ED and field language model fusion training scheme is adopted. Specifically, the decoder end of the ED model structure is fused and trained with the RNN language model. Through the fusion training method, the decoder end of the model can learn the large text corpus information of the language model at the same time. As shown in Fig. Figure 3 The front autoregressive, Attention and classification lstmp modules are consistent with the decoder end of the ED, and the ColdFusion fusion training fuses the hidden layer output of the second lstmp The input of the RNN language model is the last decoding result, which is subjected to two long short-term memory modules to form an RNN-LM. The hidden layer output of the RNN-LM is used to calculate a gate (LM-gate) together with to select the field language model, and then the selected result is classified together with to obtain the final decoding result.

[0035] It is not difficult to find that the present application balances the general and medical special disease scene recognition effect by the end-to-end based super large field language model self-adaptive method, realizes the construction of the memory unit based on the dialect accent change rule by using the known training data to statistically analyze the dialect accent change of different regional patients, and detects the effective speech segment of the patient to improve the recognition accuracy.

[0036] The second embodiment of the present application relates to a voice recognition device for cognitive dysfunction screening, comprising:

[0037] An acquisition module is configured to acquire an audio signal to be detected.

[0038] The recognition module is configured to input the audio signal into a speech recognition model to obtain a recognition result, wherein the speech recognition model comprises an acoustic model part, a language model part, a feature extraction part, and a decoder part, the acoustic model part is obtained by training speech data, the language model part comprises a general language model and a disease-specific language model, the disease-specific language model is obtained according to corresponding text data collected from Alzheimer's disease and related cognitive impairment, the general language model and the disease-specific language model are fused by self-learning, the feature extraction part is configured to obtain audio features in the audio signal based on the acoustic model part, and the decoder part is configured to recognize the extracted audio features, and in the recognition, the general language model and the disease-specific language model are dynamically selected according to decoding parameters.

[0039] The decoder part comprises a first one-way long short-term memory module, an attention layer module, and a second one-way long short-term memory module connected in sequence, the output of the second one-way long short-term memory module is fused with the output of the language model part, specifically, the input of the language model part is the output of the decoder part in the last time, and the two long short-term memory modules form an RNN-LM, the output of the RNN-LM is used together with the output of the second one-way long short-term memory module to calculate a gating for selecting the general language model and the disease-specific language model, and after the selection, the output of the second one-way long short-term memory module is used together for classification to obtain a final decoding result.

[0040] The acoustic model part also performs multi-accent adaptive processing, specifically, a general recognition model is used to decode the acoustic part of the training speech data, and a residual vector based on a phoneme level is formed by using the decoding result and the annotation to represent the difference between the current speaker and the standard speaker, then a threshold is set to screen out the part with large recognition difference of the current speaker by the acoustic model as a dialect accent residual vector, and a clustering method is used to cluster the screened dialect accents and form a fixed number of dialect accent memory vectors, and an attention mechanism is used to weight the clustered dialect features to obtain accent features; after learning and refining the accent features, the accent features are integrated into the current acoustic model, and the dialect accent data is mapped to the space of the standard pronunciation.

[0041] The speech recognition device for cognitive impairment screening further comprises an effectiveness detection module configured to perform effectiveness detection on the audio signal.

[0042] The effectiveness detection module comprises a first detection unit configured to perform speech segment detection on the audio signal to obtain at least one speech segment, and a second detection unit configured to perform effectiveness detection on each speech segment based on the text content corresponding to the speech segment to determine whether the speech segment is an effective speech segment.

[0043] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, apparatus, or computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer program code thereon for use by or in connection with an instruction execution system. Program code embodied on one or more computer-usable storage media can be downloaded over a network

[0044] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing system or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 means for performing the function specified by the flowchart illustrations and / or block diagrams block or blocks.

[0045] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 means for performing the function specified by the flowchart illustrations and / or block diagrams block or blocks.

[0046] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 means for performing the function specified by the flowchart illustrations and / or block diagrams block or blocks.

[0047] While the preferred embodiments of the application have been described, additional variations and modifications can be made to these embodiments by those skilled in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, it is intended that the appended claims include all such modifications and variations as fall within the scope of the present application.

[0048] It is apparent that those skilled in the art can make various changes and modifications to the application without departing from the spirit and scope of the application. It is therefore intended that the present application cover all such changes and modifications that are within its scope.

Claims

1. A speech recognition method for screening cognitive impairment, characterized in that, Includes the following steps: Acquire the audio signal to be detected; The audio signal is input into a speech recognition model for recognition to obtain a recognition result. The speech recognition model includes an acoustic model, a language model, a feature extraction part, and a decoder part. The acoustic model is trained using speech data. The language model includes a general language model and a disease-specific language model. The disease-specific language model is obtained based on corresponding text data collected from Alzheimer's disease and related cognitive impairments. The general language model and the disease-specific language model undergo self-learning fusion. The feature extraction part is used to obtain audio features from the audio signal based on the acoustic model. The decoder part is used to recognize the extracted audio features, dynamically selecting between the general language model and the disease-specific language model based on decoding parameters during recognition. The decoder section includes a first unidirectional long short-term memory module, an attention layer module, and a second unidirectional long short-term memory module connected in sequence. The output of the second unidirectional long short-term memory module is fused with the output of the language model section. Specifically, the input of the language model section is the output of the previous decoder section, which is combined with the two long short-term memory modules to form an RNN-LM. The output of the RNN-LM and the output of the second unidirectional long short-term memory module are used to calculate a gating mechanism to select between a general language model and a disease-specific language model. After selection, the gating mechanism is combined with the output of the second unidirectional long short-term memory module for classification to obtain the final decoding result. The acoustic model also performs multi-accent adaptive processing, specifically: a general recognition model is used to decode the acoustic part of the training speech data, and the decoding results and annotations are used to form a phoneme-level residual vector to represent the difference between the current speaker and the standard speaker. Then, by setting a threshold, the part of the difference recognized by the current acoustic model for the current speaker that exceeds the threshold is selected as the dialect accent residual vector. The selected dialect accents are clustered to form a fixed number of dialect accent memory vectors. An attention mechanism is used to weight the clustered dialect features to obtain accent features. The accent features are learned, refined, and integrated into the current acoustic model, mapping the dialect accent data to the standard pronunciation space.

2. The speech recognition method for screening cognitive impairment according to claim 1, characterized in that, It also includes validity detection of the audio signal.

3. The speech recognition method for screening cognitive impairment according to claim 2, characterized in that, The validity detection of the audio signal includes: The audio signal is subjected to speech segment detection to obtain at least one speech segment; For each audio segment, a validity check is performed on the audio segment based on the corresponding text content to determine whether the audio segment is a valid audio segment.

4. A speech recognition device for screening cognitive impairment, characterized in that, include: The acquisition module is used to acquire the audio signal to be detected; The recognition module is used to input the audio signal into a speech recognition model for recognition and obtain a recognition result. The speech recognition model includes an acoustic model, a language model, a feature extraction component, and a decoder. The acoustic model is trained using speech data. The language model includes a general language model and a disease-specific language model. The disease-specific language model is obtained based on corresponding text data collected from Alzheimer's disease and related cognitive impairments. The general language model and the disease-specific language model undergo self-learning fusion. The feature extraction component is used to obtain audio features from the audio signal based on the acoustic model. The decoder is used to recognize the extracted audio features, dynamically selecting between the general language model and the disease-specific language model based on decoding parameters during recognition. The decoder section includes a first unidirectional long short-term memory module, an attention layer module, and a second unidirectional long short-term memory module connected in sequence. The output of the second unidirectional long short-term memory module is fused with the output of the language model section. Specifically, the input of the language model section is the output of the previous decoder section, which is combined with the two long short-term memory modules to form an RNN-LM. The output of the RNN-LM and the output of the second unidirectional long short-term memory module are used to calculate a gating mechanism to select between a general language model and a disease-specific language model. After selection, the gating mechanism is combined with the output of the second unidirectional long short-term memory module for classification to obtain the final decoding result. The acoustic model also performs multi-accent adaptive processing, specifically: a general recognition model is used to decode the acoustic part of the training speech data, and the decoding results and annotations are used to form a phoneme-level residual vector to represent the difference between the current speaker and the standard speaker. Then, by setting a threshold, the part of the difference recognized by the current acoustic model for the current speaker that exceeds the threshold is selected as the dialect accent residual vector. The selected dialect accents are clustered to form a fixed number of dialect accent memory vectors. An attention mechanism is used to weight the clustered dialect features to obtain accent features. The accent features are learned, refined, and integrated into the current acoustic model, mapping the dialect accent data to the standard pronunciation space.

5. The speech recognition device for screening cognitive impairment according to claim 4, characterized in that, Also includes: The validity detection module is used to detect the validity of the audio signal.

6. The speech recognition device for screening cognitive impairment according to claim 5, characterized in that, The validity detection module includes: a first detection unit, used to perform speech segment detection on the audio signal to obtain at least one speech segment; and a second detection unit, used to perform validity detection on each speech segment based on the text content corresponding to the speech segment, and determine whether the speech segment is a valid speech segment.

Images