Method and device for identifying Alzheimer's disease patient based on voice signal

Abstract

The invention discloses a method and device for recognizing an Alzheimer's disease patient based on voice signals. The method comprises the following steps: acquiring a voice signal of a target object, extracting and processing a shallow feature and a deep feature of the voice signal by using a first model to obtain a first result, segmenting and processing the voice signal by using a second model to obtain a second result, and processing the voice signal and text information by using a third model to obtain a second result. The text information is obtained by transcribing the voice signal, and the first result, the second result and the third result are all used for indicating whether the target object is the Alzheimer's disease patient; and jointly determining whether the target object is the Alzheimer's disease patient according to the first result, the second result and the third result. According to the method and the device, the technical problems that a method for detecting the Alzheimer's disease in related technologies does not support wide-range application and the detection result is inaccurate are solved.

Description

Technical Field

[0001] The present application relates to the technical field of natural speech processing, and in particular, to a method and device for identifying Alzheimer's disease patients based on speech signals. Background Art

[0002] In the related technology, intelligent assisted screening for Alzheimer's disease based on neuroimaging, the diagnostic method based on neuroimaging examination requires equipment to collect images, but the equipment for collecting images is very expensive and cannot be used on a large scale. Therefore, there is a lack of image data collected from Alzheimer's patients, which cannot support the training of the model based on neuroimaging for detecting Alzheimer's disease, resulting in inaccurate test results when using intelligent assisted screening for Alzheimer's disease based on neuroimaging.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a method and device for identifying Alzheimer's disease patients based on voice signals, so as to at least solve the technical problems in the related art that the method for detecting Alzheimer's disease does not support large-scale application and the detection results are inaccurate.

[0005] According to one aspect of an embodiment of the present application, a method for identifying Alzheimer's patients based on speech signals is provided, comprising: acquiring a speech signal of a target object, and using a first model to extract and process shallow features and deep features of the speech signal to obtain a first result, using a second model to segment and process the speech signal to obtain a second result, and using a third model to process the speech signal and text information to obtain a third result, wherein the text information is obtained by transcribing the speech signal, and the first result, the second result, and the third result are all used to indicate whether the target object is an Alzheimer's patient; and jointly determining whether the target object is an Alzheimer's patient based on the first result, the second result, and the third result.

[0006] Optionally, a first model is used to extract and process shallow features and deep features of a speech signal to obtain a first result, including: extracting shallow features of a speech signal, wherein the shallow features are used to describe the acoustic features of the speech signal, and the acoustic features include: frequency, timbre, and volume; generating a first type of vector based on the shallow features, and generating a second type of vector based on the first type of vector, wherein the dimension of the second type of vector is smaller than the dimension of the first type of vector, and the second type of vector is used to represent the deep features of the speech signal, and the deep features are used to describe the correlation between the acoustic features; analyzing and processing the target feature vector generated based on the first type of vector and the second type of vector to output the first result.

[0007] Optionally, generating a second type of vector based on the first type of vector includes: for each element in the first type of vector, determining the similarity between the element and other elements in the first type of vector, and obtaining multiple similarities corresponding to each element; determining the product of each similarity and the element, and determining the new element used to replace the element by the sum of the multiple products; and determining the vector composed of the multiple new elements as the second type of vector.

[0008] Optionally, the second model is trained by the following method: obtaining a speech segment, wherein the speech segment is generated based on a speech signal of an Alzheimer's patient; dividing the speech segment into multiple short speech segments in sequence, wherein the similarity between a previous short speech segment and a subsequent short speech segment is greater than or equal to a preset similarity, and each short speech segment contains a speech signal of a preset length; using multiple short speech segments to iteratively train the neural network model until the number of iterations reaches a preset number, and outputting a second model, wherein the second model learns the speech features of the speech signal of the Alzheimer's patient during the training process.

[0009] Optionally, a third model is used to process the speech signal and text information to obtain a third result, including: performing speech processing on the speech signal to obtain speech features of the speech signal, wherein the speech features include: the number of silences and the proportion of silences, and silences are speech signals whose volume value is less than a preset volume value and whose duration is less than a preset duration; converting the speech signal into text, and performing text processing on the text to obtain first-category text features, wherein the first-category text features include: the number of verbs, the proportion of verbs, the number of nouns, the proportion of nouns, the number of quantifiers, the proportion of quantifiers, and the number of pause words; and extracting second-category text features from the text, wherein the second-category text features are used to indicate the semantic content of the speech signal; fusing the speech features, the first-category text features, and the second-category text features to obtain a fusion result; analyzing and processing the fusion result to obtain a third result.

[0010] Optionally, speech processing is performed on the speech signal to obtain speech features of the speech signal, including: determining the duration of each silent sound and determining the cumulative sum of the durations of multiple silent sounds; and determining the ratio of the cumulative sum to the duration of the speech signal as the ratio of the silent sounds.

[0011] Optionally, determining whether the target object is an Alzheimer's patient based on the first result, the second result, and the third result includes: determining a first number of results in the first result, the second result, and the third result indicating that the target object is an Alzheimer's patient, and determining a second number of results in the first result, the second result, and the third result that the target object is not an Alzheimer's patient; when the first number is greater than the second number, determining that the target object is an Alzheimer's patient; and when the first number is less than the second number, determining that the target object is not an Alzheimer's patient.

[0012] According to another aspect of an embodiment of the present application, a device for identifying Alzheimer's patients based on speech signals is also provided, including: a processing module, used to obtain a speech signal of a target object, and use a first model to extract and process shallow features and deep features of the speech signal to obtain a first result, use a second model to segment and process the speech signal to obtain a second result, and use a third model to process the speech signal and text information to obtain a third result, wherein the text information is obtained by transcribing the speech signal, and the first result, the second result and the third result are all used to indicate whether the target object is an Alzheimer's patient; a determination module, used to jointly determine whether the target object is an Alzheimer's patient based on the first result, the second result and the third result.

[0013] According to another aspect of an embodiment of the present application, a non-volatile storage medium is provided, in which a computer program is stored, wherein the method for identifying Alzheimer's patients based on voice signals is executed by running the computer program on a device where the non-volatile storage medium is located.

[0014] According to another aspect of an embodiment of the present application, an electronic device is also provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned method for identifying Alzheimer's patients based on voice signals through the computer program.

[0015] According to another aspect of an embodiment of the present application, a computer program product is also provided, including computer instructions, which, when executed by a processor, implement the steps of the above-mentioned method for identifying Alzheimer's patients based on speech signals.

[0016] In an embodiment of the present application, a speech signal of a target object is acquired, and a first model is used to extract and process shallow features and deep features of the speech signal to obtain a first result, a second model is used to segment and process the speech signal to obtain a second result, and a third model is used to process the speech signal and text information to obtain a third result, wherein the text information is obtained by transcribing the speech signal, and the first result, the second result, and the third result are all used to indicate whether the target object is an Alzheimer's patient; a method of jointly determining whether the target object is an Alzheimer's patient based on the first result, the second result, and the third result is used simultaneously to identify the target object. The method of detecting Alzheimer's disease patients by voice recognition and text recognition adopts different models to perform text transcription, feature extraction and recognition on the speech signal of the detection object to determine whether the detection object is an Alzheimer's disease patient. Finally, whether the detection object is an Alzheimer's disease patient is determined based on the determination results of different models, thereby achieving the purpose of determining whether the detection object is an Alzheimer's disease patient based on voice recognition, thereby achieving the technical effect of expanding the application scope of the Alzheimer's disease detection method and improving the accuracy of the detection results, and further solving the technical problems in the related technology that the method for detecting Alzheimer's disease does not support large-scale application and the detection results are inaccurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 is a hardware structure block diagram of a computer terminal for implementing a method for identifying Alzheimer's patients based on voice signals according to an embodiment of the present application;

[0019] Figure 2 is a flowchart of steps for identifying Alzheimer's patients based on speech signals according to an embodiment of the present application;

[0020] Figure 3 is a workflow diagram of a first model according to an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of processing a speech signal according to a third model of an embodiment of the present application;

[0022] Figure 5 1 is a schematic diagram of a network structure of a bidirectional encoder based on a transformer (BERT) model according to an embodiment of the present application;

[0023] Figure 6 is a structural diagram of a device for identifying Alzheimer's patients based on speech signals according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] In the related art, the detection of Alzheimer's patients based on speech signals requires a high degree of manual participation. Researchers collect the speech of the test object through some fixed tasks, such as picture description and language fluency detection, and extract semantic and syntactic features from the collected data to determine whether the test object is an Alzheimer's patient. The above scheme requires manual extraction and annotation of semantic and syntactic features, so there are problems such as low accuracy of the test results and inability to be widely used. In order to solve this problem, the relevant solution is provided in the embodiment of the present application, which is described in detail below.

[0027] According to an embodiment of the present application, a method embodiment for identifying Alzheimer's patients based on speech signals is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0028] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal for implementing a method for identifying Alzheimer's patients based on voice signals is shown. Figure 1As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0029] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0030] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method for identifying Alzheimer's patients based on voice signals in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned method for identifying Alzheimer's patients based on voice signals is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0031] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0032] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0033] The embodiment of the present application provides a method for identifying Alzheimer's disease patients based on speech signals that can be run in the above-mentioned operating environment. Figure 2 is a flowchart of steps for identifying Alzheimer's disease patients based on speech signals according to an embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:

[0034] Step S202, obtain the speech signal of the target object, and use the first model to extract and process the shallow features and deep features of the speech signal to obtain a first result, use the second model to segment and process the speech signal to obtain a second result, and use the third model to process the speech signal and text information to obtain a third result, wherein the text information is obtained by transcribing the speech signal, and the first result, the second result and the third result are all used to indicate whether the target object is an Alzheimer's patient.

[0035] The embodiment of the present application uses different models with speech recognition function to detect and identify the speech signal input by the detection object (i.e., the target object), and determines whether the detected speech signal has the acoustic characteristics of an Alzheimer's patient. In step S202, the received speech signal is input into different models (first model, second model, and third model) respectively. In this embodiment, although the models applied all have speech recognition function, they are different when processing the speech signal. For example, in step S202, the process of processing the speech signal by the first model is: extracting the shallow features and deep features of the speech signal, fusing the shallow features and the deep features, and outputting the recognition result (i.e., the first result) indicating whether the detection object is an Alzheimer's patient by analyzing and processing the features fused by the shallow features and the deep features. The processing process of the second model is to first segment the speech signal, expand the data set, analyze and process each segment of the speech signal after segmentation, and finally output the recognition result (i.e., the second result) indicating whether the detection object is an Alzheimer's patient. The third model processes the speech signal and the text information transcribed from the speech signal separately, uses the text features extracted from the text information as supplementary information to the acoustic features extracted from the speech signal, and jointly determines whether the test object has the pathological characteristics of an Alzheimer's patient based on the acoustic features and the text features, and outputs a recognition result indicating whether the test object is an Alzheimer's patient (i.e., the third result).

[0036] The first model, the second model and the third model mentioned above may be included in the same neural network model, or may be models that run independently. When the first model, the second model and the third model are included in the same neural network model, in step S202, the neural network model may be loaded into the memory, for example, the raw data of the neural network model may be loaded from the non-volatile memory into the volatile memory so that the processor runs the neural network model. The raw data of the neural network model refers to unprocessed data, and generally includes parameters and structural data of the neural network model. The structural data may be a calculation relationship based on parameters, such as a forward propagation calculation relationship between intermediate layers and neurons. Specifically, the structural data may include code related to the structure of the neural network model, such as code for performing related calculations between intermediate layers and neurons. If the first model, the second model and the third model run independently, the first model, the second model and the third model are loaded into the memory respectively.

[0037] In one embodiment, an area for loading a neural network model that includes the first model, the second model, and the third model at the same time, or an area for loading the first model, the second model, and the third model respectively, can be divided in the memory, and the area can include a structure data storage area and a parameter storage area. The structure data storage area is used to store structure-related codes, and the parameters referenced by it can point to the address of specific parameters in the parameter storage area through a pointer. During the training process of the neural network model that includes the first model, the second model, and the third model at the same time, or during the training process of the first model, the second model, and the third model respectively, the model may need to frequently update the parameters, and the parameter values ​​in the parameter storage area can be updated.

[0038] According to an optional embodiment of the present application, a first model is used to extract and process shallow features and deep features of a speech signal to obtain a first result, including: extracting shallow features of the speech signal, wherein the shallow features are used to describe the acoustic features of the speech signal, and the acoustic features include: frequency, timbre, and volume; generating a first type of vector based on the shallow features, and generating a second type of vector based on the first type of vector, wherein the dimension of the second type of vector is smaller than the dimension of the first type of vector, and the second type of vector is used to represent the deep features of the speech signal, and the deep features are used to describe the correlation between the acoustic features; analyzing and processing the target feature vector generated according to the first type of vector and the second type of vector to output the first result.

[0039] The shallow features of speech signals are information related to the acoustic features of speech signals. For example, the shallow features of speech signals include: the frequency of sound waves, such as the fundamental frequency representing the number of vibrations of sound waves per unit time, timbre, volume, the zero-crossing rate representing the number of times the speech signal model crosses the zero point in a time period, short-time energy (representing the energy of the speech signal in a preset time window), frequency cepstral coefficients (Mel-scale Frequency Cepstral Coefficients, MFCC) and other simple information. It is not comprehensive enough to use only these simple information to represent the pathological characteristics of Alzheimer's patients. Therefore, in this embodiment, the deep features and shallow features of the speech signal are fused and analyzed. Figure 3 is the workflow diagram of the first model, such as Figure 3As shown, the first model first extracts shallow features from the speech signal, and then mines the deep features of the speech signal from the extracted shallow features, and uses the fusion of shallow and deep features to retain the global information of the speech signal, and finally obtains a multi-level acoustic representation; by analyzing and processing the fused acoustic representation, it outputs "a label representing that the object to be detected is an Alzheimer's patient" (i.e., the first result) or "a label representing that the object to be detected is not an Alzheimer's patient" (i.e., the first result). In the above scheme, the first model extracts the shallow features of the input speech signal to obtain a vector representing the shallow features (i.e., a first-class vector); further, by extracting the dependency relationship between the shallow features, a lower-dimensional vector representing the deep features (i.e., a second-class vector) is generated; the first model can be a Transformer encoder or other neural network models with natural language processing capabilities. As Figure 3 As shown, when analyzing and processing the fused acoustic representation, the following method can be used: input the acoustic representation into the bidirectional long short-term memory neural network (BiLSTM) model for analysis. The BiLSTM model contains a forward long short-term memory neural network (LSTM) network and a backward long short-term memory neural network. The advantage of this structural method is that when processing the information at the current moment, the information at the future moment can be used, and the superposition of future and past information makes the network's learning ability more outstanding; therefore, in the process of processing the acoustic representation, BiLSTM uses a forward long short-term memory neural network to forwardly process the acoustic features of each moment in the order from the earliest moment to the latest moment; and uses a backward long short-term memory neural network to reversely process the acoustic features of each moment in the order from the latest moment to the earliest moment, and merges the output of the forward long short-term memory neural network with the output of the backward long short-term memory neural network to obtain complete context information. Still as Figure 3 As shown, the complete context information of the above output is input into the classifier (Softmax) for classification. Softmax classifies the acoustic features that are consistent with Alzheimer's patients and the acoustic features that are not consistent with Alzheimer's patients, and outputs the probability that the detection object is an Alzheimer's patient and the probability that the detection object is not an Alzheimer's patient. The first model will output the one with a higher probability value as the first result.

[0040] Optionally, generating a second type of vector based on the first type of vector includes: for each element in the first type of vector, determining the similarity between the element and other elements in the first type of vector, and obtaining multiple similarities corresponding to each element; determining the product of each similarity and the element, and determining the new element used to replace the element by the sum of the multiple products; and determining the vector composed of the multiple new elements as the second type of vector.

[0041] In this embodiment, the following method can be used to mine deep features from shallow features. For the vector representation of shallow features (i.e., the first type of vector), the similarity between each element and the remaining other elements in the (first type) vector is determined, and each element in the first type of vector corresponds to multiple similarities; further, the result of multiplying the element with each similarity and then adding them is the new element used to replace the element; through the above method, each element in the first type of vector is replaced by its corresponding new element to obtain a vector representing the deep features (i.e., the second type of vector).

[0042] According to another optional embodiment of the present application, the second model is trained by the following method: obtaining a speech segment, wherein the speech segment is generated based on a speech signal of an Alzheimer's patient; dividing the speech segment into a plurality of short speech segments in sequence, wherein a similarity between a previous short speech segment and a subsequent short speech segment is greater than or equal to a preset similarity, and each short speech segment contains a speech signal of a preset duration; iteratively training the neural network model using the plurality of short speech segments until the number of iterations reaches a preset number, and outputting a second model, wherein the second model learns the speech features of the speech signal of the Alzheimer's patient during the training process.

[0043] In order to avoid inaccurate recognition results caused by too little data of the input voice signal, in this embodiment, other voice recognition models different from the first model (for example, Wav2vec2.0) are also used to segment the voice signal and extract acoustic features from each segmented voice signal to determine whether the detected object is an Alzheimer's patient. After the second model is trained, it will know how to segment the speech signal when processing the speech signal. In this embodiment, during the training process, the second model arranges the input speech signal in the order of receiving time from early to late, generates a speech segment with time sequence, and segments the speech segment into multiple short speech segments with equal speech duration and shorter duration than the input speech segment (for example, all 4 seconds, or all 6 seconds), and during segmentation, it is ensured that the same speech signal exists in two speech segments adjacent in the segmentation order, so as to ensure that the similarity of two speech segments adjacent in the segmentation order is greater than the preset similarity (for example, 50%). Based on this, while expanding the data set, each speech segment retains transition information. Therefore, the second model can accurately identify the contextual relationship of the speech segment when extracting and analyzing the features of the speech segment; wherein, when segmenting, the transition information can be retained by using a 50% overlapping step before and after. The multiple short speech segments segmented by the above method are used as training data to iteratively train the second model until the number of iterations reaches the preset number, and it is determined that the training of the second model is completed. The training process of the second model includes two stages: the pre-training stage and the fine-tuning stage. In the pre-training stage, the second model first performs convolution and pooling operations on the original waveform of each audio clip, converting it into a high-dimensional feature vector. These high-dimensional feature vectors will be input into the quantizer and the context network respectively. The quantizer will convert the input continuous speech representation into a discrete sequence; and before the speech representation is sent to the context network, 50% of the speech representation will be masked, so that the previous short speech segment and the next speech segment have a 50% overlap effect. In the fine-tuning stage: The fine-tuning stage is to add an average pooling layer and a fully connected layer after the model. The average pooling layer performs pooling operations on the output of the second model, and the fully connected layer can convert the features into label representations (i.e., the second result).

[0044] According to some optional embodiments of the present application, a third model is used to process speech signals and text information to obtain a third result, including: performing speech processing on the speech signal to obtain speech features of the speech signal, wherein the speech features include: the number of silent sounds and the proportion of silent sounds, and the silent sound is a speech signal with a volume value less than a preset volume value and a duration less than a preset duration; converting the speech signal into text, and performing text processing on the text to obtain a first type of text features, wherein the first type of text features include: the number of verbs, the proportion of verbs, the number of nouns, the proportion of nouns, the number of quantifiers, the proportion of quantifiers, and the number of pause words; and extracting a second type of text features from the text, wherein the second type of text features are used to indicate the semantic content of the speech signal; fusing the speech features, the first type of text features, and the second type of text features to obtain a fusion result; analyzing and processing the fusion result to obtain a third result.

[0045] In the embodiment of the present application, the speech signal is also converted into text to perform text feature extraction, wherein the third model is a model different from the first model and the second model and has a natural language processing function. Figure 4 This is a schematic diagram of the third model processing speech signals, such as Figure 4 As shown, in this embodiment, the third model is used to perform speech recognition and text processing on the speech signal respectively. When the third model processes the speech signal, it mainly performs silence detection and extracts pause speech features in the speech signal, wherein silence is a speech signal with a volume value less than a preset volume value (for example, -50 decibels) and a duration less than a preset duration (for example, 500 milliseconds), and the pause speech features include: the number of silences contained in the speech signal and the proportion of silences. The text features extracted during the text processing process include: part-of-speech features (i.e., the first type of text features), text features representing the speech content of speech signals (a collection of the second type of text features), wherein the above-mentioned part-of-speech features include: the number of verbs, nouns, and quantifiers in each text, the number of pause words (such as "en"), and the proportion of verbs, nouns, and quantifiers in each text, and the proportion of pause words (such as "en"); silent pauses and filled pauses are considered to be important signs for distinguishing Alzheimer's patients. Although some errors may occur in the transcribed text, filler words such as "hmm" and "ah" generally do not make mistakes, so they can be extracted as part-of-speech features. Text features (a collection of the second type of text features) are in the form of a vector, which are used to represent the discourse-level text features of speech signals, such as sentence frequency, sentence length, word meaning, context, etc. Still like Figure 4 As shown, the third model combines the above-mentioned text features, part-of-speech features and voice features and then analyzes them to determine whether the detected object is an Alzheimer's patient.

[0046] Optionally, speech processing is performed on the speech signal to obtain speech features of the speech signal, including: determining the duration of each silent sound and determining the cumulative sum of the durations of multiple silent sounds; and determining the ratio of the cumulative sum to the duration of the speech signal as the ratio of the silent sounds.

[0047] In the above embodiment, when performing silence detection and extracting pause-like speech features in a speech signal, the silence ratio determined is the ratio of the sum of the durations of all silences to the duration of the entire speech.

[0048] The third model in the embodiment of the present application can be a bidirectional encoder based on a transformer (BidirectionalEncoder Representation from Transformers, BERT). Before the third model is applied to process speech signals, it needs to be pre-trained. For example, when the third model is BERT, the steps of its pre-training are as follows: The pre-training stage of BERT is completed by training on large-scale unlabeled text data, which comes from Alzheimer's patients. The pre-training stage can be divided into two stages: the first stage is to use an unsupervised method to train the language model, that is, given the front part of a paragraph of text, predict the words behind the paragraph of text; the second stage is to use a masked language model for pre-training, that is, randomly replace some words in a paragraph of text with special mask tags, and the model predicts these replaced words based on the context. Figure 5 This is a schematic diagram of the network structure of the BERT model, such as Figure 5As shown in the figure, the input of the BERT model consists of three parts: token embeddings (TE) encoding, segment embeddings (SE) encoding, and position embeddings (PE) encoding; TE, SE, and PE are used to represent the semantic information of words, the distinction information of sentences, and the position information of words, respectively; token embedding (TE) is the most basic embedding in the BERT model, which maps each word (token) to a high-dimensional vector space. This mapping process is usually performed using word vectors. When performing word embedding, BERT splits words into multiple subwords and maps each subword to a word vector. For a word, its token embedding is obtained by averaging the word vectors of all its subwords. Paragraph embedding (SE) is used to distinguish words in two sentences. When training the BERT model, it is usually necessary to concatenate two sentences into an input sequence. In this case, paragraph embedding (SE) is needed to identify which sentence each word belongs to. Specifically, the BERT model maps all the words in the first sentence to a vector in the vector space, maps all the words in the second sentence to another vector in the vector space, and then adds these two vectors to the token embedding of each word. Position embedding (PE) is used to represent the position information of each word in the input sequence. Since the BERT model is based on Transformer ( Figure 5 The Trm) model in it does not have the ability of sequential modeling like the recurrent neural network (RNN), so the position information needs to be embedded in the input vector. Specifically, for each word, the BERT model assigns it a vector representing its position in the input sequence, and the value of this vector is related to the position of the word in the input sequence. When the third model is applied for text processing, since the length of each sentence is inconsistent, in this embodiment, the third model is set to average all word-level features of each sentence, and finally obtain the discourse-level text features of the sentence, and send it to the BiLSTM network for learning, and finally classify it through Softmax to output the recognition result (ie, the third result).

[0049] Step S204: determine whether the target object is an Alzheimer's disease patient based on the first result, the second result and the third result.

[0050] In step S204, it is determined whether the object to be detected is an Alzheimer's patient based on the results output by different models. For example, a voting method can be adopted. When the number of recognition results indicating that the object to be detected is an Alzheimer's patient is greater than the number of recognition results indicating that the object to be detected is not an Alzheimer's patient, it is determined that the object to be detected is an Alzheimer's patient. Alternatively, a scoring method is adopted to pre-set corresponding weights for different models, add the weights of the models with the same output recognition results as the score of the recognition result, and determine the recognition result with a higher score as the final result.

[0051] Optionally, determining whether the target object is an Alzheimer's patient based on the first result, the second result, and the third result includes: determining a first number of results in the first result, the second result, and the third result indicating that the target object is an Alzheimer's patient, and determining a second number of results in the first result, the second result, and the third result that the target object is not an Alzheimer's patient; when the first number is greater than the second number, determining that the target object is an Alzheimer's patient; and when the first number is less than the second number, determining that the target object is not an Alzheimer's patient.

[0052] The method provided in the embodiment of the present application adopts a voting method to finally determine whether the object to be detected is an Alzheimer's patient according to multiple results output by multiple models. Specifically, in this embodiment, when determining the final result according to the number of recognition results, the following method can be adopted: classify according to the type of results output by each model, classify the type of "indicating that the object to be detected is an Alzheimer's patient" into one group, and classify the type of "indicating that the object to be detected is not an Alzheimer's patient" into another group; determine the final result by comparing the number of different types of results, for example, the number of results containing more The result corresponding to that category is determined as the final result. For example, when the (first) number of results included in the group "indicating that the object to be detected is a patient with Alzheimer's disease" is greater than the (second) number of results included in the group "indicating that the object to be detected is a patient with Alzheimer's disease", it is determined that the object to be detected is a patient with Alzheimer's disease; otherwise, if the (first) number of results included in the group "indicating that the object to be detected is a patient with Alzheimer's disease" is less than the (second) number of results included in the group "indicating that the object to be detected is not a patient with Alzheimer's disease", it is determined that the object to be detected is not a patient with Alzheimer's disease.

[0053] Through the above steps, the speech modality and text modality of the input speech signal are studied independently and integrated respectively. Among them, when the speech modality is studied independently, different speech recognition models are used. One speech recognition model obtains multi-dimensional pathological features by combining shallow information with deep information, so as to avoid the problem of inaccurate recognition caused by the shallow acoustic features being insufficient to fully express the oral damage information of Alzheimer's patients; another speech recognition model adopts the method of retaining transition information before and after to segment long audio, thereby expanding the data level and improving the accuracy of recognition results; and the model for the fusion processing of speech modality and text modality extracts text features such as syntax, part of speech, silent pauses, and filled pauses in the speech signal to complete the text information, thereby avoiding the problem of inaccurate discourse-level text features caused by transcription errors in the transcribed text; finally, in terms of the fusion of speech modality and text modality, the recognition of multiple models is voted on in a majority-minority manner to obtain the final result, thereby improving the accuracy of the recognition results and the robustness of the model.

[0054] Figure 6 is a structural diagram of a device for identifying Alzheimer's disease patients based on speech signals according to an embodiment of the present application, such as Figure 6 As shown, the device for identifying Alzheimer's patients based on voice signals includes: a processing module 60, which is used to obtain the voice signal of the target object, and use a first model to extract and process the shallow features and deep features of the voice signal to obtain a first result, use a second model to segment and process the voice signal to obtain a second result, and use a third model to process the voice signal and text information to obtain a third result, wherein the text information is obtained by transcribing the voice signal, and the first result, the second result and the third result are all used to indicate whether the target object is an Alzheimer's patient; a determination module 62, which is used to jointly determine whether the target object is an Alzheimer's patient based on the first result, the second result and the third result.

[0055] It should be noted that Figure 6 The preferred implementation of the illustrated embodiment can be found in Figure 2 The relevant description of the illustrated embodiment will not be repeated here.

[0056] An embodiment of the present application also provides a non-volatile storage medium, in which a computer program is stored, wherein the above method for identifying Alzheimer's patients based on voice signals is executed on a device where the non-volatile storage medium is located by running the computer program.

[0057] The above-mentioned non-volatile storage medium is used to store a program that performs the following functions: obtaining a speech signal of a target object, and using a first model to extract and process shallow features and deep features of the speech signal to obtain a first result, using a second model to segment and process the speech signal to obtain a second result, and using a third model to process the speech signal and text information to obtain a third result, wherein the text information is obtained by transcribing the speech signal, and the first result, the second result, and the third result are all used to indicate whether the target object is an Alzheimer's patient; and jointly determining whether the target object is an Alzheimer's patient based on the first result, the second result, and the third result.

[0058] In another aspect of an embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above method for identifying Alzheimer's patients based on speech signals through the computer program.

[0059] The processor in the above-mentioned electronic device is used to run a program that performs the following functions: obtain a speech signal of a target object, and use a first model to extract and process shallow features and deep features of the speech signal to obtain a first result, use a second model to segment and process the speech signal to obtain a second result, and use a third model to process the speech signal and text information to obtain a third result, wherein the text information is obtained by transcribing the speech signal, and the first result, the second result, and the third result are all used to indicate whether the target object is an Alzheimer's patient; and jointly determine whether the target object is an Alzheimer's patient based on the first result, the second result, and the third result.

[0060] The embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the above method for identifying Alzheimer's patients based on speech signals.

[0061] It should be noted that the various modules in the above-mentioned device for identifying Alzheimer's patients based on voice signals can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0062] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0063] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0064] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0065] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0066] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0067] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0068] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for identifying Alzheimer's disease patients based on speech signals, characterized in that: include: Acquire a speech signal of a target object, and use a first model to extract and process shallow features and deep features of the speech signal to obtain a first result, use a second model to segment and process the speech signal to obtain a second result, and use a third model to process the speech signal and text information to obtain a third result, wherein the text information is obtained by transcribing the speech signal, and the first result, the second result, and the third result are all used to indicate whether the target object is an Alzheimer's patient; It is determined whether the target object is an Alzheimer's disease patient based on the first result, the second result and the third result.

2. The method according to claim 1, characterized in that The first model is used to extract and process the shallow features and deep features of the speech signal to obtain a first result, including: Extracting shallow features of the speech signal, wherein the shallow features are used to describe acoustic features of the speech signal, and the acoustic features include: frequency, timbre, and volume; Generate a first type of vector according to the shallow features, and generate a second type of vector according to the first type of vector, wherein the dimension of the second type of vector is smaller than the dimension of the first type of vector, and the second type of vector is used to represent the deep features of the speech signal, and the deep features are used to describe the correlation between the acoustic features; The target feature vector generated according to the first type of vectors and the second type of vectors is analyzed and processed to output the first result.

3. The method according to claim 2, characterized in that Generating a second type of vector according to the first type of vector includes: For each element in the first type of vector, determine the similarity between the element and other elements in the first type of vector, and obtain multiple similarities corresponding to each element; Determine a product of each of the similarities and the element, and determine a new element for replacing the element by a sum of a plurality of the products; A vector consisting of a plurality of the new elements is determined as the second-type vector.

4. The method according to claim 1, characterized in that: The second model is trained by the following method: Acquire a speech segment, wherein the speech segment is generated according to a speech signal of an Alzheimer's disease patient; Dividing the speech segment into a plurality of short speech segments in sequence, wherein the similarity between a previous short speech segment and a subsequent short speech segment is greater than or equal to a preset similarity, and each short speech segment contains a speech signal of a preset duration; The neural network model is iteratively trained using the multiple short speech segments until the number of iterations reaches a preset number, and the second model is output, wherein the second model learns the speech features of the speech signals of Alzheimer's patients during the training process.

5. The method according to claim 1, characterized in that The voice signal and the text information are processed using a third model to obtain a third result, including: Performing voice processing on the voice signal to obtain voice features of the voice signal, wherein the voice features include: the number of silences and the proportion of silences, wherein the silences are voice signals whose volume value is less than a preset volume value and whose duration is less than a preset duration; Converting the speech signal into text and performing text processing on the text to obtain first-category text features, wherein the first-category text features include: the number of verbs, the proportion of verbs, the number of nouns, the proportion of nouns, the number of quantifiers, the proportion of quantifiers, and the number of pause words; and Extracting a second type of text feature from the text, wherein the second type of text feature is used to indicate the semantic content of the speech signal; fusing the speech feature, the first type of text feature, and the second type of text feature to obtain a fusion result; The fusion result is analyzed and processed to obtain the third result.

6. The method according to claim 5, characterized in that Performing speech processing on the speech signal to obtain speech features of the speech signal includes: Determine the duration of each of the silent sounds, and determine the cumulative sum of the durations of multiple silent sounds; The ratio of the accumulated sum to the duration of the speech signal is determined as the ratio of the silence.

7. The method according to claim 1, characterized in that Determining whether the target object is an Alzheimer's disease patient according to the first result, the second result, and the third result includes: Determine a first number of results indicating that the target object is an Alzheimer's disease patient among the first result, the second result, and the third result, and determine a second number of results indicating that the target object is not an Alzheimer's disease patient among the first result, the second result, and the third result; When the first number is greater than the second number, determining that the target object is an Alzheimer's disease patient; When the first number is smaller than the second number, it is determined that the target object is not an Alzheimer's disease patient.

8. A device for identifying Alzheimer's disease patients based on speech signals, characterized in that: include: A processing module, used to obtain a speech signal of a target object, and use a first model to extract and process shallow features and deep features of the speech signal to obtain a first result, use a second model to segment and process the speech signal to obtain a second result, and use a third model to process the speech signal and text information to obtain a third result, wherein the text information is obtained by transcribing the speech signal, and the first result, the second result, and the third result are all used to indicate whether the target object is an Alzheimer's patient; A determination module is used to determine whether the target object is an Alzheimer's disease patient based on the first result, the second result and the third result.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, wherein the device where the non-volatile storage medium is located executes the method for identifying Alzheimer's disease patients based on speech signals as described in any one of claims 1 to 7 by running the computer program.

10. An electronic device comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is configured to execute the method for identifying Alzheimer's disease patients based on speech signals according to any one of claims 1 to 7 through the computer program.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method for identifying Alzheimer's disease patients based on speech signals as described in any one of claims 1 to 7 are implemented.

Images