A method, system, device and storage medium for monitoring neurodegenerative diseases

Abstract

The application discloses a kind of neurodegenerative disease monitoring method, system, device and storage medium, wherein method includes: obtaining the speech information of speaker;According to the identity information of speaker identified by speech information, and the emotion of speaker is identified, speech information is converted into text information;According to the speech emotion obtained by identification and text information generation dialogue text, convert dialogue text into speech and output;According to the identity information obtained by identification, speech information is handled, and the speech information of same speaker is stored in same speech storage module;According to identity information, the speech corpus of corresponding speaker is obtained in speech storage module to diagnose neurodegenerative disease, and whether speaker is ill is judged.The application combines neurodegenerative disease screening with smart home, uses the problems that occur in the pronunciation of patient speech caused by neurodegenerative disease to detect disease, and can be widely applied in disease identification technical field.

Description

Technical Field

[0001] This invention relates to the field of disease identification technology, and in particular to a method, system, device, and storage medium for monitoring neurodegenerative diseases. Background Technology

[0002] Neurodegenerative diseases are caused by damage to neurons and their myelin sheaths. Once diagnosed, these diseases are often incurable, and the condition gradually worsens, causing enormous emotional and economic losses to patients and their families. However, early detection and treatment can significantly slow disease progression and reduce the socioeconomic costs (direct medical expenses, direct non-medical expenses, and indirect costs) borne by patients and society.

[0003] Currently, screening for neurodegenerative diseases requires specialized medical testing equipment and hospital visits, which is a complex and expensive process, making it unsuitable for early large-scale screening. Summary of the Invention

[0004] To at least partially address one of the technical problems existing in the prior art, the present invention aims to provide a method for monitoring neurodegenerative diseases based on voice dialogue interaction.

[0005] The technical solution adopted in this invention is:

[0006] A method for monitoring neurodegenerative diseases includes the following steps:

[0007] Obtain the speaker's voice information;

[0008] Based on the speech information, the speaker's identity information and the speaker's emotions are identified, and the speech information is converted into text information;

[0009] Based on the identified speech emotions and text information, a dialogue text is generated, which is then converted into speech and output.

[0010] The voice information is processed based on the identity information obtained from the identification, and the voice information of the same speaker is stored in the same voice storage module;

[0011] Based on the identity information, the corresponding speaker's voice data is retrieved from the voice storage module to diagnose neurodegenerative diseases, and to determine whether the speaker has a neurodegenerative disease and the severity of the disease.

[0012] Furthermore, the step of identifying the speaker's identity information based on voice information includes:

[0013] The pre-trained model WavLM-Large is used to extract speaker information features from speech information;

[0014] The extracted information features are input into the transformer model. The self-attention mechanism in the transformer model captures key information reflecting the speaker's identity from the information features, which is then used as the speaker's identity information to identify the speaker's identity.

[0015] Furthermore, based on the voice information, the speaker's emotions are identified, including:

[0016] The pre-trained model WavLM-Large is used to extract emotional information features from speech information; the emotional information features are the output features of the 12th layer transformer encoder in WavLM-Large.

[0017] The extracted emotional information features are input into the transformer model to perform global relation modeling on the features in order to identify the speaker's emotions;

[0018] The identified emotions include four types: calm, happiness, anger, and sadness.

[0019] Furthermore, the conversion of voice information into text information includes:

[0020] MFCC features are obtained from the speech information. First-order and second-order difference operations are performed on the MFCC features to obtain the first dynamic feature and the second dynamic feature. The MFCC features, the first dynamic feature and the second dynamic feature are concatenated to obtain the speech features.

[0021] The speech features are input into a transformer-based encoder-decoder model to encode and decode the speech features, and finally identify the text information corresponding to the speech.

[0022] Furthermore, the step of generating dialogue text based on the identified speech emotions and text information, converting the dialogue text into speech, and outputting it includes:

[0023] The identified emotional states and textual information are input into a BERT-based model to generate dialogue text for responses, and then the dialogue text is converted into speech output.

[0024] Furthermore, the voice storage module is used to store voice files within a preset time period;

[0025] The step of processing the voice information based on the identified identity information and storing the voice information of the same speaker in the same voice storage module includes:

[0026] Based on the identity information obtained through identification, the acquired voice information is classified and stored in the corresponding voice storage module.

[0027] Furthermore, the step of obtaining the corresponding speaker's speech data from the speech storage module based on identity information for neurodegenerative disease diagnosis, and determining whether the speaker suffers from a neurodegenerative disease and the severity of the disease, includes:

[0028] Based on the identity information obtained from the identification, the speaker's recent speech data is retrieved from the speech storage module;

[0029] The corpus data is segmented into multiple segments of speech data with preset durations;

[0030] Speech features are extracted from each segment of speech data using an open-source pre-trained model;

[0031] The extracted speech features are used to model global relationships using a transformer model to capture speech pronunciation problems caused by neurodegenerative diseases;

[0032] The results of multiple short speech recognitions are used to obtain the final judgment result, thereby identifying whether the speaker suffers from a neurodegenerative disease and the severity of the disease.

[0033] Another technical solution adopted in this invention is:

[0034] A neurodegenerative disease monitoring system, comprising:

[0035] The voice acquisition module is used to acquire the speaker's voice information;

[0036] The speech recognition module is used to identify the speaker's identity information and emotions based on the speech information, and to convert the speech information into text information.

[0037] The dialogue generation module is used to generate dialogue text based on the recognized speech emotions and text information, convert the dialogue text into speech and output it.

[0038] The voice storage module is used to process voice information based on the identity information obtained from recognition, and to store the voice information of the same speaker in the same voice storage module;

[0039] The speech analysis module is used to retrieve the corresponding speaker's speech data from the speech storage module based on the speaker's identity information to diagnose neurodegenerative diseases and determine whether the speaker has a neurodegenerative disease and the severity of the disease.

[0040] Another technical solution adopted in this invention is:

[0041] A neurodegenerative disease monitoring device, comprising:

[0042] At least one processor;

[0043] At least one memory for storing at least one program;

[0044] When the at least one program is executed by the at least one processor, the at least one processor implements the method described above.

[0045] Another technical solution adopted in this invention is:

[0046] A computer-readable storage medium storing a processor-executable program, which, when executed by a processor, performs the method described above.

[0047] The beneficial effects of this invention are as follows: This invention combines neurodegenerative disease screening with smart home technology. It utilizes the deterioration of the cognitive and language systems in the brain caused by neurodegenerative diseases, which leads to problems in the patient's speech pronunciation. It innovatively proposes to collect the speaker's natural speech data in a non-contact and unintentional manner through voice dialogue interaction. Then, it analyzes the speaker's recent speech data through a deep learning model to capture pronunciation problems in the speaker's speech, thereby improving the recognition accuracy. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a schematic diagram of the structure of a neurodegenerative disease monitoring system based on voice dialogue interaction in an embodiment of the present invention;

[0050] Figure 2 This is a flowchart illustrating the steps of a neurodegenerative disease monitoring method based on voice dialogue interaction in an embodiment of the present invention. Detailed Implementation

[0051] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0052] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0053] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0054] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0055] In the early stages of neurodegenerative diseases, the cognitive and language areas of the brain degenerate and atrophy, thereby affecting the patient's language and cognitive functions. Therefore, this invention proposes an early neurodegenerative disease monitoring system based on voice dialogue interaction. The advantage of this system is that it is suitable for large-scale screening and monitoring in communities and homes.

[0056] like Figure 1 As shown, this embodiment provides a neurodegenerative disease monitoring system based on voice dialogue interaction, including a voice acquisition module, a voice recognition module, a dialogue generation module, a voice storage module, and a voice analysis module.

[0057] The voice acquisition module is used to acquire the speaker's voice signal.

[0058] The speech recognition module identifies the speaker's identity information and emotions, and converts the speaker's speech content into text.

[0059] The text generation module uses the speaker's emotions and text content to generate dialogue content, and then outputs the dialogue content as speech to the speaker.

[0060] The voice storage module is responsible for storing the voice information collected by the voice acquisition module. The voice processing module is responsible for processing the speaker's recent voice recordings to determine whether the speaker suffers from a neurodegenerative disease.

[0061] As an optional implementation, the speaker identification unit in the speech recognition module first uses the open-source pre-trained model WavLM-Large to extract speaker information features from the speech signal collected by the speech acquisition module, and then inputs these features into the transformer model. The self-attention mechanism in the transformer can capture key information reflecting the speaker's identity in the features, thereby identifying the speaker's identity.

[0062] The speaker identification unit in the speech recognition module is trained on an open-source dataset. The model with the best generalization ability and recognition performance is selected as the speaker identification unit model of this system.

[0063] As an optional implementation, the emotion recognition unit in the speech recognition module first uses the open-source pre-trained model WavLM-Large to extract emotional information features from the speech signal acquired by the speech acquisition module. The speech features utilize the output of the 12th layer transformer encoder in WavLM-Large. This emotional feature is input into the transformer model to perform global relational modeling of the features, thereby recognizing the emotional information of the incoming speech.

[0064] The emotion recognition unit in the speech recognition module first prepares speech data of calm, happy, sad, and angry states as training and testing data for the model. All training data is input into the constructed model for training, and the model that performs best on the testing data is selected as the emotion recognition unit of this system.

[0065] As an optional implementation, the speech-to-text unit in the speech recognition module first extracts features from the input speech. In this unit, the extracted speech features are Mel Frequency Cepstrum Coefficients (MFCCs). MFCC parameters combine the auditory perception features of the human ear with the speech generation mechanism.

[0066] First, the input speech is framed with a frame length of 25 milliseconds and a frame shift of 10 milliseconds. The segmented speech becomes many smaller segments. Pre-emphasis processing is then applied to the segmented speech. The pre-emphasis formula is as follows:

[0067] a1[n]=s[n]-αs[n-1](0.9<α<1)

[0068] Where s[n] is the speech signal after framing, a1[n] is the speech signal after pre-emphasis, and α is the pre-emphasis coefficient, which is used to remove the influence of lip radiation and enhance the high-frequency resolution of speech.

[0069] Because computer signal processing can only handle signals of finite length, the original speech signal needs to be windowed to transform it into a finite-length speech signal. The pre-emphasized speech is convolved with a window function, specifically a Hamming window, as shown in the following formula:

[0070]

[0071] The formula for the speech windowing process is expressed as follows:

[0072]

[0073] Where a2[n] is the windowed speech signal.

[0074] After windowing the speech signal, the spectrum of the speech signal is calculated using the short-time Fourier transform. The short-time Fourier transform of signal a2[n] is defined as:

[0075]

[0076]

[0077] In the formula, w[n] is the Hamming window function.

[0078] After obtaining the spectrum, calculate the energy spectrum of the speech signal a2[n]. The energy spectrum is the square of the spectrum. The calculation formula is:

[0079]

[0080] After obtaining the power spectrum of the input speech, a Mel filter bank is designed to filter the energy spectrum. The parameters of the Mel filter bank are calculated as follows:

[0081] First, select a minimum frequency of 300Hz and a maximum frequency of 8kHz. Then, convert the minimum and maximum frequencies to their corresponding Mel-scale frequencies. The conversion formulas are as follows:

[0082]

[0083] After selecting the number L of filters in the Mel filter bank, L+2 points are needed to evenly distribute the frequency band in the Mel frequency space, and the Mel frequencies at these points are then converted back to their corresponding frequencies. The conversion formula is as follows:

[0084]

[0085] After obtaining the frequencies, for the i-th filter in the Mel filter, its starting point is the frequency of the i-th point, its peak point is the frequency of the (i+1)-th point, and its ending point is the frequency of the (i+2)-th point. The specific formula is expressed as:

[0086]

[0087] The energy spectrum is filtered using a Mel filter bank, and the calculation formula is as follows:

[0088]

[0089] Taking the logarithm of the output of each Mel-spectrum filter yields the logarithmic power spectrum at the corresponding frequency, followed by an inverse discrete cosine transform to obtain L MFCC coefficients. The calculation formula is as follows:

[0090]

[0091] The obtained MFCC features are subjected to first-order and second-order difference operations to obtain the dynamic features of MFCC. Finally, these three features are concatenated together to form the speech features input to this unit.

[0092] The extracted features are processed by an encoder-decoder model based on transformers to encode and decode the input speech features, ultimately identifying the text sequence of the input speech.

[0093] As an optional implementation, the text generation module uses the text data and sentiment tags obtained from the analysis of the input speech in the speech recognition module, inputs them into a BERT-based model, generates the text information of the response, and converts the text information into speech output.

[0094] The text generation module is trained on an open-source dataset, and the model with the best generalization performance is selected as the model used by the text generation module of this system.

[0095] As an optional implementation, the speech analysis module uses the speech recognition module to analyze the input speech to obtain the speaker's identity information, and then calls the speech storage module to read the speaker's recent speech data. The acquired speech data is then first subjected to data augmentation.

[0096] The data augmentation operation is as follows: the acquired speech data is first segmented into short speech data with a step size of 3 seconds and an interval of 6 seconds.

[0097] The obtained short speech data is processed using a pre-trained model to extract deep speech features. This module uses the WavLM-Large model as the pre-trained model, and the output of the 12th layer transformer encoder of the WavLM-Large model is used as the feature of the speech data to be recognized. For a 6-second short speech data set, the time dimension of the deep features obtained after processing by this pre-trained model is 299, and the feature dimension is 1024. In other words, a total of 299 tokens are extracted.

[0098] The acquired deep speech features are input into the model designed in this module for speech feature analysis and diagnosis of neurodegenerative diseases. The model designed in this module is based on the transformer.

[0099] The transformer model includes a self-attention mechanism and a feedforward network. The self-attention mechanism models from a global perspective, enabling the machine to notice the correlations between different parts of the input, thereby highlighting key local information. Patients with neurodegenerative diseases exhibit syllable stress shifts, high repetition at the word and character levels, and decreased fluency at the sentence level. These problems occur in localized areas of the patient's speech corpus. The application of the self-attention mechanism helps the machine locate these important regions.

[0100] The self-attention mechanism is calculated as follows:

[0101] The input speech features are projected and mapped into three matrices Q, K, and V.

[0102] For each token in the speech features, calculate its correlation with all tokens in the speech features. The calculation formula is:

[0103]

[0104] Where dh is the scaling factor to avoid problems caused by excessively large values; Softmax maps the relevance of each token to all tokens to a range between [0, 1]. Softmax calculation formula:

[0105]

[0106] Among them, z i Let C be the relevance result of the i-th token, and C be the total number of tokens.

[0107] After the correlation calculation, the correlation matrix and V are weighted and summed. The calculation formula is:

[0108]

[0109] After the self-attention mechanism is computed, the features are input into a feedforward network (FFN). The feedforward network introduces a non-linear transformation, thereby enhancing the model's fitting ability. The formula for FFN is as follows:

[0110] FFN(x)=FC(ReLU(FC(x)))

[0111] Where FC stands for fully connected layer, and ReLU is a type of activation function, expressed as follows:

[0112]

[0113] The complete formula for the transformer model is as follows:

[0114] x1 = LayerNorm(x + Attention(xW) Q ,xW K ,xW V ))

[0115] Transformer(x1)=LayerNorm(x1+FFN(x1))

[0116] Where x represents the input speech features, W Q W K W V The LayerNorm matrix represents the projection mapping. It removes the size relationships between different samples but preserves the size relationships between different features within a single sample. By stacking multiple layers of transformer models, the model can learn the temporal relationships of the input features.

[0117] After passing through multiple transformer layers, the input features are averaged over time using average pooling layers to obtain the global information feature x. mean The global information features are then input into a three-layer multilayer perceptron. The multilayer perceptron is responsible for classifying the learned features. The formula for the first-layer perceptron is as follows:

[0118] H1(x mean =ReLU(W1x) mean +B1)

[0119] Where W1 is a learnable matrix of dimension 512×1024, B1 is a learnable vector of dimension 512, and ReLU is the activation function.

[0120] The formula for the second-layer perceptron is as follows:

[0121] H2(H1) = ReLU(W2H1 + B2)

[0122] Where W2 is a learnable matrix of dimension 256×512, and B2 is a learnable vector of dimension 256.

[0123] The formula for the third-layer perceptron is as follows:

[0124] H3(H2) = ReLU(W3H2 + B3)

[0125] Where W3 is a learnable matrix of dimension 7×256, and B3 is a learnable vector of dimension 7.

[0126] After passing through a multilayer perceptron, the input speech signal is classified accordingly as: normal, early Alzheimer's disease, mid-to-late Alzheimer's disease, early Parkinson's disease, mid-to-late Parkinson's disease, early Huntington's disease, and mid-to-late Huntington's disease.

[0127] For the speech analysis module of the system, speech data from normal individuals, early-stage Alzheimer's patients, mid-to-late-stage Alzheimer's patients, early-stage Parkinson's patients, mid-to-late-stage Parkinson's patients, early-stage Huntington's patients, and mid-to-late-stage Huntington's patients are prepared during the training phase, and then processed according to the module operation described above. Model training updates system parameters using gradient descent. The model's loss function is the cross-entropy loss function, expressed as follows:

[0128]

[0129] Where d represents the d-th category; M represents the number of categories; i represents the i-th sample; N represents the number of samples; y id This is a sign function, taking values ​​of 0 or 1. It returns 1 if the label of the i-th sample is the same as the label of the d-th sample, and 0 if the i-th sample is not in the d-th sample category; p id Let be the predicted probability that observed sample i belongs to category d. The model with the best generalization ability and the best performance on the test data is selected as the model for the speech analysis module of this system after training.

[0130] In actual testing, after performing the above analysis on multiple short speech segments from the speaker, the majority voting method was used to determine the speaker's health status. The majority voting method involves: counting which category each of the short speech segments belongs to, counting the number of each category, and using the category with the most entries as the speaker's diagnosis result.

[0131] This invention discloses a voice-based interactive neurodegenerative disease monitoring system, primarily designed for large-scale community and family screening for neurodegenerative diseases, particularly in the early stages of the disease. For example, elderly individuals can converse with the system. During the conversation, the system automatically identifies the elderly person and provides appropriate responses based on their speech data, fulfilling a companionship function. Simultaneously, the system automatically performs comprehensive analysis of the elderly person's recent speech in the background to arrive at a diagnosis, thus fulfilling the disease monitoring function. The advantage of this system lies in its organic combination of companionship and disease monitoring functions. It enables contactless disease monitoring while the conversational approach alleviates the elderly person's anxiety and resistance to diagnosis, thus monitoring the speaker's disease implicitly.

[0132] like Figure 2 As shown, based on the above system, this embodiment also provides a method for monitoring neurodegenerative diseases based on voice dialogue interaction, including the following steps:

[0133] S1. Obtain the speaker's voice information;

[0134] S2. Based on the voice information, identify the speaker's identity information and the speaker's emotions, and convert the voice information into text information;

[0135] S3. Generate dialogue text based on the recognized speech emotion and text information, convert the dialogue text into speech and output it.

[0136] S4. Process the voice information based on the identity information obtained from the recognition, and store the voice information of the same speaker in the same voice storage module;

[0137] S5. Based on the identity information, retrieve the corresponding speaker's voice data from the voice storage module to diagnose neurodegenerative diseases, and determine whether the speaker has a neurodegenerative disease and the severity of the disease.

[0138] The neurodegenerative disease monitoring method based on voice dialogue interaction in this embodiment corresponds to the aforementioned neurodegenerative disease monitoring system based on voice dialogue interaction, and therefore possesses the corresponding functions and beneficial effects of that system.

[0139] This embodiment also provides a neurodegenerative disease monitoring device, including:

[0140] At least one processor;

[0141] At least one memory for storing at least one program;

[0142] When the at least one program is executed by the at least one processor, the at least one processor performs the following: Figure 2The method shown.

[0143] This embodiment of a neurodegenerative disease monitoring device can execute a voice-based interactive neurodegenerative disease monitoring method provided in the method embodiment of the present invention. It can execute any combination of the implementation steps of the method embodiment and has the corresponding functions and beneficial effects of the method.

[0144] This application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform... Figure 2 The method shown.

[0145] This embodiment also provides a storage medium storing a method embodiment of the present invention that can execute the provided method embodiment. Figure 2 The instructions or program of the method shown can execute any combination of implementation steps of the method embodiment when the instructions or program are run, and have the corresponding functions and beneficial effects of the method.

[0146] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0147] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0148] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0149] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0150] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0151] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0152] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0153] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0154] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for monitoring neurodegenerative diseases, characterized in that, Includes the following steps: Obtain the speaker's voice information; Based on the speech information, the speaker's identity information and the speaker's emotions are identified, and the speech information is converted into text information; The dialogue text is generated based on the identified emotional state and text information, converted into speech and output; the speech information is processed based on the identified identity information, and the speech information of the same speaker is stored in the same speech storage module. Based on the identity information, the speech data of the corresponding speaker is retrieved from the speech storage module for the diagnosis of neurodegenerative diseases. And determine whether the speaker suffers from a neurodegenerative disease and the severity of the disease; Identifying the speaker's emotions based on voice information includes: The pre-trained model WavLM-Large is used to extract emotional information features from speech information; the emotional information features are the output features of the 12th layer transformer encoder in WavLM-Large. The extracted emotional information features are input into the transformer model to perform global relation modeling on the features in order to identify the speaker's emotions; Among them, the identified emotions include four types: calm, happiness, anger, and sadness. The process of generating dialogue text based on the identified emotional state and text information, converting the dialogue text into speech, and outputting it includes: The identified emotional states and textual information are input into a BERT-based model to generate dialogue text for responses, and then the dialogue text is converted into speech output.

2. The method for monitoring neurodegenerative diseases according to claim 1, characterized in that, The step of identifying the speaker's identity information based on voice information includes: Pre-trained models are used to extract speaker information features from speech information; The extracted information features are input into the transformer model. The self-attention mechanism in the transformer model captures key information reflecting the speaker's identity from the information features, which is then used as the speaker's identity information to identify the speaker's identity.

3. The method for monitoring neurodegenerative diseases according to claim 1, characterized in that, The process of converting speech information into text information includes: MFCC features are obtained from the speech information. First-order and second-order difference operations are performed on the MFCC features to obtain the first dynamic feature and the second dynamic feature. The MFCC features, the first dynamic feature and the second dynamic feature are concatenated to obtain the speech features. The speech features are input into a transformer-based encoder-decoder model to encode and decode the speech features, and finally identify the text information corresponding to the speech.

4. The method for monitoring neurodegenerative diseases according to claim 1, characterized in that, The step of processing the voice information based on the identified identity information and storing the voice information of the same speaker in the same voice storage module includes: Based on the identity information obtained through identification, the acquired voice information is classified and stored in the corresponding voice storage module.

5. The method for monitoring neurodegenerative diseases according to claim 1, characterized in that, The step of retrieving the corresponding speaker's speech data from the speech storage module based on identity information for neurodegenerative disease diagnosis, and determining whether the speaker suffers from a neurodegenerative disease and the severity of the disease, includes: Based on the identity information obtained from the recognition, the speech data of the speaker for a preset time is retrieved from the speech storage module; The corpus data is segmented into multiple segments of speech data with preset durations; Speech features are extracted from each segment of speech data using an open-source pre-trained model; The extracted speech features are used to model global relationships using a transformer model to capture speech pronunciation problems caused by neurodegenerative diseases; The results of multiple short speech recognitions are used to obtain the final judgment result, thereby identifying whether the speaker suffers from a neurodegenerative disease and the severity of the disease.

6. A neurodegenerative disease monitoring system, applied to the neurodegenerative disease monitoring method according to any one of claims 1-5, characterized in that, include: The voice acquisition module is used to acquire the speaker's voice information; The speech recognition module is used to identify the speaker's identity information and emotions based on the speech information, and to convert the speech information into text information. The dialogue generation module is used to generate dialogue text based on the recognized speech emotions and text information, convert the dialogue text into speech and output it. The voice storage module is used to process voice information based on the identity information obtained from recognition, and to store the voice information of the same speaker in the same voice storage module; The speech analysis module is used to retrieve the corresponding speaker's speech data from the speech storage module based on the speaker's identity information to diagnose neurodegenerative diseases and determine whether the speaker has a neurodegenerative disease and the severity of the disease.

7. A monitoring device for neurodegenerative diseases, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method of any one of claims 1-5.

8. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the method as described in any one of claims 1-5.

Images