Cerebral stroke rehabilitation evaluation method and system based on voice multi-task learning
By combining voice multi-task learning with the recognition of psychological emotions and physiological functions, a multi-task learning mechanism is constructed, which solves the problem of neglecting the patient's subjective feelings in existing technologies, and improves the individual accuracy of stroke rehabilitation assessment and user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- THE FIRST AFFILIATED HOSPITAL OF WANNAN MEDICAL COLLEGE (YIJISHAN HOSPITAL OF WANNAN MEDICAL COLLEGE)
- Filing Date
- 2023-12-07
- Publication Date
- 2026-04-21
AI Technical Summary
Existing voice-based stroke rehabilitation assessment methods rely solely on objective information for functional evaluation, neglecting patients' subjective feelings. This leads to reduced individual accuracy in rehabilitation assessments and insufficient user experience.
A speech-based multi-task learning approach is adopted, which combines psychological and emotional recognition with physiological function recognition to construct a multi-task learning mechanism that adaptively combines subjective and objective assessment tasks to generate comprehensive rehabilitation assessment results.
This approach allows for a balance between patients' subjective and objective feelings during rehabilitation assessments, improving the individual accuracy of the assessments and enhancing the user experience.
Smart Images
Figure CN121890936A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of rehabilitation assessment technology, specifically to a method and system for stroke rehabilitation assessment based on speech multitasking learning. Background Technology
[0002] Stroke is an acute cerebrovascular disease caused by the rupture or blockage of blood vessels in the brain, resulting in brain nerve damage. It has a high incidence and disability rate. Surveys show that stroke is the leading cause of death in my country, and the number of stroke deaths in my country accounts for about one-third of the global total. Stroke patients often experience symptoms such as unclear speech and incoherent speech, severely impacting their daily lives.
[0003] Existing speech-based stroke detection methods mainly rely on feature engineering, which generates time alignment information between audio and text through a pre-trained speech recognition model, thereby calculating features that reflect the accuracy and fluency of pronunciation, such as the quality of speech pronunciation and the number of syllables per unit time.
[0004] However, in existing technologies, stroke rehabilitation evaluation based on patient voice using deep learning models is an objective assessment that mines the patient's physiological language function reflected in the voice. This type of functional assessment, which is based solely on objective voice information, ignores the patient's subjective feelings. This makes it difficult for the patient rehabilitation assessment to perceive the pain and discomfort experienced by the stroke patient, ultimately resulting in reduced individual accuracy of stroke rehabilitation assessment and an unsatisfactory user experience. Summary of the Invention
[0005] The purpose of this invention is to provide a stroke rehabilitation assessment method and system based on speech multi-task learning, in order to solve the technical problem that the existing functional assessments based only on objective speech information ignore the patient's subjective feelings, making it difficult for the patient rehabilitation assessment to perceive the pain and discomfort of the stroke patient, and ultimately reducing the individual accuracy of stroke rehabilitation assessment.
[0006] To solve the above-mentioned technical problems, the present invention specifically provides the following technical solution:
[0007] A stroke rehabilitation assessment method based on speech multi-task learning includes the following steps:
[0008] Acquire the speech data of the assessment subject, and perform specific extraction on the speech data to obtain assessment speech data for rehabilitation assessment;
[0009] By performing psychological and emotional recognition on the assessment voice data, a subjective assessment task for stroke rehabilitation is obtained to perceive the subjective rehabilitation feelings of the assessment subjects.
[0010] By performing physiological function recognition on the assessment voice data, an objective assessment task for stroke rehabilitation is obtained to perceive the objective rehabilitation status of the assessment subject.
[0011] A stroke rehabilitation assessment model was obtained by adaptively combining the weights of subjective and objective assessment tasks for stroke rehabilitation using a multi-task learning mechanism.
[0012] Finally, the stroke rehabilitation assessment model outputs the rehabilitation assessment results of the assessed subjects.
[0013] As a preferred embodiment of the present invention, the speech data is specifically extracted to obtain assessment speech data for rehabilitation assessment, including:
[0014] The voice data is divided into multiple voice segments according to a preset length;
[0015] Multiple speech segment data are clustered to obtain multiple speech segment sets;
[0016] The speech segment data that is closest to the center of each speech segment set is selected as the specific speech data;
[0017] The specific speech data of each speech segment set is processed sequentially into Mel spectrograms to obtain the Mel spectrograms of the specific speech data.
[0018] The Mel spectrograms of specific speech data are processed by first-order and second-order difference to obtain the first-order and second-order difference results of the Mel spectrograms of specific speech data.
[0019] The Mel spectrogram of the specific speech data, the first-order difference, and the second-order difference results of the Mel spectrogram of the specific speech data are superimposed to obtain the three-channel Mel spectrogram of the specific speech data.
[0020] The three-channel Mel spectrogram of the specific speech data is used as the evaluation speech data.
[0021] As a preferred embodiment of the present invention, a stroke rehabilitation subjective assessment task is obtained by performing psychological and emotional recognition on the assessment voice data, which is used to perceive the subjective rehabilitation feelings of the assessment subject, including:
[0022] A set of sample speech is obtained from the speech database, and each sample speech is labeled according to the emotional state of the speech representation to obtain the emotional label of each sample speech.
[0023] The three-channel Mel-spectrum of the sample speech was used as the input to a CNN neural network, and the emotion label of the sample speech was used as the output. The CNN neural network was then trained on the input and output to construct an emotion recognition model.
[0024] The emotion recognition model is as follows:
[0025] Label_e = CNN(G_3channel);
[0026] In the formula, Label_e is the emotion label, G_3channel is the three-channel Mel spectrogram of the sample speech, and CNN is the CNN neural network;
[0027] The loss function of the emotion recognition model is: loss_subj=MSE(Label_e,Label_e_real), where loss_subj is the loss function of the emotion recognition model, MSE is the mean squared error function, Label_e is the emotion label output by the emotion recognition model, and Label_e_real is the true value of the emotion label of the sample speech.
[0028] An emotion recognition model is used to identify the psychological emotions in the assessment speech data as a subjective assessment task for stroke rehabilitation. The output of the emotion recognition model is the stroke rehabilitation assessment result of the assessment subject in the subjective assessment task for stroke rehabilitation.
[0029] As a preferred embodiment of the present invention, by performing physiological function recognition on the assessment voice data, an objective assessment task for stroke rehabilitation, used to perceive the objective rehabilitation status of the assessment subject, is obtained, including:
[0030] A set of sample speech is obtained from the speech library, and each sample speech is labeled according to the functional state of the speech representation to obtain the functional label of each sample speech.
[0031] The three-channel Mel-spectrum of the sample speech is used as the input to the softmax classifier, and the function label of the sample speech is used as the output of the softmax classifier. The softmax classifier is then used to train the input and output of the softmax classifier to construct a function recognition model.
[0032] The functional recognition model is as follows:
[0033] Label_e=sofrmax(G_3channel);
[0034] In the formula, Label_f is the functional label, G_3channel is the three-channel Mel spectrogram of the sample speech, and softmax is the softmax classifier;
[0035] The loss function of the function recognition model is: loss_obj=MSE(Label_f,Label_f_real), where loss_obj is the loss function of the function recognition model, MSE is the mean squared error function, Label_f is the function label output by the function recognition model, and Label_f_real is the true value of the function label of the sample speech.
[0036] The functional recognition model performs physiological function recognition on the assessment speech data as an objective assessment task for stroke rehabilitation. The output of the functional recognition model is the stroke rehabilitation assessment result of the assessment subject by the objective assessment task for stroke rehabilitation.
[0037] As a preferred embodiment of the present invention, a multi-task learning mechanism is used to adaptively combine the weights of subjective assessment tasks and objective assessment tasks for stroke rehabilitation to obtain a stroke rehabilitation assessment model, including:
[0038] By utilizing a multi-task learning mechanism, the input layer and bottom network layer of the subjective assessment task and the objective assessment task of stroke rehabilitation are shared. By adaptively combining the loss function with weights, a stroke rehabilitation assessment model with dynamic task priority is obtained.
[0039] The stroke rehabilitation assessment model is as follows:
[0040]
[0041] In the formula, G_3channel is the three-channel Mel spectrogram of the sample speech, Label_e is the emotion label, Label_f is the function label, softmax is the softmax classifier, and CNN is the CNN neural network.
[0042] The loss function of the stroke rehabilitation assessment model is:
[0043] loss=∑[w_e(t)*loss_subj(t)+w_f(t)*loss_obj(t)];
[0044] w_e(t)=-[1-K_e(t)] r_e logK_e(t);
[0045] w_f(t)=-[1-K_f(t)] r_f logK_f(t);
[0046] In the formula, loss is the loss function of the stroke rehabilitation assessment model, w_e(t) is the weight of the loss function of the emotion recognition model when the stroke rehabilitation assessment model has been trained for t iterations, w_f(t) is the weight of the loss function of the function recognition model when the stroke rehabilitation assessment model has been trained for t iterations, loss_subj(t) is the loss function of the emotion recognition model when the stroke rehabilitation assessment model has been trained for t iterations, loss_obj(t) is the loss function of the function recognition model when the stroke rehabilitation assessment model has been trained for t iterations, K_e(t) is the model KPI of the emotion recognition model when the stroke rehabilitation assessment model has been trained for t iterations, K_f(t) is the model KPI of the function recognition model when the stroke rehabilitation assessment model has been trained for t iterations, r_e is the manually adjusted parameter of w_e(t), r_f is the manually adjusted parameter of w_f(t), t is the number of training iterations, and the model KPI is at least one of model accuracy, model precision, model recall, F1 score and ROC curve.
[0047] As a preferred embodiment of the present invention, the stroke rehabilitation assessment model outputs the rehabilitation assessment results of the assessed subject, including:
[0048] The assessment voice data is input into the stroke rehabilitation assessment model, the number of iterations is preset, and the stroke rehabilitation assessment model is iteratively trained to obtain the stroke rehabilitation assessment results of the subjective assessment task and the stroke rehabilitation assessment results of the objective assessment task.
[0049] The stroke rehabilitation assessment results of the subjects were averaged using both the subjective assessment task and the objective assessment task to obtain the rehabilitation assessment results of the subjects.
[0050] As a preferred embodiment of the present invention, the size of the three-channel Mel spectrum is 64*64 pixels.
[0051] As a preferred embodiment of the present invention, the present invention provides an assessment system applying the aforementioned speech-based multi-task learning-based stroke rehabilitation assessment method, comprising:
[0052] A data preprocessing unit is used to acquire the speech data of the assessment subject and perform specific extraction on the speech data to obtain assessment speech data for rehabilitation assessment.
[0053] The multi-task learning unit is used to obtain a stroke rehabilitation subjective assessment task for perceiving the subjective rehabilitation feelings of the assessment subject by performing psychological and emotional recognition on the assessment voice data.
[0054] By performing physiological function recognition on the assessment voice data, an objective assessment task for stroke rehabilitation is obtained to perceive the objective rehabilitation status of the assessment subject.
[0055] A stroke rehabilitation assessment model was obtained by adaptively combining the weights of subjective and objective assessment tasks for stroke rehabilitation using a multi-task learning mechanism.
[0056] The assessment unit is used to output the rehabilitation assessment results of the assessment subject using the stroke rehabilitation assessment model.
[0057] As a preferred embodiment of the present invention, the multi-task learning unit performs psychological and emotional recognition on the assessment speech data to obtain a stroke rehabilitation subjective assessment task for perceiving the subjective rehabilitation feelings of the assessment subject, including:
[0058] A set of sample speech is obtained from the speech database, and each sample speech is labeled according to the emotional state of the speech representation to obtain the emotional label of each sample speech.
[0059] The three-channel Mel-spectrum of the sample speech was used as the input to a CNN neural network, and the emotion label of the sample speech was used as the output. The CNN neural network was then trained on the input and output to construct an emotion recognition model.
[0060] The emotion recognition model is as follows:
[0061] Label_e = CNN(G_3channel);
[0062] In the formula, Label_e is the emotion label, G_3channel is the three-channel Mel spectrogram of the sample speech, and CNN is the CNN neural network;
[0063] The loss function of the emotion recognition model is: loss_subj=MSE(Label_e,Label_e_real), where loss_subj is the loss function of the emotion recognition model, MSE is the mean squared error function, Label_e is the emotion label output by the emotion recognition model, and Label_e_real is the true value of the emotion label of the sample speech.
[0064] An emotion recognition model is used to identify the psychological emotions in the assessment speech data as a subjective assessment task for stroke rehabilitation. The output of the emotion recognition model is the stroke rehabilitation assessment result of the assessment subject in the subjective assessment task for stroke rehabilitation.
[0065] The multi-task learning unit performs physiological function recognition on the assessment speech data to obtain an objective assessment task for stroke rehabilitation, used to perceive the objective rehabilitation status of the assessment subject, including:
[0066] A set of sample speech is obtained from the speech library, and each sample speech is labeled according to the functional state of the speech representation to obtain the functional label of each sample speech.
[0067] The three-channel Mel-spectrum of the sample speech is used as the input to the softmax classifier, and the function label of the sample speech is used as the output of the softmax classifier. The softmax classifier is then used to train the input and output of the softmax classifier to construct a function recognition model.
[0068] The functional recognition model is as follows:
[0069] Label_e=sofrmax(G_3channel);
[0070] In the formula, Label_f is the functional label, G_3channel is the three-channel Mel spectrogram of the sample speech, and softmax is the softmax classifier;
[0071] The loss function of the function recognition model is: loss_obj=MSE(Label_f,Label_f_real), where loss_obj is the loss function of the function recognition model, MSE is the mean squared error function, Label_f is the function label output by the function recognition model, and Label_f_real is the true value of the function label of the sample speech.
[0072] The functional recognition model performs physiological function recognition on the assessment speech data as an objective assessment task for stroke rehabilitation. The output of the functional recognition model is the stroke rehabilitation assessment result of the assessment subject by the objective assessment task for stroke rehabilitation.
[0073] As a preferred embodiment of the present invention, the multi-task learning unit utilizes a multi-task learning mechanism to adaptively combine the weights of subjective assessment tasks and objective assessment tasks for stroke rehabilitation, thereby obtaining a stroke rehabilitation assessment model, including:
[0074] By utilizing a multi-task learning mechanism, the input layer and bottom network layer of the subjective assessment task and the objective assessment task of stroke rehabilitation are shared. By adaptively combining the loss function with weights, a stroke rehabilitation assessment model with dynamic task priority is obtained.
[0075] The stroke rehabilitation assessment model is as follows:
[0076]
[0077] In the formula, G_3channel is the three-channel Mel spectrogram of the sample speech, Label_e is the emotion label, Label_f is the function label, softmax is the softmax classifier, and CNN is the CNN neural network.
[0078] The loss function of the stroke rehabilitation assessment model is:
[0079] loss=∑[w_e(t)*loss_subj(t)+w_f(t)*loss_obj(t)];
[0080] w_e(t)=-[1-K_e(t)] r_e logK_e(t);
[0081] w_f(t)=-[1-K_f(t)] r_f logK_f(t);
[0082] In the formula, loss is the loss function of the stroke rehabilitation assessment model, w_e(t) is the weight of the loss function of the emotion recognition model when the stroke rehabilitation assessment model has been trained for t iterations, w_f(t) is the weight of the loss function of the function recognition model when the stroke rehabilitation assessment model has been trained for t iterations, loss_subj(t) is the loss function of the emotion recognition model when the stroke rehabilitation assessment model has been trained for t iterations, loss_obj(t) is the loss function of the function recognition model when the stroke rehabilitation assessment model has been trained for t iterations, K_e(t) is the model KPI of the emotion recognition model when the stroke rehabilitation assessment model has been trained for t iterations, K_f(t) is the model KPI of the function recognition model when the stroke rehabilitation assessment model has been trained for t iterations, r_e is the manually adjusted parameter of w_e(t), r_f is the manually adjusted parameter of w_f(t), t is the number of training iterations, and the model KPI is at least one of model accuracy, model precision, model recall, F1 score and ROC curve.
[0083] Compared with the prior art, the present invention has the following advantages:
[0084] This invention utilizes a multi-task learning mechanism to combine a subjective assessment task for stroke rehabilitation, which is used to perceive the subjective rehabilitation experience of the assessment subject, with an objective assessment task for stroke rehabilitation, which is used to perceive the objective rehabilitation status of the assessment subject. This allows for joint rehabilitation assessment, taking into account both the objective functional status and subjective emotional experience reflected in the patient's speech during the assessment. It enables the perception of the pain and discomfort experienced by stroke patients during functional assessment, thereby improving the individual accuracy of stroke rehabilitation assessment and enhancing the user experience. Attached Figure Description
[0085] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.
[0086] Figure 1 A flowchart of a stroke rehabilitation assessment method based on speech multitasking learning provided in an embodiment of the present invention;
[0087] Figure 2 A system block diagram of the evaluation method provided in the embodiments of the present invention. Detailed Implementation
[0088] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0089] like Figure 1 As shown, this invention provides a stroke rehabilitation assessment method based on speech multi-task learning, comprising the following steps:
[0090] Acquire the speech data of the assessment subjects and perform specific extraction on the speech data to obtain assessment speech data for rehabilitation assessment;
[0091] By performing psychological and emotional recognition on the assessment voice data, a subjective assessment task for stroke rehabilitation is obtained to perceive the subjective rehabilitation feelings of the assessment subjects.
[0092] By performing physiological function recognition on the assessment voice data, an objective assessment task for stroke rehabilitation is obtained to perceive the objective rehabilitation status of the assessment subject.
[0093] A stroke rehabilitation assessment model was obtained by adaptively combining the weights of subjective and objective assessment tasks for stroke rehabilitation using a multi-task learning mechanism.
[0094] Finally, the stroke rehabilitation assessment model outputs the rehabilitation assessment results of the assessed subjects.
[0095] In order to objectively evaluate the rehabilitation of the language function of stroke patients, this invention can also perceive the autonomous feelings of stroke patients and learn about the pain or discomfort they experience during the rehabilitation process. This allows for a comprehensive assessment of stroke rehabilitation, respecting the subjective feelings of stroke patients while understanding the objective situation of their speech function rehabilitation, and achieving individualized rehabilitation evaluation. This approach considers both subjective and objective aspects of stroke rehabilitation evaluation.
[0096] Specifically, if we only use objective language function to assess stroke rehabilitation, for example, if a patient's language rehabilitation is very good, but the patient also reports strong discomfort or pain, the evaluation based solely on language function would give the patient a very high rehabilitation rating. However, in reality, the patient's strong discomfort and pain indicate that the patient's rehabilitation is not good, or that it is not as good as the language function suggests. Therefore, from a practical perspective, using only objective language function to assess stroke rehabilitation has the flaw of incomplete assessment, ultimately limiting the accuracy of the individual assessment. Similarly, using only subjective feelings to assess stroke rehabilitation also has the flaw of incomplete assessment, ultimately limiting the accuracy of the individual assessment.
[0097] In summary, this invention, which conducts stroke rehabilitation assessment at both the objective and subjective levels, is of practical significance and can make the rehabilitation assessment more comprehensive and accurate.
[0098] Furthermore, in order to combine the objective and subjective levels in stroke rehabilitation assessment, this invention sets up assessment tasks at both the subjective and objective levels. Specifically, there is a subjective assessment task for stroke rehabilitation to perceive the subjective rehabilitation experience of the assessment subject, and an objective assessment task for stroke rehabilitation to perceive the objective rehabilitation status of the assessment subject. Objective functional assessment and subjective experience assessment of stroke are carried out using two assessment task lines respectively, thereby achieving stroke rehabilitation assessment that combines the objective and subjective levels.
[0099] Specifically, this invention utilizes a multi-task learning mechanism to fuse two assessment task lines. Multi-task learning trains the two assessment tasks simultaneously, improving the performance of each task through parameter sharing and interactive learning. Furthermore, multi-task learning combines the two assessment tasks into a single model, reducing the training and inference burden of multiple independent models, improving computational efficiency, and ultimately enhancing the generalization ability and efficiency of rehabilitation assessment.
[0100] Furthermore, this invention employs a dynamic task priority approach to combine two evaluation tasks. When both tasks are trained simultaneously, the more difficult task is given a higher learning priority, enhancing the learning adaptability of both evaluation tasks. Adjustments are made based on the learning stage, difficulty level, and even learning effect of different tasks, ultimately improving the learning effect of each evaluation task simultaneously.
[0101] This invention, when conducting stroke rehabilitation assessments at both the objective and subjective levels, identifies psychological and emotional states and physiological functions from the patient's speech. Therefore, whether conducting stroke rehabilitation assessments at the objective or subjective level, it is necessary to analyze and recognize speech data. To enhance stroke rehabilitation assessments at both the objective and subjective levels, it is necessary to obtain effective speech data for identifying psychological and emotional states and physiological functions (i.e., assessment speech data), as detailed below:
[0102] Specific extraction is performed on the speech data to obtain assessment speech data for rehabilitation evaluation, including:
[0103] The voice data is divided into multiple voice segments according to a preset length.
[0104] Multiple speech segment data are clustered to obtain multiple speech segment sets;
[0105] The speech segment data that is closest to the center of each speech segment set is selected as the specific speech data;
[0106] The specific speech data of each speech segment set is processed sequentially into Mel spectrograms to obtain the Mel spectrograms of the specific speech data.
[0107] The Mel spectrograms of specific speech data are processed by first-order and second-order difference to obtain the first-order and second-order difference results of the Mel spectrograms of specific speech data.
[0108] The Mel spectrogram of the specific speech data, the first-order difference, and the second-order difference results of the Mel spectrogram of the specific speech data are superimposed to obtain the three-channel Mel spectrogram of the specific speech data.
[0109] Three-channel Mel spectrograms of specific speech data were used as evaluation speech data.
[0110] This invention utilizes clustering for speech data preprocessing, which can effectively identify speech data capable of psychological and emotional recognition and physiological function recognition. From a data processing perspective, this enhances the accuracy of subsequent objective and subjective stroke rehabilitation assessments.
[0111] In order to balance the objective and subjective aspects of stroke rehabilitation assessment, this invention sets assessment tasks at both the subjective and objective levels, as follows:
[0112] By performing psycho-emotional recognition on the assessment speech data, a subjective assessment task for stroke rehabilitation, used to perceive the subjective rehabilitation experience of the assessment subject, is obtained, including:
[0113] A set of sample speech is obtained from the speech database, and each sample speech is labeled according to the emotional state of the speech representation to obtain the emotional label of each sample speech.
[0114] The three-channel Mel-spectrum of the sample speech was used as the input to a CNN neural network, and the emotion label of the sample speech was used as the output. The CNN neural network was then trained on the input and output to construct an emotion recognition model.
[0115] The emotion recognition model is as follows:
[0116] Label_e = CNN(G_3channel);
[0117] In the formula, Label_e is the emotion label, G_3channel is the three-channel Mel spectrogram of the sample speech, and CNN is the CNN neural network;
[0118] The loss function of the emotion recognition model is: loss_subj=MSE(Label_e,Label_e_real), where loss_subj is the loss function of the emotion recognition model, MSE is the mean squared error function, Label_e is the emotion label output by the emotion recognition model, and Label_e_real is the true value of the emotion label of the sample speech.
[0119] The emotion recognition model is used to identify the psychological emotions in the assessment speech data as a subjective assessment task for stroke rehabilitation. The output of the emotion recognition model is the stroke rehabilitation assessment result of the assessment subject in the subjective assessment task for stroke rehabilitation.
[0120] By performing physiological function recognition on the assessment speech data, an objective assessment task for stroke rehabilitation, used to perceive the objective rehabilitation status of the assessment subject, is obtained, including:
[0121] A set of sample speech is obtained from the speech library, and each sample speech is labeled according to the functional state of the speech representation to obtain the functional label of each sample speech.
[0122] The three-channel Mel-spectrum of the sample speech is used as the input to the softmax classifier, and the function label of the sample speech is used as the output of the softmax classifier. The softmax classifier is then used to train the input and output of the softmax classifier to construct a function recognition model.
[0123] The functional recognition model is as follows:
[0124] Label_e=sofrmax(G_3channel);
[0125] In the formula, Label_f is the functional label, G_3channel is the three-channel Mel spectrogram of the sample speech, and softmax is the softmax classifier;
[0126] The loss function of the function recognition model is: loss_obj=MSE(Label_f,Label_f_real), where loss_obj is the loss function of the function recognition model, MSE is the mean squared error function, Label_f is the function label output by the function recognition model, and Label_f_real is the true value of the function label of the sample speech.
[0127] The functional recognition model is used to identify physiological functions in the assessment speech data as an objective assessment task for stroke rehabilitation. The output of the functional recognition model is the stroke rehabilitation assessment result of the assessment subject in the objective assessment task for stroke rehabilitation.
[0128] This invention employs a dynamic task prioritization approach to combine two evaluation tasks. When both tasks are trained simultaneously, the more difficult task receives a higher learning priority, enhancing the adaptive learning capabilities of both evaluation tasks. Adjustments are made based on the learning stage, difficulty level, and even the learning outcome of each task, ultimately improving the learning performance of each evaluation task simultaneously. Specifically:
[0129] A stroke rehabilitation assessment model is obtained by adaptively combining the weights of subjective and objective assessment tasks using a multi-task learning mechanism, including:
[0130] By utilizing a multi-task learning mechanism, the input layer and bottom network layer of the subjective assessment task and the objective assessment task of stroke rehabilitation are shared. By adaptively combining the loss function with weights, a stroke rehabilitation assessment model with dynamic task priority is obtained.
[0131] The stroke rehabilitation assessment model is as follows:
[0132]
[0133] In the formula, G_3channel is the three-channel Mel spectrogram of the sample speech, Label_e is the emotion label, Label_f is the function label, softmax is the softmax classifier, and CNN is the CNN neural network.
[0134] G_3channel—CNN—Label_e represents the task line for the subjective assessment task of stroke rehabilitation, and G_3channel—softmax—Label_f represents the task line for the objective assessment task of stroke rehabilitation. The stroke rehabilitation assessment model contains both the task lines for the objective assessment task and the task lines for the subjective assessment task, thereby achieving multi-task learning of both objective and subjective assessment tasks of stroke rehabilitation, and combining objective and subjective levels of stroke rehabilitation assessment.
[0135] The loss function of the stroke rehabilitation assessment model is:
[0136] loss=∑[w_e(t)*loss_subj(t)+w_f(t)*loss_obj(t)];
[0137] w_e(t)=-[1-K_e(t)] r_e logK_e(t);
[0138] w_f(t)=-[1-K_f(t)] r_f logK_f(t);
[0139] In the formula, loss is the loss function of the stroke rehabilitation assessment model, w_e(t) is the weight of the loss function of the emotion recognition model when the stroke rehabilitation assessment model has been trained for t iterations, w_f(t) is the weight of the loss function of the function recognition model when the stroke rehabilitation assessment model has been trained for t iterations, loss_subj(t) is the loss function of the emotion recognition model when the stroke rehabilitation assessment model has been trained for t iterations, loss_obj(t) is the loss function of the function recognition model when the stroke rehabilitation assessment model has been trained for t iterations, K_e(t) is the model KPI of the emotion recognition model when the stroke rehabilitation assessment model has been trained for t iterations, K_f(t) is the model KPI of the function recognition model when the stroke rehabilitation assessment model has been trained for t iterations, r_e is the manually adjusted parameter of w_e(t), r_f is the manually adjusted parameter of w_f(t), t is the number of training iterations, and the model KPI is at least one of model accuracy, model precision, model recall, F1 score, and ROC curve.
[0140] This invention uses model KPIs to measure the difficulty of task learning; the higher the KPI, the lower the learning difficulty. It also uses the weights of the loss function to quantify the priority of tasks; higher weights correspond to higher priority. Therefore, by mapping KPIs to the weight calculation of the loss function, tasks with high KPIs are easier to learn, resulting in lower weights; conversely, tasks with lower KPIs have higher weights. This achieves a learning adaptive capability where more difficult tasks have higher weights and higher priority, adjusting the two evaluation tasks based on their learning stage, difficulty level, and even learning effectiveness.
[0141] The stroke rehabilitation assessment model outputs the rehabilitation assessment results of the assessed subjects, including:
[0142] The assessment voice data is input into the stroke rehabilitation assessment model. The number of iterations is preset, and the stroke rehabilitation assessment model is trained iteratively to obtain the stroke rehabilitation assessment results of the subjective assessment task and the stroke rehabilitation assessment results of the objective assessment task.
[0143] The stroke rehabilitation assessment results of the subjects were averaged using both the subjective assessment task and the objective assessment task to obtain the rehabilitation assessment results of the subjects.
[0144] The three-channel Mel spectrum is 64*64 pixels in size.
[0145] like Figure 2 As shown, this invention provides an assessment system for a stroke rehabilitation assessment method based on speech multi-task learning, comprising:
[0146] The data preprocessing unit is used to acquire the speech data of the assessment subject and perform specific extraction on the speech data to obtain assessment speech data for rehabilitation assessment.
[0147] The multi-task learning unit is used to obtain a stroke rehabilitation subjective assessment task for perceiving the subjective rehabilitation feelings of the assessment subject by performing psychological and emotional recognition on the assessment voice data.
[0148] By performing physiological function recognition on the assessment voice data, an objective assessment task for stroke rehabilitation is obtained to perceive the objective rehabilitation status of the assessment subject.
[0149] A stroke rehabilitation assessment model was obtained by adaptively combining the weights of subjective and objective assessment tasks for stroke rehabilitation using a multi-task learning mechanism.
[0150] The assessment unit is used to output the rehabilitation assessment results of the assessed subjects using the stroke rehabilitation assessment model.
[0151] The multi-task learning unit performs psycho-emotional recognition on assessment speech data to obtain a subjective assessment task for stroke rehabilitation, used to perceive the subjective rehabilitation experience of the assessment subject, including:
[0152] A set of sample speech is obtained from the speech database, and each sample speech is labeled according to the emotional state of the speech representation to obtain the emotional label of each sample speech.
[0153] The three-channel Mel-spectrum of the sample speech was used as the input to a CNN neural network, and the emotion label of the sample speech was used as the output. The CNN neural network was then trained on the input and output to construct an emotion recognition model.
[0154] The emotion recognition model is as follows:
[0155] Label_e = CNN(G_3channel);
[0156] In the formula, Label_e is the emotion label, G_3channel is the three-channel Mel spectrogram of the sample speech, and CNN is the CNN neural network;
[0157] The loss function of the emotion recognition model is: loss_subj=MSE(Label_e,Label_e_real), where loss_subj is the loss function of the emotion recognition model, MSE is the mean squared error function, Label_e is the emotion label output by the emotion recognition model, and Label_e_real is the true value of the emotion label of the sample speech.
[0158] The emotion recognition model is used to identify the psychological emotions in the assessment speech data as a subjective assessment task for stroke rehabilitation. The output of the emotion recognition model is the stroke rehabilitation assessment result of the assessment subject in the subjective assessment task for stroke rehabilitation.
[0159] The multi-task learning unit performs physiological function recognition on the assessment speech data to obtain an objective assessment task for stroke rehabilitation, used to perceive the objective rehabilitation status of the assessment subject, including:
[0160] A set of sample speech is obtained from the speech library, and each sample speech is labeled according to the functional state of the speech representation to obtain the functional label of each sample speech.
[0161] The three-channel Mel-spectrum of the sample speech is used as the input to the softmax classifier, and the function label of the sample speech is used as the output of the softmax classifier. The softmax classifier is then used to train the input and output of the softmax classifier to construct a function recognition model.
[0162] The functional recognition model is as follows:
[0163] Label_e=sofrmax(G_3channel);
[0164] In the formula, Label_f is the functional label, G_3channel is the three-channel Mel spectrogram of the sample speech, and softmax is the softmax classifier;
[0165] The loss function of the function recognition model is: loss_obj=MSE(Label_f,Label_f_real), where loss_obj is the loss function of the function recognition model, MSE is the mean squared error function, Label_f is the function label output by the function recognition model, and Label_f_real is the true value of the function label of the sample speech.
[0166] The functional recognition model is used to identify physiological functions in the assessment speech data as an objective assessment task for stroke rehabilitation. The output of the functional recognition model is the stroke rehabilitation assessment result of the assessment subject in the objective assessment task for stroke rehabilitation.
[0167] The multi-task learning unit utilizes a multi-task learning mechanism to adaptively combine the weights of subjective and objective assessment tasks in stroke rehabilitation, resulting in a stroke rehabilitation assessment model, including:
[0168] By utilizing a multi-task learning mechanism, the input layer and bottom network layer of the subjective assessment task and the objective assessment task of stroke rehabilitation are shared. By adaptively combining the loss function with weights, a stroke rehabilitation assessment model with dynamic task priority is obtained.
[0169] The stroke rehabilitation assessment model is as follows:
[0170]
[0171] In the formula, G_3channel is the three-channel Mel spectrogram of the sample speech, Label_e is the emotion label, Label_f is the function label, softmax is the softmax classifier, and CNN is the CNN neural network.
[0172] G_3channel—CNN—Label_e represents the task line for the subjective assessment task of stroke rehabilitation, and G_3channel—softmax—Label_f represents the task line for the objective assessment task of stroke rehabilitation. The stroke rehabilitation assessment model contains both the task lines for the objective assessment task and the task lines for the subjective assessment task, thereby achieving multi-task learning of both objective and subjective assessment tasks of stroke rehabilitation, and combining objective and subjective levels of stroke rehabilitation assessment.
[0173] The loss function of the stroke rehabilitation assessment model is:
[0174] loss=∑[w_e(t)*loss_subj(t)+w_f(t)*loss_obj(t)];
[0175] w_e(t)=-[1-K_e(t)] r_e logK_e(t);
[0176] w_f(t)=-[1-K_f(t)] r_f logK_f(t);
[0177] In the formula, loss is the loss function of the stroke rehabilitation assessment model, w_e(t) is the weight of the loss function of the emotion recognition model when the stroke rehabilitation assessment model has been trained for t iterations, w_f(t) is the weight of the loss function of the function recognition model when the stroke rehabilitation assessment model has been trained for t iterations, loss_subj(t) is the loss function of the emotion recognition model when the stroke rehabilitation assessment model has been trained for t iterations, loss_obj(t) is the loss function of the function recognition model when the stroke rehabilitation assessment model has been trained for t iterations, K_e(t) is the model KPI of the emotion recognition model when the stroke rehabilitation assessment model has been trained for t iterations, K_f(t) is the model KPI of the function recognition model when the stroke rehabilitation assessment model has been trained for t iterations, r_e is the manually adjusted parameter of w_e(t), r_f is the manually adjusted parameter of w_f(t), t is the number of training iterations, and the model KPI is at least one of model accuracy, model precision, model recall, F1 score, and ROC curve.
[0178] This invention utilizes a multi-task learning mechanism to combine a subjective assessment task for stroke rehabilitation, which is used to perceive the subjective rehabilitation experience of the assessment subject, with an objective assessment task for stroke rehabilitation, which is used to perceive the objective rehabilitation status of the assessment subject. This allows for joint rehabilitation assessment, taking into account both the objective functional status and subjective emotional experience reflected in the patient's speech during the assessment. It enables the perception of the pain and discomfort experienced by stroke patients during functional assessment, thereby improving the individual accuracy of stroke rehabilitation assessment and enhancing the user experience.
[0179] The above embodiments are merely exemplary embodiments of this application and are not intended to limit this application. The scope of protection of this application is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to this application within its substance and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of this application.
Claims
1. A stroke rehabilitation assessment method based on speech multi-task learning, characterized in that: Includes the following steps: Acquire the speech data of the assessment subject, and perform specific extraction on the speech data to obtain assessment speech data for rehabilitation assessment; By performing psychological and emotional recognition on the assessment voice data, a subjective assessment task for stroke rehabilitation is obtained to perceive the subjective rehabilitation feelings of the assessment subjects. By performing physiological function recognition on the assessment voice data, an objective assessment task for stroke rehabilitation is obtained to perceive the objective rehabilitation status of the assessment subject. A stroke rehabilitation assessment model was obtained by adaptively combining the weights of subjective and objective assessment tasks for stroke rehabilitation using a multi-task learning mechanism. Finally, the stroke rehabilitation assessment model outputs the rehabilitation assessment results of the assessed subjects.
2. The stroke rehabilitation assessment method based on speech multi-task learning according to claim 1, characterized in that: The speech data is specifically extracted to obtain assessment speech data for rehabilitation evaluation, including: The voice data is divided into multiple voice segments according to a preset length; Multiple speech segment data are clustered to obtain multiple speech segment sets; The speech segment data that is closest to the center of each speech segment set is selected as the specific speech data; The specific speech data of each speech segment set is processed sequentially into Mel spectrograms to obtain the Mel spectrograms of the specific speech data. The Mel spectrograms of specific speech data are processed by first-order and second-order difference to obtain the first-order and second-order difference results of the Mel spectrograms of specific speech data. The Mel spectrogram of the specific speech data, the first-order difference, and the second-order difference results of the Mel spectrogram of the specific speech data are superimposed to obtain the three-channel Mel spectrogram of the specific speech data. The three-channel Mel spectrogram of the specific speech data is used as the evaluation speech data.
3. The stroke rehabilitation assessment method based on speech multi-task learning according to claim 2, characterized in that: By performing psycho-emotional recognition on the assessment speech data, a subjective assessment task for stroke rehabilitation, used to perceive the subjective rehabilitation experience of the assessment subject, is obtained, including: A set of sample speech is obtained from the speech database, and each sample speech is labeled according to the emotional state of the speech representation to obtain the emotional label of each sample speech. The three-channel Mel-spectrum of the sample speech was used as the input to a CNN neural network, and the emotion label of the sample speech was used as the output. The CNN neural network was then trained on the input and output to construct an emotion recognition model. The emotion recognition model is as follows: Label_e = CNN(G_3channel); In the formula, Label_e is the emotion label, G_3channel is the three-channel Mel spectrogram of the sample speech, and CNN is the CNN neural network; The loss function of the emotion recognition model is: loss_subj=MSE(Label_e,Label_e_real), where loss_subj is the loss function of the emotion recognition model, MSE is the mean squared error function, Label_e is the emotion label output by the emotion recognition model, and Label_e_real is the true value of the emotion label of the sample speech. An emotion recognition model is used to identify the psychological emotions in the assessment speech data as a subjective assessment task for stroke rehabilitation. The output of the emotion recognition model is the stroke rehabilitation assessment result of the assessment subject in the subjective assessment task for stroke rehabilitation.
4. The stroke rehabilitation assessment method based on speech multi-task learning according to claim 3, characterized in that: By performing physiological function recognition on the assessment speech data, an objective assessment task for stroke rehabilitation is obtained to perceive the objective rehabilitation status of the assessment subject, including: A set of sample speech is obtained from the speech library, and each sample speech is labeled according to the functional state of the speech representation to obtain the functional label of each sample speech. The three-channel Mel-spectrum of the sample speech is used as the input to the softmax classifier, and the function label of the sample speech is used as the output of the softmax classifier. The softmax classifier is then used to train the input and output of the softmax classifier to construct a function recognition model. The functional recognition model is as follows: Label_e=sofrmax(G_3channel); In the formula, Label_f is the functional label, G_3channel is the three-channel Mel spectrogram of the sample speech, and softmax is the softmax classifier; The loss function of the function recognition model is: loss_obj=MSE(Label_f,Label_f_real), where loss_obj is the loss function of the function recognition model, MSE is the mean squared error function, Label_f is the function label output by the function recognition model, and Label_f_real is the true value of the function label of the sample speech. The functional recognition model performs physiological function recognition on the assessment speech data as an objective assessment task for stroke rehabilitation. The output of the functional recognition model is the stroke rehabilitation assessment result of the assessment subject by the objective assessment task for stroke rehabilitation.
5. A stroke rehabilitation assessment method based on speech multi-task learning according to claim 4, characterized in that: A stroke rehabilitation assessment model is obtained by adaptively combining the weights of subjective and objective assessment tasks using a multi-task learning mechanism, including: By utilizing a multi-task learning mechanism, the input layer and bottom network layer of the subjective assessment task and the objective assessment task of stroke rehabilitation are shared. By adaptively combining the loss function with weights, a stroke rehabilitation assessment model with dynamic task priority is obtained. The stroke rehabilitation assessment model is as follows: In the formula, G_3channel is the three-channel Mel spectrogram of the sample speech, Label_e is the emotion label, Label_f is the function label, softmax is the softmax classifier, and CNN is the CNN neural network. The loss function of the stroke rehabilitation assessment model is: loss=∑[w_e(t)*loss_subj(t)+w_f(t)*loss_obj(t)]; w_e(t)=-[1-K_e(t)] r_e logK_e(t); w_f(t)=-[1-K_f(t)] r_f logK_f(t); In the formula, loss is the loss function of the stroke rehabilitation assessment model, w_e(t) is the weight of the loss function of the emotion recognition model when the stroke rehabilitation assessment model has been trained for t iterations, w_f(t) is the weight of the loss function of the function recognition model when the stroke rehabilitation assessment model has been trained for t iterations, loss_subj(t) is the loss function of the emotion recognition model when the stroke rehabilitation assessment model has been trained for t iterations, loss_obj(t) is the loss function of the function recognition model when the stroke rehabilitation assessment model has been trained for t iterations, K_e(t) is the model KPI of the emotion recognition model when the stroke rehabilitation assessment model has been trained for t iterations, K_f(t) is the model KPI of the function recognition model when the stroke rehabilitation assessment model has been trained for t iterations, r_e is the manually adjusted parameter of w_e(t), r_f is the manually adjusted parameter of w_f(t), t is the number of training iterations, and the model KPI is at least one of model accuracy, model precision, model recall, F1 score and ROC curve.
6. The stroke rehabilitation assessment method based on speech multi-task learning according to claim 5, characterized in that: The stroke rehabilitation assessment model outputs the rehabilitation assessment results of the assessed subjects, including: The assessment voice data is input into the stroke rehabilitation assessment model, the number of iterations is preset, and the stroke rehabilitation assessment model is iteratively trained to obtain the stroke rehabilitation assessment results of the subjective assessment task and the stroke rehabilitation assessment results of the objective assessment task. The stroke rehabilitation assessment results of the subjects were averaged using both the subjective assessment task and the objective assessment task to obtain the rehabilitation assessment results of the subjects.
7. The stroke rehabilitation assessment method based on speech multi-task learning according to claim 2, characterized in that: The three-channel Mel spectrum has a size of 64*64 pixels.
8. An assessment system applying the speech-based multi-task learning-based stroke rehabilitation assessment method according to any one of claims 1-7, characterized in that, include: A data preprocessing unit is used to acquire the speech data of the assessment subject and perform specific extraction on the speech data to obtain assessment speech data for rehabilitation assessment. The multi-task learning unit is used to obtain a stroke rehabilitation subjective assessment task for perceiving the subjective rehabilitation feelings of the assessment subject by performing psychological and emotional recognition on the assessment voice data. By performing physiological function recognition on the assessment voice data, an objective assessment task for stroke rehabilitation is obtained to perceive the objective rehabilitation status of the assessment subject. A stroke rehabilitation assessment model was obtained by adaptively combining the weights of subjective and objective assessment tasks for stroke rehabilitation using a multi-task learning mechanism. The assessment unit is used to output the rehabilitation assessment results of the assessment subject using the stroke rehabilitation assessment model.
9. The evaluation system according to claim 8, characterized in that: The multi-task learning unit performs psycho-emotional recognition on assessment speech data to obtain a subjective assessment task for stroke rehabilitation, used to perceive the subjective rehabilitation experience of the assessment subject, including: A set of sample speech is obtained from the speech database, and each sample speech is labeled according to the emotional state of the speech representation to obtain the emotional label of each sample speech. The three-channel Mel-spectrum of the sample speech was used as the input to a CNN neural network, and the emotion label of the sample speech was used as the output. The CNN neural network was then trained on the input and output to construct an emotion recognition model. The emotion recognition model is as follows: Label_e = CNN(G_3channel); In the formula, Label_e is the emotion label, G_3channel is the three-channel Mel spectrogram of the sample speech, and CNN is the CNN neural network; The loss function of the emotion recognition model is: loss_subj=MSE(Label_e,Label_e_real), where loss_subj is the loss function of the emotion recognition model, MSE is the mean squared error function, Label_e is the emotion label output by the emotion recognition model, and Label_e_real is the true value of the emotion label of the sample speech. An emotion recognition model is used to identify the psychological emotions in the assessment speech data as a subjective assessment task for stroke rehabilitation. The output of the emotion recognition model is the stroke rehabilitation assessment result of the assessment subject in the subjective assessment task for stroke rehabilitation. The multi-task learning unit performs physiological function recognition on the assessment speech data to obtain an objective assessment task for stroke rehabilitation, used to perceive the objective rehabilitation status of the assessment subject, including: A set of sample speech is obtained from the speech library, and each sample speech is labeled according to the functional state of the speech representation to obtain the functional label of each sample speech. The three-channel Mel-spectrum of the sample speech is used as the input to the softmax classifier, and the function label of the sample speech is used as the output of the softmax classifier. The softmax classifier is then used to train the input and output of the softmax classifier to construct a function recognition model. The functional recognition model is as follows: Label_e=sofrmax(G_3channel); In the formula, Label_f is the functional label, G_3channel is the three-channel Mel spectrogram of the sample speech, and softmax is the softmax classifier; The loss function of the function recognition model is: loss_obj=MSE(Label_f,Label_f_real), where loss_obj is the loss function of the function recognition model, MSE is the mean squared error function, Label_f is the function label output by the function recognition model, and Label_f_real is the true value of the function label of the sample speech. The functional recognition model performs physiological function recognition on the assessment speech data as an objective assessment task for stroke rehabilitation. The output of the functional recognition model is the stroke rehabilitation assessment result of the assessment subject by the objective assessment task for stroke rehabilitation.
10. The evaluation system according to claim 9, characterized in that: The multi-task learning unit utilizes a multi-task learning mechanism to adaptively combine the weights of subjective and objective assessment tasks in stroke rehabilitation, resulting in a stroke rehabilitation assessment model, including: By utilizing a multi-task learning mechanism, the input layer and bottom network layer of the subjective assessment task and the objective assessment task of stroke rehabilitation are shared. By adaptively combining the loss function with weights, a stroke rehabilitation assessment model with dynamic task priority is obtained. The stroke rehabilitation assessment model is as follows: In the formula, G_3channel is the three-channel Mel spectrogram of the sample speech, Label_e is the emotion label, Label_f is the function label, softmax is the softmax classifier, and CNN is the CNN neural network. The loss function of the stroke rehabilitation assessment model is: loss=∑[w_e(t)*loss_subj(t)+w_f(t)*loss_obj(t)]; w_e(t)=-[1-K_e(t)] r_e logK_e(t); w_f(t)=-[1-K_f(t)] r_f logK_f(t); In the formula, loss is the loss function of the stroke rehabilitation assessment model, w_e(t) is the weight of the loss function of the emotion recognition model when the stroke rehabilitation assessment model has been trained for t iterations, w_f(t) is the weight of the loss function of the function recognition model when the stroke rehabilitation assessment model has been trained for t iterations, loss_subj(t) is the loss function of the emotion recognition model when the stroke rehabilitation assessment model has been trained for t iterations, loss_obj(t) is the loss function of the function recognition model when the stroke rehabilitation assessment model has been trained for t iterations, K_e(t) is the model KPI of the emotion recognition model when the stroke rehabilitation assessment model has been trained for t iterations, K_f(t) is the model KPI of the function recognition model when the stroke rehabilitation assessment model has been trained for t iterations, r_e is the manually adjusted parameter of w_e(t), r_f is the manually adjusted parameter of w_f(t), t is the number of training iterations, and the model KPI is at least one of model accuracy, model precision, model recall, F1 score and ROC curve.