Parkinson's disease patient demand identification method based on language enhancement technology

Abstract

The invention provides a Parkinson's disease patient demand identification method based on a language enhancement technology, and belongs to the technical field of deep learning language processing. Firstly, continuous vowel sounds and repeated syllable sounds of a Parkinson patient are collected; preprocessing the voice data to obtain standard input data; inputting the data into a model obtained by training, and outputting a demand category of the patient; the designed demand identification model comprises an encoder module, a decoder module and a patient demand identification module; inputting the continuous vowel data and the repeated syllable data into a trained encoder module to obtain fusion features; inputting the fusion features into a trained patient demand identification module to output a patient demand category; the decoder module is used for obtaining reconstructed features, and the reconstructed features utilize a loss function to restrain and guide the expression of the encoder output features. According to the language enhancement technology designed by the invention, through a series of innovative methods, the accuracy and efficiency of identifying the requirements of the Parkinson's disease patients are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning language processing, and in particular relates to a method for identifying needs of Parkinson's patients based on language enhancement technology. Background Art

[0002] Parkinson's disease is a degenerative disease of the nervous system, and its patients often face the challenge of motor dysfunction, which is not only manifested as slow limb movements, but also may affect the ability to express speech, leading to symptoms such as unclear speech, slow speech speed, and weak voice. These communication barriers make it difficult for Parkinson's patients to express their needs and emotions smoothly in daily life, which makes them prone to loneliness and depression. In order to improve this dilemma of Parkinson's patients and improve their quality of life, the present invention needs to explore and implement a more effective demand identification method.

[0003] Understanding the needs of Parkinson's patients involves not only their physical needs, but also psychological concerns. With the advancement of science and technology, especially the rapid development of artificial intelligence and speech recognition technology, there is an opportunity to use speech recognition technology to convert patients' voices into text, or to predict and meet their needs through intelligent analysis. In addition, improving the ability to recognize the needs of Parkinson's patients will help them integrate into social life more smoothly, which is of great significance to promoting social inclusiveness and harmonious coexistence.

[0004] When developing a method for identifying Parkinson's disease patients' needs based on language enhancement technology, the technical challenges faced mainly include the following aspects:

[0005] Speech recognition accuracy: Parkinson's disease patients may have tremors, low volume, unclear pronunciation and other problems in their speech, which increases the difficulty of speech recognition. It is necessary to develop recognition algorithms that can adapt to these special speech characteristics.

[0006] Noise and environmental interference: In actual usage environments, background noise and other interference sources may affect the clarity of speech signals.

[0007] Natural language understanding: Parkinson's disease patients may lack fluency and standardization in their speech, which poses a challenge to natural language processing (NLP). It is necessary to develop NLP technologies that can understand non-standard language expressions and ambiguous sentences. Summary of the invention

[0008] In view of the above problems, the present invention aims to utilize advanced language enhancement technology to effectively solve the problem of Parkinson's disease patients in demand identification.

[0009] The present invention provides a method for identifying needs of Parkinson's disease patients based on language enhancement technology, comprising the following steps:

[0010] The continuous vowel phonation and repeated syllable phonation of Parkinson's patients were collected;

[0011] Preprocess the speech data to obtain standard input data for training the patient demand recognition model, including sustained vowel data and repeated syllable data;

[0012] Inputting the sustained vowel data and the repeated syllable data into the trained patient demand recognition model, and outputting the patient's demand category;

[0013] The patient demand recognition model includes an encoder module, a decoder module and a patient demand recognition module; continuous vowel data and repeated syllable data are input into the trained encoder module to obtain fusion features; the fusion features are input into the trained patient demand recognition module to output the patient demand category; the decoder module is used to obtain reconstructed vowel features and syllable features, and the reconstructed features use a loss function to constrain and guide the expression of the output features of the encoder module.

[0014] Preferably, the number of the patient's demand categories is N, and the multi-classification results are represented by a one-dimensional array structure, wherein the results of each demand category are represented by 1 to indicate that the patient's demand includes this category, and by 0 to indicate that the patient's demand does not fall into this category.

[0015] Preferably, the encoder module includes a unique representation learning submodule and a multi-feature fusion submodule, and the representation learning submodule includes a K-means clustering unit, a feature grouping unit and a dual-branch representation learning unit; the K-means clustering unit is used to cluster the continuous vowel data and repeated syllable data input by the model to obtain the vowel fine-grained feature F _v and syllable fine-grained features F _p The feature grouping unit is used to group F according to the correlation of the features within and between groups. _v and F _p The dual-branch representation learning unit is based on a multi-layer feedforward neural network, including a dual-branch parallel first feature extraction network and a second feature extraction network, respectively obtaining a specific representation V of a continuous vowel v and repeated syllable-specific representation V p ;

[0016] The multi-feature fusion submodule includes a common feature extraction unit and a feature splicing unit. First, the feature extraction unit is used to group the F _v and F _p Multi-feature fusion is performed to extract the common information features of sustained vowels and repeated syllables, and then the common information features and the specific representation V of sustained vowels are combined through the feature concatenation unit. v and repeated syllable-specific representation V p The three are spliced ​​together to obtain the multivariate fusion feature V containing shared informationcla ; The final output V of the encoder module cla As input to the patient needs identification and detection module.

[0017] Preferably, the two feature extraction networks in the dual-branch representation learning unit both include three layers of feedforward neural network layer, ReLU activation function layer, fully connected layer, pooling layer and Softmax activation function layer; the vowel grouping feature is input into the first feature extraction network to first obtain the vowel local feature, and the syllable grouping feature is input into the second feature extraction network to first obtain the syllable local feature. The specific calculation formula is as follows:

[0018] F_v′=FNN(FNN(FNN(F_v)))

[0019] F_p′=FNN(FNN(FNN(F_p)))

[0020] Among them, F_v′ and F_p′ are the local features output after the three-layer feedforward neural network, and FNN(·) is a fully connected layer; then the ReLU activation function layer is used for feature processing, followed by a fully connected layer and a pooling layer, and the SoftMax function is used to distribute the features. The specific formula is as follows:

[0021] V v =SoftMax(Max(FC(RELU(F′_v))))

[0022] V p =SoftMax(Max(FC(RELU(F′_p))))

[0023] Among them, Max(·) indicates that the pooling operation is the maximum pooling, FC(·) indicates the fully connected layer, SoftMax(·) indicates the SoftMax function, V v and V p It is the sustained vowel-specific representation and repeated syllable-specific representation after the double-branch representation learning unit.

[0024] Preferably, the common feature extraction unit includes a Transformer encoding layer and a linear mapping layer; the specific processing flow is:

[0025] First, the grouped F i,v and F i,p Pairing, the kth feature group and the kth feature group Paired into k-th feature subgroup Secondly, the kth feature subgroup is passed Input linear mapping layer to obtain features The information of each sub-feature group is aligned in the feature space; finally, after all the feature sub-groups are input into the linear mapping layer, the output of the linear mapping layer is Form a U×K feature matrix, where K is the number of feature subgroups and U is the feature dimension of the feature subgroup after linear mapping;

[0026] Then, randomly initialize a U-dimensional classification vector As a classification token, it is concatenated with the feature matrix to obtain the concatenated feature

[0027] Then, the stitching features Input the Transformer encoding layer and loop 6 times to obtain the common information feature V fusion , the Transformer encoding layer includes a normalization layer, a multi-head attention mechanism, and a feedforward neural network. The input of the Transformer encoding layer network and the output of the multi-head attention mechanism use a residual connection, and the output of the multi-head attention mechanism and the output of the feedforward neural network use a residual connection; the specific calculation formula is as follows:

[0028]

[0029] e′ j =MSA(LN(e j-1 ))+e j-1

[0030] e j =FNN(LN(e j-1 ))+e′ j

[0031]

[0032] Among them, V fusion That is, the common information features obtained by the distribution and fusion of multi-source data. To obtain the classification representation, it contains all the input vectors The target information extracted from for The output vector after feature interaction with other vectors; e0 represents the initial input of the transformer encoding network layer, that is, e0 = x fusion ; e′ j represents the output of the multi-head attention mechanism layer after j cycles, LN(·) represents the normalization layer, and MSA(·) represents the multi-head attention mechanism layer; e j represents the output of the feedforward neural network after j cycles, and FNN(·) is the feedforward neural network.

[0033] Preferably, the decoder module converts the V output by the Transformer encoding network into fusion The encoder module output V v and V p After splicing, the feature vector V spc_v and V spc_p They are respectively used as inputs of the decoder feature reconstruction network, which contains two single-source speech feature branches corresponding to the encoder structure;

[0034] The single-source speech feature branch consists of a 3-layer feedforward neural network, using the ReLu activation function; the output is the original acoustic feature of the single-source speech After the full connection and pooling operations, it is input into the SoftMax function for feature activation, and then the enhanced continuous vowel feature x is obtained through the Top-K operation. v and enhanced repeated syllable features x p , expressed as:

[0035] x v =Dec v (V spc_v ; W rec_v )

[0036] x p =Dec p (V spc_p ; W rec_p )

[0037] Among them, W rec_p and W rec_v The module parameter x v and x p is the reconstructed feature vector; through the decoder Dec corresponding to the encoder v and Dec p Get the decoded and reconstructed enhanced continuous vowel feature x v and repeated syllable feature x p , and use loss optimization to constrain the expression of original features.

[0038] Preferably, the patient demand recognition module is composed of a feedforward neural network with three hidden layers, the number of neurons in each layer is 32, 16 and 2 respectively, and ReLu is used as the activation function; the real label of the sample y∈[0,1], when y is 0, it means that the patient has no such demand, and when y is 1, it means that the patient has such demand; the classification output of the patient demand detection module is:

[0039]

[0040] Among them, W cla For module parameters, the classification loss is calculated by the predicted value It is defined by the cross entropy loss between y and the true label value y.

[0041] Preferably, the decoder module uses the Smooth L1-loss loss function to calculate the reconstruction error, which is expressed as:

[0042]

[0043] Among them, x,x p / v are the original features and the output of the model reconstruction network respectively, and the final reconstruction loss is:

[0044]

[0045] where x i ,x i,rec They represent the feature representation of the i-th sample and the reconstructed features output by the reconstruction network respectively, and N is the total number of samples.

[0046] Preferably, orthogonal constraints are applied to the common information representation and the unique information representation obtained by the encoder module to reduce the information redundancy between the common information representation and the unique information representation. The difference loss between the features is calculated by orthogonal constraints. The feature difference loss function in feature orthogonalization is as follows:

[0047]

[0048] Among them, the matrix H is the fusion feature of the common information of multi-source speech data As a matrix composed of rows, the matrix S m is the single-source speech-specific information representation V extracted from the m-th single-source speech data v or V p As a matrix composed of rows, ||.|| is the squared Ferrobenius norm.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] (1) Multimodal data processing: By considering both vowels and syllables, the present invention can more comprehensively capture the subtle changes in the speech of Parkinson's patients, thereby improving the accuracy of demand recognition. Multimodal input enables the model to maintain good recognition performance even when single-modal data is of poor quality or missing. Different patients may have different degrees of difficulty in pronouncing vowels or syllables. Multimodal data processing can better adapt to individual differences and improve the universal applicability of the model.

[0051] (2) Feature fusion and complementary enhancement: By fusing different features, the present invention can achieve information complementarity, so that the model can understand the patient's speech characteristics from multiple angles and improve the accuracy of recognition. The patient demand recognition model designed by the present invention helps to build a richer and more powerful feature representation through feature fusion, which helps the neural network to better learn and classify complex speech patterns. Feature fusion can reduce the errors that may be caused by a single feature and improve the stability and reliability of the model in demand recognition tasks.

[0052] (3) Encoder-decoder architecture reduces the amount of training: The encoder-decoder architecture designed in the present invention can effectively process sequence data, is suitable for the timing characteristics of speech signals, and helps to improve the continuity and fluency of recognition. This architecture supports end-to-end learning, simplifies the training process, and enables the model to learn useful information directly from the original speech data without the need for complex preprocessing steps. The encoder can capture the contextual information of the speech signal, and the decoder can use this information to accurately decode, thereby better understanding the patient's needs. The encoder-decoder structure is highly flexible and can be easily extended to other types of speech recognition tasks, providing broad space for future research and applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a schematic diagram of the overall model data processing flow of the present invention.

[0054] Figure 2 This is a structural diagram of the unique representation learning submodule of the present invention.

[0055] Figure 3 Schematic diagram of the multivariate data feature fusion submodule of the present invention.

[0056] Figure 4 Schematic diagram of the decoder module network of the present invention.

[0057] Figure 5 This is a diagram of the experimental results of the present invention. DETAILED DESCRIPTION

[0058] The overall concept of the present invention is: construct a language data set S, including the collection and annotation of continuous vowel data and repeated syllable data. The data is a sample of the patient's voice recording after editing, which is divided into N demand categories, for example, a total of 5 demand categories, namely eating, drinking, going to the toilet, resting, and exercising. The multi-classification result y∈[0,1] marked as the demand category, y is a one-dimensional array structure, the array size is the number of categories 5, y=1 indicates that the patient's demand is in this category, and y=0 indicates that the patient's demand is not in this category; construct a patient demand recognition model, including an encoder module, a decoder module and a patient demand recognition module;

[0059] The encoder module includes a unique representation learning submodule and a multivariate data fusion submodule. The unique representation learning submodule performs K-means clustering on the constructed language dataset S, and then extracts and groups features to obtain the original continuous vowel feature F _v and repeated syllable feature F _p ; Then, a dual-branch feature learning unit based on a multi-layer feedforward neural network is constructed to learn the unique representation of continuous vowel data and the unique representation of repeated syllable data; then, the multivariate data fusion submodule fuses the two unique representations. First, the common information of the multivariate features is extracted in the common feature extraction unit, and then the multivariate features are spliced; then, the patient needs are classified in the patient needs identification module to obtain the patient needs classification results. Finally, the complete model is trained using multi-loss optimization technology through multi-loss optimization. The decoder module is used to obtain the reconstructed vowel features and syllable features. The reconstructed features use the loss function to constrain the expression of the output features of the encoder module.

[0060] The invention will be further described below in conjunction with specific embodiments.

[0061] 1. Initial Dataset Construction

[0062] The present invention collects speech samples of sustained vowels and repeated syllables, sustained vowel pronunciation (such as continuous pronunciation of vowels such as "a", "i", "u") and syllable repetition tasks (such as repetition of syllables such as "ba-ba-ba", "ma-ma-ma", etc.), which are composed of speech data of 68 patients and 17 healthy people screened out of the Parkinson's disease and movement disorder clinic of the hospital, among which there are 57 male subjects (including 49 Parkinson's disease patients (PD) and 8 healthy people (HC)), aged from 46 to 88 years old; there are 28 female subjects (including 19 Parkinson's disease patients and 9 healthy people), aged from 56 to 84 years old. Three pronunciation features of the subjects' frequency perturbation, amplitude perturbation, and harmonic noise ratio are extracted to form the sustained vowel data set S1; two pronunciation features of the subjects' Mel cepstral coefficients and Bark band energy are extracted to form the repeated syllable data set S2. Construct a language dataset S = {S1, S2, label}, where each data value in label is 1 or 0, 1 represents that the current demand matches the subject's demand, and 0 represents that the demand does not match the subject's demand. For example, the patient's demand label is y = [0, 1, 0, 0, 0], indicating that the patient has a need for drinking water. Store the constructed dataset S in CSV or HDF5 format to facilitate subsequent model training and evaluation.

[0063] The initial label of the dataset S is a manually annotated multi-classification label y∈[0,1], where y=0 means that the class does not belong to the category, and y=1 means that the class belongs to the category.

[0064] 2. Construction of Patient Needs Identification Model

[0065] 1. Encoder module

[0066] Since the input and output of the language enhancement model are the same-sized language data before and after enhancement, the language enhancement network designed by the present invention adopts an encoder and decoder structure. The encoder structure includes a unique representation learning submodule and a multi-feature fusion submodule, and the constructed and annotated data set S is used as the original input.

[0067] (1) Unique representation learning submodule

[0068] The unique representation learning submodule structure is as follows Figure 2 As shown in FIG. 1 , the system includes a K-means clustering unit, a feature grouping unit, and a dual-branch representation learning unit; the K-means clustering unit is used to cluster the continuous vowel data and repeated syllable data input by the model to obtain the vowel fine-grained feature F _v and syllable fine-grained features F _p The feature grouping unit is used to group F according to the correlation of intra-group and inter-group features. _v and F _p The dual-branch representation learning unit is based on a multi-layer feedforward neural network, including a dual-branch parallel first feature extraction network and a second feature extraction network, which respectively obtain the specific representation V of the continuous vowel v and repeated syllable-specific representation V p .

[0069] K-means clustering unit: Considering that the acoustic features extracted from the continuous vowel dataset S1 and the repeated syllable dataset S2 often have large feature redundancy, in order to analyze the data features in a more fine-grained manner, the present invention uses the K-means clustering method to cluster S1 and S2 respectively to obtain the vowel fine-grained features F _v and syllable fine-grained features F _p , the specific calculation formula is as follows:

[0070] F _v =Kmeans(S1)

[0071] F _p =Kmeans(S2)

[0072] where Kmeans(·) represents the K-means clustering operation.

[0073] Feature grouping unit: The feature grouping unit is based on the principle that the correlation of the features within the feature data group is as large as possible and the correlation of the features between groups is relatively weak. _v and F _p The specific calculation formula is as follows:

[0074]

[0075] Where d represents the feature dimension of each subgroup, and k represents the grouping identifier of the feature. i,v is the grouped F _v , F i,p is the grouped F _p .

[0076] Dual-branch representation learning unit: The dual-branch representation learning unit is a dual-branch parallel first feature extraction network structure and a second feature extraction network structure. Its main function is to learn speech-specific Parkinson's pathological representations with high-level semantic expression from high-dimensional low-order semantic features. The design of the present invention fully considers the acoustic changes caused by Parkinson's disease, especially the clinical pathological information contained in the continuous vowel features and repeated syllable features. The first unique representation learning network is used to learn the vowel grouping feature F_ output by the feature grouping unit. v The second unique representation learning network is used to learn the repeated syllable grouping feature F_ output by the feature grouping unit p The local features of the unique representation learning network include three layers of feedforward neural network (FNN), ReLU activation function layer, fully connected layer, pooling layer and Softmax activation function layer; the vowel grouping features are input into the first unique representation learning network to obtain the vowel local features, and the syllable grouping features are input into the second unique representation learning network to obtain the syllable local features. The specific calculation formula is as follows:

[0077] F_v′=FNN(FNN(FNN(F_v)))

[0078] F_p′=FNN(FNN(FNN(F_p)))

[0079] Among them, F_v′ and F_p′ are the local features output after three layers of feedforward neural network, FNN(·) is a fully connected layer, and then the ReLU activation function is used for feature processing, and a fully connected layer and a pooling layer are connected, and the SoftMax function is used to distribute the features. The specific formula is as follows:

[0080] V v =SoftMax(Max(FC(RELU(F′_v))))

[0081] V p =SoftMax(Max(FC(RELU(F′_p))))

[0082] Among them, Max() indicates that the pooling operation is the maximum pooling, FC indicates the fully connected layer, SoftMax(·) indicates the SoftMax function, V v and V p It is the sustained vowel-specific representation and repeated syllable-specific representation after the double-branch representation learning unit.

[0083] (2) Multi-feature fusion submodule

[0084] The multi-feature fusion submodule includes a common feature extraction unit and a feature concatenation unit; the continuous vowel-specific representation V v and repeated syllable-specific representation V p There is a large redundancy, and the two are not at the same semantic level. v and V p If the splicing is performed, a lot of invalid information will be introduced, which will affect the performance of the model. To solve the above problem, the feature extraction unit first extracts the grouped F output by the feature grouping unit. _v and F _p Multi-feature fusion is performed to extract the common information features of sustained vowels and repeated syllables, and then the common information features and the specific representation V of sustained vowels are combined through the feature concatenation unit. v and repeated syllable-specific representation V p The three are spliced ​​together to obtain the multivariate fusion feature V containing shared information cla ; The final output V of the encoder module cla As input to the patient needs identification and detection module.

[0085] Common feature extraction units:

[0086] The common feature extraction unit includes Transformer encoding layer, feedforward neural network layer, normalization layer, linear mapping layer and multi-head attention mechanism layer. The network structure is as follows: Figure 3 shown.

[0087] First, the grouped F i,v and F i,p To pair, The kth feature group in and The kth feature group in Paired into k-th feature subgroup Secondly, the kth feature subgroup is passed Input linear mapping layer to obtain features The common features of the highly correlated features in the group are extracted, redundancy is removed, and the information of each sub-feature group is aligned in the feature space; finally, after all the feature sub-groups are input into the linear mapping layer, the output of the linear mapping layer is Form a U×K feature matrix, where K is the number of feature subgroups and U is the feature dimension of the feature subgroup after linear mapping;

[0088] Randomly initialize a U-dimensional classification vector As a classification token, it is concatenated with the feature matrix to obtain the concatenated feature

[0089] Join features Input the Transformer encoding layer and loop 6 times to obtain the common information feature V fusion , the Transformer encoding layer includes a normalization layer, a multi-head attention mechanism, and a feedforward neural network. The input of the Transformer encoding layer network and the output of the multi-head attention mechanism use a residual connection, and the output of the multi-head attention mechanism and the output of the feedforward neural network use a residual connection; the specific calculation formula is as follows:

[0090]

[0091] e′ j =MSA(LN(e j-1 ))+e j-1 ; j = 1, 2, ..., 5,

[0092] e j =FNN(LN(e j-1 ))+e′ j ; j = 1, 2, ..., 5

[0093]

[0094] Among them, V fusion That is, the common information features obtained by the distribution and fusion of multi-source data. To obtain the classification representation, it contains all the input vectors The target information extracted from for The output vector after feature interaction with other vectors; e0 represents the initial input of the transformer encoding network, that is, e0 = x fusion ; e′ j represents the output of the multi-head attention mechanism layer after j cycles, LN(·) represents the normalization layer, and MSA(·) represents the multi-head attention mechanism layer; e jrepresents the output of the feedforward neural network after j cycles, and FNN(·) is the feedforward neural network. The purpose of introducing the normalization layer and residual connection in the transformer encoding network is to accelerate the training of the network; the transformer encoding network will learn the fusion interaction information between each feature subgroup and complete the information fusion across feature subgroups. The specific calculation formula of the multi-head attention mechanism layer is as follows:

[0095] MSA(x)=Concat(head1,...,head h )W j

[0096] Where x is the feature sequence of the input MSA, head1,...head h It is the attention mechanism block in the multi-head self-attention mechanism. j The linear mapping network can aggregate the outputs of h attention mechanism blocks. h It is the core module of information fusion, which is composed of the self-attention mechanism SA network. The calculation method is as follows:

[0097] head h =SA(XW h,Q ,XW h,K ,XW h,V )

[0098] Among them, W h,Q ,W h,K ,W h,V For head h The parameters of the three projection matrices are responsible for mapping the input feature sequence to the query, key, and value vector spaces.

[0099] Feature splicing unit:

[0100] The common information feature V output by the Transformer encoding network fusion The encoder module output V v and V p Splice and splice into vector group, V cla =[V v ; V fusion ; V p ], the concatenated result is used as the fusion feature V cla , in order to achieve the complete expression of multi-source speech information, V cla As input to the Parkinson's detection module.

[0101] 2. Decoder module

[0102] The present invention converts the V output by the Transformer encoding network intofusion The encoder module output V v and V p After splicing, the feature vector V spc_v and V spc_p They are used as the input of the two decoder branches for reconstructing single-source speech features.

[0103] The decoder module is designed as a feature reconstruction network as follows Figure 4 As shown in Figure 2, after feature concatenation and feature orthogonalization in the multi-feature fusion submodule, the unique vowel feature V is obtained. spc_v and the unique syllable feature V spc_p As the input of the feature reconstruction module.

[0104] The decoder structure is the same as the encoder structure, which is also a dual-branch structure with shared structure. The continuous vowel feature reconstruction network is used to reconstruct the feature vector extracted from the continuous vowel, and the repeated syllable feature reconstruction network is used to reconstruct the acoustic feature vector extracted from the repeated syllable. The feature reconstruction network consists of a 3-layer feedforward neural network and uses the ReLu activation function. The output is the original acoustic feature of the single source speech. After the full connection and pooling operations, it is input into the SoftMax function for feature activation, and then the enhanced continuous vowel feature x is obtained through the Top-K operation. v and enhanced repeated syllable features x p , which can be expressed as:

[0105] x v =Dec v (V spc_v ; W rec_v )

[0106] x p =Dec p (V spc_p ; W rec_p )

[0107] Among them, W rec_p and W rec_v The module's parameters, x v and x p is the reconstructed feature vector. Through the decoder Dec corresponding to the encoder v and Dec p Get the decoded and reconstructed enhanced continuous vowel feature x v and repeated syllable feature x p , and use loss optimization to constrain the expression of original features. The specific loss function is described in detail in the following content.

[0108] 3. Patient needs identification module

[0109] The patient demand recognition module is a multi-layer neural network classifier that outputs the recognition category of the current demand. cla As the input of the Parkinson's disease demand identification module, the needs of Parkinson's disease patients are identified through the corresponding classifier. The patient demand identification module consists of a feedforward neural network with three hidden layers. The number of neurons in each layer is 32, 16 and 2 respectively, and ReLu is used as the activation function. The true label of the sample y∈[0,1], when y is 0, it means that the subject has no such need, and when y is 1, it means that the subject has such need. The classification output of the patient demand detection module is:

[0110]

[0111] Among them, W cla For module parameters, the classification loss is calculated by the predicted value It is defined by the cross entropy loss between y and the true label value y.

[0112] 3. Model Training

[0113] The decoder module uses the Smooth L1-loss loss function to calculate the reconstruction error, which is expressed as:

[0114]

[0115] Among them, x,x p / v are the original features and the output of the model reconstruction network respectively. The final reconstruction loss is:

[0116]

[0117] where x i ,x i,rec They represent the feature representation of the i-th sample and the reconstructed features output by the reconstruction network respectively, and N is the total number of samples.

[0118] Feature orthogonalization constraint: To further ensure that the encoder can extract the common information and unique representations in the continuous vowel feature Feat_v and the repeated syllable feature Feat_p, the present invention imposes orthogonal constraints on the common information representation and the unique representation obtained by the encoder, reduces the information redundancy between the common information representation and the unique information representation, and calculates the difference loss between the features through the orthogonal constraint. The feature difference loss function in feature orthogonalization is as follows:

[0119]

[0120] Among them, the matrix H is the fusion feature of the common information of multi-source speech data As a matrix composed of rows, the matrix S mis the single-source speech-specific information representation V extracted from the m-th single-source speech data v or V p As a matrix composed of rows, ||.|| is the squared Ferrobenius norm.

[0121] The language enhancement model proposed in the present invention is composed of multiple submodules, among which the patient demand recognition module will use the cross entropy loss function, and the decoder module will use the Smooth L1-loss function. In order to make full use of the label information in the data set, the final model loss of the present invention is:

[0122] L total =λ1L rec_v +λ2L rec_p +λ3L cla +λ4L diff

[0123] Among them, L rec_v is the loss for reconstructing sustained vowel speech in the decoder module, L rec_p is the loss for reconstructing repeated syllable speech in the decoder module, L cla is the classification loss of the Parkinson's disease need recognition module, L diff is the difference loss between the common information representation and the unique information representation. Here, since the loss of single-source speech feature reconstruction is obviously much larger than the loss of the Parkinson's disease demand recognition module, in order to avoid the optimization of multiple losses together, the model tends to optimize in the direction of large losses due to scale inconsistency, resulting in a decrease in the performance of other modules. Four hyperparameters λ1, λ2, λ3, and λ4 are preset, and the impact of scale inconsistency on the model is reduced by weighting each loss. It is worth noting that in order to avoid the proposed model focusing on optimizing the feature reconstruction loss and ignoring the Parkinson's disease demand recognition module when updating parameters, the inputs of the Parkinson's disease demand classification module and the feature reconstruction branch in the decoder are different. The above designs can avoid the problem of weight imbalance in the optimization process, and can also help the proposed model learn a more compact fusion representation.

[0124] IV. Experimental Results

[0125] In order to verify the effectiveness of the method of the present invention, a test set consisting of 50 Parkinson's patients and their needs was used to verify the method proposed by the present invention. Figure 5 As shown in the figure, sample numbers 1-50 represent 50 Parkinson's patients, blue circles represent the actual demand of Parkinson's patients, and red triangles represent the predicted demand of Parkinson's patients. Figure 5 It can be seen that the number of samples predicted incorrectly by the method of the present invention is 4, which shows that the method of the present invention has a high prediction accuracy.

[0126] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0127] Although the above describes the specific implementation methods of the present invention, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for identifying needs of Parkinson's patients based on language enhancement technology, characterized in that: The following steps are involved: The continuous vowel phonation and repeated syllable phonation of Parkinson's patients were collected; Preprocess the speech data to obtain standard input data for training the patient demand recognition model, including sustained vowel data and repeated syllable data; Inputting the sustained vowel data and the repeated syllable data into the trained patient demand recognition model, and outputting the patient's demand category; The patient demand recognition model includes an encoder module, a decoder module and a patient demand recognition module; continuous vowel data and repeated syllable data are input into the trained encoder module to obtain fusion features; the fusion features are input into the trained patient demand recognition module to output the patient demand category; the decoder module is used to obtain reconstructed vowel features and syllable features, and the reconstructed features use a loss function to constrain and guide the expression of the output features of the encoder module.

2. A method for identifying Parkinson's disease patient needs based on language enhancement technology as claimed in claim 1, characterized in that: The number of demand categories of the patient is N, and the multi-classification result is represented by a one-dimensional array structure, in which the result of each demand category is 1 to indicate that the patient's demand includes this category, and 0 to indicate that the patient's demand does not belong to this category.

3. The method for identifying needs of Parkinson's patients based on language enhancement technology as claimed in claim 1, characterized in that: The encoder module includes a unique representation learning submodule and a multi-feature fusion submodule, wherein the representation learning submodule includes a K-means clustering unit, a feature grouping unit and a dual-branch representation learning unit; the K-means clustering unit is used to cluster the continuous vowel data and repeated syllable data input by the model to obtain the vowel fine-grained feature F _v and syllable fine-grained features F _p The feature grouping unit is used to group F according to the correlation of the features within and between groups. _v and F _p The dual-branch representation learning unit is based on a multi-layer feedforward neural network, including a dual-branch parallel first feature extraction network and a second feature extraction network, respectively obtaining a specific representation V of a continuous vowel v and repeated syllable-specific representation V p ; The multi-feature fusion submodule includes a common feature extraction unit and a feature splicing unit. First, the feature extraction unit is used to group the F _v and F _p Multi-feature fusion is performed to extract the common information features of sustained vowels and repeated syllables, and then the common information features and the specific representation V of sustained vowels are combined through the feature concatenation unit. v and repeated syllable-specific representation V p The three are spliced ​​together to obtain the multivariate fusion feature V containing shared information cla ; The final output V of the encoder module cla As input to the patient needs identification and detection module.

4. A method for identifying Parkinson's disease patient needs based on language enhancement technology as claimed in claim 3, characterized in that: The two feature extraction networks in the dual-branch representation learning unit both include three layers of feedforward neural network layers, ReLU activation function layers, fully connected layers, pooling layers, and Softmax activation function layers; the vowel grouping features are input into the first feature extraction network to first obtain the vowel local features, and the syllable grouping features are input into the second feature extraction network to first obtain the syllable local features. The specific calculation formula is as follows: F_v′=FNN(FNN(FNN(F_v))) F_p′=FNN(FNN(FNN(F_p))) Among them, F_v′ and F_p′ are the local features output after the three-layer feedforward neural network, and FNN(·) is a fully connected layer; then the ReLU activation function layer is used for feature processing, followed by a fully connected layer and a pooling layer, and the SoftMax function is used to distribute the features. The specific formula is as follows: In v =SoftMax(Max(FC(RELU(F′_v)))) In p =SoftMax(Max(FC(RELU(F′_p)))) Among them, Max(·) indicates that the pooling operation is the maximum pooling, FC(·) indicates the fully connected layer, SoftMax(·) indicates the SoftMax function, V v and V p It is the sustained vowel-specific representation and repeated syllable-specific representation after the double-branch representation learning unit.

5. The method for identifying needs of Parkinson's disease patients based on language enhancement technology as claimed in claim 3, characterized in that: The common feature extraction unit includes a Transformer encoding layer and a linear mapping layer; the specific processing flow is: First, the grouped F i,v and F i,p Pairing, the kth feature group and the kth feature group Paired into k-th feature subgroup Secondly, the kth feature subgroup is passed Input linear mapping layer to obtain features The information of each sub-feature group is aligned in the feature space; finally, after all the feature sub-groups are input into the linear mapping layer, the output of the linear mapping layer is Form a U×K feature matrix, where K is the number of feature subgroups and U is the feature dimension of the feature subgroup after linear mapping; Then, randomly initialize a U-dimensional classification vector As a classification token, it is concatenated with the feature matrix to obtain the concatenated feature Then, the stitching features Input the Transformer encoding layer and loop 6 times to obtain the common information feature V fusion , the Transformer encoding layer includes a normalization layer, a multi-head attention mechanism, and a feedforward neural network. The input of the Transformer encoding layer network and the output of the multi-head attention mechanism use a residual connection, and the output of the multi-head attention mechanism and the output of the feedforward neural network use a residual connection; the specific calculation formula is as follows: And' j =MSE(LN(e j-1 ))+e j-1 have been j =FNN(LN(e j-1 ))+e′ j Among them, V fusion That is, the common information features obtained by the distribution and fusion of multi-source data. To obtain the classification representation, it contains all the input vectors The target information extracted from for The output vector after feature interaction with other vectors; e0 represents the initial input of the transformer encoding network layer, that is, e0 = x fusion ;e j ′ represents the output of the multi-head attention mechanism layer after j cycles, LN(·) represents the normalization layer, and MSA(·) represents the multi-head attention mechanism layer; e j represents the output of the feedforward neural network after j cycles, and FNN(·) is the feedforward neural network.

6. The method for identifying Parkinson's disease patients' needs based on language enhancement technology as claimed in claim 1, characterized in that: The decoder module converts the output of the Transformer encoding network into V fusion The encoder module output V v and V p After splicing, the feature vector V spc_v and V spc_p They are respectively used as inputs of the decoder feature reconstruction network, which contains two single-source speech feature branches corresponding to the encoder structure; The single-source speech feature branch consists of a 3-layer feedforward neural network, using the ReLu activation function; the output is the original acoustic feature of the single-source speech After the full connection and pooling operations, it is input into the SoftMax function for feature activation, and then the enhanced continuous vowel feature x is obtained through the Top-K operation. v and enhanced repeated syllable features x p , expressed as: x v =Dec v (V spc_v ;W rec_v ) x p =Dec p (V spc_p ;W rec_p ) Among them, W rec_p and W rec_v The module parameter x v and x p is the reconstructed feature vector; through the decoder Dec corresponding to the encoder v and Dec p Get the decoded and reconstructed enhanced continuous vowel feature x v and repeated syllable feature x p , and use loss optimization to constrain the expression of original features.

7. The method for identifying needs of Parkinson's disease patients based on language enhancement technology as claimed in claim 1, characterized in that: The patient demand recognition module is composed of a feedforward neural network with three hidden layers, with 32, 16 and 2 neurons in each layer, respectively, and ReLu is used as the activation function; the real label of the sample y∈[0,1], when y is 0, it means that the patient has no such demand, and when y is 1, it means that the patient has such demand; the classification output of the patient demand detection module is: Among them, W cla For module parameters, the classification loss is calculated by the predicted value It is defined by the cross entropy loss between y and the true label value y.

8. The method for identifying needs of Parkinson's disease patients based on language enhancement technology as claimed in claim 1, characterized in that: The decoder module uses the Smooth L1-loss loss function to calculate the reconstruction error, which is expressed as: Among them, x,x p / v are the original features and the output of the model reconstruction network respectively, and the final reconstruction loss is: where x i ,x i,rec They represent the feature representation of the i-th sample and the reconstructed features output by the reconstruction network respectively, and N is the total number of samples.

9. The method for identifying needs of Parkinson's disease patients based on language enhancement technology as claimed in claim 1, characterized in that: Orthogonal constraints are imposed on the common information representation and unique representation obtained by the encoder module to reduce the information redundancy between the common information representation and the unique information representation. The difference loss between the features is calculated through orthogonal constraints. The feature difference loss function in feature orthogonalization is as follows: Among them, the matrix H is the fusion feature of the common information of multi-source speech data As a matrix composed of rows, the matrix S m is the single-source speech-specific information representation V extracted from the m-th single-source speech data v or V p As a matrix composed of rows, ||.|| is the squared Ferrobenius norm.

Images