Speech recognition and training method for speech disorder rehabilitation patient

Abstract

The present application relates to the technical field of speech recognition, in particular to a speech recognition and training method for speech disorder rehabilitation patients. The present application first extracts the text in the speech data, and obtains the problem words and the standard scores of the text of the patient's pronunciation; further constructs the feature vector of each text, and classifies the standard pronunciation text and the text of the patient's pronunciation; further obtains the pronunciation disorder degree of each problem word of the patient according to the text difference and the feature vector difference between the pronunciation word class cluster and the corresponding standard word class cluster, in combination with the standard score of the text of the patient and the proportion of the problem word; finally, according to the distribution of the problem word in the sentence, in combination with the pronunciation disorder degree and the appearance frequency of the problem word, the pronunciation difficulty of each problem word is obtained, the personalized pronunciation disorder of the patient is accurately identified, and personalized training schemes are formulated for relevant personnel, thereby providing support for improving the intervention effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech recognition, in particular to a speech recognition and training method for speech disorder rehabilitation patients. BACKGROUND

[0002] For neurosurgery patients, due to the diversity of speech disorders, the causes, pathological mechanisms and effects on pronunciation of each type are different, which in turn leads to different effects on pronunciation.

[0003] When conducting speech training for neurosurgery patients with speech disorders, due to the different brain injury conditions of neurosurgery patients, if only the traditional standardized process is used to train each type of pronunciation word by word, the pronunciation characteristics and speech error patterns of different patients will be ignored, which is difficult to meet the actual needs of patients and is not conducive to effective intervention for differentiated speech disorders, thus greatly reducing the training effect. SUMMARY

[0004] In order to solve the technical problem that the conventional training method cannot accurately identify the individualized speech disorders of patients and affects the rehabilitation process, the purpose of the present application is to provide a speech recognition and training method for speech disorder rehabilitation patients, and the technical solution adopted is as follows:

[0005] Obtain the speech data of the neurosurgery patient for the predetermined text and the speech data of the standard speech, and split them into individual words; use a pre-trained neural network to filter out the problem words of the patient's pronunciation and obtain the standard score of the pronunciation of each word;

[0006] Construct a feature vector for each word based on the number of phonemes, speech duration and pronunciation intensity of each word in the speech data; classify the words according to the differences between the feature vectors of different words of the standard speech to obtain standard word class clusters; classify the patient's words according to the differences between the feature vectors of the patient's words and the feature vectors of the words in the standard word class clusters to obtain pronunciation word class clusters;

[0007] According to the differences in words and feature vectors between the pronunciation word class clusters and the corresponding standard word class clusters, combined with the standard score of the patient's words and the proportion of the problem words, obtain the pronunciation disorder degree of each problem word of the patient; according to the distribution of the problem words in the sentence, combined with the pronunciation disorder degree and the frequency of the problem words, obtain the pronunciation difficulty of each problem word.

[0008] Further, the method for obtaining the pronunciation disorder degree comprises:

[0009] According to the quantity difference and category difference of the included characters of the pronunciation character cluster and the corresponding standard character cluster, the classification accuracy of the pronunciation character cluster of the patient is obtained;

[0010] Any of the problem characters of the patient is selected as a target character, and according to the difference between the feature vector of the target character and the feature vector of each character in the corresponding standard character cluster, the pronunciation abnormality degree of the target character is obtained;

[0011] According to the classification accuracy of the pronunciation character cluster where the target character is located, the proportion of the problem characters and the standard score of all characters, and in combination with the pronunciation abnormality degree of the target character, the pronunciation disorder degree of the target character of the patient is obtained; the classification accuracy is negatively correlated with the pronunciation disorder degree.

[0012] Further, the classification accuracy obtaining method comprises:

[0013] According to the absolute value of the difference between the number of characters of the pronunciation character cluster and the number of characters of the corresponding standard character cluster, and in combination with the intersection-union ratio of the pronunciation character cluster and the corresponding standard character cluster, the classification accuracy of the characters of the patient is obtained.

[0014] Further, the pronunciation difficulty obtaining method comprises:

[0015] According to the distribution of the problem characters in each sentence, the continuous pronunciation difficulty of each sentence is obtained.

[0016] The difficulty weight of each problem character is obtained by fusing the frequency of each problem character appearing in the predetermined text and the pronunciation disorder degree.

[0017] For each sentence where any of the problem characters is located, the pronunciation sub-difficulty corresponding to each sentence is obtained according to the continuous pronunciation difficulty and the difficulty weight of all problem characters in the sentence; and the mean value of the pronunciation sub-difficulty of all sentences where the problem characters are located is taken as the pronunciation difficulty corresponding to the problem characters.

[0018] Further, the continuous pronunciation difficulty obtaining method comprises:

[0019] According to the number of problem characters in each sentence and the maximum continuous number of problem characters, the continuous pronunciation difficulty of each sentence is obtained.

[0020] Further, the standard character cluster obtaining method comprises:

[0021] According to the Euclidean distance between the feature vectors of different characters, the characters are clustered to obtain a standard character cluster.

[0022] Further, the pronunciation character cluster obtaining method comprises:

[0023] obtaining the mean value of the feature vectors of all characters in each standard character class cluster as a representative feature vector;

[0024] the standard character class cluster with the most similar representative feature vector of the feature vector of the character of the patient as the standard character class cluster matched by the corresponding character of the patient, and all characters of the patient matched by the same standard character class cluster are classified into a category to obtain a pronunciation character class cluster.

[0025] Further, the mean value of the amplitude value in the speech signal of the character is taken as the pronunciation strength of the corresponding character.

[0026] Further, the character splitting method comprises: converting the speech signal in the speech data into a character sequence using Wav2Vec, and splitting the character sequence into individual characters.

[0027] Further, the obtaining of the pronunciation difficulty of each problem character further comprises:

[0028] According to the pronunciation difficulty and the pronunciation difficulty of the problem character, the preset training times of the problem character are adjusted to obtain a corrected training times.

[0029] The present application has the following beneficial effects:

[0030] The present application first extracts characters and problem characters in speech data, which facilitates the analysis of the pronunciation disorder performance of different characters of the patient, obtains the standard score of the character to quantify the pronunciation standard degree, and provides a basis for subsequent analysis of pronunciation disorder; further, the feature vector of each character is constructed based on the number of phonemes, the speech duration and the pronunciation strength, which represents the pronunciation characteristics of the character and provides a basis for character classification; further, the standard pronunciation characters are classified to obtain standard character class clusters, and the patient's pronunciation characters are classified to obtain pronunciation character class clusters, which facilitates the analysis of the pronunciation deviation of the patient for a similar pronunciation character, and accurately identifies the type and mode of the pronunciation deviation of the patient; further, according to the character difference and the feature vector difference between the pronunciation character class cluster and the corresponding standard character class cluster and the proportion of the problem character, the actual pronunciation deviation of the patient is reflected from the overall perspective of a type of character, the pronunciation deviation of the patient is reflected from the local perspective of a single character according to the standard score of the character of the patient, and the pronunciation disorder degree is obtained comprehensively to comprehensively evaluate the pronunciation disorder of the patient; finally, the distribution of the problem character in the sentence and the occurrence frequency of the problem character are combined to supplement the pronunciation disorder degree, obtain the pronunciation difficulty of each problem character, accurately identify the personalized pronunciation disorder of the patient, and provide support for relevant personnel to develop personalized training programs and improve intervention effect. BRIEF DESCRIPTION OF DRAWINGS

[0031] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 A flowchart illustrating a speech recognition and training method for patients with speech disorders, provided as an embodiment of the present invention;

[0033] Figure 2 A flowchart illustrating a method for obtaining speech impairment according to an embodiment of the present invention;

[0034] Figure 3 A flowchart illustrating a method for obtaining pronunciation difficulty according to an embodiment of the present invention. Detailed Implementation

[0035] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a speech recognition and training method for patients with speech disorders proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0037] The following description, in conjunction with the accompanying drawings, details a specific scheme for a speech recognition and training method for patients with speech disorders provided by this invention.

[0038] Please see Figure 1 The diagram illustrates a flowchart of a speech recognition and training method for patients with speech disorders, provided by an embodiment of the present invention, specifically including:

[0039] Step S1: Obtain the speech data of the neurosurgical patient for the predetermined text and the speech data of the standard speech, and break them down into individual characters; use a pre-trained neural network to filter out the problematic words in the patient's pronunciation and obtain a standard score for the pronunciation of each character.

[0040] Due to brain nerve damage, neurosurgical patients often experience complex and diverse speech disorders. The degree and type of impairment vary among patients depending on the area and extent of brain damage, resulting in different pronunciations of different words. Therefore, it is necessary to compare the patient's speech with standard pronunciation to identify problematic words with significant pronunciation deviations. This helps to pinpoint the specific pronunciation defects caused by nerve damage, accurately identify the patient's individualized speech disorders, provide reliable information for relevant personnel, and facilitate the development of targeted training programs.

[0041] For example, if it is determined that the patient has a common pronunciation problem with retroflex consonants, retroflex consonants should be the focus of attention and training in subsequent training.

[0042] In one embodiment of the present invention, a predetermined text and voice data of a standard speech of the predetermined text are prepared.

[0043] During the data collection process, a quiet room with good sound insulation should be selected as the collection location to reduce external noise interference and ensure that the collected speech samples are clear and identifiable.

[0044] Selection of acquisition equipment: Voice acquisition equipment such as voice recorders and professional microphones can be used to ensure that the acquired voice has high sound quality and clarity. When using it, the microphone should be placed in a suitable position, generally about 20-40 cm away from the patient's mouth, to avoid popping sound due to being too close or too far away, resulting in too low sound.

[0045] For the collected speech data of neurosurgical patients, professional audio editing software is used to remove background noise from the collected speech, so as to minimize noise interference and improve the clarity and recognizability of the speech while preserving the patient's speech characteristics. Then, silent parts in the speech are identified and deleted, and the speech is organized and stored according to the patient's name, collection time, etc.

[0046] Voice data is converted into digital signals through analog-to-digital conversion.

[0047] Preferably, in one embodiment of the present invention, Wav2Vec is used to convert the speech signal in the speech data into a text sequence, and the text sequence is split into individual characters; at the same time, the speech data of the predetermined text is also split to obtain the corresponding text.

[0048] Furthermore, Mel-frequency cepstral coefficients (MFCC) are used to extract acoustic feature vectors for the pronunciation of each word in the patient's and the predetermined text; a pre-trained neural network is then used to match the patient's pronunciation of each word with the standard pronunciation of the predetermined text.

[0049] The model's input consists of the acoustic feature vector of each word spoken by the patient and the acoustic feature vector of the standard pronunciation of the given text. The model's output is the judgment result ("yes" or "no") of whether the patient's pronunciation of each word matches the standard pronunciation, as well as the standard score corresponding to each word, and the standard score is normalized to the range of 0-1. The pre-trained neural network is a convolutional neural network; 0 represents a complete mismatch and the pronunciation is completely non-standard, and 1 represents a complete match and the pronunciation is completely standard. The words with a no matching result are the problematic words in the patient's pronunciation.

[0050] It should be noted that the methods of Wav2Vec, Mel frequency cepstral coefficients, and training neural network contrast vectors are all well-known techniques in the art. The selection of the predefined text can be chosen by the implementer based on factors such as the age and education level of the neurosurgical patient, without any restrictions.

[0051] Step S2: Construct a feature vector for each character based on the number of phonemes, speech duration, and pronunciation intensity of each character in the speech data; classify the characters to obtain standard character clusters based on the differences between the feature vectors of different characters in the standard speech; classify the patient's characters to obtain pronunciation character clusters based on the differences between the feature vectors of each character in the patient and the feature vectors of characters in the standard character clusters.

[0052] Considering that the acoustic features, pitch, and phonemes of each character in the pre-defined text are not exactly the same, but the pronunciation of different characters may be similar, the characters are first classified based on the pronunciation features of characters in standard speech, so as to analyze the pronunciation deviation of patients for a class of similar pronunciation characters and accurately identify the type and pattern of the patient's pronunciation deviation.

[0053] Considering that phonemes are the smallest units that make up syllables and represent the complexity of speech structure; speech duration can reflect the duration of text pronunciation, and pronunciation intensity can reflect the volume control when pronouncing text. Therefore, we first construct the feature vector of each character based on the number of phonemes, speech duration, and pronunciation intensity of each character in the speech data to characterize the pronunciation features of the character and provide a basis for character classification; then, based on the differences between the feature vectors of different characters in standard speech, we classify the characters to obtain standard character clusters.

[0054] In one embodiment of the present invention, considering that Euclidean distance can measure the difference between feature vectors of different characters, Euclidean distance is used as the clustering distance. Characters are clustered according to the Euclidean distance between feature vectors of different characters to obtain standard character clusters.

[0055] It should be noted that before calculating the Euclidean distance between feature vectors, the values ​​of each dimension of the feature vectors can be normalized, such as by linear normalization, to avoid the influence of differences in the range of data values ​​of different dimensions, so that each dimension has the same impact on the clustering distance. When clustering, the K-means clustering algorithm can be used. Both K-means and Euclidean distance are existing technologies and will not be elaborated further.

[0056] It should be noted that, in one embodiment of the present invention, when dividing the text, the speech signal is also divided, and the average amplitude value of the speech signal of the text is used as the pronunciation intensity of the corresponding text.

[0057] After classifying the texts with normal pronunciation, the texts of neurosurgical patients can be compared and matched with standard character clusters. By using the classification results of normal similar pronunciations, the patient's pronounced texts can be classified, which facilitates the subsequent identification of the deviation type of the patient's actual pronunciation and quantifies the pronunciation difficulty of the texts. Considering that the difference between the feature vector of the patient's pronounced texts and the feature vector of the texts in the standard character clusters represents the degree of matching between the two, the patient's texts are classified into pronounced character clusters based on the difference between the feature vector of each text in the patient and the feature vector of the texts in the standard character clusters.

[0058] Preferably, in one embodiment of the present invention, considering that the mean of all feature vectors within a cluster represents the overall pronunciation features of all characters, the mean of all feature vectors within each standard character class cluster is obtained as the representative feature vector to provide a matching basis for patient pronunciation character classification;

[0059] Considering the existence of multiple standard character clusters and multiple representative feature vectors, the higher the similarity between the feature vector of the patient's pronunciation and a certain representative feature vector, the more the actual pronunciation of the text matches the pronunciation of this type of standard text. Therefore, the standard character cluster of the representative feature vector that is most similar to the feature vector of the patient's text is taken as the standard character cluster that matches the patient's corresponding text. All the texts of the patient matched by the same standard character cluster are classified into one category to obtain the pronunciation character cluster.

[0060] As an example, the similarity between the feature vector and the representative feature vector is measured by Euclidean distance. The smaller the Euclidean distance, the higher the similarity. This represents the difference between the feature vector of each character in the patient's speech and the feature vector of the characters in the standard character class. The standard character class with the smallest Euclidean distance is selected as the standard character class that matches the patient's corresponding character, thus completing the classification of the patient's actual pronunciation. At this time, the patient's pronunciation character class corresponds one-to-one with the standard character class.

[0061] In other embodiments of the present invention, the implementer may also choose to match the feature vector of the patient's pronunciation with the feature vector of the standard pronunciation of each character, and select the standard character class cluster in which the standard pronunciation of the character with the smallest Euclidean distance is located as the standard character class cluster matched with the patient's corresponding character.

[0062] Step S3: Based on the differences in character and feature vector between the pronunciation character cluster and the corresponding standard character cluster, combined with the patient's standard score and the proportion of problem characters, obtain the pronunciation difficulty of each problem character; based on the distribution of problem characters in the sentences they belong to, combined with the pronunciation difficulty and frequency of occurrence of problem characters, obtain the pronunciation difficulty of each problem character.

[0063] Considering the differences in character and feature vector between the patient's actual pronunciation character cluster and the corresponding standard character cluster, this represents the deviation of the patient's actual pronunciation character classification result from the standard pronunciation classification result. Combined with the proportion of problematic characters within the pronunciation character cluster, this reflects the patient's actual pronunciation deviation from the overall perspective of a character category.

[0064] The standard score of the patient's text represents the standard score of the neural network for the patient's text pronunciation. It reflects the patient's pronunciation deviation from the local perspective of a single text. Therefore, based on the differences in text and feature vectors between the pronunciation character cluster and the corresponding standard character cluster, combined with the patient's standard score and the proportion of problematic characters, the degree of pronunciation impairment of each problematic character of the patient is obtained. This comprehensively assesses the patient's pronunciation impairment and improves the accuracy and interpretability of pronunciation problem identification for neurosurgical language disorder patients.

[0065] Preferably, in one embodiment of the present invention, please refer to Figure 2 The flowchart illustrates a method for obtaining speech impairment according to an embodiment of the present invention, specifically including:

[0066] Step S301: Based on the differences in the number and types of characters contained in the pronunciation character cluster and the corresponding standard character cluster, obtain the classification accuracy of the patient's pronunciation character cluster.

[0067] When a patient's pronunciation is completely standard, the feature vector of the actual pronounced text is highly similar to the feature vector of the standard pronounced text, and the classification result is close to the ideal of complete consistency, which is reflected in the complete consistency of the number and types of text. Conversely, the greater the difference in the number and types of text, the lower the accuracy of classifying the patient's text based on the feature vector and the standard character cluster, indicating that the patient's pronunciation deviates more and the patient's pronunciation disorder is greater, thus providing a basis for obtaining the degree of pronunciation disorder.

[0068] In one embodiment of the present invention, the classification accuracy of the patient's characters is obtained by combining the absolute value of the difference between the number of characters in the pronunciation character cluster and the number of characters in the corresponding standard character cluster, and the intersection-union ratio of the pronunciation character cluster and the corresponding standard character cluster.

[0069] As an example: the absolute value of the difference between the number of characters in the pronunciation character cluster and the number of characters in the corresponding standard character cluster, plus the sum of the preset positive parameter of 0.1 divided by zero, is used as the first denominator; the intersection and union of the characters contained in the pronunciation character cluster and the corresponding standard character cluster are obtained, the ratio of the number of characters in the intersection to the number of characters in the union is used as the intersection-union ratio, the intersection-union ratio is used as the first numerator, and the ratio of the fraction corresponding to the first denominator and the first numerator is used as the classification accuracy.

[0070] The difference in the number of characters contained in the phonetic character cluster and the corresponding standard character cluster is represented by the absolute value of the difference; the difference in the number of characters is represented by the intersection-union ratio; and the classification accuracy is obtained by combining the two to represent the difference in the number of characters in the phonetic character cluster and the corresponding standard character cluster.

[0071] In other embodiments of the present invention, the implementer may also normalize the reciprocals of the first numerator and the first denominator respectively, and then weight and sum the normalization results to obtain the classification accuracy.

[0072] Step S302: Select any problematic character from the patient as the target character; obtain the pronunciation abnormality degree of the target character based on the difference between the feature vector of the target character and the feature vector of each character in the corresponding standard character cluster.

[0073] First, select any problematic character from the patient as the target character for analysis. Considering that the greater the difference between the feature vector of the target character and the feature vector of each character in the standard character cluster, the more the target character deviates from the standard character cluster, and the corresponding standard character cluster is the most matching cluster, the greater the degree of deviation, the more it indicates that the pronunciation of the target character is less standard and the greater the degree of pronunciation abnormality.

[0074] In one embodiment of the present invention, the difference between feature vectors is quantified by Euclidean distance, and the average value of the Euclidean distance between the feature vector of the target character and the feature vector of each character in the corresponding standard character cluster is used as the pronunciation abnormality of the target character.

[0075] Step S303: Based on the classification accuracy of the target word's pronunciation cluster, the proportion of problematic words, and the standard scores of all words, combined with the pronunciation abnormality of the target word, obtain the patient's pronunciation disorder of the target word; classification accuracy is negatively correlated with pronunciation disorder.

[0076] Considering that the lower the standard score of the target word itself, the less standard the pronunciation is judged; the greater the pronunciation abnormality, the less standard the pronunciation of the target word is, and the greater the pronunciation obstacle.

[0077] Considering that when the proportion of problem words in the pronunciation word class cluster where the target word is located is higher, it indicates that the patient has obvious pronunciation difficulties for this type of words. For example, if the patient has difficulty accurately pronouncing the voiceless alveolar fricative, there will be a large number of problem words in the corresponding pronunciation word class cluster. At this time, the higher the pronunciation disorder degree; at the same time, the lower the classification accuracy, it indicates that the deviation degree of the patient's pronunciation is greater, and the greater the patient's pronunciation disorder.

[0078] As an example, take the number of problem words in the pronunciation word class cluster where the target word is located as the numerator, the number of all words as the denominator, and the ratio as the proportion of problem words; after linearly normalizing the product of the reciprocal of the classification accuracy, the proportion of problem words, and the pronunciation abnormality degree, the normalized result is used as the pronunciation disorder degree of the target word.

[0079] In another embodiment of the present invention, a weighted summation method can also be used to fuse the reciprocal of the classification accuracy, the proportion of problem words, and the pronunciation abnormality degree to obtain the pronunciation disorder degree of the target word.

[0080] It should be noted that when the problem word does not exist in the standard word class cluster corresponding to it, it indicates that the pronunciation accuracy of this problem word is relatively low. It may be a word in other clustering clusters that is classified into this clustering cluster due to a large pronunciation error. Therefore, this problem word is recorded as a high-difficulty word. Since the pronunciation disorder degree of the problem word is normalized to the range of 0-1 above, the pronunciation disorder degree of the high-difficulty word is directly recorded as 1 here. When a problem word exists in multiple pronunciation word class clusters, since each pronunciation word class cluster corresponds to a pronunciation disorder degree, the maximum value among them is finally taken as the pronunciation disorder degree of the problem word. [[ID=~13]]

[0081] It should be noted that considering that some words are polysemous words, such as "ci": serving and waiting for an opportunity; "can": participating, uneven, and ginseng; since the different pronunciations of polysemous words have large differences, the implementer can regard each pronunciation of the polysemous word as a separate word.

[0082] For a given training text, in order to fully reflect the language disorders of the patient in language expression scenarios such as daily communication and reading aloud, it is usually not the content of a single word one by one, but some simple phrases or sentences with different words in different sentences. The structure and context of the sentences will also affect the pronunciation of each word. Even for the same word, in different sentences, due to factors such as the connection of the preceding and following words and the distribution of sentence stress, the accuracy of pronunciation may be different. Therefore, the distribution characteristics of problem words within a sentence will also affect the patient's pronunciation;

[0083] Meanwhile, considering that the frequency of occurrence of problem words also reflects the degree of pronunciation difficulty of the problem words, the pronunciation difficulty of each problem word is obtained based on its distribution in the sentence, combined with the degree of pronunciation impairment and frequency of occurrence. This allows for a detailed quantification of the patient's pronunciation difficulty for different problem words, accurate identification of the patient's individual pronunciation impairment, and provides a basis for relevant personnel to develop personalized and targeted training programs, thereby facilitating the improvement of training effectiveness.

[0084] Preferably, in one embodiment of the present invention, please refer to Figure 3 The diagram illustrates a flowchart of a method for obtaining pronunciation difficulty according to an embodiment of the present invention, specifically including:

[0085] Step S311: Based on the distribution of problem words in each sentence, obtain the continuous pronunciation difficulty of each sentence.

[0086] Considering that the more problematic words there are in a sentence, and the greater the maximum consecutive number of problematic words, the more difficult it is for the patient to pronounce the sentence in a continuous and standard manner, the continuous pronunciation difficulty of each sentence is obtained based on the number of problematic words in each sentence and the maximum consecutive number of problematic words.

[0087] As an example, the product of the number of question words in each sentence and the maximum consecutive number of question words is used as the consecutive pronunciation difficulty of the corresponding sentence.

[0088] It should be noted that sentences expressed independently are directly regarded as a single sentence. In one embodiment of the present invention, for the division of sentences expressed continuously, periods, question marks, exclamation marks, and semicolons are used as the dividing criteria, and the text between these symbols is considered as a single sentence; in other embodiments of the present invention, since colons, commas, dashes, and ellipses can also cause pauses in pronunciation, they can also be used as dividing criteria.

[0089] Step S312: Combine the frequency of each question word in the predetermined text with the degree of pronunciation difficulty to obtain the difficulty weight of each question word.

[0090] Since the problematic words may appear in different sentences, and the distribution of problematic words in different sentences is different, we first analyze the difficulty weight of each problematic word, so as to quantify the pronunciation difficulty of the sentence from the perspective of the problematic words, which is convenient for subsequent comprehensive analysis of the pronunciation difficulty of the problematic words in all sentences.

[0091] Considering that the same problem word may appear in different sentences, the higher the frequency of its appearance in the pre-defined text, the more difficult it is for the patient to pronounce the problem word; the greater the degree of pronunciation impairment, the more difficulty the patient exhibits in pronouncing the word, and the higher the difficulty of pronunciation.

[0092] Based on this, as an example, the product of the frequency of occurrence of the problem word in the text and the pronunciation disorder degree is used as its own difficulty weight.

[0093] As another example, considering that the higher the probability of the character corresponding to the problem word being the problem word, the higher the difficulty of the problem word and the more difficult it is for the patient to pronounce accurately; it is also possible to use the proportion of the problem word in the characters corresponding to the problem word as the basis for calculating the difficulty weight, and use the product of the proportion of the problem word, the frequency of occurrence in the text, and the pronunciation disorder degree as its own difficulty weight.

[0094] It is also possible to fuse the frequency and the pronunciation disorder degree by means of addition or weighted summation.

[0095] It should be noted that not all the characters corresponding to the problem words are necessarily mispronounced. For example, the character "程" corresponding to the problem word appears 10 times in total, and 5 of them are problem words, so the proportion is 1 / 2; the text has a total of 400 characters, so the frequency is 5 / 400.

[0096] Step S313: For each sentence where any problem word is located, obtain the pronunciation sub-difficulty corresponding to each sentence according to the continuous pronunciation difficulty and the difficulty weight of all the problem words in the sentence; use the average value of the pronunciation sub-difficulties of all the sentences where the problem word is located as the pronunciation difficulty corresponding to the problem word.

[0097] Since the problem word may appear in multiple sentences, each sentence where the problem word is located is analyzed one by one;

[0098] Considering that in a certain sentence where the target word is located, the greater the continuous pronunciation difficulty of the sentence, the greater the deviation between the actual pronunciation of the sentence and the standard pronunciation, indicating that the pronunciation difficulty of the target word is greater from the overall perspective of the sentence; the greater the difficulty weight of all the problem words in the sentence, the less standard the pronunciation of the sentence where the target word is located from the perspective of the characters, so the pronunciation sub-difficulty is obtained in this way;

[0099] When the pronunciation sub-difficulties of all the sentences where the target word is located are greater, it indicates that the pronunciation difficulty presented by the sentence is more likely to be caused by the target word, and the pronunciation difficulty of the target word is greater. Therefore, use the average value of the pronunciation sub-difficulties of all the sentences where the problem word is located as the pronunciation difficulty corresponding to the problem word.

[0100] As an example, use the product of the sum value of the difficulty weights of all the problem words in the sentence and the continuous pronunciation difficulty of the sentence itself as the pronunciation sub-difficulty of the sentence, and use the average value of the pronunciation sub-difficulties of all the sentences where the problem word is located as the pronunciation difficulty corresponding to the problem word.

[0101] This approach comprehensively assesses the pronunciation difficulty of problem words from multiple perspectives, including their own scores, differences between actual and standard classification clusters, deviations in feature vectors, distribution of problem words in sentences, and frequency in text. This allows for accurate identification of patients' individualized pronunciation disorders, providing support for relevant personnel to develop personalized training programs and improve intervention effectiveness.

[0102] In another embodiment of the present invention, after obtaining the pronunciation difficulty of each problem word, the method further includes:

[0103] Considering that the greater the pronunciation difficulty and degree of pronunciation of the problem words, the more emphasis and the more training sessions are required, the preset number of training sessions for the problem words is adjusted according to their pronunciation difficulty and degree of pronunciation difficulty to obtain a corrected number of training sessions.

[0104] As an example, the default training iterations are 5. The formula for calculating the number of training iterations includes:

[0105]

[0106] Where C i The training iteration count for the i-th question word is indicated by `norm()`, which represents the linear normalization function. Z represents the number of training iterations for the i-th question word. i N represents the pronunciation difficulty of the i-th problematic word; i This indicates the pronunciation difficulty of the i-th question word; This indicates rounding up to the nearest integer.

[0107] The formula for calculating the number of corrective training sessions combines the degree of pronunciation impairment and the degree of pronunciation difficulty through multiplication, and then limits the correction range through normalization. The more pronounced the pronunciation impairment and the more difficult the pronunciation of the problem word, the more corrective training sessions are obtained, providing a reference for relevant personnel to formulate training programs.

[0108] For words with significant pronunciation difficulties, adjust the number of training sessions appropriately and practice them multiple times to allow the vocal organs to gradually become familiar with the correct pronunciation method and position, thereby producing the target sound more accurately. At the same time, repeated training of the pronunciation of specific words can enhance muscle memory and reduce mispronunciation.

[0109] For words without pronunciation problems, the number of training sessions can be reduced to avoid over-fatigue. Repeatedly practicing what has already been mastered may make patients feel bored or resistant, reducing their enthusiasm and focus on training. By reducing the need to correct pronunciations without problems, more energy can be concentrated on words with pronunciation difficulties, thus improving overall training efficiency.

[0110] In summary, to address the technical problem that conventional training methods struggle to accurately identify patients' individualized pronunciation disorders, thus hindering the rehabilitation process, this invention proposes a speech recognition and training method for patients with speech disorders. This invention first acquires speech data, extracts the text, and obtains the standard scores for the problematic words and texts pronounced by the patient. Next, it constructs feature vectors for each text based on phoneme count, speech duration, and pronunciation intensity, classifying texts with standard pronunciation to obtain standard character clusters. Then, based on the differences in feature vectors between each patient's text and texts within the standard character clusters, it obtains pronunciation character clusters. Finally, based on the differences in texts and feature vectors between the pronunciation character clusters and their corresponding standard character clusters, combined with the patient's standard text scores and the proportion of problematic words, it obtains the pronunciation disorder degree of each problematic word. Finally, based on the distribution of problematic words within their respective sentences, combined with the pronunciation disorder degree and frequency of occurrence, it obtains the pronunciation difficulty of each problematic word, accurately identifying the patient's individualized pronunciation disorders and providing support for relevant personnel to develop personalized training plans and improve intervention effectiveness.

[0111] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0112] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A method for speech recognition and training in patients with speech disorders undergoing rehabilitation, characterized in that, The method includes: Acquire speech data of neurosurgical patients for a predetermined text and speech data of standard speech, and break them down into individual characters; use a pre-trained neural network to filter out problematic words in the patients' pronunciation and obtain a standard score for the pronunciation of each character; A feature vector for each character is constructed based on the number of phonemes, speech duration, and pronunciation intensity of each character in the speech data; the characters are classified into standard character clusters based on the differences between the feature vectors of different characters in standard speech; and the patient's characters are classified into pronunciation character clusters based on the differences between the feature vectors of each character in the patient and the feature vectors of characters in the standard character clusters. Based on the differences in character features between the pronunciation character cluster and the corresponding standard character cluster, and the differences in feature vectors, combined with the standard score of the patient's text and the proportion of the problematic characters, the pronunciation difficulty of each problematic character is obtained; based on the distribution of the problematic characters in the sentences they belong to, and combined with the pronunciation difficulty and frequency of occurrence of the problematic characters, the pronunciation difficulty of each problematic character is obtained.

2. The speech recognition and training method for patients with speech disorders according to claim 1, characterized in that, The method for obtaining the degree of speech impairment includes: Based on the differences in the number and types of characters contained in the pronounced character cluster and the corresponding standard character cluster, the classification accuracy of the patient's pronounced character cluster is obtained; Select any of the patient's problematic characters as the target characters; obtain the pronunciation abnormality degree of the target characters based on the difference between the feature vector of the target characters and the feature vector of each character in the corresponding standard character cluster; Based on the classification accuracy of the target character in the pronunciation character cluster, the proportion of the problem character, and the standard score of all characters, combined with the pronunciation abnormality of the target character, the patient's pronunciation disorder of the target character is obtained; the classification accuracy is negatively correlated with the pronunciation disorder.

3. The method for speech recognition and training of patients with speech disorders according to claim 2, characterized in that, The method for obtaining the classification accuracy includes: The patient's character classification accuracy is obtained by combining the absolute value of the difference between the number of characters in the pronunciation character cluster and the number of characters in the corresponding standard character cluster, and the intersection-union ratio of the pronunciation character cluster and the corresponding standard character cluster.

4. The method for speech recognition and training of patients with speech disorders according to claim 1, characterized in that, The methods for obtaining the pronunciation difficulty include: Based on the distribution of the problematic words in each sentence, obtain the continuous pronunciation difficulty of each sentence; The difficulty weight of each problem word is obtained by combining the frequency of its occurrence in the predetermined text with the degree of pronunciation difficulty. For each sentence containing any of the aforementioned problem words, the pronunciation sub-difficulty of each sentence is obtained based on the continuous pronunciation difficulty and the difficulty weight of all the aforementioned problem words in the sentence; the average of the pronunciation sub-difficulties of all sentences containing the problem word is taken as the pronunciation difficulty of the corresponding problem word.

5. The method for speech recognition and training of patients with speech disorders according to claim 4, characterized in that, The method for obtaining the difficulty of continuous pronunciation includes: The continuous pronunciation difficulty of each sentence is obtained based on the number of problematic words in each sentence and the maximum consecutive number of problematic words.

6. The speech recognition and training method for patients with speech disorders according to claim 1, characterized in that, The method for obtaining the standard character class cluster includes: The characters are clustered based on the Euclidean distance between their feature vectors to obtain standard character clusters.

7. The method for speech recognition and training of patients with speech disorders according to claim 1, characterized in that, The method for obtaining the pronunciation character cluster includes: The mean value of the feature vectors of all characters within each standard character class cluster is obtained as the representative feature vector; The standard character class cluster that is most similar to the feature vector of the patient's text is taken as the standard character class cluster that matches the patient's corresponding text; all the texts of the patient matched by the same standard character class cluster are classified into one category to obtain the pronunciation character class cluster.

8. The speech recognition and training method for patients with speech disorders according to claim 1, characterized in that, The average amplitude value in the speech signal of a text is taken as the pronunciation intensity of the corresponding text.

9. The method for speech recognition and training of patients with speech disorders according to claim 1, characterized in that, The text splitting method includes: using Wav2Vec to convert the speech signal in the speech data into a text sequence, and splitting the text sequence into individual characters.

10. A method for speech recognition and training of patients with speech disorders according to claim 1, characterized in that, After obtaining the pronunciation difficulty of each of the problem words, the process also includes: Based on the pronunciation obstacle level and pronunciation difficulty of the problem word, the preset number of training sessions for the problem word is adjusted to obtain a corrected number of training sessions.

Images