Method, device, equipment and storage medium for speech synthesis of hearing-impaired users

Abstract

The present application discloses a method, device, equipment and storage medium for speech synthesis of hearing-impaired users. When it is determined that the pronunciation quality of the target hearing-impaired user meets the first pronunciation quality condition, a first speech synthesis model is obtained. The training data during model training is synthesized with the timbre, text and similar hearing-impaired rhythm of the pronunciation data of healthy people. The similar hearing-impaired rhythm is the rhythm of the pronunciation data of the hearing-impaired user that is similar to the rhythm of the pronunciation data of healthy people. The model trained based on this can learn the pronunciation style of healthy people. Further, the coding features of the rhythm coding module in the model are fixed to the rhythm features of the target hearing-impaired user, and the coding features of the timbre coding module are fixed to the timbre features of the target hearing-impaired user. The processed speech synthesis model obtained is used as a personalized speech synthesis model for the target hearing-impaired user, which can synthesize personalized speech. Because the model has learned the pronunciation style of healthy people, the final synthesized speech is smoother and clearer.

Description

Technical Field

[0001] The present application relates to the field of speech synthesis technology, and more specifically, to a speech synthesis method, apparatus, device and storage medium for hearing-impaired users. Background Art

[0002] There are two types of hearing-impaired people. One type has hearing problems since infancy, which leads to communication barriers. The other type is acquired hearing-impaired people. Due to long-term lack of language communication, the vocal organs become rigid and the articulation organs function degenerate, which greatly affects the control of breathing during the vocalization process, the pitch and strength of the voice, the coordination of the resonance cavity, and the rhythm of intonation, making it difficult to maintain normal timbre and rhythm.

[0003] The existing auxiliary pronunciation systems for hearing-impaired people generally use a unified voice robot to synthesize text to speech, and cannot provide personalized synthesized speech for different hearing-impaired users. The existing voice-changing technology is a high-precision reproduction of the speaker's voice, and does not take into account the actual situation of hearing-impaired people's inaccurate pronunciation. Therefore, the existing voice-changing technology cannot be applied to speech synthesis scenarios for hearing-impaired people. Summary of the invention

[0004] In view of the above problems, the present application is proposed to provide a method, apparatus, device and storage medium for speech synthesis for hearing-impaired users, so as to achieve the purpose of synthesizing personalized and clear speech for hearing-impaired users.

[0005] The specific plan is as follows:

[0006] In a first aspect, a method for speech synthesis for a hearing-impaired user is provided, comprising:

[0007] Obtain the pronunciation quality of the target hearing-impaired user;

[0008] If the pronunciation quality meets the set first pronunciation quality condition, a preconfigured first speech synthesis model corresponding to the first pronunciation quality condition is obtained, the first speech synthesis model includes a rhythm encoding module, a text encoding module, a timbre encoding module and a decoding module, the model is trained using the pronunciation data of a simulated hearing-impaired user as training data, the pronunciation data of the simulated hearing-impaired user is synthesized by the timbre, text and similar hearing-impaired rhythm of the pronunciation data of a healthy person, wherein the similar hearing-impaired rhythm is the rhythm of the pronunciation data of the hearing-impaired user that meets the set rhythm similarity condition with the rhythm of the pronunciation data of the healthy person;

[0009] Acquire the timbre characteristics and prosody characteristics of the target hearing-impaired user, process the first speech synthesis model, and use the processed model as the personalized speech synthesis model for the target hearing-impaired user. The processing includes: fixing the coding characteristics of the prosody coding module as the prosody characteristics of the target hearing-impaired user, and fixing the coding characteristics of the timbre coding module as the timbre characteristics of the target hearing-impaired user.

[0010] In a second aspect, a speech synthesis device for a hearing-impaired user is provided, comprising:

[0011] A pronunciation quality acquisition unit, used to acquire the pronunciation quality of a target hearing-impaired user;

[0012] a first speech synthesis model acquisition unit, configured to acquire a preconfigured first speech synthesis model corresponding to the first pronunciation quality condition if the pronunciation quality satisfies a set first pronunciation quality condition, wherein the first speech synthesis model includes a rhythm encoding module, a text encoding module, a timbre encoding module and a decoding module, and the model is trained using pronunciation data of a simulated hearing-impaired user as training data, wherein the pronunciation data of the simulated hearing-impaired user is synthesized using the timbre, text and similar hearing-impaired rhythm of the pronunciation data of a healthy person, wherein the similar hearing-impaired rhythm is the rhythm of the pronunciation data of the hearing-impaired user that satisfies a set rhythm similarity condition with the rhythm of the pronunciation data of the healthy person;

[0013] The first speech synthesis model processing unit is used to obtain the timbre characteristics and prosody characteristics of the target hearing-impaired user, process the first speech synthesis model, and use the processed model as the personalized speech synthesis model of the target hearing-impaired user, wherein the processing includes: fixing the coding characteristics of the prosody coding module as the prosody characteristics of the target hearing-impaired user, and fixing the coding characteristics of the timbre coding module as the timbre characteristics of the target hearing-impaired user.

[0014] In a third aspect, a speech synthesis device for a hearing-impaired user is provided, comprising: a memory and a processor;

[0015] The memory is used to store programs;

[0016] The processor is used to execute the program to implement the various steps of the method for synthesizing speech for hearing-impaired users as described above.

[0017] In a fourth aspect, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the various steps of the method for synthesizing speech for hearing-impaired users as described above are implemented.

[0018] By means of the above technical solution, the present application first determines the pronunciation quality of the target hearing-impaired user, and when it is determined that the first pronunciation quality condition is met, a pre-configured first speech synthesis model corresponding to the condition is obtained. The first speech synthesis model pre-configured in the present application includes a rhythm encoding module, a text encoding module, a timbre encoding module and a decoding module. It can be seen that the present application can decouple the rhythm and timbre of the speech by setting three encoding modules of rhythm, timbre and text. At the same time, the training data used in the training of the first speech synthesis model is synthesized with the timbre, text and similar hearing-impaired rhythm of the pronunciation data of healthy people. The similar hearing-impaired rhythm is the rhythm of the pronunciation data of the hearing-impaired user that meets the set rhythm similarity condition with the rhythm of the pronunciation data of the healthy person. That is, the training data retains the rhythm of the hearing-impaired user and transfers the pronunciation style of the healthy person. The first speech synthesis model trained based on the training data can learn the pronunciation style of the healthy person. Based on the first speech synthesis model, the present application obtains the timbre characteristics and rhythmic characteristics of the target hearing-impaired user, fixes the coding characteristics of the rhythmic coding module in the first speech synthesis model to the rhythmic characteristics of the target hearing-impaired user, and fixes the coding characteristics of the timbre coding module to the timbre characteristics of the target hearing-impaired user. The processed speech synthesis model obtained thereby is used as a personalized speech synthesis model for the target hearing-impaired user, thereby realizing the synthesis of personalized speech for the target hearing-impaired user. Moreover, since the model has learned the pronunciation style of healthy people, the final synthesized speech is smoother and clearer. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0020] Figure 1 A schematic diagram of a flow chart of a method for synthesizing speech for hearing-impaired users provided in an embodiment of the present application;

[0021] Figure 2 An example of a schematic diagram of the training process of an autoencoder neural network model is shown;

[0022] Figure 3 A schematic diagram of the structure of a text processing device provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of the structure of a text processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] The present application provides a speech synthesis solution for hearing-impaired users, which can be applied to scenarios where speech synthesis is performed for hearing-impaired users.

[0026] The present application scheme can be implemented based on a terminal with data processing capabilities, which can be a voice recorder, mobile phone, computer, server, cloud, etc.

[0027] Next, combine Figure 1 The method for synthesizing speech for hearing-impaired users of the present application may include the following steps:

[0028] Step S100: Acquire the pronunciation quality of the target hearing-impaired user.

[0029] Among them, the target hearing-impaired user is the hearing-impaired user who needs to perform speech synthesis. The pronunciation quality can be evaluated in the form of pronunciation level, pronunciation score, etc. This application can pre-set different pronunciation quality conditions, and different pronunciation quality conditions represent different requirements for pronunciation quality. In this step, the pronunciation quality of the target hearing-impaired user is obtained, and the pronunciation quality conditions satisfied by the user can be determined. Different speech synthesis strategies can be set for different pronunciation quality conditions.

[0030] Step S110: If the pronunciation quality meets a set first pronunciation quality condition, a preconfigured first speech synthesis model corresponding to the first pronunciation quality condition is obtained.

[0031] The first speech synthesis model may include a rhythm encoding module, a text encoding module, a timbre encoding module and a decoding module. The first speech synthesis model is trained using the pronunciation data of a simulated hearing-impaired user as training data. The pronunciation data of the simulated hearing-impaired user is synthesized using the timbre, text and similar hearing-impaired rhythm of the pronunciation data of a healthy person. The similar hearing-impaired rhythm is the rhythm of the pronunciation data of the hearing-impaired user that meets the set rhythm similarity condition with the rhythm of the pronunciation data of the healthy person.

[0032] In this embodiment, the first speech synthesis model is configured to include three modules: prosody coding, text coding and timbre coding, thereby decoupling prosody and timbre.

[0033] Furthermore, in order to enable the first speech synthesis model to synthesize clearer and smoother speech, in this embodiment, when training the first speech synthesis model, a rhythm similar to the rhythm of the pronunciation data of healthy people is selected from the pronunciation set of the collected hearing-impaired users, and then the selected rhythm, as well as the timbre and transcribed text of the pronunciation data of the healthy people are used to synthesize the speech. In this embodiment, the synthesized speech is defined as the pronunciation data of simulated hearing-impaired users. The pronunciation data of the simulated hearing-impaired users is used to train the first speech synthesis model, so that the trained first speech synthesis model can be transferred to learn the pronunciation style of healthy people.

[0034] Step S120: Acquire the timbre characteristics and rhythmic characteristics of the target hearing-impaired user, and based on the timbre characteristics and rhythmic characteristics of the target hearing-impaired user, process the first speech synthesis model to obtain a personalized speech synthesis model for the target hearing-impaired user.

[0035] In this step, the process of obtaining the timbre characteristics of the target hearing-impaired user can be based on a pre-trained timbre characterization model (timbre can also be understood as voiceprint, so the timbre characterization model can also be called a voiceprint characterization model) to extract the timbre of the pronunciation data of the target hearing-impaired user.

[0036] The process of obtaining the prosodic features of the target hearing-impaired user may be to perform prosodic extraction on the pronunciation data of the target hearing-impaired user based on a pre-trained prosodic representation model.

[0037] In this step, the process of processing the first speech synthesis model using the timbre characteristics and prosody characteristics of the target hearing-impaired user may include:

[0038] The coding features of the prosody coding module in the first speech synthesis model are fixed as the prosody features of the target hearing-impaired user, and the coding features of the timbre coding module are fixed as the timbre features of the target hearing-impaired user.

[0039] Among them, the prosodic features of the target hearing-impaired user may be a prosodic distribution space, that is, the coding features of the prosodic coding module in the first speech synthesis model are fixed in the prosodic distribution space of the target hearing-impaired user. When performing speech synthesis on different texts, the prosodic coding module may select specific prosodic features in the prosodic distribution space of the target hearing-impaired user according to the different texts to be synthesized, and then perform speech synthesis on the text according to the selected prosodic features and the timbre features fixed by the timbre coding module, so as to obtain a synthesized speech that meets the personality of the target hearing-impaired user.

[0040] The method provided in the embodiment of the present application first determines the pronunciation quality of the target hearing-impaired user, and when it is determined that the first pronunciation quality condition is met, a pre-configured first speech synthesis model corresponding to the condition is obtained. The first speech synthesis model pre-configured in the present application includes a rhythm encoding module, a text encoding module, a timbre encoding module and a decoding module. It can be seen that the present application can decouple the rhythm and timbre of the speech by setting three encoding modules of rhythm, timbre and text. At the same time, the training data used in the training of the first speech synthesis model is synthesized with the timbre, text and similar hearing-impaired rhythm of the pronunciation data of healthy people. The similar hearing-impaired rhythm is the rhythm of the pronunciation data of the hearing-impaired user that meets the set rhythm similarity condition with the rhythm of the pronunciation data of the healthy person. That is, the training data retains the rhythm of the hearing-impaired user and transfers the pronunciation style of the healthy person. The first speech synthesis model trained based on the training data can learn the pronunciation style of the healthy person. Based on the first speech synthesis model, the present application obtains the timbre characteristics and rhythmic characteristics of the target hearing-impaired user, fixes the coding characteristics of the rhythmic coding module in the first speech synthesis model to the rhythmic characteristics of the target hearing-impaired user, and fixes the coding characteristics of the timbre coding module to the timbre characteristics of the target hearing-impaired user. The processed speech synthesis model obtained thereby is used as a personalized speech synthesis model for the target hearing-impaired user, thereby realizing the synthesis of personalized speech for the target hearing-impaired user. Moreover, since the model has learned the pronunciation style of healthy people, the final synthesized speech is smoother and clearer.

[0041] In some embodiments of the present application, the process of obtaining the pronunciation quality of the target hearing-impaired user in the above step S100 is introduced.

[0042] As an optional implementation method, the present application can obtain the pronunciation quality of the target hearing-impaired user given by a professional pronunciation quality evaluation agency, such as the pronunciation quality of the target hearing-impaired user evaluated by a specialist doctor or researcher.

[0043] As another optional implementation method, the present application can obtain the pronunciation data of the target hearing-impaired user, and then perform a pronunciation quality evaluation on the pronunciation data to obtain the pronunciation quality of the target hearing-impaired user.

[0044] Specifically, in the embodiment of the present application, the pronunciation data of the target hearing-impaired user for the specified text content under the guidance of the designated reading user can be obtained. For example, the present application can select several professional reading persons with different voices to read the specified text content and obtain a recording. The target hearing-impaired user can select a recording of a professional reading person according to the voice, and read the specified text content under the guidance of the recording. The present application obtains the pronunciation data of the target hearing-impaired user for the specified text content.

[0045] Furthermore, the process of evaluating the pronunciation quality of the pronunciation data to obtain the pronunciation quality of the target hearing-impaired user may include:

[0046] The pronunciation data of the target hearing-impaired user is evaluated using the pre-trained pronunciation quality evaluation model to obtain the pronunciation quality of the target hearing-impaired user.

[0047] The pronunciation quality evaluation model may be trained based on the pronunciation data of hearing-impaired users annotated with pronunciation quality evaluation results, wherein the pronunciation quality evaluation results are determined based on the transcription accuracy of the pronunciation data and the acoustic likelihood compared with the standard acoustic model.

[0048] Specifically, the process of establishing a pronunciation quality evaluation model may include:

[0049] S1. Obtain a standard acoustic model.

[0050] The standard acoustic model can be established based on the pronunciation data of ordinary users who are not hearing-impaired. In this embodiment, a large amount of pronunciation data of ordinary people (such as 100,000 or other optional values) can be collected, and the acoustic features of each pronunciation data can be obtained by using speech signal processing technology, and then the characteristics of each acoustic feature are identified according to statistical modeling to obtain a standard acoustic model. The standard acoustic model can be a Gaussian model or other probability statistical models.

[0051] S2. Obtain a speech database of the hearing-impaired population.

[0052] The hearing-impaired database contains a plurality of pieces of pronunciation data of hearing-impaired users and corresponding annotated texts. The annotated text corresponding to each piece of pronunciation data may be the text content corresponding to the pronunciation data that is manually annotated.

[0053] S3. Identify the speech transcription text of each pronunciation data in the hearing-impaired population database, and compare it with the corresponding annotated text, and determine the transcription accuracy of each pronunciation data based on the comparison result.

[0054] Specifically, the transcription text of each pronunciation data can be identified by the speech transcription model. Further, the identified transcription text is compared with the annotated text corresponding to the pronunciation data, and the transcription accuracy of each pronunciation data is determined based on the comparison result. The transcription accuracy can characterize the expression clarity of the pronunciation data.

[0055] S4. Compare each piece of pronunciation data in the database with the standard acoustic model to obtain the acoustic likelihood of each piece of pronunciation data.

[0056] Specifically, each piece of pronunciation data may be input into a standard acoustic model to obtain the acoustic likelihood of each piece of pronunciation data output by the standard acoustic model, and the acoustic likelihood reflects the gap between the acoustic features of the pronunciation data and the acoustic features of the standard pronunciation.

[0057] S5. Determine the quality evaluation result of each piece of pronunciation data by combining the transcription accuracy and acoustic likelihood of each piece of pronunciation data.

[0058] Specifically, the transcription accuracy and acoustic likelihood can be combined in a linear weighted manner, and the result is used as the quality evaluation result of the pronunciation data. Of course, in addition to this, other fusion methods can also be used to combine the transcription accuracy and acoustic likelihood of the pronunciation data to obtain the quality evaluation result of the pronunciation data.

[0059] The quality evaluation result may be in the form of quality grade, quality score, etc. In this embodiment, the quality grade is used as an example for explanation:

[0060] The pronunciation quality can be divided into several levels from high to low, such as 1-16. Among them, 1-5 can be defined as good pronunciation quality, 6-13 can be defined as medium pronunciation quality, and 14-16 can be defined as poor pronunciation quality. Of course, the above-mentioned good, medium and poor pronunciation quality are all divided according to the standards defined by the user.

[0061] The foregoing Figure 1 In the corresponding embodiment, the pronunciation quality satisfies the first pronunciation quality condition, wherein the first pronunciation quality condition may be that the pronunciation level is between level 6 and level 13, that is, the first pronunciation quality condition may be a condition indicating that the pronunciation quality is medium.

[0062] Further, the present application will also mention the second pronunciation quality condition and the third pronunciation quality condition hereinafter. Taking the division method of pronunciation quality level in the present embodiment as an example, the corresponding second pronunciation quality condition may be that the pronunciation level is at level 1-5, that is, the second pronunciation quality condition may be a condition indicating good pronunciation quality. The third pronunciation quality condition may be that the pronunciation level is at level 14-16, that is, the third pronunciation quality condition may be a condition indicating poor pronunciation quality.

[0063] S6. Using each piece of pronunciation data in the database and its corresponding quality evaluation result as training data, a pronunciation quality evaluation model is trained.

[0064] Specifically, according to the processing of the aforementioned steps, the corresponding quality evaluation results are determined for each pronunciation data in the hearing-impaired population database. Furthermore, in this step, each pronunciation data marked with the quality evaluation results can be used as training data to train the pronunciation quality evaluation model. The trained pronunciation quality evaluation model can be used to predict the quality evaluation results of the pronunciation data.

[0065] In some embodiments of the present application, the preconfigured first speech synthesis model corresponding to the first pronunciation quality condition obtained in the above step S110 is introduced.

[0066] Specifically, the process of establishing the first speech synthesis model can refer to the following steps:

[0067] S11. Selecting healthy person pronunciation data from a set healthy person acoustic database, and extracting the timbre features and transcribed text of the healthy person pronunciation data.

[0068] Specifically, the present application can collect a large amount of pronunciation data of healthy people in advance to form a healthy person acoustic database. Further, the pronunciation data of healthy people are randomly selected from the healthy person acoustic database to extract the timbre characteristics of the pronunciation data and the corresponding transcription text.

[0069] Furthermore, the present application can also collect pronunciation data of hearing-impaired users in advance to form an acoustic database for the hearing-impaired population.

[0070] For the convenience of description, the sound person pronunciation data selected from the sound person acoustic database is defined as the source pronunciation data.

[0071] S12, using a pre-trained prosody characterization model to extract a first prosody feature of the sound person's pronunciation data, and using the prosody characterization model to extract a second prosody feature of the pronunciation data of each hearing-impaired user in a set hearing-impaired group acoustic database.

[0072] Specifically, the present application can pre-train a prosody characterization model for extracting prosody features of pronunciation data. On this basis, the prosody characterization model is used to extract the first prosody features of the source pronunciation data, and the second prosody features of the pronunciation data of each hearing-impaired user in the hearing-impaired acoustic database are extracted.

[0073] S13. Among the second prosodic features, select a similar hearing-impaired prosodic feature that is most similar to the first prosodic feature, and synthesize the pronunciation data of a simulated hearing-impaired user using the timbre features of the healthy person's pronunciation data, the transcribed text, and the similar hearing-impaired prosodic features.

[0074] Specifically, the previous step obtains the second prosodic feature of each hearing-impaired user's pronunciation data in the hearing-impaired acoustic database. On this basis, the similarity between each second prosodic feature and the first prosodic feature of the source pronunciation data extracted in step S2 can be calculated respectively. The calculation process can use Euclidean distance or other forms to represent the similarity. Then, the second prosodic feature with the highest similarity is determined, and the second prosodic feature with the highest similarity is defined as a similar hearing-impaired prosodic feature.

[0075] After obtaining similar hearing-impaired prosodic features, speech is synthesized using the timbre features of the source pronunciation data selected in step S1, the transcribed text, and similar hearing-impaired prosodic features, and the synthesized speech is defined as pronunciation data of a simulated hearing-impaired user.

[0076] It can be seen from the above scheme that the synthesized pronunciation data of the simulated hearing-impaired user in this embodiment retains the rhythmic characteristics of the hearing-impaired user and at the same time transfers the speaking style of healthy people.

[0077] S14. Training a first speech synthesis model based on the pronunciation data of the simulated hearing-impaired user.

[0078] Since the pronunciation data of the simulated hearing-impaired user retains the rhythm of the hearing-impaired user and transfers the pronunciation style of the healthy person, the first speech synthesis model trained based on the training data can learn the pronunciation style of the healthy person.

[0079] In some other embodiments of the present application, another process for establishing the first speech synthesis model is introduced, which can refer to the following steps:

[0080] S21. Mix the acoustic database of healthy people and the acoustic database of hearing-impaired people, and train the autoencoder neural network model based on the mixed database.

[0081] Among them, the acoustic database of healthy people and the acoustic database of hearing-impaired people can refer to the introduction in the previous article.

[0082] Reference Figure 2 As shown, the autoencoding neural network model may include a rhythm encoding module, a text encoding module, a timbre encoding module and a decoding module.

[0083] Among them, the text encoding module and the timbre encoding module are pre-configured, and the network parameters remain unchanged during the training process.

[0084] That is, in the training process of the autoencoder neural network model, it is only necessary to extract the prosodic features of the input pronunciation data x through the prosodic coding module. The text feature c of the pronunciation data x can be determined by the preconfigured text coding module, and the timbre feature s can be determined by the preconfigured timbre coding module. During the training process, the prosodic feature z, the text feature c and the timbre feature s can be constrained by mutual information (MMI). According to the prior assumption, the input speech data mainly contains the timbre feature s and the text feature c, and the rest is only the prosodic feature z. Among them, the timbre feature s and the text feature c are determined by the preconfigured text coding module and the timbre coding module, so the pronunciation data can be decoupled through the mutual information constraint, and the corresponding prosodic features can be obtained using the prosodic coding module.

[0085] The decoding module can decode the pronunciation data x using the rhythm feature z, text feature c and timbre feature s.

[0086] According to the above training method, the prosody encoding module in the trained autoencoder neural network model has the ability to extract prosody features from pronunciation data, so it can be used as a prosody representation model for use in the following step S23.

[0087] S22. Selecting healthy person pronunciation data from a set healthy person acoustic database, and extracting the timbre features and transcribed text of the healthy person pronunciation data.

[0088] S23, using a pre-trained prosody characterization model to extract a first prosody feature of the sound person's pronunciation data, and using the prosody characterization model to extract a second prosody feature of the pronunciation data of each hearing-impaired user in a set hearing-impaired group acoustic database.

[0089] Specifically, the prosody representation model mentioned in this step may adopt the prosody encoding module trained in the above step S21.

[0090] S24. Among each of the second prosodic features, select a similar hearing-impaired prosodic feature that is most similar to the first prosodic feature, input the timbre features of the healthy person's pronunciation data, the transcribed text and the similar hearing-impaired prosodic feature into the decoding module in the trained autoencoder neural network model, and obtain the pronunciation data of the simulated hearing-impaired user synthesized after decoding.

[0091] Specifically, the decoding module in the trained autoencoder neural network model in step S21 has the ability to decode the input prosodic features z, text features c and timbre features s to obtain the pronunciation data x. Therefore, in this step, the timbre features of the pronunciation data of a healthy person, the transcribed text and the similar hearing-impaired prosodic features can be input into the decoding module to obtain the pronunciation data of the simulated hearing-impaired user synthesized after decoding.

[0092] S25. Training a first speech synthesis model based on the pronunciation data of the simulated hearing-impaired user.

[0093] Specifically, after obtaining the pronunciation data of the simulated hearing-impaired user, the pronunciation data can be used as training data to retrain a speech synthesis model.

[0094] The method for establishing the first speech synthesis model introduced in this embodiment obtains a prosody encoding module and a decoding module by training an autoencoder neural network model. The prosody encoding module is used as a prosody representation model to extract the first prosody feature of the pronunciation data of healthy people, and the second prosody feature of the pronunciation data of each hearing-impaired user in the hearing-impaired acoustic database. The decoding module can be used to obtain the pronunciation data of the simulated hearing-impaired user synthesized after decoding based on the determined timbre features of the pronunciation data of healthy people, the transcribed text and the similar hearing-impaired prosody features, and finally the pronunciation data of the simulated hearing-impaired user is used to train the first speech synthesis model.

[0095] It can be understood that the prosody representation model obtained through training in this embodiment can also be used in the aforementioned step S120 to extract the prosody features of the pronunciation data of the target hearing-impaired user.

[0096] In some embodiments of the present application, another optional implementation of the method for synthesizing speech for hearing-impaired users is introduced. Based on the above embodiments, the method may further include the following steps:

[0097] If the pronunciation quality meets the set second pronunciation quality condition, a preconfigured second speech synthesis model corresponding to the second pronunciation quality condition is used as the personalized speech synthesis model for the target hearing-impaired user.

[0098] The second pronunciation quality condition has higher requirements on pronunciation quality than the first pronunciation quality condition. As described above, if the pronunciation quality is divided into 1-16 levels, the higher the numerical level, the worse the pronunciation quality, such as the pronunciation quality of level 16 is lower than the pronunciation quality of level 1.

[0099] The first pronunciation quality condition may be that the pronunciation level is between 6 and 13, and the second pronunciation quality condition may be that the pronunciation level is between 1 and 5.

[0100] The second speech synthesis model may be a speech synthesis model obtained by fine-tuning a general speech synthesis model using the pronunciation data of a target hearing-impaired user and the corresponding transcribed text.

[0101] In this embodiment, when it is determined that the pronunciation quality of the target hearing-impaired user meets the second pronunciation quality condition, that is, the pronunciation quality is considered to be good, fine-tuning can be performed on the basis of the general speech synthesis model.

[0102] The general speech synthesis model may be a model suitable for speech synthesis for healthy people. When fine-tuning it, only the timbre encoding layer and the decoding layer may be fine-tuned to ensure that the timbre of the synthesized speech output by the adjusted model is similar to the timbre of the target hearing-impaired user.

[0103] In some embodiments of the present application, another optional implementation of the method for synthesizing speech for hearing-impaired users is introduced. Based on the above embodiments, the method may further include:

[0104] If the pronunciation quality meets the set third pronunciation quality condition, the personalized speech synthesis model of the target hearing-impaired user is obtained in the following manner in this embodiment.

[0105] Among them, the third pronunciation quality condition has lower requirements on pronunciation quality than the first pronunciation quality condition.

[0106] According to the above description, if the pronunciation quality is divided into 1-16 levels, the first pronunciation quality condition may be that the pronunciation level is 6-13, the second pronunciation quality condition may be that the pronunciation level is 1-5, and the third pronunciation quality condition may be that the pronunciation level is 14-16.

[0107] Specifically, when the third pronunciation quality condition is met, the process of obtaining the personalized speech synthesis model of the target hearing-impaired user may include the following steps:

[0108] S1. Acquire the timbre characteristics of the target hearing-impaired user.

[0109] Specifically, the timbre characteristics of the target hearing-impaired user can be extracted through a pre-trained timbre representation model.

[0110] S2. In a preconfigured mixed timbre space, select a number of timbre features that meet a set timbre similarity condition with the timbre features of the target hearing-impaired user, and calculate an average timbre feature of the selected number of timbre features.

[0111] The mixed timbre space can be established based on the pronunciation data of healthy users and hearing-impaired users. Specifically, the collected pronunciation data of healthy users and the collected pronunciation data of hearing-impaired users can be mixed together, and the timbre features of each mixed pronunciation data are extracted respectively, and the extracted timbre features are combined into a mixed timbre space.

[0112] By mixing the timbre features of healthy users and hearing-impaired users to construct a mixed timbre space, it is possible to ensure that the timbre features that are sufficiently similar to the target hearing-impaired user can be selected. By averaging several selected timbre features that meet the similarity conditions, an average timbre feature is obtained, thereby achieving the migration of the timbre of the target hearing-impaired user.

[0113] The selected timbre features satisfying the set timbre similarity condition may be one or more timbre features.

[0114] S3. Obtain the prosodic features of one or more other users' pronunciations, and synthesize each prosodic feature obtained with the average timbre feature and the set text to obtain a synthesized speech corresponding to each prosodic feature.

[0115] Specifically, when the pronunciation quality of the target hearing-impaired user meets the third pronunciation quality condition, it can be understood that the pronunciation quality of the target hearing-impaired user is poor, which means that the prosodic features extracted based on the pronunciation data of the target hearing-impaired user can no longer be used. If speech synthesis is performed based on the prosodic features, it may cause the synthesized speech to be unclear. For this reason, in this step, the prosodic features of the pronunciation of other users are selected to replace the prosodic features of the pronunciation of the target hearing-impaired user for speech synthesis. Among them, other users may be other users whose pronunciation quality exceeds the requirements of the third pronunciation quality condition. For example, users who meet the requirements of the second pronunciation quality condition can be used as other users in this step. On this basis, the prosodic features of the pronunciation of other users obtained are synthesized with the average timbre features and the set text to ensure the clarity of the synthesized speech.

[0116] In addition, in order to allow the target hearing-impaired user to have more choices, the prosodic features of the pronunciation of one or more other users can be obtained in this step, and the synthesized speech corresponding to each prosodic feature can be obtained. Exemplarily, the prosodic features of the pronunciations of 6 different users can be selected in this step, and then the synthesized speech corresponding to each of the 6 different prosodic features can be obtained.

[0117] S4. Outputting the synthesized speech corresponding to each prosodic feature, and in response to the target hearing-impaired user's operation of selecting a target synthesized speech, determining the target prosodic feature corresponding to the target synthesized speech.

[0118] Specifically, the target hearing-impaired user can select a target synthesized speech that he or she likes from the output synthesized speech according to his or her own preferences, and then determine the target prosodic features corresponding to the target synthesized speech in this step.

[0119] S5. Obtain a third speech synthesis model including a prosody encoding module, a text encoding module, a timbre encoding module and a decoding module, process the third speech synthesis model, and use the processed model as a personalized speech synthesis model for the target hearing-impaired user.

[0120] The processing includes: fixing the coding feature of the prosody coding module as the target prosody feature, and fixing the coding feature of the timbre coding module as the average timbre feature.

[0121] Specifically, in order to obtain a personalized speech synthesis model corresponding to the target hearing-impaired user, in this step, the coding features of the prosody coding module in the third speech synthesis model are fixed to the target prosody features determined in the previous step, and the coding features of the timbre coding module are fixed to the average timbre features determined in the above step.

[0122] The target prosodic feature may be a prosodic distribution space, that is, the coding feature of the prosodic coding module in the third speech synthesis model is fixed in the prosodic distribution space corresponding to the target prosodic feature. When performing speech synthesis on different texts, the prosodic coding module may select specific prosodic features in the prosodic distribution space according to the different texts to be synthesized, and then perform speech synthesis on the text according to the selected prosodic features and the timbre features fixed by the timbre coding module, so as to obtain a synthesized speech that meets the personality of the target hearing-impaired user.

[0123] Optionally, the synthesized speech obtained in the above step S3 may result in a situation where the timbre of some synthesized speech is too different from the timbre of the target hearing-impaired user because the synthesized speech is based on different prosodic features. If such synthesized speech is output, the probability of being selected by the target hearing-impaired user is very low, and it will affect the selection operation of the target hearing-impaired user. To this end, the following step can be added between the above steps S3 and S4 in the embodiment of the present application:

[0124] S6. Acquire the timbre features of the synthesized speech corresponding to each prosodic feature, calculate the similarity between the timbre features of each synthesized speech and the timbre features of the target hearing-impaired user, and select the top N synthesized speech with the highest similarity.

[0125] On this basis, the process of outputting the synthesized speech corresponding to each prosodic feature in step S4 specifically includes:

[0126] Output the first N synthesized speech sounds selected in the above step S6.

[0127] In some embodiments of the present application, an optional method for establishing the preconfigured mixed timbre space in S2 is introduced, and the specific process is as follows:

[0128] 1) For each pronunciation data in the database containing pronunciation data of healthy users and hearing-impaired users, the timbre features are extracted respectively.

[0129] 2) Cluster the extracted timbre features, and form a mixed timbre space from the clusters obtained by clustering.

[0130] Specifically, according to the method of this embodiment, the mixed timbre space is composed of a number of clusters, each of which contains a number of timbre features. It can be understood that the similarity of the timbre features in the same cluster is relatively high.

[0131] On this basis, the process of selecting a plurality of timbre features that meet the set timbre similarity condition with the timbre features of the target hearing-impaired user in the preconfigured mixed timbre space in step S2 may include:

[0132] S21, calculating the distance between the timbre feature of the target hearing-impaired user and the cluster center of each cluster in the mixed timbre space, and determining the target cluster with the shortest distance.

[0133] S22. Taking each timbre feature contained in the target cluster as the timbre feature that satisfies the set timbre similarity condition, or, calculating the timbre feature that is most similar to the timbre feature of the target hearing-impaired user in the target cluster as the timbre feature that satisfies the set timbre similarity condition, or, taking the timbre feature corresponding to the cluster center of the target cluster as the timbre feature that satisfies the set timbre similarity condition.

[0134] The following is a description of a speech synthesis device for hearing-impaired users provided in an embodiment of the present application. The speech synthesis device for hearing-impaired users described below and the speech synthesis method for hearing-impaired users described above can refer to each other.

[0135] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of a speech synthesis device for hearing-impaired users disclosed in an embodiment of the present application.

[0136] like Figure 3 As shown, the device may include:

[0137] A pronunciation quality acquisition unit 11, used to acquire the pronunciation quality of a target hearing-impaired user;

[0138] A first speech synthesis model acquisition unit 12 is used to acquire a preconfigured first speech synthesis model corresponding to the first pronunciation quality condition if the pronunciation quality meets the set first pronunciation quality condition, wherein the first speech synthesis model includes a rhythm encoding module, a text encoding module, a timbre encoding module and a decoding module, and the model is trained using pronunciation data of a simulated hearing-impaired user as training data, wherein the pronunciation data of the simulated hearing-impaired user is synthesized using the timbre, text and similar hearing-impaired rhythm of the pronunciation data of a healthy person, wherein the similar hearing-impaired rhythm is the rhythm of the pronunciation data of the hearing-impaired user that meets the set rhythm similarity condition with the rhythm of the pronunciation data of the healthy person;

[0139] The first speech synthesis model processing unit 13 is used to obtain the timbre characteristics and prosody characteristics of the target hearing-impaired user, and process the first speech synthesis model to use the processed model as the personalized speech synthesis model of the target hearing-impaired user. The processing includes: fixing the coding characteristics of the prosody coding module as the prosody characteristics of the target hearing-impaired user, and fixing the coding characteristics of the timbre coding module as the timbre characteristics of the target hearing-impaired user.

[0140] Optionally, the device of the present application may further include:

[0141] A second speech synthesis model acquisition unit is used to use a preconfigured second speech synthesis model corresponding to the second pronunciation quality condition as a personalized speech synthesis model for the target hearing-impaired user if the pronunciation quality meets a set second pronunciation quality condition, wherein the second speech synthesis model is a speech synthesis model obtained by fine-tuning a general speech synthesis model using the pronunciation data of the target hearing-impaired user and the corresponding transcribed text;

[0142] The second pronunciation quality condition has higher requirements on pronunciation quality than the first pronunciation quality condition.

[0143] Optionally, the device of the present application may further include:

[0144] The third speech synthesis model acquisition unit is used to acquire and process the third speech synthesis model according to the following processing method if the pronunciation quality meets the set third pronunciation quality condition, so as to obtain a personalized speech synthesis model for the target hearing-impaired user, and the processing method includes:

[0145] Acquiring the timbre characteristics of the target hearing-impaired user, wherein the third pronunciation quality condition has lower requirements on pronunciation quality than the first pronunciation quality condition;

[0146] In a preconfigured mixed timbre space, a plurality of timbre features that meet a set timbre similarity condition with the timbre features of the target hearing-impaired user are selected, and an average timbre feature of the selected plurality of timbre features is calculated, wherein the mixed timbre space is established based on pronunciation data of a mixed sound user and a hearing-impaired user;

[0147] Acquire the prosodic features of one or more other users' pronunciations, and synthesize each acquired prosodic feature with the average timbre feature and the set text to obtain a synthesized speech corresponding to each prosodic feature;

[0148] Outputting a synthesized speech corresponding to each prosodic feature, and in response to an operation by a target hearing-impaired user to select a target synthesized speech, determining a target prosodic feature corresponding to the target synthesized speech;

[0149] A third speech synthesis model including a prosody coding module, a text coding module, a timbre coding module and a decoding module is obtained, and the third speech synthesis model is processed to use the processed model as a personalized speech synthesis model for the target hearing-impaired user. The processing includes: fixing the coding features of the prosody coding module as the target prosody features, and fixing the coding features of the timbre coding module as the average timbre features.

[0150] Optionally, the device of the present application may further include:

[0151] The first speech synthesis model establishing unit is used to establish a first speech synthesis model. The establishing process may include:

[0152] Selecting healthy person pronunciation data from a set healthy person acoustic database, extracting the timbre features and transcribed text of the healthy person pronunciation data, and using a pre-trained prosody characterization model to extract a first prosody feature of the healthy person pronunciation data, and using the prosody characterization model to extract a second prosody feature of the pronunciation data of each hearing-impaired user in a set hearing-impaired group acoustic database;

[0153] Selecting a similar hearing-impaired prosodic feature that is most similar to the first prosodic feature from among the second prosodic features, and synthesizing the pronunciation data of a simulated hearing-impaired user using the timbre features of the sound person's pronunciation data, the transcribed text, and the similar hearing-impaired prosodic features;

[0154] Based on the pronunciation data of the simulated hearing-impaired user, a first speech synthesis model is trained.

[0155] Optionally, before extracting the timbre and transcription text of the healthy person's pronunciation data, the first speech synthesis model building unit may further include:

[0156] The sound person acoustic database and the hearing-impaired person acoustic database are mixed, and an autoencoder neural network model is trained based on the mixed database, wherein the autoencoder neural network model includes a prosody encoding module, a text encoding module, a timbre encoding module and a decoding module, wherein the text encoding module and the timbre encoding module are pre-configured, and the network parameters remain unchanged during the training process; the prosody encoding module in the trained autoencoder neural network model is used as the prosody representation model;

[0157] On this basis, the process of synthesizing the pronunciation data of the simulated hearing-impaired user by the first speech synthesis model building unit using the timbre characteristics of the sound person's pronunciation data, the transcribed text and the similar hearing-impaired rhythm characteristics may include:

[0158] The timbre characteristics, transcribed text and similar hearing-impaired rhythmic characteristics of the healthy person's pronunciation data are input into the decoding module in the trained autoencoder neural network model to obtain the pronunciation data of the simulated hearing-impaired user synthesized after decoding.

[0159] Optionally, the device of the present application may further include:

[0160] The mixed timbre space establishing unit is used to establish the mixed timbre space. The process includes:

[0161] Extracting timbre features from each piece of pronunciation data in a database containing pronunciation data of healthy users and hearing-impaired users;

[0162] The extracted timbre features are clustered, and the clusters obtained by clustering form a mixed timbre space. On this basis, the process of the third speech synthesis model acquisition unit selecting a number of timbre features that meet the set timbre similarity condition with the timbre features of the target hearing-impaired user in the pre-configured mixed timbre space may include:

[0163] Calculating the distance between the timbre feature of the target hearing-impaired user and the cluster center of each cluster in the mixed timbre space, and determining the target cluster with the shortest distance;

[0164] The timbre features contained in the target cluster are used as the timbre features that meet the set timbre similarity condition, or the timbre features most similar to the timbre features of the target hearing-impaired user are calculated in the target cluster and used as the timbre features that meet the set timbre similarity condition.

[0165] Optionally, the process of the third speech synthesis model acquisition unit acquiring the prosodic features of one or more other user's pronunciations may include:

[0166] Among other users whose pronunciation quality exceeds the third pronunciation quality condition, one or more prosodic features of the pronunciation of other users are selected.

[0167] Optionally, before outputting the synthesized speech corresponding to each prosodic feature, the third speech synthesis model acquisition unit may further include:

[0168] The timbre features of the synthesized speech corresponding to each prosodic feature are obtained respectively, and the similarity between the timbre features of each synthesized speech and the timbre features of the target hearing-impaired user is calculated; the first N synthesized speech with the highest similarity is selected. On this basis, the process in which the third speech synthesis model acquisition unit outputs the synthesized speech corresponding to each prosodic feature may include:

[0169] Output the first N synthesized speech.

[0170] Optionally, the process of the pronunciation quality acquisition unit of the present application acquiring the pronunciation quality of the target hearing-impaired user may include:

[0171] Obtain pronunciation data of target hearing-impaired users;

[0172] Pronunciation quality evaluation is performed on the pronunciation data of the target hearing-impaired user to obtain the pronunciation quality of the target hearing-impaired user.

[0173] Optionally, the process in which the pronunciation quality acquisition unit performs pronunciation quality evaluation on the pronunciation data of the target hearing-impaired user to obtain the pronunciation quality of the target hearing-impaired user may include:

[0174] Using a pre-trained pronunciation quality evaluation model, the pronunciation data of the target hearing-impaired user is evaluated to obtain the pronunciation quality of the target hearing-impaired user;

[0175] The pronunciation quality evaluation model is trained based on the pronunciation data of hearing-impaired users marked with pronunciation quality evaluation results, wherein the pronunciation quality evaluation results are determined based on the transcription accuracy of the pronunciation data and the acoustic likelihood compared with the standard acoustic model.

[0176] The speech synthesis apparatus for hearing-impaired users provided in the embodiments of the present application can be applied to speech synthesis devices for hearing-impaired users, such as voice recorders, mobile phones, computers, etc. Optionally, Figure 4 The hardware structure diagram of the speech synthesis device for hearing-impaired users is shown in FIG. Figure 4 , the hardware structure of the speech synthesis device for hearing-impaired users may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;

[0177] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0178] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

[0179] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0180] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to:

[0181] Obtain the pronunciation quality of the target hearing-impaired user;

[0182] If the pronunciation quality meets the set first pronunciation quality condition, a preconfigured first speech synthesis model corresponding to the first pronunciation quality condition is obtained, the first speech synthesis model includes a rhythm encoding module, a text encoding module, a timbre encoding module and a decoding module, the model is trained using the pronunciation data of a simulated hearing-impaired user as training data, the pronunciation data of the simulated hearing-impaired user is synthesized by the timbre, text and similar hearing-impaired rhythm of the pronunciation data of a healthy person, wherein the similar hearing-impaired rhythm is the rhythm of the pronunciation data of the hearing-impaired user that meets the set rhythm similarity condition with the rhythm of the pronunciation data of the healthy person;

[0183] Acquire the timbre characteristics and prosody characteristics of the target hearing-impaired user, process the first speech synthesis model, and use the processed model as the personalized speech synthesis model for the target hearing-impaired user. The processing includes: fixing the coding characteristics of the prosody coding module as the prosody characteristics of the target hearing-impaired user, and fixing the coding characteristics of the timbre coding module as the timbre characteristics of the target hearing-impaired user.

[0184] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0185] The embodiment of the present application further provides a storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0186] Obtain the pronunciation quality of the target hearing-impaired user;

[0187] If the pronunciation quality meets the set first pronunciation quality condition, a preconfigured first speech synthesis model corresponding to the first pronunciation quality condition is obtained, the first speech synthesis model includes a rhythm encoding module, a text encoding module, a timbre encoding module and a decoding module, the model is trained using the pronunciation data of a simulated hearing-impaired user as training data, the pronunciation data of the simulated hearing-impaired user is synthesized by the timbre, text and similar hearing-impaired rhythm of the pronunciation data of a healthy person, wherein the similar hearing-impaired rhythm is the rhythm of the pronunciation data of the hearing-impaired user that meets the set rhythm similarity condition with the rhythm of the pronunciation data of the healthy person;

[0188] Acquire the timbre characteristics and prosody characteristics of the target hearing-impaired user, process the first speech synthesis model, and use the processed model as the personalized speech synthesis model for the target hearing-impaired user. The processing includes: fixing the coding characteristics of the prosody coding module as the prosody characteristics of the target hearing-impaired user, and fixing the coding characteristics of the timbre coding module as the timbre characteristics of the target hearing-impaired user.

[0189] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0190] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0191] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0192] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for synthesizing speech for hearing-impaired users, characterized in that: include: Obtain the pronunciation quality of the target hearing-impaired user; If the pronunciation quality meets the set first pronunciation quality condition, a preconfigured first speech synthesis model corresponding to the first pronunciation quality condition is obtained, the first speech synthesis model includes a rhythm encoding module, a text encoding module, a timbre encoding module and a decoding module, the model is trained using the pronunciation data of a simulated hearing-impaired user as training data, the pronunciation data of the simulated hearing-impaired user is synthesized by the timbre, text and similar hearing-impaired rhythm of the pronunciation data of a healthy person, wherein the similar hearing-impaired rhythm is the rhythm of the pronunciation data of the hearing-impaired user that meets the set rhythm similarity condition with the rhythm of the pronunciation data of the healthy person; Acquire the timbre characteristics and prosody characteristics of the target hearing-impaired user, process the first speech synthesis model, and use the processed model as the personalized speech synthesis model for the target hearing-impaired user. The processing includes: fixing the coding characteristics of the prosody coding module as the prosody characteristics of the target hearing-impaired user, and fixing the coding characteristics of the timbre coding module as the timbre characteristics of the target hearing-impaired user.

2. The method according to claim 1, characterized in that Also includes: If the pronunciation quality meets the set second pronunciation quality condition, a preconfigured second speech synthesis model corresponding to the second pronunciation quality condition is used as the personalized speech synthesis model for the target hearing-impaired user, wherein the second speech synthesis model is a speech synthesis model obtained by fine-tuning a general speech synthesis model using the pronunciation data of the target hearing-impaired user and the corresponding transcribed text; The second pronunciation quality condition has higher requirements on pronunciation quality than the first pronunciation quality condition.

3. The method according to claim 1, characterized in that Also includes: If the pronunciation quality satisfies a set third pronunciation quality condition, then acquiring the timbre characteristics of the target hearing-impaired user, wherein the third pronunciation quality condition has lower requirements on pronunciation quality than the first pronunciation quality condition; In a preconfigured mixed timbre space, a plurality of timbre features that meet a set timbre similarity condition with the timbre features of the target hearing-impaired user are selected, and an average timbre feature of the selected plurality of timbre features is calculated, wherein the mixed timbre space is established based on pronunciation data of a mixed sound user and a hearing-impaired user; Acquire prosodic features of pronunciations of more than one other user, and synthesize each acquired prosodic feature with the average timbre feature and the set text to obtain a synthesized speech corresponding to each prosodic feature; wherein the other user includes a user whose pronunciation quality exceeds the third pronunciation quality condition requirement; Outputting a synthesized speech corresponding to each prosodic feature, and in response to an operation by a target hearing-impaired user to select a target synthesized speech, determining a target prosodic feature corresponding to the target synthesized speech; A third speech synthesis model including a prosody coding module, a text coding module, a timbre coding module and a decoding module is obtained, and the third speech synthesis model is processed to use the processed model as a personalized speech synthesis model for the target hearing-impaired user. The processing includes: fixing the coding features of the prosody coding module as the target prosody features, and fixing the coding features of the timbre coding module as the average timbre features.

4. The method according to claim 1, characterized in that: The process of establishing the first speech synthesis model includes: Selecting healthy person pronunciation data from a set healthy person acoustic database, extracting the timbre features and transcribed text of the healthy person pronunciation data, and using a pre-trained prosody characterization model to extract a first prosody feature of the healthy person pronunciation data, and using the prosody characterization model to extract a second prosody feature of the pronunciation data of each hearing-impaired user in a set hearing-impaired group acoustic database; Selecting a similar hearing-impaired prosodic feature that is most similar to the first prosodic feature from among the second prosodic features, and synthesizing the pronunciation data of a simulated hearing-impaired user using the timbre features of the sound person's pronunciation data, the transcribed text, and the similar hearing-impaired prosodic features; Based on the pronunciation data of the simulated hearing-impaired user, a first speech synthesis model is trained.

5. The method according to claim 4, characterized in that Before extracting the timbre and transcription text of the sound person's pronunciation data, the method further includes: The sound person acoustic database and the hearing-impaired person acoustic database are mixed, and an autoencoder neural network model is trained based on the mixed database, wherein the autoencoder neural network model includes a prosody encoding module, a text encoding module, a timbre encoding module and a decoding module, wherein the text encoding module and the timbre encoding module are pre-configured, and the network parameters remain unchanged during the training process; the prosody encoding module in the trained autoencoder neural network model is used as the prosody representation model; The method of synthesizing the pronunciation data of a simulated hearing-impaired user by utilizing the timbre characteristics of the pronunciation data of the healthy person, the transcribed text and the similar hearing-impaired rhythm characteristics includes: The timbre characteristics, transcribed text and similar hearing-impaired rhythmic characteristics of the healthy person's pronunciation data are input into the decoding module in the trained autoencoder neural network model to obtain the pronunciation data of the simulated hearing-impaired user synthesized after decoding.

6. The method according to claim 4, characterized in that The process of obtaining the prosodic features of the target hearing-impaired user includes: The prosodic features of the pronunciation data of the target hearing-impaired user are extracted using a pre-trained prosodic representation model.

7. The method according to claim 3, characterized in that The process of establishing the mixed timbre space includes: Extracting timbre features from each piece of pronunciation data in a database containing pronunciation data of healthy users and hearing-impaired users; The extracted timbre features are clustered, and the clusters obtained by clustering form a mixed timbre space; The step of selecting, in the preconfigured mixed timbre space, a plurality of timbre features that meet a set timbre similarity condition with the timbre features of the target hearing-impaired user includes: Calculating the distance between the timbre feature of the target hearing-impaired user and the cluster center of each cluster in the mixed timbre space, and determining the target cluster with the shortest distance; The timbre features contained in the target cluster are used as the timbre features that meet the set timbre similarity condition, or the timbre features most similar to the timbre features of the target hearing-impaired user are calculated in the target cluster and used as the timbre features that meet the set timbre similarity condition.

8. The method according to claim 3, characterized in that The obtaining of the prosodic features of the pronunciations of more than one other user comprises: Among other users whose pronunciation quality exceeds the third pronunciation quality condition, one or more prosodic features of the pronunciation of other users are selected.

9. The method according to claim 3, characterized in that: Before outputting the synthesized speech corresponding to each prosodic feature, the method further comprises: Acquire the timbre features of the synthesized speech corresponding to each prosodic feature, and calculate the similarity between the timbre features of each synthesized speech and the timbre features of the target hearing-impaired user; Select the first N synthesized speech with the highest similarity; The outputting of the synthesized speech corresponding to each prosodic feature comprises: Output the first N synthesized speech.

10. The method according to any one of claims 1 to 9, characterized in that: The obtaining of the pronunciation quality of the target hearing-impaired user includes: Obtain pronunciation data of target hearing-impaired users; Pronunciation quality evaluation is performed on the pronunciation data of the target hearing-impaired user to obtain the pronunciation quality of the target hearing-impaired user.

11. The method according to claim 10, characterized in that Performing a pronunciation quality evaluation on the pronunciation data of the target hearing-impaired user to obtain the pronunciation quality of the target hearing-impaired user includes: Using a pre-trained pronunciation quality evaluation model, the pronunciation data of the target hearing-impaired user is evaluated to obtain the pronunciation quality of the target hearing-impaired user; Among them, the pronunciation quality evaluation model is trained based on the pronunciation data of hearing-impaired users marked with pronunciation quality evaluation results, wherein the pronunciation quality evaluation results are determined based on the transcription accuracy of the pronunciation data and the acoustic likelihood compared with the standard acoustic model.

12. A speech synthesis device for hearing-impaired users, characterized in that: include: A pronunciation quality acquisition unit, used to acquire the pronunciation quality of a target hearing-impaired user; a first speech synthesis model acquisition unit, configured to acquire a preconfigured first speech synthesis model corresponding to the first pronunciation quality condition if the pronunciation quality satisfies a set first pronunciation quality condition, wherein the first speech synthesis model includes a rhythm encoding module, a text encoding module, a timbre encoding module and a decoding module, and the model is trained using pronunciation data of a simulated hearing-impaired user as training data, wherein the pronunciation data of the simulated hearing-impaired user is synthesized using the timbre, text and similar hearing-impaired rhythm of the pronunciation data of a healthy person, wherein the similar hearing-impaired rhythm is the rhythm of the pronunciation data of the hearing-impaired user that satisfies a set rhythm similarity condition with the rhythm of the pronunciation data of the healthy person; The first speech synthesis model processing unit is used to obtain the timbre characteristics and prosody characteristics of the target hearing-impaired user, process the first speech synthesis model, and use the processed model as the personalized speech synthesis model of the target hearing-impaired user, wherein the processing includes: fixing the coding characteristics of the prosody coding module as the prosody characteristics of the target hearing-impaired user, and fixing the coding characteristics of the timbre coding module as the timbre characteristics of the target hearing-impaired user.

13. A speech synthesis device for hearing-impaired users, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the method for synthesizing speech for hearing-impaired users according to any one of claims 1 to 11.

14. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the method for synthesizing speech for a hearing-impaired user as claimed in any one of claims 1 to 11 is implemented.

Images