Speech recognition system training method, electronic device, and storage medium

Abstract

The application discloses a speech recognition system training method, an electronic device and a storage medium, wherein the speech recognition system training method comprises the following steps: for a speech scene to be synthesized by a speech synthesis model, using real speech-text data homologous to the speech scene to train a phoneme classification model; inputting real speech data into the trained phoneme classification model to obtain training data of the speech synthesis model, and using the training data to train the speech synthesis model; inputting at least text data into the trained speech synthesis model to obtain synthesized speech, and using the text data and the synthesized speech to form a synthesized speech-text data pair; and using at least the synthesized speech-text data pair to train a speech recognition model. By training a phoneme speech recognition classifier to label speech recognition data, the difference between synthesized speech data and real speech data can be reduced.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of speech recognition, and in particular, to a speech recognition system training method, an electronic device, and a storage medium. BACKGROUND

[0002] In the prior art, a speech synthesis system for generating speech data is generally trained by adding various improvement techniques in the process of constructing the system, so that the obtained speech synthesis system is more suitable for subsequent speech recognition tasks. Generally, considering the robustness of the model and the subsequent recognition application, the system is a multi-speaker speech synthesis system. Then, the text corresponding to the target scene data to be synthesized is selected. Then, the text is input into the trained speech synthesis system to generate speech, and at this time, the speech-text data pair is obtained. Finally, the obtained speech-text data pair is input into the speech recognition system to be trained / optimized, and is trained separately or jointly trained with real speech-text data to obtain an optimized speech recognition system.

[0003] The defect is that the training of the speech synthesis system usually requires phoneme-level audio annotation, and the annotation cost of such data is high, and it is usually difficult to obtain a large amount of data, so the amount of data used for training the synthesis system is usually small, and thus the synthesized speech effect of the trained speech synthesis system usually has a certain gap with real audio. Moreover, since the synthesized data still has a certain gap with the real data, the recognition accuracy improvement brought by the final is limited, especially when there is a certain amount of real data and synthesized data for joint training, compared with only using real data, many times the accuracy improvement cannot be observed. This greatly restricts the application of this technology in real scenarios, because in real scenarios there is usually a certain amount (for example, several hundred hours) of real data, and it is hoped that more (for example, thousands or tens of thousands of hours) of synthesized data can be generated through speech synthesis technology under this condition, and then joint training is performed to improve the speech recognition effect. SUMMARY

[0004] Embodiments of the present application provide a speech recognition system training method, apparatus and electronic device, which are used to at least solve one of the above technical problems.

[0005] In a first aspect, an embodiment of the present application provides a speech recognition system training method, the speech recognition system comprising a phoneme classification model, a speech synthesis model and a speech recognition model, comprising: training the phoneme classification model using real speech-text data homologous to a speech scene to be synthesized by the speech synthesis model for the speech scene; inputting real speech data into the trained phoneme classification model to obtain training data of the speech synthesis model, and training the speech synthesis model using the training data; inputting at least text data into the trained speech synthesis model to obtain synthesized speech, and using the text data and the synthesized speech to form a synthesized speech-text data pair; and using at least the synthesized speech-text data pair for training of the speech recognition model.

[0006] In a second aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor, and a memory connected to the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any of the speech recognition system training methods described above.

[0007] In a third aspect, an embodiment of the present application provides a storage medium, the storage medium storing one or more programs comprising execution instructions, the execution instructions being readable and executable by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to perform any of the speech recognition system training methods described above.

[0008] In a fourth aspect, an embodiment of the present application further provides a computer program product, the computer program product comprising a computer program stored on a storage medium, the computer program comprising program instructions, when the program instructions are executed by a computer, causing the computer to perform any of the speech recognition system training methods described above.

[0009] The method of the present application annotates speech recognition data by training a phoneme speech recognition classifier, obtains data that can be used for speech synthesis model training, and then directly uses the real speech recognition data for automatic modeling of the speech synthesis system, which can greatly improve the data used for speech synthesis model training, so as to reduce the difference between synthesized speech data and real data in a data-driven manner, and then obtain the speech synthesis model of each speaker through transfer learning, so as to be more suitable for speech recognition tasks; finally, the adaptive modeling method is introduced, which models real speech and synthesized speech while sharing main parameters, and introduces a certain amount of independent adjustable parameters, so that the model can learn the common text information of the two kinds of audio, and also learn the difference between the two kinds of audio; further, the independent adjustable parameters are not introduced when modeling real speech, and the model structure remains the same as that of the joint training without using synthesized data, so as to avoid introducing additional calculation amount in speech recognition decoding. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0011] Figure 1 A flow chart of a speech recognition system training method provided by an embodiment of the present application is shown in FIG. 1.

[0012] Figure 2 A flow chart of another speech recognition system training method provided by an embodiment of the present application is shown in FIG. 2.

[0013] Figure 3 A speech recognition system training method based on speech synthesis and adaptive modeling of a specific example of the speech recognition system training method provided by an embodiment of the present application is shown in FIG. 3.

[0014] Figure 4 A phoneme classification model and a speech synthesis flow chart of a specific example of the speech recognition system training method provided by an embodiment of the present application is shown in FIG. 4.

[0015] Figure 5 An adaptive modeling speech recognition model structure diagram of a specific example of the speech recognition system training method provided by an embodiment of the present application is shown in FIG. 5.

[0016] Figure 6FIG. 1 shows an adaptive residual connection layer and an adaptive module diagram in a speech recognition model of one specific example of a speech recognition system training method provided by an embodiment of the present application;

[0017] Figure 7 FIG. 1 shows an adaptive residual connection layer and an adaptive module diagram in a speech recognition model of one specific example of a speech recognition system training method provided by an embodiment of the present application; DETAILED DESCRIPTION

[0018] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings of the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort belong to the scope of protection of the present application.

[0019] Please refer to Figure 1 FIG. 1 shows a flowchart of a speech recognition system training method provided by an embodiment of the present application, wherein the speech recognition system comprises a phoneme classification model, a speech synthesis model and a speech recognition model.

[0020] As shown in FIG. 1, in step 101, the phoneme classification model is trained using real speech-text data pairs homologous to a speech scene to be synthesized by the speech synthesis model; Figure 1

[0021] In step 102, real speech data is input into the trained phoneme classification model to obtain training data of the speech synthesis model, and the speech synthesis model is trained using the training data;

[0022] In step 103, at least text data is input into the trained speech synthesis model to obtain synthesized speech, and the text data and the synthesized speech are combined to form a synthesized speech-text data pair;

[0023] In step 104, at least the synthesized speech-text data pair is used for training of the speech recognition model.

[0024] In this embodiment, for this step 101, the speech recognition system training device trains the phoneme classification model using collected real speech-text data pairs homologous to a speech scene to be synthesized by the speech synthesis model, for example, if the target is to synthesize Cantonese, a certain amount of Cantonese speech recognition data can be collected to train the phoneme classification model.

[0025] ​Then, for step 102, the voice recognition system training device inputs real voice data into the trained phoneme classification model to obtain training data of the voice synthesis model, and trains the voice synthesis model using the training data. For example, several thousand or even tens of thousands of hours of Cantonese voice data without text annotation collected by various means is input into the phoneme classification model to obtain recognized phoneme sequences and corresponding timestamp information. These data can be referred to as voice-annotation data that can be used for synthesis model training. Then, the voice-annotation data that can be used for synthesis model training is used to train a multi-speaker voice synthesis system.

[0026] Subsequently, for step 103, the voice recognition system training device at least inputs text data into the trained voice synthesis model to obtain synthesized voice, and forms a synthesized voice-text data pair by combining the text data and the synthesized voice. For example, when the text data is input into the trained voice synthesis model to obtain synthesized voice, reference voice data is also input into the trained voice synthesis model, so that the synthesized audio contains not only the text content but also the characteristics of the reference audio, such as emotion, environmental noise, etc.

[0027] Finally, for step 104, the voice recognition system training device at least uses the synthesized voice-text data pair for training of a voice recognition model. In a specific embodiment, the synthesized voice-text data and real voice-text data pairs collected from the same voice scene can be used jointly for training of a self-adaptive modeling voice recognition model. For example, modeling is performed for real voice and synthesized voice, while keeping the main parameters shared, a certain amount of independently adjustable parameters are introduced, so that the model can learn not only the common text information in the two types of audio but also the differences between the two types of audio.

[0028] The method of the embodiment labels voice recognition data by training a phoneme voice recognition classifier to obtain data that can be used for training of a voice synthesis model, and then directly uses the real voice recognition data for modeling of a voice synthesis system, thereby reducing the difference between synthesized voice data and real voice data.

[0029] Further reference is made to Figure 2 which shows another voice recognition system training method provided by an embodiment of the application. The flowchart mainly shows the flow of Figure 1 the flow of the step of further limiting the flow of "training the voice synthesis model using the training data".

[0030] As Figure 2 indicated, in step 201, a multi-speaker voice synthesis model is trained using the training data.

[0031] In step 202, the multi-speaker speech synthesis model is iterated using a transfer learning method with training data of multiple specific speakers respectively, to obtain a plurality of specific speaker speech synthesis models corresponding thereto.

[0032] In the embodiment, for step 201, the speech recognition system training apparatus trains the multi-speaker speech synthesis model using training data, in a specific embodiment, the speaker information is represented as a plurality of vectors, and then the information is added to the speech synthesis model through a model fusion method, so that the multi-speaker speech synthesis model has little difference in sound perception when synthesizing the speech of a specific speaker, compared with a specific speaker speech synthesis model.

[0033] Then, for step 202, the speech recognition system training apparatus iterates the multi-speaker speech synthesis model using a transfer learning method with training data of multiple specific speakers respectively, to obtain a plurality of specific speaker speech synthesis models corresponding thereto, in a specific embodiment, the effect of the speech generated by the multi-speaker synthesis model on the training of the recognition model has a certain gap with the speech generated by the specific speaker speech synthesis model, further, on the basis of the multi-speaker speech synthesis model, a small amount of real data is used to iterate the multi-speaker synthesis model using a transfer learning method, to obtain a specific speaker speech synthesis system, repeating the step can generate a plurality of independent specific speaker speech synthesis systems, wherein the specific speaker speech synthesis model can generate data that is more fitted to the real audio distribution of the speaker, and the data that is more fitted to the distribution is better for training the speech recognition model, thereby being more suitable for the speech recognition task.

[0034] The method of the embodiment can obtain a speech synthesis model for each speaker through a transfer learning method, so that the speech synthesis data generated by the specific speaker speech synthesis can be more suitable for the speech recognition task than directly using a multi-speaker speech synthesis system.

[0035] In some optional embodiments, the method further comprises:

[0036] The speech recognition system training apparatus inputs the text data and the reference speech data into the trained speech synthesis model to obtain the synthesized speech, so that the synthesized audio contains the characteristics of the reference speech, for example, the synthesized audio can contain the text content and the characteristics of the specific speaker, and also contain the characteristics of the reference speech such as emotion and environmental noise.

[0037] The method of the embodiment can realize that the synthesized audio contains the characteristics of the reference speech by inputting the reference speech data into the speech synthesis model.

[0038] In some optional embodiments, the at least using the synthetic speech-text data pair for training of the speech recognition model comprises:

[0039] The speech recognition system training apparatus jointly uses the real speech-text data pair and the synthetic speech-text data pair for training of an adaptively modeled speech recognition model, wherein the adaptively modeled speech recognition model models real speech and synthetic speech through an adaptive modeling method, while keeping main parameters shared, a certain amount of independently adjustable parameters are introduced, so that the model can learn both common text information in the two types of audio and differences between the two types of audio.

[0040] In some optional embodiments, the real speech and the synthetic speech are formed into different vector encodings through an adaptive encoding layer, all real speeches share the same vector encoding, and all synthetic speeches share the same vector encoding.

[0041] In some optional embodiments, the improvement of the adaptively modeled speech recognition model comprises: improving all network layers in the speech recognition model into adaptive residual connection layers, wherein the model layers are represented as network weights, an adaptive module is introduced for each layer, the input of the adaptive module is the output of the previous layer and the adaptive vector encoding of the real speech and / or the synthetic speech, after the adaptive vector encoding is subjected to network transformation, layer regularization and sigmoid activation function, the result is multiplied by the result of normal input subjected to network transformation to obtain the output result of the adaptive module, the output of the adaptive module is added to the output subjected to network weight transformation and the input to form the output of the adaptive residual connection layer, in specific embodiments, the model layer is part of the model selected as an adaptive layer, which can be all layers or selected part according to the situation, which is not limited in this application.

[0042] The method of the embodiment can realize that the model can learn both common text information in the two types of audio and differences between the two types of audio by modeling real speech and synthetic speech while keeping main parameters shared and introducing a certain amount of independently adjustable parameters.

[0043] In some optional embodiments, the method further comprises: setting the adaptive vector encoding of the real data to a full 0 vector to not introduce additional calculation amount in actual decoding process, for example, setting the adaptive vector encoding of the real data to a full 0 vector in a manually set manner, so that the result after network transformation, layer regularization and sigmoid activation function remains a full 0 vector, and the result of the dot product of any vector and the full 0 vector is also a full 0 vector, so that the output of the adaptive module corresponding to the real data is a full 0 vector, which is the same as the original model structure, and no additional calculation amount is introduced in actual decoding process.

[0044] In any of the above embodiments, the phoneme model can obtain the phoneme sequence contained in the speech and the timestamp information corresponding to the phoneme sequence. In specific embodiments, the phoneme sequence and the phoneme timestamp information can be used to train the acoustic model and the vocoder of the speech synthesis system after being processed by the front-end of the speech synthesis system.

[0045] Please refer to Figure 3 , which shows a specific example of a speech recognition system training method based on speech synthesis and adaptive modeling provided by an embodiment of the present application.

[0046] As shown in Figure 3 , step 1: For the speech scene to be synthesized, use the available homologous speech-text speech recognition data pairs (for example, if the target to be synthesized is Cantonese, use a certain amount of Cantonese speech recognition data that can be collected) to train a phoneme classification model. Through this model, the purpose of obtaining the phoneme sequence contained in the speech and the corresponding timestamp information can be achieved. These information can be used to train the speech synthesis system.

[0047] Step 2: After completing the training of step 1, send the massive real speech data (for example, several thousand or even tens of thousands of hours of text-unannotated Cantonese speech data) collected by various means into the phoneme classification model to obtain the recognized phoneme sequence and the corresponding timestamp information. These data are referred to as "speech-annotation data that can be used for synthesis model training".

[0048] Step 3: Use the large amount of data obtained in step 2 to train a multi-speaker speech synthesis system. Typically, in this system, the speaker information will be represented as a vector, and then these information will be added to the speech synthesis model through model fusion. Such a multi-speaker speech synthesis system is not much different from a specific speaker speech synthesis system in terms of listening when synthesizing the speech of a specific speaker. However, we found that for the speech recognition task, the effect of the speech generated by the multi-speaker synthesis system for the recognition model training is different from that of the speech generated by the specific speaker speech synthesis system.

[0049] Step 4: Based on the multi-speaker speech synthesis system obtained in step 3, use the specific speaker training data (a small amount of real data) to iteratively train the multi-speaker synthesis system using the transfer learning method to obtain a specific speaker speech synthesis system. Repeating this step can generate multiple independent specific speaker speech synthesis systems.

[0050] Step 5: Select a specific speaker speech synthesis system and use the text data to be synthesized to generate synthesized speech.

[0051] Step 6: In this process, the reference speech data is also input to the speech synthesis system, so that the synthesized audio contains the characteristics of the reference audio (such as emotion, environmental noise, etc.) in addition to the text content and the characteristics of the specific speaker. Repeating this step can obtain multi-speaker speech synthesis data, which is referred to as "synthetic speech-text data for recognition model training" together with the corresponding text.

[0052] Step 7: The "synthetic speech-text data for recognition model training" obtained in the previous step is combined with the collected target scene real speech-text data for adaptive modeling speech recognition model training. The speech recognition model introduces an adaptive modeling method, models real speech and synthesized speech, while maintaining the sharing of main parameters, introduces a certain amount of independently adjustable parameters, so that the model can learn both the common text information in the two types of audio and the differences between the two types of audio.

[0053] Please refer to Figure 4 , which shows a phoneme classification model and a speech synthesis flowchart of one specific example of the speech recognition system training method provided by an embodiment of the present application.

[0054] As Figure 4 shown, after training the phoneme classification model, the phoneme sequence and phoneme timestamp information of the speech data passing through the model can be obtained, which can be used to train the acoustic model and vocoder of the speech synthesis system after being processed by the front end of the speech synthesis system.

[0055] Please refer to Figure 5 , which shows a structure diagram of an adaptive modeling speech recognition model of one specific example of the speech recognition system training method provided by an embodiment of the present application.

[0056] As Figure 5 described, the real data and the synthesized data will pass through the adaptive coding layer to form different vector coding representations, and all real data share the same vector coding, and all synthesized data also share the same vector coding. Then the original model structure in the speech recognition model is improved, and all network layers are improved into adaptive residual connection layers.

[0057] Please refer to Figure 6 , which shows an adaptive residual connection layer and an adaptive module diagram in the speech recognition model of one specific example of the speech recognition system training method provided by an embodiment of the present application.

[0058] As Figure 6As shown, the original model layer is represented as network weight, and an adaptive module is introduced to each layer. The model layer is the part of the model selected as an adaptive layer, which can be all layers or selected layers according to the situation. The adaptive module has two inputs, the first is the output of the last layer (the first layer is the input feature data), and the second is the real data / synthetic data adaptive vector encoding. After network transformation, layer regularization and sigmoid activation function, the adaptive vector encoding is multiplied by the result of the normal input after network transformation to obtain the output result of the adaptive module. The output of the adaptive module is added to the output of the network weight transformation and the input to form the output of the final adaptive residual connection layer. In this way, real speech and synthesized speech are modeled while maintaining the sharing of main parameters. A certain amount of independent adjustable parameters is introduced, so that the model can learn the common text information of the two kinds of audio and the difference between the two kinds of audio. LayerNorm is used to stabilize the distribution of a certain layer by normalizing the Hidden size dimension. Reference is defined as follows:

[0059] https: / / pytorch.org / docs / stable / generated / torch.nn.LayerNorm.html.

[0060] Further, the adaptive vector encoding of the real data is fixed as a full 0 vector in a manually set manner. The result after network transformation, layer regularization and sigmoid activation function remains a full 0 vector. The dot product of any vector and a full 0 vector is also a full 0 vector. Therefore, the output of the adaptive module corresponding to the real data is a full 0 vector, which is the same as the original model structure. In the actual decoding process, no additional calculation is introduced.

[0061]

[0062] Table 1

[0063] As shown in Table 1, on the open source speech recognition data task AISHELL-1 (Chinese Mandarin), whether to train a phoneme speech recognition classifier is compared, speech recognition data is labeled, and the experiment of expanding the amount of speech synthesis model training data is performed. It can be seen that only using 180 hours of AISHELL-1 data to train a speech synthesis model, then generating synthesized speech recognition training data, and performing model training, the word error rate on the development set and the test set is 18.3% and 23.5% respectively. While using the method to expand AISHELL-2 (1000 hours of Chinese Mandarin data) data to join in the training of the speech synthesis model, and then generating synthesized speech recognition training data, and performing model training, the word error rate on the development set and the test set is 15.7% and 20.1% respectively, which can obtain obvious accuracy improvement.

[0064]

[0065] Table 2

[0066] As shown in Table 2, on the open source speech recognition data task AISHELL-1 (Chinese Mandarin), whether to use transfer learning to migrate the multi-speaker synthesis system to a multi-specific person synthesis system is compared, and then the synthesized speech data for speech recognition task training is synthesized, and the speech recognition model is trained. It can be seen that compared with the version without transfer learning, the word error rate is obviously reduced after using transfer learning.

[0067]

[0068] Table 3

[0069] As shown in Table 3, on the open source speech recognition data task AISHELL-1 (Chinese Mandarin), the speech recognition effect in the case of jointly using synthesized data and real data is compared with the case of only using real data. It can be seen that compared with the word error rate in the case of only using real data, the modeling method of jointly using real data and synthesized data through adaptive modeling can obtain better recognition effect, and the recognition word error rate is obviously reduced.

[0070] In some other embodiments, the embodiment of the present application also provides a non-volatile computer storage medium, the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the speech recognition system training method in any method embodiment described above;

[0071] As an implementation manner, the non-volatile computer storage medium of the present application stores computer executable instructions, and the computer executable instructions are set to:

[0072] For the speech scene to be synthesized by the speech synthesis model, the phoneme classification model is trained using real speech-text data from the same source as the speech scene;

[0073] Real speech data is input into a trained phoneme classification model to obtain training data for the speech synthesis model, and the speech synthesis model is trained using the training data.

[0074] At least the text data is input into the trained speech synthesis model to obtain synthesized speech, and the text data and the synthesized speech are combined to form a synthesized speech-text data pair;

[0075] The synthesized speech-text data pairs are used at least for training the speech recognition model.

[0076] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the speech recognition system training device, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely disposed relative to the processor, and these remote memories may be connected to the speech recognition system training device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0077] This invention also provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer, cause the computer to perform any of the above-described speech recognition system training methods.

[0078] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present invention, such as... Figure 7 As shown, the device includes: one or more processors 710 and memory 720. Figure 7 Taking a processor 710 as an example, the device for training a speech recognition system may further include an input device 730 and an output device 740. The processor 710, memory 720, input device 730, and output device 740 can be connected via a bus or other means. Figure 7The bus connection is taken as an example. The memory 720 is the nonvolatile computer readable storage medium described above. The processor 710 performs various functions of the server and data processing by running the nonvolatile software programs, instructions and modules stored in the memory 720, that is, implements the voice recognition system training method of the method embodiment described above. The input device 730 can receive input digital or character information, and generate key signal input related to user settings and function control of the communication compensation device. The output device 740 can include a display device such as a display screen.

[0079] The product described above can perform the method provided by the embodiment of the application, has the corresponding function modules and beneficial effects of performing the method. Technical details not described in detail in the embodiment can be referred to the method provided by the embodiment of the application.

[0080] As an implementation form, the electronic device described above is applied to a voice recognition system training apparatus, and includes at least one processor and a memory in communication connection with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0081] training the phoneme classification model using real speech-text data homologous to a speech scene to be synthesized by the speech synthesis model;

[0082] inputting real speech data into the trained phoneme classification model to obtain training data of the speech synthesis model, and training the speech synthesis model using the training data;

[0083] inputting at least text data into the trained speech synthesis model to obtain synthesized speech, and composing the text data and the synthesized speech into a synthesized speech-text data pair;

[0084] using at least the synthesized speech-text data pair for training of the speech recognition model.

[0085] The electronic device of the embodiment of the application exists in various forms, including but not limited to:

[0086] (1) Mobile communication device: This type of device is characterized by having mobile communication function and mainly providing voice and data communication. This type of terminal includes: smart phone (such as iPhone), multimedia phone, functional phone, and low-end phone, etc.

[0087] (2) Ultra-mobile personal computer device: This type of device belongs to the category of personal computers, has computing and processing functions, and generally has mobile Internet features. This type of terminal includes: PDA, MID and UMPC devices, such as iPad.

[0088] (3) Portable entertainment device: This kind of device can display and play multimedia content. This kind of device includes: audio, video player (such as iPod), palm game machine, electronic book, and smart toy and portable car navigation device.

[0089] (4) Server: The device providing computing services, the server structure includes processor, hard disk, memory, system bus, etc. The server and the general computer architecture are similar, but because it is necessary to provide high reliable service, therefore in the processing capacity, stability, reliability, security, scalability, manageability, etc. The requirement is higher.

[0090] (5) Other electronic devices with data interaction function.

[0091] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to the actual needs. Those skilled in the art can understand and implement without creative labor.

[0092] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course, they can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including a plurality of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute the method of each embodiment or some part of the embodiment.

[0093] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training a speech recognition system, the speech recognition system comprising a phoneme classification model, a speech synthesis model and a speech recognition model, the method comprising: training the phoneme classification model using real speech-text data pairs homologous to a speech scenario to be synthesized by the speech synthesis model; inputting real speech data into the trained phoneme classification model to obtain a time-stamped phoneme sequence, the phoneme sequence being training data for the speech synthesis model; training a multi-speaker speech synthesis model using the training data and iteratively obtaining a plurality of specific speaker speech synthesis models by a transfer learning method; inputting at least text data and reference speech data into the trained specific speaker speech synthesis model to obtain synthesized speech, and forming a synthesized speech-text data pair by combining the text data and the synthesized speech; and inputting at least the synthesized speech-text data pair and real speech-text data pairs into the speech recognition model for training, wherein the speech recognition model is an adaptively modeled speech recognition model, the adaptively modeled speech recognition model is modeled for real speech and synthesized speech by an adaptive modeling method, while keeping the main parameters shared, a certain amount of independently adjustable parameters are introduced, so that the model can learn the common text information in the two kinds of audio and the difference between the two kinds of audio. The inputting at least text data into the trained speech synthesis model to obtain synthesized speech comprises: inputting text data and reference speech data into the trained speech synthesis model to obtain synthesized speech so that the synthesized audio contains the characteristics of the reference speech. Real speech and synthesized speech will pass through an adaptive coding layer to form different vector coding representations, all real speech shares the same vector coding, and all synthesized speech shares the same vector coding. The improvement of the adaptively modeled speech recognition model comprises: improving all network layers in the speech recognition model into adaptive residual connection layers, wherein the model layers are represented as network weights, an adaptive module is introduced for each layer, the input of the adaptive module is the output of the previous layer and the adaptive vector coding of real speech and / or synthesized speech, the adaptive vector coding is transformed by a network, layer regularization and a sigmoid activation function, and then multiplied by the result of the normal input transformed by the network to obtain the output result of the adaptive module, the output of the adaptive module is added to the output and input transformed by the network weight transformation to form the output of the adaptive residual connection layer. The method further comprises: setting the adaptive vector coding of real data to all 0 vectors to not introduce additional computational complexity in actual decoding process. The phoneme model can obtain the phoneme sequence contained in the speech and the time stamp information corresponding to the phoneme sequence.

2. The method of claim 1, wherein, ​ ​ 3. The method of claim 1, wherein, ​ 4. The method of claim 3, wherein, ​ 5. The method of claim 4, wherein, ​ ​ 6. The method of any one of claims 1-5, wherein, ​ 7. An electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executed by the at least one processor, and the instructions executed by the at least one processor enable the at least one processor to perform the steps of the method of any one of claims 1 to 6.

8. A storage medium having stored thereon a computer program, characterized in that The program, when executed by a processor, implements the steps of the method of any one of claims 1 to 6.

Images