Speech phoneme recognition method, medium, apparatus, and computing device

Abstract

Embodiments of the present disclosure provide a speech phoneme recognition method, medium, device and computing device. The method comprises: inputting speech data to be recognized into a pre-trained phoneme recognition model, outputting a phoneme sequence corresponding to the speech data, and the phoneme sequence comprising the occurrence order of each phoneme in the speech data. The present disclosure solves the problem that speech recognition in the related art cannot effectively match the lip movement of a virtual character image and speech. The speech data is disassembled into a set of phonemes that can be represented by the lip movement of AI, and is represented in a sequence form, so that the AI can perform representation through lip movement in turn according to the duration of each phoneme by reading the phoneme sequence, thereby achieving accurate matching of lip movement and speech, and significantly improving the experience of the audience.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of Internet, and more particularly, embodiments of the present disclosure relate to a speech phoneme recognition method, medium, device and computing device. BACKGROUND

[0002] This section is intended to provide background information to the embodiments of the present disclosure recited in the claims. The description herein does not constitute admission that the information provided herein is prior art to the present disclosure.

[0003] In the prior art, with the development of artificial intelligence technology and human-computer interaction technology, functions such as live broadcast and travel information introduction are realized through virtual characters, AI or virtual anchors, and are applied more and more widely. By pre-producing the content to be broadcast, and then automatically broadcasting by a virtual character image or AI, the human cost can be significantly saved. In order to ensure the user experience of the audience, the lip movement of such virtual character image when broadcasting needs to be well synchronized with the pre-produced content voice, i.e. the lip-driven scene. Therefore, the AI corresponding speech recognition algorithm needs to determine the time when each word or finer unit appears, so as to ensure that the lip movement of the virtual character image can be directly synchronized with the voice in the lip-driven scene.

[0004] The existing speech recognition algorithm can usually only determine the speech of the whole sentence, and the granularity is insufficient, which leads to the fact that the lip movement of the virtual character image cannot be effectively matched with the voice, and the audience experience is also poor. SUMMARY

[0005] The present disclosure provides a speech phoneme recognition method, medium, device and computing device to solve the problem that the speech recognition in the related art cannot effectively match the lip movement of the virtual character image with the voice.

[0006] In a first aspect of the embodiments of the present disclosure, a speech phoneme recognition method is provided, comprising:

[0007] inputting the speech data to be recognized into a pre-trained phoneme recognition model, and outputting a phoneme sequence corresponding to the speech data, the phoneme sequence comprising the order of occurrence of each phoneme in the speech data.

[0008] In a second aspect of the embodiments of the present disclosure, a speech phoneme recognition model training method is provided, comprising:

[0009] collecting sample speech data and sample phoneme sequences corresponding to the sample speech data for training;

[0010] inputting the sample speech data into the phoneme recognition model, and training the phoneme recognition model based on the predicted phoneme sequence output and the sample phoneme sequence.

[0011] In a third aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, comprising:

[0012] The computer readable storage medium stores computer execution instructions, which, when executed by a processor, are used to implement the index data storage method in the first aspect of the present disclosure, and / or, when executed by the processor, are used to implement the speech phoneme recognition model training method in the second aspect of the present disclosure.

[0013] In a fourth aspect of the embodiments of the present disclosure, a speech phoneme recognition device is provided, comprising:

[0014] The recognition module is configured to input the speech data to be recognized into the pre-trained phoneme recognition model, and output a phoneme sequence corresponding to the speech data, the phoneme sequence comprising an order of occurrence of each phoneme in the speech data.

[0015] In a fifth aspect of the embodiments of the present disclosure, a speech phoneme recognition model training device is provided, comprising:

[0016] The sample module is configured to collect sample speech data and a sample phoneme sequence corresponding to the sample speech data for training;

[0017] The training module is configured to input the sample speech data into the phoneme recognition model, and train the phoneme recognition model based on a comparison between the predicted phoneme sequence output by the phoneme recognition model and the sample phoneme sequence.

[0018] In a sixth aspect of the embodiments of the present disclosure, a computing device is provided, comprising: at least one processor;

[0019] and a memory in communication connection with the at least one processor;

[0020] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the computing device to perform the speech phoneme recognition method in the first aspect of the present disclosure, and / or to cause the computing device to perform the speech phoneme recognition model training method in the second aspect of the present disclosure.

[0021] According to the speech phoneme recognition method, medium, device and computing device of the embodiments of the present disclosure, by inputting the speech data to be recognized into the pre-trained phoneme recognition model, a phoneme sequence corresponding to the speech data is output. In this way, the speech data can be disassembled into a set of phonemes that can be represented by the lip movements of AI, and represented in a sequence form, so that the AI can sequentially represent the lip movements according to the duration of each phoneme by reading the phoneme sequence, thereby achieving accurate matching between the lip movements and the speech, and further significantly improving the experience of the audience. BRIEF DESCRIPTION OF DRAWINGS

[0022] The above and other objects, features and advantages of the present disclosure exemplary embodiments will be more apparent from the following detailed description read in conjunction with the accompanying drawings, in which:

[0023] Figure 1 A diagram of an application scenario according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 2 A flowchart of a speech phoneme recognition method according to another embodiment of the present disclosure is schematically shown;

[0025] Figure 3 A flowchart of a speech phoneme recognition model training method according to yet another embodiment of the present disclosure is schematically shown;

[0026] Figure 4 A flowchart of a speech phoneme recognition model training method according to still another embodiment of the present disclosure is schematically shown;

[0027] Figure 5 A structure diagram of a storage medium according to still another embodiment of the present disclosure is schematically shown; Figure 4 A flowchart of a method of outputting a predicted phoneme sequence in the illustrated embodiment is schematically shown;

[0028] Figure 6 A structure diagram of a speech phoneme recognition apparatus according to still another embodiment of the present disclosure is schematically shown;

[0029] Figure 7 A structure diagram of a speech phoneme recognition model training apparatus according to still another embodiment of the present disclosure is schematically shown;

[0030] Figure 8 A structure diagram of a speech phoneme recognition model training apparatus according to still another embodiment of the present disclosure is schematically shown;

[0031] Figure 9 A structure diagram of a computing device according to still another embodiment of the present disclosure is schematically shown.

[0032] In the drawings, the same or corresponding parts are denoted by the same or corresponding reference numerals. DETAILED DESCRIPTION

[0033] The principles and spirits of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are given only so that those skilled in the art can better understand and implement the present disclosure, and do not limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided so that the present disclosure is more thorough and complete, and the scope of the present disclosure is fully conveyed to those skilled in the art.

[0034] Those skilled in the art understand that the embodiments of the present disclosure can be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure can be embodied in the form of entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0035] According to the embodiments of the present disclosure, a speech phoneme recognition method, medium, device and computing equipment are provided.

[0036] The following is an explanation of the terms involved in the present disclosure:

[0037] Lip-sync driving: that is, in a virtual character live broadcast, the lips of the virtual character image or AI are matched with the played voice, so that the audience produces the scene that the voice is generated by the movement of the lips (and the oral cavity) of the virtual character image or AI. The lip-sync driving has higher fidelity than the non-lip-sync driving scene, and will not make the audience feel that the sound and picture are out of sync, and can significantly improve the experience of the user watching the virtual character live broadcast.

[0038] Phoneme: a phoneme is the smallest unit of speech, one pronunciation action constitutes a phoneme, and a piece of audio is composed of a sequence of multiple phonemes.

[0039] In this paper, it should be understood that the terms involved are only used for convenience and do not represent a limitation on the meaning. In addition, any number of elements in the drawings is used for example and not limitation, and any naming is only used for distinction and does not have any limiting meaning.

[0040] In addition, the data involved in the present disclosure can be data authorized by the user or fully authorized by all parties, and the collection, dissemination, use, etc. of the data meet the requirements of relevant national laws and regulations. The embodiments / examples of the present disclosure can be combined with each other. SUMMARY

[0042] The present inventors found that in the prior art, the content to be broadcast is pre-produced, and then a virtual character image or AI is used to automatically broadcast, which can significantly save labor costs. In order to ensure the user experience of the audience, the lips of such virtual character image need to be synchronized with the pre-produced content voice when broadcasting, that is, the lip-sync driving scene. Therefore, the corresponding speech recognition algorithm of the AI needs to determine the time when each word or finer unit appears, so as to ensure that the lip movement of the virtual character image can be directly synchronized with the voice in the lip-sync driving scene. The existing speech recognition algorithm can usually only determine the voice of the whole sentence, and cannot accurately represent the complex structure inside the voice. Even if deep learning is used to recognize the voice, it can only recognize the voice of each word, which has insufficient granularity, resulting in inconsistent time between the lip movement of the virtual character image and the voice, and poor audience experience.

[0043] In the scheme, the speech data is recognized by the pre-trained phoneme recognition model, and the occurrence order of each phoneme is output. Compared with the existing recognition method, the granularity is significantly improved, the recognition accuracy is increased, the matching effect of virtual character lip movement and speech is significantly improved, and the experience of the audience is improved.

[0044] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure will be specifically introduced.

[0045] OVERVIEW OF APPLICATION SCENARIOS

[0046] First, referring to Figure 1 , in the speech phoneme recognition process, the server inputs the speech data 100 into the phoneme recognition model 110, and can output the phoneme sequence 120, so as to send the phoneme sequence 120 to the virtual character live broadcast program 130 for broadcasting, thereby completing the speech phoneme recognition process.

[0047] It should be noted that, Figure 1 , the speech data, the phoneme recognition model, the phoneme sequence and the virtual character live broadcast program in the scenario are only exemplarily illustrated by one, but the present disclosure is not limited thereto, that is, the number of speech data, phoneme recognition model, phoneme sequence and virtual character live broadcast program can be arbitrary.

[0048] Exemplary Method

[0049] Next, combined with Figure 1 application scenarios, the speech phoneme recognition method according to the exemplary embodiments of the present disclosure is described with reference to Figure 2 , and the speech phoneme recognition model training method according to the exemplary embodiments of the present disclosure is described with reference to Figures 3-5 . It should be noted that the above application scenarios are only shown for the purpose of facilitating understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0050] Figure 2 The flowchart of the speech phoneme recognition method provided by an embodiment of the present disclosure is shown in Figure 2 . The speech phoneme recognition method provided by the embodiment includes the following steps:

[0051] Step S201, input the speech data to be recognized into the pre-trained phoneme recognition model, and output the phoneme sequence corresponding to the speech data.

[0052] The phoneme sequence includes the occurrence order of each phoneme in the speech data.

[0053] Specifically, the voice data is audio content that needs to be played by the virtual character in the live broadcast, which is usually voice-over content generated by voice-over personnel or automatically generated by a program and based on specific text. Therefore, the voice data usually corresponds to specific text.

[0054] Since the virtual character itself is only an animated image driven by a program, it cannot automatically identify the text content in the voice data. Therefore, it is necessary to identify the voice data and convert it into content that the corresponding program of the virtual character can express (through lip movements). That is, to identify the phonemes in the voice data.

[0055] Because a complete word may involve multiple lip movements, such as the word "tong", which requires the lips to first open and then close, there are at least two actions at this time. If only single words in the voice data are identified, it may not be known when the lip movement should change, and the audience may also think that there is a problem of out-of-sync audio and video. The word "tong" actually has three phonemes (t, o, ng), and the lip state is different under different phonemes (the lips are slightly open at t sound, the lips are fully open at o sound, and the lips are closed at ng sound). If the single word is split into each phoneme and each action is corresponded, the change of lip movement can be accurately expressed.

[0056] As can be seen, when the phoneme changes, the lip movement of a normal person will change accordingly. Therefore, as long as the lip movement of the virtual character image changes synchronously with the change of the phoneme in the voice data, the audience will have the perception of audio-visual synchronization (the audience will unconsciously think that the voice is read by the virtual character), thereby ensuring the audience experience.

[0057] Therefore, the voice data is converted into a sequence of phonemes by a phoneme recognition model, so that the corresponding program of the virtual character determines the order and time point of the occurrence of each phoneme in the voice data, and switches the corresponding lip movement of the phoneme at the time point in order, thereby realizing the effect of lip-driven.

[0058] In an embodiment of the present disclosure, the voice data is Chinese voice, and the phoneme sequence includes: at least one interval character between each Chinese character.

[0059] Specifically, in actual application, since the audience of the virtual character live broadcast mainly comes from the Chinese-speaking world, the voice data is mainly Chinese voice. At this time, since there is no space or other connector between Chinese characters, in order to identify the interval between single characters in the phoneme sequence, one or more interval characters are added between single characters (i.e. between each Chinese character). When there is only one interval character, it is used to represent the distinction between single characters. When there are continuous multiple interval characters, they can be used to represent the length of the pause time between single characters, so as to express the order and time length (and time point) of the phonemes through the phoneme sequence, thereby accurately splitting the voice data and ensuring high accuracy of the matching between the lip movements of the virtual character and the phonemes, thereby significantly improving the viewing experience of the user.

[0060] According to the voice phoneme recognition method of the embodiment of the present disclosure, the voice data to be recognized is input into the pre-trained phoneme recognition model, and the phoneme sequence corresponding to the voice data is output. In this way, the voice data can be disassembled into a set of phonemes that can be represented by the lip movements of the AI, and represented in a sequence form, so that the AI can perform representation through the lip movements in the order of the duration of each phoneme by reading the phoneme sequence, thereby realizing accurate matching between the lip movements and the voice, and further significantly improving the experience of the audience.

[0061] Figure 3 The flowchart of the voice phoneme recognition model training method provided by an embodiment of the present disclosure is shown in FIG. 1. As shown in FIG. 1, the voice phoneme recognition model training method provided by the embodiment includes the following steps: Figure 3

[0062] Step S301, collect sample voice data and sample phoneme sequence corresponding to the sample voice data for training.

[0063] Specifically, the voice phoneme recognition model is a phoneme recognition model. When training the phoneme recognition model, the language of the training sample (i.e. sample voice data) of the phoneme recognition model needs to be determined according to the corresponding language in actual use, and the corresponding phoneme sequence (i.e. sample phoneme sequence) in the sample voice data is identified based on the language.

[0064] The specific identification method can identify the time points of the occurrence of each phoneme in the sample voice data in order and time interval. The time interval can be represented by the number of interval characters. The more interval characters, the longer the time interval between phonemes. In actual application, the corresponding factor sequence (i.e. sample factor sequence) can be identified by manual annotation.

[0065] Step S302, input the sample voice data into the phoneme recognition model, and train the phoneme recognition model based on the predicted phoneme sequence output and the sample phoneme sequence. ​

[0066] Specifically, the sample speech data is combined with the sample phoneme sequence to obtain a training sample of the phoneme recognition model, wherein the sample speech data is used for input into the phoneme recognition model for training, an output of a predicted phoneme sequence (i.e., a predicted phoneme sequence), and a loss function can be calculated based on a comparison between the output result and the corresponding sample phoneme sequence (a classical loss function can be directly selected, which is not limited herein), and parameters in the phoneme recognition model are updated and optimized based on the loss function, so as to realize the training process.

[0067] When the phoneme recognition model is trained, in a scenario requiring lip-driven speech, the speech data to be used can be directly input into the trained phoneme recognition model for recognition and inference, and the corresponding phoneme sequence is output, which is then sent to a program corresponding to the virtual character live broadcast, so that the lip-driven speech can be realized while ensuring the viewing experience of the user. For related content, please refer to Figure 2 The content in the embodiments shown herein will not be repeated herein.

[0068] According to the speech phoneme recognition model training method of the embodiments of the present disclosure, sample speech data used for training and sample phoneme sequences corresponding to the sample speech data are collected, and then the sample speech data is input into the phoneme recognition model, and the phoneme recognition model is trained based on the predicted phoneme sequence output and the sample phoneme sequence. Thus, the trained phoneme recognition model can recognize the phonemes in the speech data and the sequence in which the phonemes appear, so as to input the phoneme sequence into the program of the virtual character live broadcast, thereby improving the lip-driven audio-visual synchronization effect during the virtual character live broadcast and improving the user experience.

[0069] Figure 4 A flowchart of the speech phoneme recognition model training method provided by an embodiment of the present disclosure is shown in FIG. 4. As shown in FIG. 4, the speech phoneme recognition model training method provided by the present embodiment includes the following steps: Figure 4

[0070] Step S401, determining a sample Chinese character sequence corresponding to sample speech data used for training.

[0071] Specifically, in the present embodiment, the sample speech data used for training is taken as an example of Chinese speech data (i.e., a scenario in which the virtual character live broadcast is performed using Chinese Mandarin or a dialect).

[0072] Therefore, the Chinese text used in the sample speech data needs to be obtained first, and the sample Chinese character sequence is extracted therefrom.

[0073] ​The Chinese text corresponding to the sample voice data usually includes non-Hanzi content such as punctuation marks, spaces, line breaks and the like for easy reading, but these non-Hanzi content will not be pronounced in the voice data, so it is necessary to filter these non-Hanzi content, extract the corresponding Hanzi and its order, and then obtain the Hanzi sequence (i.e. the sample Hanzi sequence).

[0074] The method of filtering non-Hanzi content can use a character filtering algorithm first, which is not limited here.

[0075] In an embodiment of the present disclosure, the training data can use the Chinese Mandarin Lip Reading (CMLR) data set, which contains 102076 sentences of 11 news anchors, each sentence has at most 29 Hanzi, and does not contain English, Arabic numerals and rare punctuation. When used for training, the recognition effect is good and the training efficiency is high.

[0076] Step S402, inserting a sample interval character between each adjacent sample character.

[0077] Specifically, if the voice data is directly parsed into a sequence composed entirely of phonemes, it will be difficult to distinguish where to break the sentence and where to pause when live streaming a virtual character. For example, the phoneme sequence "n i h a o" may correspond to "nǐ hǎo" or "nǐ hā oh", which will result in inaccurate lip movement (the corresponding lip movements are different). Therefore, the interval between Hanzi and Hanzi needs to be identified during training, so that the phoneme recognition model trained subsequently can also recognize the interval between Hanzi and Hanzi.

[0078] Therefore, an interval character (i.e. a sample interval character) needs to be inserted between each character in the sample Hanzi sequence. The interval character used can be any non-Hanzi symbol (usually it also needs to avoid commonly used numbers and letters), such as "-" and "·".

[0079] Step S403, converting the sample Hanzi into sample pinyin.

[0080] Specifically, in Chinese, the pronunciation of Hanzi is represented by pinyin, and the initial, medial and final of pinyin are determined based on phonemes (one final may contain multiple factors, such as ong which contains o and ng). Therefore, on the basis of adding interval characters, each Hanzi in the sample Hanzi sequence needs to be converted into corresponding sample pinyin, such as "nǐ-hǎo" converted into "ni-hao", so as to be further converted into a sequence of phonemes subsequently.

[0081] Step S404, convert the sequence containing sample pinyin and sample interval character into a sample phoneme sequence.

[0082] Specifically, the sample pinyin is split according to the relationship between pinyin and phoneme, so that the original sample pinyin sequence can be converted into a sample phoneme sequence. The split phonemes can be distinguished by a space symbol or other symbols different from the sample interval character.

[0083] Through steps S401 to S404, the Chinese in the sample speech data is converted into a sample phoneme sequence with interval characters, and the interval characters are inserted to enable the trained phoneme recognition model to align the output phoneme sequence with the corresponding speech data in time slices (i.e., align the phonemes with the time nodes at which the speech data occurs), thereby ensuring the synchronization of lip movements and speech in the application.

[0084] Step S405, input the sample speech data into the phoneme recognition model to output a predicted phoneme sequence.

[0085] Specifically, the phoneme recognition model can include a convolutional neural network CNN, a bidirectional long short-term memory network BiLSTM, and a neural network-based time sequence class classifier CTC connected in sequence. The CNN is used to convert the speech data into an audio feature parameter sequence, then the BiLSTM is used to extract phoneme features therefrom, and the CTC is used to ensure that the output predicted phoneme sequence is aligned in time with the sample speech data. Through the results of the three, the effect of converting the speech data into a phoneme sequence is achieved.

[0086] In an embodiment of the present disclosure, as shown in Figure 5 The method flowchart for outputting a predicted phoneme sequence specifically includes the following steps:

[0087] Step S4051, convert the sample speech data into spectrogram data and input it into the convolutional neural network to output speech feature data corresponding to the sample speech data.

[0088] Specifically, the process of converting the sample speech data into spectrogram data is a standard process in speech recognition, which can be realized by existing conventional audio processing programs or directly completed by the encapsulated convolutional neural network CNN.

[0089] The convolutional neural network CNN automatically outputs the speech feature data corresponding to the sample speech data for further processing.

[0090] Step S4052, input the speech feature data into the bidirectional long short-term memory network to input time sequence feature data corresponding to the sample speech data.

[0091] Specifically, by processing the voice feature data through the bidirectional long and short memory network BiLSTM, the relationship and corresponding information of each word in the sentence can be fully extracted, and then the text and phonemes in the sentence can be effectively recognized, and data containing timing characteristics (i.e., timing feature data) can be obtained.

[0092] Step S4053, input the timing feature data into the neural network-based timing type classifier, and output the predicted phoneme sequence.

[0093] Specifically, by using the neural network-based timing type classifier CTC, the timing feature data can be mapped to a specific time point, and by generating different numbers of interval characters, each phoneme obtained can be aligned with the time point of the corresponding sound of the sample voice data, and then the final output of the predicted phoneme sequence containing interval characters can be obtained.

[0094] Step S406, based on the output predicted phoneme sequence and the sample phoneme sequence, determine the correctness of the predicted phoneme sequence.

[0095] Specifically, by comparing the output predicted phoneme sequence with the sample phoneme sequence, the correctness of the predicted phoneme sequence can be determined. Since the number of interval characters in the predicted phoneme sequence is automatically generated by the phoneme recognition model and can align the phonemes with the time slices of the sample voice data, the number of interval characters in the predicted phoneme sequence will usually differ from the number of interval characters in the sample phoneme sequence, which usually does not affect the correctness of the predicted phoneme sequence.

[0096] In an embodiment of the present disclosure, the method for determining the correctness of the predicted phoneme sequence includes the following cases:

[0097] Case one, if the same phonemes in the predicted phoneme sequence and the sample phoneme sequence have the same order of appearance, and there is at least one interval character between the adjacent Chinese characters corresponding to the predicted phoneme sequence, it is determined that the predicted phoneme sequence is correct.

[0098] Specifically, since there is only one interval character in the artificially annotated sample phoneme sequence, and there may be multiple consecutive interval characters in the predicted phoneme sequence, i.e., the interval characters in the predicted phoneme sequence are used to separate the phonemes corresponding to the Chinese characters and to align the time points of the phonemes, therefore, under the premise that the same phonemes have the same order of appearance, as long as the interval characters meet the function of separating Chinese characters, it can be considered that the correctness of the predicted phoneme sequence meets the requirements.

[0099] Case two, if the same phonemes in the predicted phoneme sequence and the sample phoneme sequence have different orders of appearance, or there is no interval character between the adjacent Chinese characters corresponding to the predicted phoneme sequence, it is determined that the predicted phoneme sequence is incorrect.

[0100] Specifically, if the order of the phonemes in the sample phoneme sequence and the predicted phoneme sequence is different, or the function of separating Chinese characters is not played, it is obvious that the prediction result cannot be used in the lip-driven scene, and therefore these cases need to be considered as not meeting the requirement of correctness.

[0101] In step S407, a loss function of the phoneme recognition model is calculated based on the correctness of the predicted phoneme sequence.

[0102] Specifically, the loss function can be directly selected as any conventional CTC loss function, and the corresponding loss function is calculated based on the correctness of the predicted phoneme sequence (i.e. the accuracy, recognition accuracy or recognition probability, which can be calculated in step S406 when the correctness is identified, the recognition probability of the interval character and the phoneme for separating Chinese characters is calculated, and the recognition probability is 100% if all are recognized).

[0103] In step S408, the phoneme recognition model is trained based on the result of the loss function.

[0104] Specifically, the network gradient is calculated based on the result of the loss function, and the parameters in the phoneme recognition model are updated based on the calculated network gradient, so as to realize the training process of the phoneme recognition model.

[0105] According to the speech phoneme recognition method of the embodiment of the present disclosure, by determining the sample Chinese character sequence corresponding to the sample speech data for training, inserting a sample interval character between each adjacent sample character, converting the sample Chinese characters into sample pinyin, converting the sequence containing the sample pinyin and the sample interval character into a sample phoneme sequence, then inputting the sample speech data into the phoneme recognition model, outputting a predicted phoneme sequence, determining the correctness of the predicted phoneme sequence based on the output predicted phoneme sequence and the sample phoneme sequence, calculating the loss function of the phoneme recognition model based on the correctness of the predicted phoneme sequence, and finally training the phoneme recognition model based on the result of the loss function. Thus, by using the speech data of the required language as the sample speech data, and processing to obtain the corresponding sample phoneme sequence for training, the phoneme recognition model can accurately recognize the phoneme sequence of the corresponding language, thereby effectively ensuring the synchronization of the lip movement and the speech of the virtual character in live broadcast, and improving the audience experience.

[0106] Exemplary Medium

[0107] After introducing the method of the exemplary embodiment of the present disclosure, next, with reference to Figure 6 The storage medium of the exemplary embodiment of the present disclosure is described.

[0108] Reference is made to Figure 6As shown, a program product 60 for implementing the above method according to the embodiments of the present disclosure is described, which can adopt a portable compact disc read-only memory (CD-ROM) and include program codes, and can run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto.

[0109] The program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0110] The readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, in which readable program codes are borne. Such a propagated data signal can take on various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The readable signal medium can also be any readable medium other than the readable storage medium.

[0111] The program code for performing the operations of the present disclosure can be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, C++, etc., and a conventional procedural programming language such as the "C" language or a similar programming language. The program code can be executed entirely on a user computing device, partially on a user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN).

[0112] Exemplary Apparatus

[0113] After introducing the medium of the exemplary embodiments of the present disclosure, next, with reference to Figure 7 The voice phoneme recognition device of the exemplary embodiments of the present disclosure is described, for implementing the voice phoneme recognition method in any of the above method embodiments, the implementation principle and technical effects are similar to the corresponding method embodiments described above, and will not be described here.

[0114] The voice phoneme recognition device 700 provided by the present disclosure comprises:

[0115] The recognition module 710 is configured to input the voice data to be recognized into a pre-trained phoneme recognition model, and output a phoneme sequence corresponding to the voice data, the phoneme sequence including an occurrence order of each phoneme in the voice data.

[0116] In an example embodiment of the present disclosure, the recognition module 710 includes: the voice data is Chinese voice, and the phoneme sequence includes: there is at least one interval character between each Chinese character.

[0117] In an example embodiment of the present disclosure, the voice phoneme recognition device 700 further includes a training module 720 configured to train the phoneme recognition model by: collecting sample voice data and a sample phoneme sequence corresponding to the sample voice data for training; inputting the sample voice data into the phoneme recognition model, and training the phoneme recognition model based on a predicted phoneme sequence output by the phoneme recognition model and the sample phoneme sequence.

[0118] In an example embodiment of the present disclosure, the training module 720 is specifically configured to: determine a sample Chinese character sequence corresponding to the sample voice data for training; insert an interval character between each adjacent sample Chinese character; convert the sample Chinese character into a sample pinyin; and convert the sequence including the sample pinyin and the interval character into the sample phoneme sequence.

[0119] In an example embodiment of the present disclosure, the training module 720 is specifically configured to: input the sample voice data into the phoneme recognition model, and output a predicted phoneme sequence; determine the correctness of the predicted phoneme sequence based on the predicted phoneme sequence output by the phoneme recognition model and the sample phoneme sequence; calculate a loss function of the phoneme recognition model based on the correctness of the predicted phoneme sequence; and train the phoneme recognition model based on a result of the loss function.

[0120] In an example embodiment of the present disclosure, the training module 720 is specifically configured to: if the occurrence order of the same phonemes in the predicted phoneme sequence and the sample phoneme sequence is the same, and there is at least one interval character between adjacent Chinese characters corresponding to the predicted phoneme sequence, determine that the predicted phoneme sequence is a correct result; and if the occurrence order of the same phonemes in the predicted phoneme sequence and the sample phoneme sequence is different, or there is no interval character between adjacent Chinese characters corresponding to the predicted phoneme sequence, determine that the predicted phoneme sequence is an incorrect result.

[0121] In an example embodiment of the present disclosure, the training module 720 is specifically configured to: if the phoneme recognition model comprises a convolutional neural network, a bidirectional long short-term memory network, and a neural network-based time series class classifier connected in sequence, convert the sample speech data into spectrogram data and input the spectrogram data into the convolutional neural network to output speech feature data corresponding to the sample speech data; input the speech feature data into the bidirectional long short-term memory network to input time series feature data corresponding to the sample speech data; and input the time series feature data into the neural network-based time series class classifier to output a predicted phoneme sequence.

[0122] Next, reference is made to Figure 8 The speech phoneme recognition model training apparatus of the example implementation of the present disclosure is described, which is configured to implement the speech phoneme recognition model training method in any of the above method embodiments, and has similar implementation principles and technical effects to the corresponding method embodiments described above, which will not be described herein again.

[0123] The speech phoneme recognition model training apparatus 800 provided by the present disclosure comprises:

[0124] The sample module 810 is configured to collect sample speech data and a sample phoneme sequence corresponding to the sample speech data for training.

[0125] The training module 820 is configured to input the sample speech data into the phoneme recognition model, and train the phoneme recognition model based on a predicted phoneme sequence output by the phoneme recognition model and the sample phoneme sequence.

[0126] In an example embodiment of the present disclosure, the sample module 810 is specifically configured to: determine a sample Chinese character sequence corresponding to the sample speech data for training; insert a sample interval character between each adjacent sample Chinese character; convert the sample Chinese characters into sample pinyin; and convert a sequence comprising the sample pinyin and the sample interval character into a sample phoneme sequence.

[0127] In an example embodiment of the present disclosure, the training module 820 is specifically configured to: input the sample speech data into the phoneme recognition model to output a predicted phoneme sequence; determine the correctness of the predicted phoneme sequence based on the predicted phoneme sequence output by the phoneme recognition model and the sample phoneme sequence; calculate a loss function of the phoneme recognition model based on the correctness of the predicted phoneme sequence; and train the phoneme recognition model based on a result of the loss function.

[0128] In an example embodiment of the present disclosure, the training module 820 is specifically configured to: if the appearance order of the same phonemes in the predicted phoneme sequence and the sample phoneme sequence is the same, and there is at least one interval character between the adjacent Chinese characters corresponding to the predicted phoneme sequence, determine that the predicted phoneme sequence is a correct result; if the appearance order of the same phonemes in the predicted phoneme sequence and the sample phoneme sequence is different, or there is no interval character between the adjacent Chinese characters corresponding to the predicted phoneme sequence, determine that the predicted phoneme sequence is an incorrect result.

[0129] In an example embodiment of the present disclosure, the training module 820 is specifically configured to: if the phoneme recognition model comprises a convolutional neural network, a bidirectional long short-term memory network and a neural network-based time series class classifier connected in sequence, convert the sample speech data into spectrogram data and input into the convolutional neural network to output the speech feature data corresponding to the sample speech data; input the speech feature data into the bidirectional long short-term memory network to input the time series feature data corresponding to the sample speech data; input the time series feature data into the neural network-based time series class classifier to output the predicted phoneme sequence.

[0130] Exemplary Computing Device

[0131] After introducing the method, medium and device of the example implementation of the present disclosure, next, with reference to Figure 9 The computing device of the example implementation of the present disclosure is described.

[0132] Figure 9 The displayed computing device 90 is only an example, and should not bring any limitation to the function and use range of the example implementation of the present disclosure.

[0133] As shown in Figure 9 The computing device 90 is in the form of a general-purpose computing device. The components of the computing device 90 can include, but are not limited to, the above-mentioned at least one processing unit 901, the above-mentioned at least one storage unit 902, and a bus 903 connecting different system components, including the processing unit 901 and the storage unit 902.

[0134] The bus 903 includes a data bus, a control bus and an address bus.

[0135] The storage unit 902 can include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 9021 and / or a cache memory 9022, and can further include a readable medium in the form of a non-volatile memory, such as a read-only memory (ROM) 9023.

[0136] The storage unit 902 also includes a number of program modules 9024 that are stored in the memory 9025, that include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or a combination of which may include implementation of a network environment.

[0137] The computing device 90 can also communicate with one or more external devices 904 such as a keyboard or pointing device, through Input / Output (I / O) interfaces 905. Further, the computing device 90 can communicate with one or more networks, such as a local area network (LAN), a wide area network (WAN), and / or the public network, such as the Internet, through a network adapter 906. As Figure 9 illustrated, the network adapter 906 is in communication with the other modules of the computing device 90 through the bus 903. It should be appreciated that although the network adapter 906 is illustrated as a single component, the network adapter 906 can comprise a plurality of components that work in cooperation to send and receive data other hardware and / or software modules, including but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage

[0138] It should be noted that although several units / modules or sub-units / modules of the supply chain policy determination apparatus and the object scoring model training apparatus are mentioned in the above detailed description, such division is merely exemplary and not mandatory. Indeed, according to embodiments of the present disclosure, features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, features and functions of one unit / module described above can be further divided into embodied by a plurality of units / modules.

[0139] Further, although operations of the methods of the present disclosure are described in a particular order in the drawings, this is not required or implied in any way as to a particular order or that all illustrated operations be performed to achieve desirable results. Additionally or alternatively, certain steps can be omitted, combined into a single step, and / or separated into multiple steps.

[0140] While the spirit and principles of the present disclosure have been described with reference to several specific embodiments, it is to be understood that the present disclosure is not limited to the precise embodiments disclosed and that various modifications can be made in aspects without departing from the scope of the present disclosure. The present disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method of speech phoneme recognition, characterized by, The method comprises: inputting the voice data to be identified into a pre-trained phoneme recognition model, outputting a phoneme sequence corresponding to the voice data, the phoneme sequence comprising the order of occurrence of each phoneme in the voice data; The phoneme recognition model is trained by the following method: determining a sample Chinese character sequence corresponding to the sample voice data for training; inserting a sample interval character between each adjacent sample character; converting the sample Chinese characters into sample pinyin; splitting the sample pinyin according to the relationship between pinyin and phonemes, and distinguishing the split phonemes by using space symbols; convert the sequence containing the sample pinyin and the sample interval character into a sample phoneme sequence; inputting the sample voice data into the phoneme recognition model to output a predicted phoneme sequence; determining the correctness of the predicted phoneme sequence based on the output predicted phoneme sequence and the sample phoneme sequence; based on the correctness of the predicted phoneme sequence, calculate the loss function of the phoneme recognition model; based on the result of the loss function, train the phoneme recognition model; The phoneme recognition model comprises a convolutional neural network, a bidirectional long short-term memory network and a neural network-based time sequence class classifier connected in turn, and the inputting of the sample voice data into the phoneme recognition model to output the predicted phoneme sequence comprises: convert the sample voice data into spectrogram data and input it into the convolutional neural network to output voice feature data corresponding to the sample voice data; input the voice feature data into the bidirectional long short-term memory network to input time sequence feature data corresponding to the sample voice data; input the time sequence feature data into the neural network-based time sequence class classifier, which maps the time sequence feature data to specific time points, aligns each phoneme with the time point of the corresponding sound in the sample voice data by generating different numbers of interval characters, and obtains and outputs the predicted phoneme sequence containing interval characters.

2. The speech phoneme recognition method of claim 1, wherein, The voice data is Chinese voice, and the phoneme sequence comprises at least one interval character between each Chinese character.

3. The speech phoneme recognition method of claim 1, wherein, The correctness of the predicted phoneme sequence is determined based on the output predicted phoneme sequence and the sample phoneme sequence, comprising: if the order of occurrence of the same phonemes in the predicted phoneme sequence and the sample phoneme sequence is the same, and there is at least one interval character between the adjacent Chinese characters corresponding to the predicted phoneme sequence, it is determined that the predicted phoneme sequence is correct; if the order of occurrence of the same phonemes in the predicted phoneme sequence and the sample phoneme sequence is different, or there is no interval character between the adjacent Chinese characters corresponding to the predicted phoneme sequence, it is determined that the predicted phoneme sequence is incorrect.

4. A method for training a speech phoneme recognition model, the method comprising: comprising: determining a sample Chinese character sequence corresponding to the sample voice data for training; inserting a sample interval character between each adjacent sample character; converting the sample Chinese characters into sample pinyin; splitting the sample pinyin according to the relationship between pinyin and phonemes, and distinguishing the split phonemes by using space symbols; transforming the sequence comprising the sample pinyin and the sample interval character into a sample phoneme sequence; inputting the sample speech data into the phoneme recognition model to output a predicted phoneme sequence; determining correctness of the predicted phoneme sequence based on the output predicted phoneme sequence and the sample phoneme sequence; calculating a loss function of the phoneme recognition model based on the correctness of the predicted phoneme sequence; training the phoneme recognition model based on the result of the loss function; wherein the phoneme recognition model comprises a convolutional neural network, a bidirectional long short-term memory network and a neural network-based time series class classifier connected in sequence, and the inputting the sample speech data into the phoneme recognition model to output the predicted phoneme sequence comprises: transforming the sample speech data into spectrogram data and inputting the spectrogram data into the convolutional neural network to output speech feature data corresponding to the sample speech data; inputting the speech feature data into the bidirectional long short-term memory network to input time series feature data corresponding to the sample speech data; inputting the time series feature data into the neural network-based time series class classifier, which maps the time series feature data to specific time points, and generates different numbers of interval characters to align each phoneme with the time point of the corresponding sound of the sample speech data, to obtain and output the predicted phoneme sequence comprising interval characters. 5.The voice phoneme recognition model training method of claim 4, wherein, The determining correctness of the predicted phoneme sequence based on the output predicted phoneme sequence and the sample phoneme sequence comprises: if the same phonemes in the predicted phoneme sequence and the sample phoneme sequence have the same order of appearance, and there is at least one interval character between the adjacent Chinese characters corresponding to the predicted phoneme sequence, determining that the predicted phoneme sequence is correct; if the same phonemes in the predicted phoneme sequence and the sample phoneme sequence have different orders of appearance, or there is no interval character between the adjacent Chinese characters corresponding to the predicted phoneme sequence, determining that the predicted phoneme sequence is incorrect.

6. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the speech phoneme recognition method of any one of claims 1 to 3; and / or the computer execution instructions are executed by the processor to implement the speech phoneme recognition model training method of any one of claims 4 to 5. The device comprises:

7. A speech phoneme recognition apparatus characterized by comprising: an identification module configured to input speech data to be identified into a pre-trained phoneme recognition model to output a phoneme sequence corresponding to the speech data, the phoneme sequence comprising an order of appearance of each phoneme in the speech data; characterized in that the device further comprises: a training module configured to train the phoneme recognition model by: determining a sample Chinese character sequence corresponding to sample speech data for training; inserting an interval character between each adjacent sample Chinese character; transforming the sample Chinese characters into sample pinyin; splitting the sample pinyin according to the relationship between pinyin and phonemes, and distinguishing the split phonemes by using space symbols. ​ transforming the sequence comprising the sample pinyin and the sample interval character into a sample phoneme sequence; inputting the sample voice data into the phoneme recognition model to output a predicted phoneme sequence; determining correctness of the predicted phoneme sequence based on the output predicted phoneme sequence and the sample phoneme sequence; calculating a loss function of the phoneme recognition model based on the correctness of the predicted phoneme sequence; training the phoneme recognition model based on a result of the loss function; the phoneme recognition model comprises a convolutional neural network, a bidirectional long short-term memory network and a neural network-based time series class classifier connected in sequence, and the training module is further configured to: transform the sample voice data into spectrogram data and input the spectrogram data into the convolutional neural network to output voice feature data corresponding to the sample voice data; input the voice feature data into the bidirectional long short-term memory network to input time series feature data corresponding to the sample voice data; input the time series feature data into the neural network-based time series class classifier, the neural network-based time series class classifier corresponds the time series feature data to a specific time point, generates different numbers of interval characters to align each phoneme obtained with a time point at which a sound corresponding to the sample voice data appears, and obtains and outputs the predicted phoneme sequence comprising the interval characters.

8. The speech phoneme recognition apparatus of claim 7, wherein the recognition module comprises: the voice data is Chinese voice, and the phoneme sequence comprises at least one interval character between each Chinese character.

9. The speech phoneme recognition apparatus of claim 7, wherein the training module is specifically configured to: if the same phonemes in the predicted phoneme sequence and the sample phoneme sequence appear in the same order, and there is at least one interval character between adjacent Chinese characters corresponding to the predicted phoneme sequence, determine that the predicted phoneme sequence is a correct result; if the same phonemes in the predicted phoneme sequence and the sample phoneme sequence appear in different orders, or there is no interval character between adjacent Chinese characters corresponding to the predicted phoneme sequence, determine that the predicted phoneme sequence is an incorrect result. 10.A voice phoneme recognition model training apparatus, characterized in that, comprise: a sample module configured to determine a sample Chinese character sequence corresponding to sample voice data for training; inserting a sample interval character between each adjacent sample character; transforming the sample Chinese characters into sample pinyin; splitting the sample pinyin according to the relationship between pinyin and phonemes, and distinguishing the split phonemes by using space symbols; and transforming the sequence comprising the sample pinyin and the sample interval character into a sample phoneme sequence; The training module is configured to input the sample voice data into the phoneme recognition model to output a predicted phoneme sequence, determine correctness of the predicted phoneme sequence based on the output predicted phoneme sequence and the sample phoneme sequence, calculate a loss function of the phoneme recognition model based on the correctness of the predicted phoneme sequence, train the phoneme recognition model based on a result of the loss function, convert the sample voice data into spectrogram data and input the spectrogram data into the convolutional neural network if the phoneme recognition model comprises a convolutional neural network, a bidirectional long short-term memory network and a neural network-based time series class classifier in sequence, output voice feature data corresponding to the sample voice data, input the voice feature data into the bidirectional long short-term memory network to input time series feature data corresponding to the sample voice data, input the time series feature data into the neural network-based time series class classifier, and make the neural network-based time series class classifier correspond the time series feature data to specific time points, generate different numbers of interval characters, align each phoneme obtained with a time point at which a sound corresponding to the sample voice data appears, and obtain and output the predicted phoneme sequence containing the interval characters. 11.The speech phoneme recognition model training apparatus of claim 10, wherein, The training module is specifically configured to: if the same phonemes in the predicted phoneme sequence and the sample phoneme sequence appear in the same order, and there is at least one interval character between adjacent Chinese characters corresponding to the predicted phoneme sequence, determine that the predicted phoneme sequence is a correct result; if the same phonemes in the predicted phoneme sequence and the sample phoneme sequence appear in different orders, or there is no interval character between adjacent Chinese characters corresponding to the predicted phoneme sequence, determine that the predicted phoneme sequence is an incorrect result.

12. A computing device, comprising: The method comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to cause the computing device to perform the voice phoneme recognition method of any one of claims 1 to 3; and / or the computer executes the instructions to implement the voice phoneme recognition model training method of any one of claims 4 to 5 when the instructions are executed by the processor.

Images