Phoneme-based voice playing method and phoneme prediction model training method

Abstract

The invention provides a phoneme-based voice playing method and a phoneme prediction model training method. The method comprises the steps of obtaining to-be-played text data; determining a phoneme code of the to-be-played text data based on a preset phoneme prediction model; wherein the preset phoneme prediction model comprises an embedding layer, a Conformer layer and a Decoder layer, the preset phoneme prediction model is used for outputting phoneme codes of the text data, the phoneme codes are in the form of character strings, one character in the phoneme codes represents one piece of phoneme information, and the phoneme information represents pronunciation; and performing voice playing on the to-be-played text data according to phoneme information represented by each character in the phoneme code. According to the method, the accuracy of phoneme generation is improved, and then the accuracy of voice playing is improved.

Description

Technical Field

[0001] The present application relates to the field of speech and model, and in particular to a phoneme-based speech playback method and a phoneme prediction model training method. Background Art

[0002] Phoneme generation technology is used to convert text into speech so that a device can play speech, wherein a phoneme is the smallest unit of speech.

[0003] However, phoneme generation technology generally uses large models such as the grapheme-to-phoneme (G2P) model, which consumes a lot of computing resources. On the end side, such as point-reading pen products, the use of existing phoneme generation technology is limited due to insufficient chip support and limited computing resources on the end side.

[0004] Therefore, the terminal side faces the problem of low phoneme generation accuracy and poor voice playback accuracy. Summary of the invention

[0005] The present application provides a phoneme-based speech playback method and a phoneme prediction model training method to solve the technical problem of poor accuracy in generating phonemes on the terminal side.

[0006] In a first aspect, the present application provides a phoneme-based speech playback method, comprising:

[0007] Get the text data to be played;

[0008] Based on a preset phoneme prediction model, determine the phoneme encoding of the text data to be played; wherein the preset phoneme prediction model includes an embedding layer, a convolution enhanced transformer Conformer layer, and a decoding Decoder layer, and the preset phoneme prediction model is used to output the phoneme encoding of the text data, the phoneme encoding is in the form of a string, one character in the phoneme encoding represents one phoneme information, and the phoneme information represents pronunciation;

[0009] According to the phoneme information represented by each character in the phoneme code, the text data to be played is played voice-wise.

[0010] Optionally, as in the above method, determining the phoneme encoding of the text data to be played based on a preset phoneme prediction model includes:

[0011] Determine letter identification data of the text data to be played; wherein the letter identification data includes a plurality of digital identifications, and the data identifications represent letters constituting the text data to be played;

[0012] The letter identification data is input into a preset phoneme prediction model to obtain the output phoneme encoding of the text data to be played.

[0013] Optionally, as in the above method, the letter identification data is input into a preset phoneme prediction model to obtain the output phoneme encoding of the text data to be played, including:

[0014] Through the Embedding layer in the preset phoneme prediction model, feature extraction processing is performed on each digital identifier in the letter identifier data to obtain a first feature vector of each digital identifier in the letter identifier data; wherein the first feature vector is a preset vector data of a first dimension, and the first feature vector represents context information of the digital identifier;

[0015] By using a Conformer layer in a preset phoneme prediction model, a vector conversion process is performed on the first feature vector of each digital identifier in the letter identifier data to obtain a second feature vector of each digital identifier in the letter identifier data; wherein the second feature vector is vector data of a preset second dimension, the second feature vector represents context information of the digital identifier, and the preset first dimension is smaller than the preset second dimension;

[0016] The second feature vector of each digital identifier in the letter identifier data is classified by the Decoder layer in the preset phoneme prediction model to obtain the character corresponding to each digital identifier in the letter identifier data;

[0017] The phoneme code of the text data to be played is determined according to the characters corresponding to the digital identifiers in the letter identifier data.

[0018] Optionally, as in the above method, the second feature vector of each digital identifier in the letter identifier data is classified by the Decoder layer in the preset phoneme prediction model to obtain the character corresponding to each digital identifier in the letter identifier data, including:

[0019] For each digital identifier in the letter identifier data, a probability value between the second feature vector and each preset character is determined through a Decoder layer in a preset phoneme prediction model; wherein the probability value represents the possibility that the second feature vector is a preset character;

[0020] According to the probability value, the character corresponding to the numeric identification in the letter identification data is determined.

[0021] Optionally, as in the above method, determining the character corresponding to the digital identifier in the letter identifier data according to the probability value includes:

[0022] If the probability value is less than a preset probability threshold, it is determined that there is no character corresponding to the digital identifier in the letter identifier data, and a prompt message is output; wherein the prompt message is used to prompt the user to check the text data to be played.

[0023] Optionally, as in the above method, the text data to be played is English data; obtaining the text data to be played includes:

[0024] Get the initial English data;

[0025] The letters of the first format in the initial English data are converted into letters of the second format to obtain the text data to be played; wherein the first format represents the format of uppercase English letters, and the second format represents the format of lowercase English letters.

[0026] In a second aspect, the present application provides a method for training a phoneme prediction model, comprising:

[0027] Acquire a data set to be trained; wherein the data set to be trained includes letter identification data of the data to be trained and actual phoneme encoding of the data to be trained;

[0028] According to the letter identification data of the data to be trained and the actual phoneme encoding of the data to be trained, the preset initial model is trained to obtain a trained phoneme prediction model; wherein the phoneme prediction model is used to implement any phoneme-based speech playback method in the first aspect.

[0029] Optionally, the method as described above further includes:

[0030] Acquire initial training data; wherein the initial training data is data in text form;

[0031] Determine the target length according to the text length of each initial training data; wherein the target length represents the maximum text length of the data to be trained;

[0032] According to the target length and based on the preset null characters, the initial training data is supplemented to obtain the data to be trained.

[0033] Optionally, as in the above method, a preset initial model is trained according to the letter identification data of the data to be trained and the actual phoneme encoding of the data to be trained to obtain a trained phoneme prediction model, including:

[0034] Inputting the letter identification data of the data to be trained into a preset initial model to obtain the predicted phoneme encoding of the data to be trained as output;

[0035] According to the predicted phoneme coding of the data to be trained and the corresponding actual phoneme coding, the preset initial model is trained to obtain a trained phoneme prediction model.

[0036] Optionally, as in the above method, the phoneme code includes multiple characters; according to the predicted phoneme code of the data to be trained and the corresponding actual phoneme code, the preset initial model is trained to obtain a trained phoneme prediction model, including:

[0037] Determine the number of erroneous characters according to the predicted phoneme code of the training data and the corresponding actual phoneme code; wherein the number of erroneous characters is at least one of a first number, a second number, and a third number; the first number represents the number of inconsistent characters at the same character position in the predicted phoneme code and the actual phoneme code; the second number represents the number of characters for which the predicted phoneme code is less than the actual phoneme code, and the third number represents the number of characters for which the predicted phoneme code is more than the actual phoneme code;

[0038] According to the number of erroneous characters and the number of characters in the actual phoneme encoding, an error rate is obtained; wherein the error rate represents the probability that the predicted phoneme encoding is an erroneous phoneme encoding;

[0039] If the error rate meets the preset training conditions, a trained phoneme prediction model is obtained.

[0040] In a third aspect, the present application provides a phoneme-based speech playback device, comprising:

[0041] A first acquisition unit, used for acquiring text data to be played;

[0042] A determination unit, used to determine the phoneme encoding of the text data to be played based on a preset phoneme prediction model; wherein the preset phoneme prediction model includes an embedding layer, a convolution enhanced transformer Conformer layer, and a decoding Decoder layer, and the preset phoneme prediction model is used to output the phoneme encoding of the text data, the phoneme encoding is in the form of a character string, one character in the phoneme encoding represents one phoneme information, and the phoneme information represents pronunciation;

[0043] The playing unit is used to play the text data to be played in voice according to the phoneme information represented by each character in the phoneme code.

[0044] In a fourth aspect, the present application provides a training device for a phoneme prediction model, characterized in that it includes:

[0045] A second acquisition unit is used to acquire a data set to be trained; wherein the data set to be trained includes letter identification data of the data to be trained and actual phoneme encoding of the data to be trained;

[0046] A training unit is used to train a preset initial model according to the letter identification data of the data to be trained and the actual phoneme encoding of the data to be trained, so as to obtain a trained phoneme prediction model; wherein the phoneme prediction model is used to implement any one of the phoneme-based speech playback methods in the first aspect.

[0047] In a fifth aspect, the present application provides an electronic device, including: a memory, a processor;

[0048] Memory stores computer-executable instructions;

[0049] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the first aspect and / or various possible implementations of the first aspect, the second aspect and / or various possible implementations of the second aspect as described above.

[0050] In a sixth aspect, the present application provides a computer-readable storage medium, which stores computer execution instructions. When the computer execution instructions are executed by a processor, they are used to implement the first aspect and / or various possible implementation methods of the first aspect, the second aspect and / or various possible implementation methods of the second aspect as described above.

[0051] In a seventh aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementations of the first aspect, the second aspect and / or various possible implementations of the second aspect.

[0052] The phoneme-based speech playback method and the training method of the phoneme prediction model provided in the present application obtain the text data to be played, determine the phoneme encoding of the text data to be played based on the preset phoneme prediction model, and then perform speech playback on the text data to be played according to the phoneme information represented by each character in the phoneme encoding. Among them, the preset phoneme prediction model includes an embedding layer, a convolution enhanced transformer Conformer layer, and a decoding Decoder layer. The preset phoneme prediction model is used to output the phoneme encoding of the text data. The phoneme encoding is in the form of a string. One character in the phoneme encoding represents a phoneme information, and the phoneme information represents the pronunciation. At the same time, the model structure of the preset phoneme prediction model is simple and light compared to the model with large parameters and large computational amount, and can run smoothly on the terminal side. The phoneme-based speech playback method provided in the present application improves the accuracy of phoneme generation, thereby improving the accuracy of speech playback. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0054] Figure 1 A flowchart of a phoneme-based speech playback method provided in this application;

[0055] Figure 2 A flowchart of a method for training a phoneme prediction model provided in this application;

[0056] Figure 3 A schematic diagram of the architecture of the phoneme prediction model provided for this application;

[0057] Figure 4 A phoneme-based speech playback device provided for this application;

[0058] Figure 5 Another phoneme-based speech playback device provided for this application;

[0059] Figure 6 A training device for a phoneme prediction model provided in this application;

[0060] Figure 7 A schematic diagram of the structure of the electronic device provided in this application.

[0061] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0062] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0063] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0064] Phonemes can refer to the smallest unit of speech divided according to the natural properties of speech. According to the pronunciation action in a syllable, one action can constitute a phoneme. For example, it can be understood that a phoneme is the pronunciation phonetic symbol corresponding to an English word.

[0065] Phoneme generation technology is used to convert text to speech so that the device can play the speech.

[0066] Existing phoneme generation technologies generally use large models such as the G2P model, which consumes a lot of computing resources.

[0067] On the end side, such as the talking pen product, the chip size on the end side is less than 2 terabytes (TB). Due to the insufficient chip support and limited computing resources on the end side, if the model of the phoneme generation technology with a heavy load is given to the end side, the end side faces the problem of overload and thus being unable to run. If the end side does not use the existing phoneme generation technology, the problem of poor phoneme generation accuracy will be faced, which will lead to poor accuracy of voice playback.

[0068] Therefore, the terminal side faces the problem of low phoneme generation accuracy and poor voice playback accuracy.

[0069] The present application provides a phoneme-based voice playback method and a phoneme prediction model training method, by obtaining the text data to be played, based on the preset phoneme prediction model, determining the phoneme encoding of the text data to be played, and then performing voice playback on the text data to be played according to the phoneme information represented by each character in the phoneme encoding. Among them, the preset phoneme prediction model includes an embedding layer, a convolution enhanced transformer Conformer layer, and a decoding Decoder layer. The preset phoneme prediction model is used to output the phoneme encoding of the text data. The phoneme encoding is in the form of a string. One character in the phoneme encoding represents a phoneme information, and the phoneme information represents the pronunciation. At the same time, the model structure of the preset phoneme prediction model is simple and light compared to the model with large parameters and large computational amount, and can run smoothly on the end side. Among them, the Conformer layer can be provided with a Relu activation function, and the Decoder layer includes a convolution layer, which is a convolution structure, so that the accuracy or operator used in the network layer of the phoneme prediction model during the training process or use process can be avoided. Accumulation of errors caused by the model, so that it can be better adapted to the chip on the end side.

[0070] In addition, the present application provides a method for training a sentence segmentation model, by obtaining a data set to be trained, wherein the data set to be trained includes letter identification data of the data to be trained and actual phoneme encoding of the data to be trained; according to the letter identification data of the data to be trained and the actual phoneme encoding of the data to be trained, a preset initial model is trained to obtain a trained phoneme prediction model. The phoneme prediction model is used to implement the phoneme-based speech playback method of the present application to improve the accuracy of phoneme generation, thereby improving the accuracy of speech playback.

[0071] The phoneme-based speech playback method provided in the present application improves the accuracy of generating phonemes on the terminal side, thereby improving the accuracy of speech playback.

[0072] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0073] Figure 1 A flowchart of a phoneme-based voice playback method provided in this application is provided. The execution subject of the method can be a terminal side, a server, a host or other devices, such as Figure 1 As shown, the method may include:

[0074] S101, obtaining text data to be played.

[0075] The text data to be played may refer to text information to be processed into voice.

[0076] Exemplarily, the text data may be any form of written language, including but not limited to words, sentences, paragraphs, and the like.

[0077] In a possible implementation, the text data to be played may have multiple sources, such as user input information, scanning results of a scanner, pre-stored text files, etc.

[0078] In an optional implementation manner, the text data to be played is English data.

[0079] Obtaining the text data to be played may specifically include:

[0080] Initial English data is obtained; letters in a first format in the initial English data are converted into letters in a second format to obtain text data to be played; wherein the first format represents the format of uppercase English letters, and the second format represents the format of lowercase English letters.

[0081] The initial English data may refer to English data including at least one of English letters in a first format and English letters in a second format.

[0082] The English letters in the first format may be uppercase English letters, and the English letters in the second format may be lowercase English letters.

[0083] It can be understood that the text data to be played can be obtained by converting the letters in the first format in the initial English data into letters in the second format.

[0084] Exemplarily, the initial English data may be represented as "HelloWorld", and by converting letters in the first format in the initial English data into letters in the second format, the text data to be played obtained may be represented as "helloworld".

[0085] In a possible implementation, obtaining the text data to be played may further include:

[0086] The letters in the second format in the initial English data are converted into letters in the first format to obtain text data to be played.

[0087] It can be understood that the obtained text data to be played is composed of letters in the first format, such as "HELLOWORLD".

[0088] The beneficial effect of such a setting is that letters in the first format in the initial English data are converted into letters in the second format, or letters in the second format in the initial English data are converted into letters in the first format, so that the letters in the obtained text data to be played are in a unified format, thereby reducing the complexity of the calculations in subsequent steps.

[0089] S102. Determine the phoneme encoding of the text data to be played based on a preset phoneme prediction model; wherein the preset phoneme prediction model includes an embedding layer, a convolutional enhanced transformer Conformer layer, and a decoding Decoder layer. The preset phoneme prediction model is used to output the phoneme encoding of the text data. The phoneme encoding is in the form of a character string. One character in the phoneme encoding represents one phoneme information, and the phoneme information represents pronunciation.

[0090] The phoneme code may be in the form of a character string, and the phoneme code may refer to a code corresponding to the phoneme information. One character in the phoneme code represents one phoneme information, and the phoneme information may represent pronunciation.

[0091] It can be understood that according to the text data to be played, the phoneme code of the text data to be played can be output using the preset phoneme prediction model. The preset phoneme prediction model can be a pre-built neural network model, that is, the text data to be played can be input into the preset phoneme prediction model, and the model outputs the phoneme code.

[0092] For example, the text data to be played can be represented as "epistemology", and the phoneme encoding of the text data to be played can be represented as {38, 56, 40, 58, 60, 10, 47, 5, 46, 10, 44, 41}. Among them, the phoneme information corresponding to e is represented by 38, and the phoneme information corresponding to p is represented by 56. In this category, each letter corresponds to a number.

[0093] In a possible implementation, a preset phoneme coding library exists in the storage medium of the execution subject of the method of the present application, and the phoneme coding library is used to store the association relationship between each character and phoneme information.

[0094] Taking a character "38" in the phoneme encoding as an example, "38" can represent a phoneme information "IH0", and the phoneme information "IH0" can represent the pronunciation of the letter e.

[0095] For example, the text data stored in the speech database can be expressed as "epistemology", and the speech information corresponding to the text data can be expressed as {IH0, P, IH2, S, T, AH0, M, AA1, L, AH0, JH, IY0}. Among them, the phoneme information corresponding to e is "IH0", and the phoneme information corresponding to p is "P". In this category, each letter corresponds to a phoneme information.

[0096] In a possible implementation, before determining the phoneme encoding of the to-be-played text data based on a preset phoneme prediction model, the following may be included:

[0097] Determine whether the text data to be played exists in the preset voice library;

[0098] If the text data to be played exists in the preset voice library, voice information corresponding to the text data to be played is obtained from the preset voice library, and the text data to be played is played by voice.

[0099] If the text data to be played does not exist in the preset voice library, the phoneme encoding of the text data to be played is determined based on the preset phoneme prediction model, and the text data to be played is voice played according to the phoneme information represented by each character in the phoneme encoding.

[0100] In one possible implementation, after determining the phoneme encoding of the text data to be played based on a preset phoneme prediction model, the voice information corresponding to the phoneme encoding of the text data to be played can be stored in a preset voice library to improve the richness of the content of the voice library.

[0101] By setting a preset voice library, the preset voice library can be used to store text data and voice information corresponding to the text data, so that the text data to be played exists in the preset voice library, and then the next time the phoneme-based voice playback method is performed, the voice information corresponding to the text data to be played can be directly obtained from the preset voice library to perform voice playback on the text data to be played, without determining the phoneme encoding of the text data to be played based on a preset phoneme prediction model, thereby reducing the occupancy of computing resources when using the phoneme-based voice playback method on the end side, and ensuring the smoothness of the end side operation.

[0102] In an optional implementation, determining the phoneme encoding of the text data to be played based on a preset phoneme prediction model may include:

[0103] Determine letter identification data of the text data to be played; wherein the letter identification data includes multiple digital identifications, and the data identifications represent letters that constitute the text data to be played; input the letter identification data into a preset phoneme prediction model to obtain the output phoneme encoding of the text data to be played.

[0104] The letter identification data may refer to data composed of at least one digital identification, and the digital identification may represent letters constituting the text data to be played.

[0105] In a possible implementation, a storage medium of the execution subject of the method of the present application contains a preset letter code library, and the letter code library is used to store the association relationship between letters and digital identifiers.

[0106] For example, the text data to be played can be represented as "epistemology", and the digital identifiers corresponding to the letters in the text data to be played can be represented in sequence as {6, 17, 10, 20, 21, 6, 14, 16, 13, 16, 8, 26}, that is, the letter identification data of the text data to be played. Among them, the digital identifier of the letter e is represented by 6, and the digital identifier of the letter p is represented by 17, and each letter has a corresponding digital identifier.

[0107] In an optional implementation, inputting the letter identification data into a preset phoneme prediction model to obtain the output phoneme encoding of the text data to be played may include:

[0108] Through the Embedding layer in the preset phoneme prediction model, feature extraction processing is performed on each digital identifier in the letter identifier data to obtain a first feature vector of each digital identifier in the letter identifier data; wherein the first feature vector is a preset vector data of a first dimension, and the first feature vector represents context information of the digital identifier;

[0109] By using a Conformer layer in a preset phoneme prediction model, a vector conversion process is performed on the first feature vector of each digital identifier in the letter identifier data to obtain a second feature vector of each digital identifier in the letter identifier data; wherein the second feature vector is vector data of a preset second dimension, the second feature vector represents context information of the digital identifier, and the preset first dimension is smaller than the preset second dimension;

[0110] The second feature vector of each digital identifier in the letter identifier data is classified by the Decoder layer in the preset phoneme prediction model to obtain the character corresponding to each digital identifier in the letter identifier data;

[0111] The phoneme code of the text data to be played is determined according to the characters corresponding to the digital identifiers in the letter identifier data.

[0112] It can be understood that the preset phoneme prediction model may include an embedding layer, a convolutional enhanced transformer Conformer layer, and a decoding Decoder layer.

[0113] The embedding layer may be used to perform feature extraction processing on each digital identifier in the letter identifier data to obtain a first feature vector of each digital identifier in the letter identifier data.

[0114] The feature extraction process may refer to mapping the digital identifier into a vector form, that is, mapping the digital identifier into a first feature vector, the first feature vector represents the context information of the digital identifier. The dimension of the first feature vector may be a preset first dimension of vector data.

[0115] Exemplarily, the first dimension may be 64 dimensions.

[0116] For example, the letter identification data may be the above-mentioned {6, 17, 10, 20, 21, 6, 14, 16, 13, 16, 8, 26}, and feature extraction processing is performed on each digital identification in the letter identification data, and the first feature vector of each digital identification in the letter identification data may be {[ ],[ ],[ ],[ ],[ ],[ ],[ ],[ ],[ ],[ ],[ ],[ ]},in, represents the first eigenvector of a digital identity, The subscript n represents the order of each digital identifier in the letter identifier data. The first eigenvector corresponding to the digital identifier 6 is [ ], the first eigenvector corresponding to the digital identifier 17 is [ ], in this category, each digital identifier corresponds to a first eigenvector.

[0117] For the first feature vector of each digital identifier in the letter identifier data, the dimension of the first feature vector is 64.

[0118] The convolution enhanced transformer Conformer layer can be used to perform vector conversion processing on the first feature vector of each digital identification in the letter identification data to obtain the second feature vector of each digital identification in the letter identification data.

[0119] The vector conversion process may refer to converting the first feature vector of each digital identifier in the letter identifier data into a second feature vector. The second feature vector is vector data of a preset second dimension, the second feature vector represents context information of the digital identifier, and the preset first dimension is smaller than the preset second dimension.

[0120] Exemplarily, the dimension of the second feature vector may be 256 dimensions.

[0121] For example, the first feature vector of each digital identifier in the letter identifier data may be the above-mentioned {[ ],[ ],[ ],[ ],[ ],[ ],[ ],[ ],[ ],[ ],[ ],[ ]}, after vector conversion, the second feature vector of each digital identifier in the obtained letter identifier data can be expressed as {[ ],[ ],[ ],[ ],[ ],[ ],[ ],[ ],[ ],[ ],[ ],[ ]},in, represents the second eigenvector of a digital identity, The subscript n represents the order of each digital identifier in the letter identifier data. The first eigenvector corresponding to the digital identifier 6 is [ ]The corresponding second eigenvector is [ ], the first eigenvector corresponding to the number 17 [ ] corresponds to the second eigenvector [ ], in this category, each digital identifier corresponds to a first eigenvector and a second eigenvector.

[0122] For the second feature vector of each digital identifier in the letter identifier data, the dimension of the second feature vector is 256.

[0123] The decoding Decoder layer can be used to classify the second feature vector of each digital identification in the letter identification data to obtain the character corresponding to each digital identification in the letter identification data.

[0124] The classification process may refer to classifying the second feature vector of each digital identification in the letter identification data into the character corresponding to each digital identification in the letter identification data.

[0125] Exemplarily, the second feature vector of each digital identifier in the letter identifier data can be expressed as the above {[ ],[ ],[ ],[ ],[ ],[ ],[ ],[ ],[ ],[ ],[ ],[ ]}, after classification processing, the characters corresponding to each digital identifier in the obtained letter identifier data can be represented as {38, 56, 40, 58, 60, 10, 47, 5, 46, 10, 44, 41}, among which the second eigenvector corresponding to the digital identifier 6 is [ ] The character corresponding to is represented by 38, and the second eigenvector corresponding to the digital identifier 17 is [ ] The character corresponding to is represented by 56. In this category, each digital identifier corresponds to a first eigenvector, a second eigenvector, and a character.

[0126] It can be understood that by arranging the characters corresponding to the digital identifiers in the letter identification data in sequence, the phoneme code of the text data to be played can be obtained as {38, 56, 40, 58, 60, 10, 47, 5, 46, 10, 44, 41}, and the phoneme information corresponding to the phoneme code can be expressed as {IH0, P, IH2, S, T, AH0, M, AA1, L, AH0, JH, IY0}.

[0127] It should be noted that the phoneme information corresponding to the phoneme encoding can also be expressed as {IH0 P IH2 ST AH0 MAA1 L AH0 JH IY0}. There is no limitation on the connection method of each phoneme information in the phoneme information, as long as the phoneme information can be played.

[0128] It can be understood that the input data of the preset phoneme prediction model is in digital form, for example, {6, 17, 10, 20, 21, 6, 14, 16, 13, 16, 8, 26}, and the output data of the preset phoneme prediction model is also in digital form, for example, {38, 56, 40, 58, 60, 10, 47, 5, 46, 10, 44, 41}, wherein each digital identifier in the input data corresponds to each character in the output data in a one-to-one relationship.

[0129] The beneficial effect of such a setting is that using characters in digital form as input data or output data of the preset phoneme prediction model, rather than other complex characters or high-precision characters, can reduce the pressure on computing resources used when the phoneme prediction model is running, while ensuring the accuracy of phoneme generation and the smoothness of operation on the end side.

[0130] In an optional implementation, the second feature vector of each digital identifier in the letter identifier data is classified by the Decoder layer in the preset phoneme prediction model to obtain the character corresponding to each digital identifier in the letter identifier data, which may include:

[0131] For each digital identifier in the letter identification data, the probability value between the second feature vector and each preset character is determined through the Decoder layer in the preset phoneme prediction model; wherein the probability value represents the possibility that the second feature vector is a preset character; and based on the probability value, the character corresponding to the digital identifier in the letter identification data is determined.

[0132] The preset character may refer to a character corresponding to a digital identifier that may be output by a preset phoneme prediction model.

[0133] In a possible implementation, the preset character may be any integer between 4 and 73.

[0134] The probability value between the second feature vector and each preset character can represent the correlation strength between the second feature vector and each preset character, that is, the possibility that the second vector is a preset character.

[0135] In one possible implementation, the Decoder layer in the preset phoneme prediction model is provided with a softmax function for outputting a probability value between the second feature vector and each preset character. The implementation method for outputting the probability value between the second feature vector and each preset character is not limited herein.

[0136] It can be understood that a large probability value between the second eigenvector of the digital identifier and a certain character means that the digital identifier has a higher correlation with the character among the preset characters, that is, the second eigenvector of the digital identifier has a higher matching degree with the character, and the character corresponding to the digital identifier is more likely to be the character.

[0137] Exemplarily, for a digital identifier in the letter identifier data, there are 70 probability values ​​between the second feature vector corresponding to the digital identifier and each preset character. The character corresponding to the maximum probability among the 70 distance values ​​is the character corresponding to the digital identifier in the letter identifier data.

[0138] In an optional implementation, determining the character corresponding to the digital identifier in the letter identifier data according to the probability value may include:

[0139] If the probability value is less than a preset probability threshold, it is determined that there is no character corresponding to the digital identifier in the letter identifier data, and a prompt message is output; wherein the prompt message is used to prompt the user to check the text data to be played.

[0140] Among them, the preset probability threshold is the probability threshold set in advance by the staff.

[0141] It can be understood that if the probability value is less than the preset probability threshold, it means that no character similar enough to the second feature vector is found from the preset characters.

[0142] Exemplarily, the preset probability threshold is 0.2.

[0143] Taking a digital identifier in the letter identifier data as an example, if there are 70 probability values ​​between the second feature vector corresponding to the digital identifier and each preset character, the maximum probability value among the 70 probability values ​​is taken as the target probability value, for example, the target probability value is 0.18.

[0144] At this time, the target probability value 0.18 is less than the preset probability threshold 0.2, so the character corresponding to the digital identifier cannot be matched from the preset characters.

[0145] In a possible implementation, if the character corresponding to the digital identifier cannot be matched from the preset characters, <unk>Indicates the character corresponding to the digital identifier. <unk>Indicates that the character corresponding to the digital identifier is unknown.

[0146] The prompt information is used to prompt the user to view the text data to be played. For example, the prompt information includes the characters <unk>, used to prompt the user to check the text data to be played, and then check whether the text data is input incorrectly, data is damaged, etc., or prompt the user to correct the characters according to the content of the text data to be played. <unk>The actual character corresponding to the digital identifier.

[0147] The beneficial effect of this setting is that, while ensuring smooth operation on the end side, the phoneme prediction model can not only accurately convert digital identifiers in text data into corresponding characters, but also provide feedback when encountering uncertain or unmatched situations, thereby improving the availability and user satisfaction of the entire system.

[0148] S103 , performing voice playback on the text data to be played according to the phoneme information represented by each character in the phoneme code.

[0149] It can be understood that one character in the phoneme encoding represents one phoneme information, and the phoneme information represents the pronunciation of the letters in the text data to be played.

[0150] Exemplarily, the phoneme code is {38, 56, 40, 58, 60, 10, 47, 5, 46, 10, 44, 41} mentioned above, and the phoneme information represented by each character in the phoneme code can be {IH0, P, IH2, S, T, AH0, M, AA1, L, AH0, JH, IY0}, among which the phoneme information corresponding to 38 is "IH0", and the phoneme information corresponding to 56 is "P". In this category, one character in the phoneme code corresponds to one phoneme information.

[0151] In a possible implementation, a vocoder is present in the storage medium of the execution body of the method of the present application, and the vocoder is used to convert the pronunciation represented by the phoneme information into an audio signal so as to play the audio information, that is, to perform voice playback on the text data to be played.

[0152] Exemplarily, the audio signal may be played through a speaker, earphones or other audio output devices, and the output device for playing the audio signal is not limited herein.

[0153] The present application provides a phoneme-based speech playback method, which uses a simple and lightweight phoneme prediction model to convert text data into phoneme encoding, thereby achieving more natural and accurate speech synthesis. The phoneme-based speech playback method provided by the present application improves the accuracy of generating phonemes at the terminal side, thereby improving the accuracy of speech playback.

[0154] Figure 2 A flowchart of a method for training a phoneme prediction model provided in the present application is provided. The execution subject of the method may be a terminal side, a server, a host or other device, such as Figure 2 As shown, the method may include:

[0155] S201, obtaining a data set to be trained; wherein the data set to be trained includes letter identification data of the data to be trained and actual phoneme encoding of the data to be trained.

[0156] The letter identification data may refer to data composed of at least one digital identification, and the digital identification may represent letters constituting the data to be trained.

[0157] The actual phoneme encoding may be in the form of a character string. The phoneme encoding may refer to the encoding corresponding to the phoneme information. One character in the phoneme encoding represents one phoneme information, and the phoneme information represents pronunciation.

[0158] In an optional implementation, the above method may further include:

[0159] Acquire initial training data; wherein the initial training data is data in text form; determine a target length according to the text length of each initial training data; wherein the target length represents the maximum text length of the data to be trained; according to the target length, supplement the initial training data based on preset null characters to obtain the data to be trained.

[0160] The initial training data is data in text form. For example, the initial training data can be expressed as {green} or {energy}.

[0161] The target length may refer to the maximum text length of the training data.

[0162] Exemplarily, the maximum text length of the data to be trained may be 6.

[0163] The preset null character may refer to a pre-set null character, and the null character is used to supplement the initial training data to reach the target length.

[0164] Exemplarily, the initial training data is {green}, and the data to be trained obtained by supplementing the initial training data {green} can be expressed as {green*}, where * is a null character. Of course, the null character can also be represented by other characters, which is not limited here.

[0165] In a possible implementation, the text to be trained includes multiple batches of sub-texts to be trained, and each batch of sub-texts to be trained includes at least one initial training data after supplementary processing.

[0166] Exemplarily, the sub-text to be trained includes {green*}, {energy}, wherein the length of the training data in the sub-text to be trained is the target length of 6.

[0167] In a possible implementation, the total number of characters in the sub-text to be trained may be 16.

[0168] In one possible implementation, the initial training data after supplementary processing in the sub-text to be trained also includes a starting character and an ending character. By setting the starting character and the ending character, an initial training data can be clearly defined. The characters in the initial training data have a strong contextual relationship, so as to improve the accuracy of the trained phoneme prediction model in predicting phonemes, thereby improving the accuracy of speech playback.

[0169] Exemplarily, the sub-text to be trained may include { <s> green*< / s> }, { <s> Energy Efficiency< / s> },in, <s> is the starting character,< / s> Is the end character.

[0170] In one possible implementation, the correspondence between the letters of the data to be trained and the digital identifiers in the letter identification data may include {'*':0, '':1, 'a':2, 'b':3, 'c':4, 'd':5, 'e':6, 'f':7, 'g':8, 'h':9, 'i':10, 'j':11, 'k':12, 'l':13, 'm':14, 'n':15, 'o':16, 'p':17, 'q':18, 'r':19, 's':20, 't':21, 'u':22, 'v':23, 'w':24, 'x':25, 'y':26, 'z':27}.

[0171] In a possible implementation, the correspondence between the phoneme information and the characters in the phoneme code may include {'*':0,' <unk>':1,' <s>':2，'< / s> ':3,'AA0':4,'AA1':5,'AA2':6,'AE0':7,'AE1':8,'AE2':9,'AH0':10,'AH1':11,'AH 2':12, 'AO0':13, 'AO1':14, 'AO2':15, 'AW0':16, 'AW1':17, 'AW2':18, 'AY0':19, 'AY1' :20, 'AY2':21, 'B':22, 'CH':23, 'D':24, 'DH':25, 'EH0':26, 'EH1':27, 'EH2':28, 'ER 0':29,'ER1':30,'ER2':31,'EY0':32,'EY1':33,'EY2':34,'F':35,'G':36,'HH':37,' IH0':38,'IH1':39,'IH2':40,'IY0':41,'IY1':42,'IY2':43,'JH':44,'K':45,'L':4 6, 'M': 47, 'N': 48, 'NG': 49, 'OW0': 50, 'OW1': 51, 'OW2': 52, 'OY0': 53, 'OY1': 54, 'OY2' :55, 'P':56, 'R':57, 'S':58, 'SH':59, 'T':60, 'TH':61, 'UH0':62, 'UH1':63, 'UH2':6 4, 'UW':65, 'UW0':66, 'UW1':67, 'UW2':68, 'V':69, 'W':70, 'Y':71, 'Z':72, 'ZH':73}.

[0172] Exemplarily, the letter identification data of the data to be trained can be expressed as {6, 17, 10, 20, 21, 6, 14, 16, 13, 16, 8, 26}, and the letter identification data of the data to be trained can represent "epistemology".

[0173] The actual phoneme encoding of the data to be trained can be expressed as {38, 56, 40, 58, 60, 10, 47, 5, 46, 10, 44, 41}. The actual phoneme encoding of the data to be trained can represent the phonemes {IH0, P, IH2, S, T, AH0, M, AA1, L, AH0, JH, IY0}, among which the phoneme represented by 38 is "IH0" and the phoneme represented by 56 is "P". In this category, each number corresponds to a phoneme.

[0174] S202. Train the preset initial model according to the letter identification data of the data to be trained and the actual phoneme encoding of the data to be trained to obtain a trained phoneme prediction model; wherein the phoneme prediction model is used to implement any phoneme-based speech playback method of the present application.

[0175] Among them, training can refer to inputting the letter identification data of the data to be trained into the initial model, and using the actual phoneme encoding of the data to be trained to guide the initial model to learn, and the initial model adjusts the parameters of the model by comparing the difference between each predicted phoneme encoding output by the model and the actual phoneme encoding.

[0176] If the parameters of the model meet the expected conditions, the initial model training is considered to be completed, and a trained phoneme prediction model is obtained.

[0177] In an optional implementation, when the number of model training times reaches 200, it can be considered that the initial model training is completed, and a trained phoneme prediction model is obtained.

[0178] The training completion conditions for obtaining the phoneme prediction model after the initial model training is completed are not limited here.

[0179] In an optional implementation, training a preset initial model according to letter identification data of the data to be trained and the actual phoneme encoding of the data to be trained to obtain a trained phoneme prediction model may include:

[0180] Inputting the letter identification data of the data to be trained into a preset initial model to obtain the predicted phoneme encoding of the data to be trained as output;

[0181] According to the predicted phoneme coding of the data to be trained and the corresponding actual phoneme coding, the preset initial model is trained to obtain a trained phoneme prediction model.

[0182] In a possible implementation, the preset initial model includes an embedding layer, a convolutional enhanced transformer Conformer layer, and a decoding Decoder layer.

[0183] To better describe the architecture of the phoneme prediction model, Figure 3 The schematic diagram of the architecture of the phoneme prediction model provided for this application is as follows: Figure 3 As shown in Figure 1, the architecture of the phoneme prediction model includes an Embedding layer, a Conformer layer, and a Decoder layer.

[0184] Among them, the Conformer layer can include 3 modules, namely the Multi-Headed Self-Attention module, the Convolution module, and the Feed forward module.

[0185] Exemplarily, the Conformer layer is located in a 256-dimensional feature space, so that the dimension of the second feature vector output by the Conformer layer is 256 dimensions.

[0186] For example, the Muti-Headed Self-Attention module in the Conformer layer can use 4 attention heads, and the size of the convolution kernel of the Convolution module can be 3.

[0187] In one possible implementation, a Relu (Rectified Linear Unit) activation function is provided in the Conformer layer to avoid error accumulation caused by the precision or operators used in the network layer of the model during the model training process or use process, so as to better adapt to the chip on the terminal side.

[0188] Exemplarily, each module in the Conformer layer is set with a residual rate of 0.05, which means that 5% of the neuron outputs of each module will be randomly set to zero. It can be understood that setting the residual rate is to use a regularization technique to reduce overfitting of the model and improve the generalization ability of the model.

[0189] Among them, the Decoder layer includes the first convolution layer, the layer normalization module, the second convolution layer, and the linear output layer.

[0190] Exemplarily, the dimension of data processed by the first convolution layer can be 256 dimensions, the dimension of data processed by the second convolution layer can also be 256 dimensions, the convolution kernel of the first convolution layer can be 5, and the convolution kernel of the second convolution layer can be 3.

[0191] Among them, different convolution kernel sizes in the Decoder layer can capture local dependencies of different scales. A larger convolution kernel size (such as 5) can capture contextual information in a wider range, while a smaller convolution kernel size (such as 3) can capture finer details.

[0192] It can be understood that setting a convolution layer in the decoder layer can avoid error accumulation caused by the precision or operators used in the network layer of the model during the model training process or use process, so as to better adapt to the chip on the terminal side.

[0193] The layer normalization module is used to normalize the output of the first convolutional layer to stabilize the model training process and accelerate the convergence of the model. The normalization process represents the normalization of the feature dimensions of the training data, which helps to reduce internal covariate shift and allow a higher learning rate. The layer normalization technology may include, but is not limited to, using the mean and variance of the output data of the first convolutional layer to normalize the vector of each dimension of the output data of the first convolutional layer, so that the distribution of the vector of each dimension of the output data of the first convolutional layer is close to the standard normal distribution.

[0194] It can be understood that layer normalization realizes the transition of the data to be trained from the first convolutional layer to the second convolutional layer.

[0195] In a possible implementation, an activation function is further provided after the first convolutional layer, and the activation function may be a Relu activation function to increase the nonlinear expression capability of the Decoder layer.

[0196] After being processed by two convolutional layers and a layer normalization module, the data is finally fed into the linear output layer.

[0197] The linear output layer may be a fully connected layer, and the number of units in the linear output layer may correspond to the number of preset characters, for example, 74.

[0198] In one possible implementation, the linear output layer includes a softmax activation function to convert the output of the linear output layer into a probability distribution, where each category (preset character) corresponds to a probability value, and the category with the highest probability value is the character corresponding to the digital identifier.

[0199] In a possible implementation, the linear output layer also includes a label smoothing loss function to improve the generalization ability of the model. Exemplarily, the smoothing value is set to 0.1.

[0200] In one possible implementation, during the use of a phoneme prediction model or the training of a phoneme prediction model, the data may be processed with an accuracy of fp16, so as to save computing resources on the terminal side while ensuring the accuracy of the predicted phonemes of the phoneme prediction model.

[0201] The architecture of the phoneme prediction model provided in the present application is simple and lightweight, occupies little computing power resources, and saves computing power resources on the terminal side while ensuring the accuracy of the predicted phonemes of the phoneme prediction model.

[0202] In an optional implementation, the phoneme encoding includes a plurality of characters;

[0203] According to the predicted phoneme coding of the data to be trained and the corresponding actual phoneme coding, the preset initial model is trained to obtain a trained phoneme prediction model, which may include:

[0204] Determine the number of erroneous characters according to the predicted phoneme code of the training data and the corresponding actual phoneme code; wherein the number of erroneous characters is at least one of a first number, a second number, and a third number; the first number represents the number of inconsistent characters at the same character position in the predicted phoneme code and the actual phoneme code; the second number represents the number of characters for which the predicted phoneme code is less than the actual phoneme code, and the third number represents the number of characters for which the predicted phoneme code is more than the actual phoneme code;

[0205] According to the number of erroneous characters and the number of characters in the actual phoneme encoding, an error rate is obtained; wherein the error rate represents the probability that the predicted phoneme encoding is an erroneous phoneme encoding;

[0206] If the error rate meets the preset training conditions, a trained phoneme prediction model is obtained.

[0207] It can be understood that the number of erroneous characters can be understood as the concept of edit distance (Levenshtein distance), that is, the minimum number of editing operations required to convert the predicted phoneme encoding of the training data into the actual phoneme encoding of the training data is the number of erroneous characters.

[0208] The error rate can be obtained by dividing the number of erroneous characters by the number of characters in the actual phoneme encoding.

[0209] Exemplarily, the error rate satisfies the formula:

[0210] ;

[0211] in, represents the error rate, represents the first quantity, represents the second quantity, represents the third quantity, Indicates the number of characters in the actual phoneme encoding.

[0212] The preset training condition may refer to an error rate being less than 0.1. If the error rate is less than 0.1, a trained phoneme prediction model is obtained.

[0213] Exemplarily, the predicted phoneme coding of the data to be trained is {38, 56, 40, 58, 60, 10, 47, 46, 10, 44, 42}, and the actual phoneme coding corresponding to the predicted phoneme coding of the data to be trained is {38, 56, 40, 58, 60, 10, 47, 5, 46, 10, 44, 41}.

[0214] It can be seen that the number of wrong characters in the first number is 1 (equivalent to the predicted phoneme coding number 42, which should actually be 41), the number of wrong characters in the second number is 1 (equivalent to the predicted phoneme coding number 5 is missing, which should actually not be missing), and the number of wrong characters in the first number is 0. Therefore, the number of wrong characters is 2, and the actual number of characters in the phoneme coding is 12.

[0215] Dividing the number of erroneous characters by the number of characters in the actual phoneme encoding, the error rate is 0.17. At this time, the error rate does not meet the preset training conditions, and the initial model needs to be retrained so that the error rate is less than 0.1, and then a trained phoneme prediction model is obtained.

[0216] The present application provides a method for training a phoneme prediction model, which trains a preset initial model according to letter identification data of the data to be trained and the actual phoneme encoding of the data to be trained to obtain a trained phoneme prediction model, and uses training completion conditions such as an error rate that satisfies preset training conditions or other training completion conditions, so that the trained phoneme prediction model can predict phonemes with higher accuracy, so that the phoneme prediction model can be better applied to the phoneme-based speech playback method, thereby improving the accuracy of generating phonemes on the terminal side, and thereby improving the accuracy of speech playback.

[0217] Figure 4 A phoneme-based speech playback device is provided in this application, such as Figure 4 As shown, the phoneme-based speech playback device 40 includes: a first acquisition unit 401 , a determination unit 402 , and a playback unit 403 .

[0218] A first acquisition unit 401 is used to acquire text data to be played;

[0219] The determination unit 402 is used to determine the phoneme coding of the text data to be played based on a preset phoneme prediction model; wherein the preset phoneme prediction model includes an embedding layer, a convolution enhanced transformer Conformer layer, and a decoding Decoder layer, and the preset phoneme prediction model is used to output the phoneme coding of the text data, the phoneme coding is in the form of a character string, one character in the phoneme coding represents one phoneme information, and the phoneme information represents pronunciation;

[0220] The playing unit 403 is used to perform voice playing of the text data to be played according to the phoneme information represented by each character in the phoneme code.

[0221] Figure 5 Another phoneme-based speech playback device provided by this application, such as Figure 5 As shown, the phoneme-based speech playback device 50 includes: a first acquisition unit 501 , a determination unit 502 , and a playback unit 503 , wherein the determination unit 502 further includes a first processing module 5021 .

[0222] In an optional example, the first processing module 5021 is used to determine letter identification data of the text data to be played; wherein the letter identification data includes a plurality of digital identifications, and the data identifications represent letters constituting the text data to be played;

[0223] The letter identification data is input into a preset phoneme prediction model to obtain the output phoneme encoding of the text data to be played.

[0224] In an optional example, the first processing module 5021 is further specifically used to perform feature extraction processing on each digital identifier in the letter identifier data through the Embedding layer in the preset phoneme prediction model to obtain a first feature vector of each digital identifier in the letter identifier data; wherein the first feature vector is a preset first dimension vector data, and the first feature vector represents context information of the digital identifier;

[0225] By using a Conformer layer in a preset phoneme prediction model, a vector conversion process is performed on the first feature vector of each digital identifier in the letter identifier data to obtain a second feature vector of each digital identifier in the letter identifier data; wherein the second feature vector is vector data of a preset second dimension, the second feature vector represents context information of the digital identifier, and the preset first dimension is smaller than the preset second dimension;

[0226] The second feature vector of each digital identifier in the letter identifier data is classified by the Decoder layer in the preset phoneme prediction model to obtain the character corresponding to each digital identifier in the letter identifier data;

[0227] The phoneme code of the text data to be played is determined according to the characters corresponding to the digital identifiers in the letter identifier data.

[0228] In an optional example, the first processing module 5021 is further specifically used to determine, for each digital identifier in the letter identifier data, a probability value between the second feature vector and each preset character through a Decoder layer in a preset phoneme prediction model; wherein the probability value represents the possibility that the second feature vector is a preset character;

[0229] According to the probability value, the character corresponding to the numeric identification in the letter identification data is determined.

[0230] In an optional example, the first processing module 5021 is further specifically used to determine that there is no character corresponding to the digital identifier in the letter identifier data if the probability value is less than a preset probability threshold, and output a prompt message; wherein the prompt message is used to prompt the user to check the text data to be played.

[0231] In an optional example, the text data to be played is English data; the first acquisition unit 501 is also used to acquire initial English data;

[0232] The letters of the first format in the initial English data are converted into letters of the second format to obtain the text data to be played; wherein the first format represents the format of uppercase English letters, and the second format represents the format of lowercase English letters.

[0233] Figure 6 A training device for a phoneme prediction model provided in this application, such as Figure 4 As shown, the training device 60 for the phoneme prediction model includes: a second acquisition unit 601 and a training unit 602.

[0234] The second acquisition unit 601 is used to acquire a data set to be trained; wherein the data set to be trained includes letter identification data of the data to be trained and actual phoneme encoding of the data to be trained;

[0235] The training unit 602 is used to train the preset initial model according to the letter identification data of the data to be trained and the actual phoneme encoding of the data to be trained to obtain a trained phoneme prediction model; wherein the phoneme prediction model is used to implement any phoneme-based speech playback method in the present application.

[0236] In an optional example, the second acquisition unit 601 is further used to acquire initial training data; wherein the initial training data is data in text form;

[0237] Determine the target length according to the text length of each initial training data; wherein the target length represents the maximum text length of the data to be trained;

[0238] According to the target length and based on the preset null characters, the initial training data is supplemented to obtain the data to be trained.

[0239] In an optional example, the training unit 602 is further used to input letter identification data of the data to be trained into a preset initial model to obtain the predicted phoneme encoding of the data to be trained as output;

[0240] According to the predicted phoneme coding of the data to be trained and the corresponding actual phoneme coding, the preset initial model is trained to obtain a trained phoneme prediction model.

[0241] In an optional example, the phoneme code includes a plurality of characters, and the training unit 602 is further specifically used to determine the number of erroneous characters according to the predicted phoneme code of the to-be-trained data and the corresponding actual phoneme code; wherein the number of erroneous characters is at least one of a first number, a second number, and a third number; the first number represents the number of inconsistent characters at the same character position in the predicted phoneme code and the actual phoneme code; the second number represents the number of characters for which the predicted phoneme code is less than the actual phoneme code, and the third number represents the number of characters for which the predicted phoneme code is more than the actual phoneme code;

[0242] According to the number of erroneous characters and the number of characters in the actual phoneme encoding, an error rate is obtained; wherein the error rate represents the probability that the predicted phoneme encoding is an erroneous phoneme encoding;

[0243] If the error rate meets the preset training conditions, a trained phoneme prediction model is obtained.

[0244] Figure 7 A schematic diagram of the structure of the electronic device provided in this application, such as Figure 7 As shown, the electronic device 70 provided in this embodiment includes: at least one processor 701 and a memory 702. Optionally, the device 70 further includes a communication component 703. The processor 701, the memory 702 and the communication component 703 are connected via a bus 704.

[0245] In a specific implementation process, at least one processor 701 executes the computer-executable instructions stored in the memory 702, so that at least one processor 701 executes the above method.

[0246] The specific implementation process of the processor 701 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0247] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0248] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0249] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0250] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0251] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0252] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0253] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0254] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0255] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0256] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. Unless otherwise specified, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. Unless otherwise specified, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory (RRAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), an enhanced dynamic random access memory (EDRAM), a high-bandwidth memory (HBM), a hybrid memory cube (HMC), etc.

[0257] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0258] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0259] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0260] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.< / unk> < / unk> < / unk> < / unk> < / unk>

Claims

1. A phoneme-based speech playback method, characterized in that: include: Get the text data to be played; Based on a preset phoneme prediction model, determine the phoneme encoding of the text data to be played; wherein the preset phoneme prediction model includes an embedding layer, a convolution enhanced transformer Conformer layer, and a decoding Decoder layer, and the preset phoneme prediction model is used to output the phoneme encoding of the text data, the phoneme encoding is in the form of a character string, one character in the phoneme encoding represents one phoneme information, and the phoneme information represents pronunciation; The text data to be played is played voice-wise according to the phoneme information represented by each character in the phoneme code.

2. The method according to claim 1, characterized in that Determining the phoneme encoding of the text data to be played based on a preset phoneme prediction model includes: Determine letter identification data of the text data to be played; wherein the letter identification data includes a plurality of digital identifications, and the digital identifications represent letters constituting the text data to be played; The letter identification data is input into a preset phoneme prediction model to obtain the output phoneme encoding of the text data to be played.

3. The method according to claim 2, characterized in that Inputting the letter identification data into a preset phoneme prediction model to obtain the output phoneme encoding of the text data to be played, including: Through the Embedding layer in the preset phoneme prediction model, feature extraction processing is performed on each digital identifier in the letter identifier data to obtain a first feature vector of each digital identifier in the letter identifier data; wherein the first feature vector is a preset vector data of a first dimension, and the first feature vector represents context information of the digital identifier; Performing vector conversion processing on the first feature vector of each digital identifier in the letter identifier data through the Conformer layer in the preset phoneme prediction model to obtain a second feature vector of each digital identifier in the letter identifier data; wherein the second feature vector is vector data of a preset second dimension, the second feature vector represents context information of the digital identifier, and the preset first dimension is smaller than the preset second dimension; Classify the second feature vector of each digital identifier in the letter identifier data through the Decoder layer in the preset phoneme prediction model to obtain the character corresponding to each digital identifier in the letter identifier data; The phoneme code of the text data to be played is determined according to the characters corresponding to the digital identifiers in the letter identifier data.

4. The method according to claim 3, characterized in that The second feature vector of each digital identifier in the letter identifier data is classified by the Decoder layer in the preset phoneme prediction model to obtain the character corresponding to each digital identifier in the letter identifier data, including: For each digital identifier in the letter identifier data, determine a probability value between the second feature vector and each preset character through a Decoder layer in the preset phoneme prediction model; wherein the probability value represents the possibility that the second feature vector is a preset character; According to the probability value, the character corresponding to the digital identification in the letter identification data is determined.

5. The method according to claim 4, characterized in that Determining, according to the probability value, a character corresponding to the digital identifier in the letter identifier data, comprising: If the probability value is less than a preset probability threshold, it is determined that there is no character corresponding to the digital identifier in the letter identifier data, and a prompt message is output; wherein the prompt message is used to prompt the user to check the text data to be played.

6. The method according to claim 1, characterized in that The text data to be played is English data; Get the text data to be played, including: Get the initial English data; The letters in the first format in the initial English data are converted into letters in the second format to obtain the text data to be played; wherein the first format represents the format of uppercase English letters, and the second format represents the format of lowercase English letters.

7. A method for training a phoneme prediction model, characterized in that: include: Acquire a data set to be trained; wherein the data set to be trained includes letter identification data of the data to be trained and actual phoneme encoding of the data to be trained; According to the letter identification data of the data to be trained and the actual phoneme encoding of the data to be trained, the preset initial model is trained to obtain a trained phoneme prediction model; wherein the phoneme prediction model is used to implement the phoneme-based speech playback method according to any one of claims 1-6.

8. The method according to claim 7, characterized in that Also includes: Acquire initial training data; wherein the initial training data is data in text form; Determine a target length according to the text length of each of the initial training data; wherein the target length represents the maximum text length of the data to be trained; According to the target length and based on preset null characters, the initial training data is supplemented to obtain the data to be trained.

9. The method according to claim 7, characterized in that: According to the letter identification data of the data to be trained and the actual phoneme encoding of the data to be trained, a preset initial model is trained to obtain a trained phoneme prediction model, including: Inputting the letter identification data of the data to be trained into a preset initial model to obtain the predicted phoneme encoding of the data to be trained as output; The preset initial model is trained according to the predicted phoneme coding of the data to be trained and the corresponding actual phoneme coding to obtain a trained phoneme prediction model.

10. The method according to claim 9, characterized in that The phoneme code includes a plurality of characters; the preset initial model is trained according to the predicted phoneme code of the data to be trained and the corresponding actual phoneme code to obtain a trained phoneme prediction model, including: Determine the number of erroneous characters according to the predicted phoneme code of the to-be-trained data and the corresponding actual phoneme code; wherein the number of erroneous characters is at least one of a first number, a second number, and a third number; the first number represents the number of inconsistent characters at the same character position in the predicted phoneme code and the actual phoneme code; the second number represents the number of characters for which the predicted phoneme code is less than the actual phoneme code, and the third number represents the number of characters for which the predicted phoneme code is more than the actual phoneme code; According to the number of erroneous characters and the number of characters in the actual phoneme encoding, an error rate is obtained; wherein the error rate represents the probability that the predicted phoneme encoding is an erroneous phoneme encoding; If the error rate meets the preset training conditions, a trained phoneme prediction model is obtained.

11. A phoneme-based speech playback device, characterized in that: include: A first acquisition unit, used for acquiring text data to be played; A determination unit, for determining the phoneme encoding of the to-be-played text data based on a preset phoneme prediction model; wherein the preset phoneme prediction model includes an embedding layer, a convolution enhanced transformer Conformer layer, and a decoding Decoder layer, and the preset phoneme prediction model is used to output the phoneme encoding of the text data, the phoneme encoding is in the form of a character string, one character in the phoneme encoding represents one phoneme information, and the phoneme information represents pronunciation; The playing unit is used to perform voice playing of the text data to be played according to the phoneme information represented by each character in the phoneme code.

12. A training device for a phoneme prediction model, characterized in that: include: A second acquisition unit is used to acquire a data set to be trained; wherein the data set to be trained includes letter identification data of the data to be trained and actual phoneme encoding of the data to be trained; A training unit, used to train a preset initial model according to the letter identification data of the data to be trained and the actual phoneme encoding of the data to be trained, so as to obtain a trained phoneme prediction model; wherein the phoneme prediction model is used to implement the phoneme-based speech playback method described in any one of claims 1-6.

13. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 10 when executed by a processor.

15. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 10 when being executed by a processor.

Images