Phoneme sequence determination method and device, computer equipment and storage medium

Abstract

The invention discloses a phoneme sequence determination method and device, computer equipment and a storage medium. The method comprises the following steps: inputting a target text into a first model to obtain a candidate phoneme sequence; generating a first coding feature according to words in the target text, and generating a second coding feature according to phonemes in the candidate phoneme sequence; and inputting the first coding feature and the second coding feature into a second model to obtain a target similarity, and determining a target phoneme sequence of the target text from the candidate phoneme sequences according to the target similarity. According to the embodiment of the invention, the generation relation from the text to the phoneme sequence is directly established through the preset first model, the effect of the generated candidate phoneme sequence can be ensured, the phoneme sequence is screened by using the similarity, the accuracy of the generated phoneme sequence is further ensured, and thus the effect of subsequent speech synthesis is improved.

Description

Technical Field

[0001] The present application relates to the field of speech synthesis technology, and in particular to a method, device, computer equipment and storage medium (computer-readable storage medium) for determining a phoneme sequence. Background Art

[0002] TTS (Text To Speech) is a technology that processes input text content in a certain way to obtain the pronunciation of a specific speaker. It has been widely used in voice prompts, e-books and other fields. At present, most TTS technologies are composed of two parts, the front end and the back end. The front end is mainly used to convert the input text into an unambiguous pronunciation representation, that is, phonemes, while the back end mainly synthesizes phonemes into speech for output through an acoustic model or a vocoder. It can be seen that converting text into a reasonable phoneme sequence will affect the stability of the entire text-to-speech synthesis technology.

[0003] At present, most front-end processing uses different functional modules to implement different processing steps of text. For example, the cleaning module is used to clean unpronounced characters in advance, the normalization module is used to reduce non-standard text representations, such as numerical values, to Chinese character sequence representations, and the Chinese character-pinyin conversion module is used to convert the normalized text into a pinyin sequence through a pronunciation dictionary, etc. However, the above method relies on the sequential dependency between different functional modules, and there is noise transmission between the modules, which results in the final generated phoneme sequence being inaccurate, affecting the effect of subsequent text-to-speech synthesis. Summary of the invention

[0004] In response to the above technical problems, embodiments of the present application provide a method, apparatus, computer device and storage medium for determining a phoneme sequence.

[0005] In a first aspect, an embodiment of the present application provides a method for determining a phoneme sequence, comprising:

[0006] Inputting the target text into a preset first model to obtain a candidate phoneme sequence;

[0007] Generating a first encoding feature of the target text according to words in the target text, and generating a second encoding feature of the candidate phoneme sequence according to phonemes in the candidate phoneme sequence;

[0008] Inputting the first encoding feature and the second encoding feature into a preset second model to obtain a target similarity between the target text and the candidate phoneme sequence, wherein the target similarity includes a similarity between each word in the target text and each phoneme in the candidate phoneme sequence;

[0009] A target phoneme sequence of the target text is determined from the candidate phoneme sequences according to the target similarity.

[0010] In a second aspect, an embodiment of the present application provides a device for determining a phoneme sequence, comprising:

[0011] An input module, used for inputting the target text into a preset first model to obtain a candidate phoneme sequence;

[0012] An encoding module, configured to generate a first encoding feature of the target text according to words in the target text, and to generate a second encoding feature of the candidate phoneme sequence according to phonemes in the candidate phoneme sequence;

[0013] A processing module, configured to input the first encoding feature and the second encoding feature into a preset second model to obtain a target similarity between the target text and the candidate phoneme sequence, wherein the target similarity includes a similarity between each word in the target text and each phoneme in the candidate phoneme sequence;

[0014] A screening module is used to determine a target phoneme sequence of the target text from the candidate phoneme sequences according to the target similarity.

[0015] In a third aspect, an embodiment of the present application further provides a computer device, comprising a memory storing a plurality of instructions; a processor loads instructions from the memory to execute the steps of any one of the phoneme sequence determination methods provided in the embodiments of the present application.

[0016] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a plurality of instructions suitable for loading by a processor to execute the steps of any one of the phoneme sequence determination methods provided in the embodiments of the present application.

[0017] In a fifth aspect, an embodiment of the present application further provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the steps of any one of the phoneme sequence determination methods provided in the embodiments of the present application.

[0018] The scheme of the embodiment of the application is adopted, and the target text is processed by presetting the first model to obtain multiple possible candidate phoneme sequences, and then the similarity between the target text and different candidate phoneme sequences is determined by using the encoding features of the candidate phoneme sequences and the text. Subsequently, the target phoneme sequence of the target text can be determined from the candidate phoneme sequences based on the similarity. Since the preset first model provided by the embodiment of the application directly establishes a generative relationship from text to phoneme sequence, the effect of the generated candidate phoneme sequence can be guaranteed, and the phoneme sequence is screened by using the similarity, which further guarantees the accuracy of the generated phoneme sequence, thereby improving the effect of subsequent speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 It is a schematic diagram of an application environment of the method for determining a phoneme sequence provided in an embodiment of the present application;

[0021] Figure 2 It is a schematic diagram of an embodiment of the method for determining a phoneme sequence provided in an embodiment of the present application;

[0022] Figure 3 A schematic flow chart of the steps of processing the coding features by a preset second model to obtain the target similarity in the form of a matrix provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of the steps of training a first model provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of a process flow for determining a first model loss value provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of the training process of the first model structure provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of the steps of training the second model provided in an embodiment of the present application;

[0027] Figure 8 A schematic diagram of a process flow for determining a second model loss value provided in an embodiment of the present application;

[0028] Fig. 9 A schematic diagram of the training process of the second model structure provided in an embodiment of the present application;

[0029] Fig.10 is a schematic diagram of the structure of a phoneme sequence determination device provided in an embodiment of the present application;

[0030] Fig.11 It is a schematic diagram of the internal structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application. At the same time, in the description of the embodiments of the present application, the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0032] In order to better understand the method, device, electronic device and storage medium for determining a phoneme sequence provided in the embodiments of the present application, the application environment applicable to the embodiments of the present application is described below.

[0033] See also Figure 1 , Figure 1 A schematic diagram of an application environment of the phoneme sequence determination method provided in an embodiment of the present application is shown. As an implementation mode, the phoneme sequence determination method and the model training method provided in the embodiment of the present application can be applied to the same electronic device. The electronic device can be as follows: Figure 1 The server 110 shown in the figure can be connected to the terminal device 120 through a network. The network is used to provide a medium for a communication link between the server 110 and the terminal device 120. The network can include various connection types, such as a wired communication link, a wireless communication link, etc., which are not limited in the embodiments of the present application. Optionally, in other embodiments, the electronic device can also be a smart phone, a laptop computer, etc.

[0034] It should be understood that Figure 1 The server 110, network and terminal device 120 are merely illustrative. Depending on the implementation requirements, there may be any number of servers, networks and terminal devices. For example, the server 110 may be a physical server or a server cluster composed of multiple servers, and the terminal device 120 may be a mobile phone, a tablet, a desktop computer, a laptop computer, and the like. It is understood that the embodiments of the present application may also allow multiple terminal devices 120 to access the server 110 at the same time.

[0035] In some embodiments, the terminal device 120 can extract the text information input by the user to obtain the user's text data. Further, the terminal device 120 sends the user's text data to the server 110 through the network. After the server 110 receives the text data, the text data can be processed by the phoneme sequence determination method described in the embodiment of the present application.

[0036] As another embodiment, the phoneme sequence determination method, the preset first model training method and the preset second model training method provided in the embodiment of the present application can also be applied to different electronic devices. For example, the phoneme sequence determination method is applied to electronic device A, the preset first model training method is applied to electronic device B, the preset second model training method is applied to electronic device C, and so on. In addition, any two of the phoneme sequence determination method, the preset first model training method and the preset second model training method provided in the embodiment of the present application can be applied to the same electronic device. For example, the phoneme sequence determination method and the preset first model training method are applied to electronic device A, and the preset second model training method is applied to electronic device B, or the phoneme sequence determination method and the preset second model training method are applied to electronic device A, and the preset first model training method is applied to electronic device B, or the preset first model training method and the preset second model training method are applied to electronic device A, and the phoneme sequence determination method is applied to electronic device B, and so on. The embodiment of the present application does not limit the electronic devices to which the above three methods are applied.

[0037] The following is a detailed description in conjunction with the accompanying drawings. In this embodiment, the execution subject is a terminal device as an example. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments. Although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings.

[0038] Please refer to Figure 2 The specific process of the phoneme sequence determination method can be as follows: Steps S210 to S240, wherein:

[0039] S210: Input the target text into a preset first model for processing to obtain a candidate phoneme sequence.

[0040] In the embodiments of the present application, the target text refers to the content in text form that needs to perform TTS, that is, text-to-speech conversion. Specifically, in different scenarios, the target text can come from different sources. For example, in e-book reading, the target text can be the relevant text content in the book read by the user, and in the field of intelligent customer service, the target text can refer to the answer text made by the human-computer dialogue system to the questions asked by the user. The embodiments of the present application do not limit the target text here. Any text that exists in text form and has a need for speech conversion can be regarded as the target text proposed in the embodiments of the present application.

[0041] In the embodiments of the present application, the preset first model here can adopt a model structure with generation capabilities, such as the more common codec model structure. Of course, as a feasible embodiment of the present application, the preset first model can also adopt a pre-trained generative model structure. Specifically, for the training process of the preset first model, please refer to the subsequent further related instructions.

[0042] Furthermore, unlike the conventional solution of converting text to phoneme sequence through multiple serially connected different functional modules, the embodiment of the present application provides an end-to-end text-to-phoneme sequence conversion, that is, the ability to generate text to phoneme sequence is learned from samples in advance through machine learning, so as to train a preset first model. In this way, when the target text is input into the preset first model for processing, the ability to generate text to phoneme sequence learned by the preset first model can be directly used to directly generate candidate phoneme sequences. When repeated multiple times, multiple groups of possible candidate phoneme sequences can be generated. Of course, as another feasible embodiment of the present application, in the process of inputting the target text into the preset first model for processing to obtain multiple candidate phoneme sequences, the target text can also be first input into the preset first model for processing to obtain a first phoneme sequence, and then the first phoneme sequence is associated with the target text as text prompt information and input into the preset first model for processing again. At this time, the preset first model can comprehensively consider the text features of the target text and the phoneme features of the first phoneme sequence to repeatedly output the second phoneme sequence, and the second phoneme sequence can also be associated with the target text as text prompt information and input into the preset first model for processing again. Repeating the above input process can also obtain multiple candidate phoneme sequences. That is to say, the step of inputting the target text into the preset first model to obtain multiple candidate phoneme sequences specifically includes:

[0043] Inputting the target text into a preset first model to obtain a first candidate phoneme sequence;

[0044] Associatively inputting the target text and the first candidate phoneme sequence into the preset first model to obtain a second candidate phoneme sequence;

[0045] The first candidate phoneme sequence and the second candidate phoneme sequence are determined as the candidate phoneme sequences.

[0046] S220, generating a first encoding feature of the target text according to the words in the target text, and generating a second encoding feature of the candidate phoneme sequence according to the phonemes in the candidate phoneme sequence.

[0047] In the embodiment of the present application, after the target text is processed by the aforementioned preset first model with the ability to generate text to phoneme sequences to obtain multiple possible candidate phoneme sequences, in order to further ensure the accuracy of the phoneme sequence finally determined, as another feasible embodiment of the present application, the target text and the candidate phoneme sequence are further encoded to obtain corresponding encoding features. That is, the first encoding feature of the target text is generated according to the words in the target text, and for each candidate phoneme sequence, the second encoding feature of the candidate phoneme sequence is generated according to the phonemes in the phoneme sequence, that is, the second encoding feature also includes multiple, which correspond one-to-one to the multiple candidate phoneme sequences.

[0048] Specifically, the coding features here can usually be obtained by querying a code table, in which the code values ​​corresponding to each word and each phoneme are recorded respectively. For example, as a common feasible implementation scheme, the code value range of the code table occupied by the Chinese character sentence word table is 1 to 20000, while the code value range of the code table occupied by the phoneme sequence is 1 to 5000. In other words, the first coding feature of the target text can be obtained by combining the position of each word in the text and querying the code value of each word from the code table. Similarly, the second coding feature of the candidate phoneme sequence can be obtained by combining the position of each phoneme in the phoneme sequence and querying the code value corresponding to each phoneme from the code table. To facilitate understanding, taking the target text "I love apples" as an example, assuming that the code value corresponding to "I" in the code table is "100", the code value corresponding to "love" is "101", the code value corresponding to "eat" is "102", and the code value corresponding to "apple" is "103", then the first coding feature finally generated is [100, 101, 102, 103]. Of course, in order to describe a complete text, you can also add a start code value indicating the beginning of the text and an end code value indicating the end of the text before and after the first coding feature. For example, the start code value indicating the beginning of the text can be 30001, and the end code value indicating the end of the text can be 30002. At this time, the first coding feature is [30001, 100, 101, 102, 103, 30002]. By the same logic, taking the phoneme sequence "wo ai chi ping guo" as an example, assuming that in the code table the code value corresponding to "wo" is "1", the code value corresponding to "ai" is "2", the code value corresponding to "eat" is "3", and the code value corresponding to "apple" is "4", then the second coding feature of the phoneme sequence is [1, 2, 3, 4]. Of course, by the same logic, a start code value indicating the beginning of the phoneme sequence and an end code value indicating the end of the phoneme sequence can usually be added before and after the second coding feature. For example, the start code value indicating the beginning of the phoneme sequence can be 30003, and the end code value indicating the end of the phoneme sequence can be 30004. At this time, the second coding feature of the phoneme sequence is [30003, 1, 2, 3, 14, 30004].

[0049] It can be seen that the above determination of the coding features depends on the predetermined code table. Considering that the data of Chinese text and phonemes are large, in order to better realize the encoding of text and phoneme sequences, the present application provides an implementation scheme of adding code tables based on the frequency of occurrence. For example, by counting the frequency of occurrence of two adjacent texts in the sample data set, each time, the adjacent combination with the highest frequency can be considered to be added to the text vocabulary table, and each word in the text vocabulary table is assigned a corresponding code value, thereby obtaining the corresponding code table. By repeating the above steps continuously, more frequent words can be added to the code table until the vocabulary that the code table can contain reaches the upper limit. For example, the maximum vocabulary contained in the code table can be set to about 20,000. Of course, in order to prevent the vocabulary boundary from exceeding the normal Chinese word boundary, it is usually required that the maximum character length added to the code table cannot exceed 4, that is, the characters in the vocabulary do not exceed 4.

[0050] Similarly, for phoneme sequences, each phoneme sequence can also be divided into complete phonemes, and then all samples are used as statistical ranges to count the frequency of occurrence of different adjacent phonemes, and the adjacent phonemes with the highest frequency are added to the phoneme table, and the different phonemes in the phoneme table are assigned corresponding code values ​​to obtain the corresponding code table. By repeating the above steps, more frequent phoneme groups can be added to the code table until the upper limit of the phonemes that the code table can contain is reached. For example, the maximum number of phonemes contained in the code table can be set to about 5000.

[0051] Of course, the order of words and phoneme combinations in the code table can also be rearranged according to the frequency of occurrence, for example, the higher the frequency, the higher the appearance.

[0052] S230: Input the first coding feature and the second coding feature into a preset second model to obtain a target similarity between the target text and the candidate phoneme sequence.

[0053] In the embodiment of the present application, inputting the first coding feature and the second coding feature into the preset second model generally refers to splicing each coding feature with the second coding feature corresponding to each candidate phoneme sequence, and then inputting the spliced ​​coding feature into the preset second model to obtain the target similarity between the target text and each candidate phoneme sequence. For example, taking the first coding feature as A, and the plurality of second coding features as "B1", "B2" and "B3" as an example, at this time, the first coding feature A and the second coding feature "B1" are spliced ​​and input into the preset second model, so as to obtain the target similarity between the target text and the candidate phoneme sequence corresponding to the second coding feature B1. Similarly, the first coding feature A and the second coding feature "B2" are spliced ​​and input into the preset second model, so as to obtain the target similarity between the target text and the candidate phoneme sequence corresponding to the second coding feature B2. The first coding feature A and the second coding feature "B3" are spliced ​​and input into the preset second model, so as to obtain the target similarity between the target text and the candidate phoneme sequence corresponding to the second coding feature B3, that is, to determine the target similarity between the target text and each candidate phoneme sequence.

[0054] Furthermore, the preset second model here is similar to the preset first model mentioned above, and generally refers to a neural network model with corresponding capabilities that is pre-trained based on a machine learning method. Specifically, for ease of understanding, the training process of the second model will be explained later.

[0055] Specifically, since the first coding feature and the second coding feature are generated based on the words in the target text and the phonemes in the candidate phoneme sequence, respectively, the position information of the words in the target text and the phonemes in the candidate phoneme sequence is implied. Therefore, in the scheme provided in the embodiment of the present application, the target similarity between the target text and each candidate phoneme sequence is usually composed of multiple similarities, that is, the target similarity includes the similarity between each word in the target text and each phoneme in the candidate phoneme sequence. That is, different from the conventional method of directly generating a unique numerical value of the target similarity between the target text and the candidate phoneme sequence through a preset model, the target similarity obtained by processing the first coding feature and the second coding feature through the preset second model in the present application includes multiple numerical values, which respectively correspond to the similarity between each word in the target text and each phoneme in the candidate phoneme sequence. Specifically, as a further feasible implementation scheme of the present application, the target similarity can be in the form of a matrix, that is, different matrix elements in the matrix are the similarities between different words in the target text and different phonemes in the candidate phoneme sequence. At this time, processing the first encoding feature and the second encoding feature by presetting the second model usually involves processing the encoding feature into a high-dimensional matrix feature and then operating the matrix feature to obtain the target similarity. Specifically, for easy understanding of the above scheme, please refer to Figure 3 , Figure 3 A schematic flow chart of the steps of processing the coding features by a preset second model to obtain the target similarity in the form of a matrix provided in an embodiment of the present application, specifically, comprising steps S310 to S340:

[0056] S310, inputting the first encoding feature into the first convolutional layer in the preset second model to obtain a text matrix of the target text.

[0057] In an embodiment of the present application, in order to process low-dimensional coding features into high-dimensional matrix features, the preset second model usually includes multiple convolution layers. Specifically, the convolution layer can be in the form of a high-dimensional convolution kernel. Of course, other structures can also be used. For example, the more common transformer (a neural network model structure) can be used, that is, the low-dimensional first coding features are input into the transformer layer in the preset second model for processing, and a text matrix describing the high-dimensional features of the target text can be obtained.

[0058] S320, input the second encoding feature into a second convolutional layer in a preset second model to obtain a phoneme matrix of the candidate phoneme sequence.

[0059] Similar to the way of processing the first coding feature to obtain a text matrix, in an embodiment of the present application, the preset second model may also include a second convolutional layer to process the low-dimensional second coding feature into a high-dimensional phoneme matrix, wherein the second convolutional layer here has a similar structure to the first convolutional layer, for example, a transformer network structure may also be used, but the parameters therein are usually different from those of the first convolutional layer, for example, parameters obtained based on training that can better extract high-dimensional features of phoneme sequences.

[0060] S330, performing matrix operation on the text matrix and the phoneme matrix to obtain a similarity matrix.

[0061] In the embodiment of the present application, after the high-dimensional matrix features of the target text and the high-dimensional matrix features of the phoneme sequence, that is, the text matrix and the phoneme matrix, are obtained respectively, since the target text and the phoneme sequence usually have equal lengths, the text matrix and the phoneme matrix usually also have the same size relationship. Therefore, matrix operations can be performed through the text matrix and the phoneme matrix. For example, the text matrix and the phoneme matrix are commonly multiplied to obtain the final similarity matrix, wherein the dimension of the similarity matrix is ​​related to the length of the target text and the phoneme sequence. At this time, each matrix element in the similarity matrix can be understood as the similarity between the corresponding position word in the text and the corresponding position phoneme in the phoneme sequence.

[0062] Of course, in the process of matrix multiplication, it is usually necessary to transpose one of the matrices first, for example, multiplying the transposed matrices of the text matrix and the phoneme matrix to obtain a similarity matrix, or multiplying the transposed matrix of the text matrix and the phoneme matrix to obtain a similarity matrix.

[0063] S340: Determine a target similarity between the target text and the candidate phoneme sequence according to matrix elements in the similarity matrix.

[0064] In the embodiment of the present application, combined with the above-mentioned related description, it can be known that each matrix element in the similarity matrix can be understood as the similarity between the corresponding position word in the text and the corresponding position phoneme in the phoneme sequence. Therefore, by extracting the matrix elements in the similarity matrix, the target similarity between the target text and the candidate phoneme sequence can be obtained.

[0065] Through the above-mentioned scheme, the similarity matrix obtained by processing the encoding features using the preset second model is used to describe the target similarity between the target text and the candidate phoneme sequence. Since a higher-dimensional similarity matrix is ​​used as the processing result, it contains more information between the target text and the candidate phoneme sequence, for example, the similarity between each word in the target text and each phoneme in the candidate phoneme sequence, so that when the target similarity is subsequently used to screen from the candidate phoneme sequence, a more accurate target phoneme sequence that is more consistent with the pronunciation of the target text can be determined.

[0066] S240: Determine a target phoneme sequence of the target text from the candidate phoneme sequences according to the target similarity.

[0067] In the embodiment of the present application, combined with the above-mentioned related description, it can be seen that since the target similarity is the result obtained by processing the first coding feature and the second coding feature through the preset second model obtained through training, the target similarity can describe the correlation between the target text and different candidate phoneme sequences to a certain extent. Therefore, the target similarities can be compared to determine the most likely target phoneme sequence of the target text from the candidate phoneme sequences based on the size relationship of the target similarities.

[0068] Of course, considering that the target similarity includes multiple values, such as the similarity between each word in the target text and each phoneme in the candidate phoneme sequence, therefore, in the process of comparing the target similarity to screen the target phoneme sequence from the candidate phoneme sequence based on the size relationship of the target similarity, the target similarity can be appropriately processed. For example, the diagonal elements located on the diagonal in the similarity matrix can be considered. Since these matrix elements are the similarities between the words and phonemes corresponding to the same position in the target text and the candidate phoneme sequence, the above-mentioned diagonal elements located on the diagonal can be weighted and summed, wherein the larger the calculation result, the higher the similarity correlation between the target text and the candidate phoneme sequence, and therefore, the candidate phoneme sequence with a higher result can be used as the final target phoneme sequence. In addition, for the diagonal elements of the similarity matrix that are on the diagonal, since these matrix elements are the similarities between words and phonemes corresponding to different positions in the target text and the candidate phoneme sequence, in the above process, negative weights can also be set for the non-diagonal elements outside the diagonal to perform weighted summation, thereby realizing the screening of candidate phoneme sequences.

[0069] The method for determining a phoneme sequence provided in the embodiment of the present application processes the target text by presetting a first model to obtain multiple possible candidate phoneme sequences, and then determines the similarity between the target text and different candidate phoneme sequences using the encoding features of the candidate phoneme sequences and the text, and subsequently determines the target phoneme sequence of the target text from the candidate phoneme sequences based on the similarity. Since the preset first model provided in the embodiment of the present application directly establishes a generative relationship from text to phoneme sequence, the effect of the generated candidate phoneme sequence can be guaranteed, and the phoneme sequence is screened using similarity, which further guarantees the accuracy of the generated phoneme sequence, thereby improving the effect of subsequent speech synthesis.

[0070] In order to better implement the phoneme sequence determination method provided in the embodiment of the present application, it is obviously dependent on the training of the preset first model and the preset second model mentioned above. To facilitate understanding of the above-mentioned training process of different models, the following will provide a specific training method for the model in combination with a specific embodiment.

[0071] For details, please refer to Figure 4 , Figure 4 A schematic flow chart of a step of training a preset first model provided in an embodiment of the present application, specifically, comprising steps S410 to S430:

[0072] S410, inputting the sample sequence into the initial first model to obtain a predicted sequence of the sample sequence.

[0073] In the embodiment of the present application, the training of the first model usually needs to rely on sample data, that is, the sample sequence is composed of the first sample text and the first sample phoneme sequence, wherein the first sample text and the first sample phoneme sequence usually refer to sample pairs with corresponding pronunciation relationships extracted in advance through a web crawler or from a known training sample database, that is, the first sample phoneme sequence can be understood as the pronunciation of the first sample text. For example, if the first sample text is "I love Beijing", the first pronunciation sequence is "wo3ai4 bei3 jing1".

[0074] In addition, since the preset first model has the ability to convert text into a phoneme sequence, the initial first model also needs to include corresponding functional modules. For example, more commonly, the initial first model can use the model structure of a pre-trained generative model, that is, after inputting a piece of text, it has the function of automatically continuing to generate a phoneme sequence.

[0075] After the sample sequence is input into the initial first model for processing, the model processing based on the first model is performed, and the latent vector representation of the final output is extracted, which is the predicted sequence of the final output.

[0076] S420, calculating a first model loss value of the initial first model according to the prediction sequence and the sample sequence.

[0077] In an embodiment of the present application, by comparing the predicted sequence obtained above with the sample sequence used for training, the effect of the model in generating text into corresponding phonemes can be determined, wherein the model loss value corresponding to the first model can usually be used to describe it.

[0078] Specifically, since the prediction sequence is usually composed of a text representation and a phoneme representation, in the process of determining the model loss value corresponding to the first model, the text representation in the prediction sequence can be compared with the first sample text, and the phoneme representation in the prediction sequence can be compared with the first sample phoneme sequence. For details, please refer to Figure 5 , Figure 5 Another step flow diagram of determining the model loss value of the first model provided in an embodiment of the present application, specifically, includes steps S510 to S530:

[0079] S510, dividing the prediction sequence to obtain a text prediction sequence and a phoneme prediction sequence.

[0080] In an embodiment of the present application, in the process of splicing the first sample text and the first sample phoneme sequence mentioned above to form a sample sequence, the first sample text and the first sample phoneme sequence can usually be distinguished by a position identifier. For example, the first sample text is located before the position identifier, and the first sample phoneme sequence is located after the position identifier. At this time, the prediction sequence is divided based on the position identifier, and a text prediction sequence and a phoneme prediction sequence can also be obtained. For example, the sequence in the prediction sequence located before the position identifier is used as a text prediction sequence, and the sequence in the prediction sequence located after the position identifier is used as a phoneme prediction sequence.

[0081] S520, determining a first text loss value between the text prediction sequence and the first sample text, and determining a first phoneme loss value between the phoneme prediction sequence and the first sample phoneme sequence.

[0082] In an embodiment of the present application, by comparing the text prediction sequence with the first sample text, the difference between the text prediction sequence and the first sample text can be determined, and the difference can be understood as the first text loss value between the text prediction sequence and the first sample text. Specifically, in order to quantify the difference, in an embodiment of the present application, a loss function can be used for calculation. For example, as a common feasible implementation scheme, a mean square error loss function or a cross entropy loss function can be used to calculate the first text loss value between the text prediction sequence and the first sample text. Of course, it is also feasible to use other loss functions, which will not be repeated in the embodiments of the present application.

[0083] Similarly, for the first phoneme loss value between the phoneme prediction sequence and the first sample phoneme sequence, it can also be similar to the first text loss value, and can be obtained by comparing the difference between the phoneme prediction sequence and the first sample phoneme sequence. For example, the mean square error loss function or the cross entropy loss function can also be used to calculate the phoneme prediction sequence and the first sample phoneme sequence to obtain the first phoneme loss value between the phoneme prediction sequence and the first sample phoneme sequence.

[0084] S530: Determine a first model loss value of the initial first model according to the first text loss value and the first phoneme loss value.

[0085] In the embodiment of the present application, it can be understood on the basis of the foregoing that the first text loss value and the first phoneme loss value can describe the ability of the initial first model to process the sample text into a suitable phoneme sequence. Therefore, the first model loss value of the initial first model can be obtained by weighting the first text loss value and the first phoneme loss value, wherein the higher the first model loss value, the worse the ability of the initial first model to process the sample text into a suitable phoneme sequence.

[0086] By using the model to process the sample sequence as mentioned above to obtain a predicted sequence, and using the phoneme prediction sequence and the phoneme prediction sequence in the predicted sequence to compare with the first sample text and the first sample phoneme sequence respectively, the ability of the model to process the text into a suitable phoneme sequence can be analyzed as a whole from the two dimensions of text and phonemes. The obtained model loss value is more suitable for subsequent training of the model.

[0087] S430: If the loss value of the first model meets the preset training end condition, the current first model is determined as the preset first model.

[0088] In an embodiment of the present application, after obtaining the first model loss value that describes the ability of the initial first model to process the sample text into a suitable phoneme sequence, the first model loss value can be used to determine whether it is necessary to iteratively train the initial first model. For example, when the first model loss value does not meet the preset training end condition, under normal circumstances, the initial first model can be updated based on the idea of ​​back propagation using the first model loss value to obtain an updated first model, and the sample sequence is input into the updated first model to obtain an updated first model loss value. After the updated first model loss value meets the preset training end condition, for example, it is less than a preset threshold for several consecutive times, the currently obtained first model can be determined as the preset first model, and the preset first model can accurately process the sample sequence into a relatively accurate predicted phoneme sequence.

[0089] In addition, considering that there are large differences in the content and pronunciation sequences of text forms in different business fields, as another feasible embodiment of the present application, the trained first model can also be associated with the business field corresponding to the first sample text, that is, for text data in different business fields, different sample texts and phoneme sequences can be used to generate the corresponding first model. At this time, in the process of inputting the target text into the preset first model for processing to obtain multiple candidate phoneme sequences, it is possible to first query the preset database to obtain the target model associated with the business field of the target text, and then use the target model to better realize the processing of the target text to obtain multiple candidate phoneme sequences, thereby improving the effect of text processing and generating phoneme sequences.

[0090] Specifically, in order to facilitate understanding of the implementation process of the above training second model, the following will be combined with Figure 6 The first model structure diagram provided gives the complete implementation steps for training the first model. The details are as follows:

[0091] (1) Look up the code table to obtain the code value (token) code and position code of the sample text, as well as the code value (token) code and position code of the phoneme sequence;

[0092] (2) Each code is concatenated along the length of the sequence and input into a pre-trained generative model, such as GPT2, and the latent vector representation of the last layer is extracted. Then, the latent vector representation is split in length according to the position where the text and phoneme are concatenated, and the text representation vector and the phoneme representation vector are obtained respectively;

[0093] (3) After the text representation vector and the phoneme representation vector are processed by the text prediction linear classification layer and the phoneme prediction linear classification layer respectively, the predicted text token and the predicted phoneme token can be obtained. After calculating the cross entropy loss with the aforementioned text token encoding and phoneme token encoding, the model is trained to obtain the final model.

[0094] Of course, in addition to the aforementioned embodiment of training the preset first model, the present application further provides a specific embodiment of training the preset second model. For details, please refer to Figure 7 , Figure 7 A schematic flow chart of a step of training a preset second model provided in an embodiment of the present application, specifically, comprising steps S710 to S730:

[0095] S710, inputting the second sample text and the second sample phoneme sequence into the initial second model to obtain a prediction similarity matrix.

[0096] In the embodiment of the present application, the second sample text and the second sample phoneme sequence are similar to the first sample text and the first sample phoneme sequence mentioned above, and may also refer to sample pairs with corresponding pronunciation relationships extracted in advance through a web crawler or from a known training sample database, and the embodiments of the present application will not be repeated here.

[0097] Specifically, since the preset second model has the function of encoding text and phoneme sequences and extracting features to achieve matching of text and phoneme sequences, the initial second model also includes corresponding functional modules. For example, as a common feasible implementation scheme, the initial second model includes an encoding layer (embedding), a feature extraction convolution layer and a matrix multiplication layer, that is, the second sample text and the second sample phoneme sequence are subjected to the encoding layer to obtain encoding features, and then are respectively subjected to the feature extraction convolution layer to obtain a high-dimensional text feature matrix and a phoneme feature matrix, and then the obtained text feature matrix and the phoneme feature matrix are transposed and multiplied to obtain an N×N-dimensional similarity matrix. Since the parameters in the initial second model are all initialized, the obtained predicted similarity matrix is ​​only the initialized prediction result, which describes the predicted similarity between different words and different phonemes in the second sample text and the second sample phoneme sequence.

[0098] S720: Determine the second model loss value corresponding to the initial second model according to the matrix elements in the prediction similarity matrix.

[0099] Since the matrix elements on the prediction similarity matrix describe the similarities between different words and phonemes in the text and the phoneme sequence respectively, the second model loss value corresponding to the initial second model can be determined based on the matrix elements, that is, the ability of the initial second model to calculate the similarity between the text and the phoneme sequence can be obtained. Specifically, consider the element M in the i-th row and j-th column of the prediction similarity matrix ij reflects the similarity between the i-th word in the text and the j-th phoneme in the phoneme sequence, and the input text and phoneme sequence are paired, that is, the words and phonemes at the corresponding positions of the input text and phoneme sequence should match, so the resulting prediction similarity matrix should be in a diagonal form, that is, the diagonal element value of the prediction similarity matrix is ​​maximized. Therefore, different loss values ​​of the initial second model can be calculated based on the elements at different positions on the prediction similarity matrix, for example, see Figure 8 , Figure 8 A schematic flow chart of a step of determining a second model loss value of an initial second model provided in an embodiment of the present application, wherein the embodiment of the present application is described by taking the diagonal elements and the non-diagonal elements in the prediction similarity matrix as an example, specifically, including steps S810 to S830:

[0100] S810: Determine a first loss value according to the diagonal element and a preset first threshold.

[0101] In the embodiment of the present application, combined with the above-mentioned related description, it can be known that the diagonal elements on the predicted similarity matrix should tend to be maximized. Therefore, the first loss value can be determined by calculating the difference between the diagonal elements and the preset first threshold, such as the difference from 1 or calculating the cross entropy. Specifically, the larger the difference, the higher the first loss value, which reflects that the effect of the model in matching text and phonemes is worse.

[0102] S820: Determine a second loss value according to the off-diagonal element and a preset second threshold.

[0103] In the embodiment of the present application, combined with the above-mentioned related description, it can be known that since the non-diagonal elements on the predicted similarity matrix describe the similarity between words and phonemes at different positions in the text and phoneme sequence, they should be minimized. Therefore, the second loss value can also be determined by calculating the difference between the non-diagonal elements and the preset second threshold, such as the difference from 0 or by calculating the cross entropy. Specifically, the larger the difference, the higher the second loss value, reflecting that the model's matching effect on text and phonemes is worse.

[0104] S830: Determine a first model loss value corresponding to the initial second model according to the first loss value and the second loss value.

[0105] In an embodiment of the present application, after different loss values ​​are determined based on the diagonal elements and off-diagonal elements on the predicted similarity matrix respectively, the model loss value corresponding to the initial second model can be determined based on the calculation of the loss values, such as weighted summation. Specifically, the weights of the first loss value and the second loss value can be set based on actual needs. For example, the weight of the first loss value can usually be higher than the weight of the second loss value. Of course, the weight of the first loss value and the weight of the second loss value can also be dynamically adjusted based on the iterative rounds in the training process, so that the model focuses on learning different comparison capabilities at different training stages.

[0106] By comprehensively calculating the model loss value by using the elements at different positions on the prediction similarity matrix, in addition to enabling the model to learn the ability to compare and match words and phonemes at the same position, the model can also learn the ability to compare and match words and phonemes at different positions in the text and phoneme sequence, thereby further improving the training effect of the model.

[0107] S730: If the loss value of the second model meets the preset training end condition, the current second model is determined as the preset second model.

[0108] In an embodiment of the present application, after obtaining the second model loss value that describes the ability of the initial second model to calculate the similarity between the text and the phoneme sequence, the second model loss value can be used to determine whether it is necessary to iteratively train the initial second model. For example, when the second model loss value does not meet the preset training end condition, under normal circumstances, the initial second model can be updated based on the idea of ​​back propagation using the second model loss value to obtain an updated second model, and the sample sequence is input into the updated second model to obtain an updated second model loss value. After the updated second model loss value meets the preset training end condition, for example, it is less than a preset threshold for several consecutive times, the currently obtained second model can be determined as the preset second model, and the preset second model can accurately predict the correlation similarity between the text and the phoneme sequence.

[0109] Specifically, in order to facilitate understanding of the implementation process of the above training second model, the following will be combined with Fig. 9 The provided schematic diagram of the second model structure gives the complete implementation steps for training the second model. The details are as follows:

[0110] (1) querying the code table to obtain the code value (token) encoding of each word in the first sample text in the original text form, merging it with the position encoding of each word in the text and inputting it into the text transformer block in the first model for processing. The structure of the transformer block is not described in detail in this application. Then, after the text linear layer processing, a matrix representation A×N of the text can be obtained, where A is the length of the text and N is the length of the feature vector of each word in the text after processing;

[0111] (2) Look up the code table to obtain the code value (token) encoding of each phoneme in the first phoneme sequence, merge it with the position encoding of each phoneme, and input it into the phoneme transformer block in the first model for processing. Then, after the phoneme linear layer processing, the matrix representation A×N of the phoneme can be obtained, where A is the phoneme length and N is the length of the feature vector of each phoneme in the text after processing.

[0112] (3) The matrix representation of the text and the matrix representation of the phoneme are transposed and multiplied to obtain an A×A similarity matrix, where the element value of each position in the similarity matrix describes the similarity between the corresponding position in the text and the sequence. The cross entropy of the similarity matrix and the preset matrix label is calculated, where the matrix label is usually a diagonal matrix, for example, a positive definite matrix of order A. The model is iteratively optimized by the cross entropy until the final second model is obtained.

[0113] In order to clarify the complete implementation process of the phoneme sequence determination method provided in this application, the following will provide the aforementioned Figures 1 to 9 The content provided provides a complete implementation step of determining a phoneme sequence, specifically, including the following steps:

[0114] (1) Collect a large number of labeled samples, where the samples are bigrams (text, corresponding phoneme sequence);

[0115] (2) UPE (Unit pair Enoding) is used to calculate the vocabulary, where UPE is an improvement on the traditional BPE (Byte Pair Encoding) algorithm. A BPE uses an 8-bit character as the basic unit to construct a high-frequency adjacency table. In this application, the complete Chinese pinyin is used as the basic unit, that is, the vocabulary consisting of multiple characters that appear frequently is added to the vocabulary by statistics. In order to prevent the boundaries of new vocabulary from exceeding the boundaries of normal Chinese words, the vocabulary with the highest frequency of occurrence and the maximum character length of no more than 4 can be selected for addition. Similarly, similar operations can be applied to phonemes to obtain a phoneme vocabulary;

[0116] (3) Encode the two vocabularies separately, where each unit in the vocabulary corresponds to a unique code value. In addition, a sequence start code value (token) and a sequence end code value (token) are also set.

[0117] For example:

[0118] The code table token range occupied by text vocabulary is 1 to 20000;

[0119] The code table token range occupied by phonemes is from 1 to 5000;

[0120] The text sequence start token value is 30001, and the text sequence end token value is 30002;

[0121] The phoneme sequence start token value is 30003, and the phoneme sequence end token value is 30004;

[0122] (4) Using samples for training, where the model training uses the token code values ​​of text and phonemes as input. However, in order to enhance the model's ability to distinguish the positions between tokens, a position code can be added to each token code value. The final representation of the text part is the encoding of the text code value plus the text position code, while the representation of the phoneme sequence is the encoding of the phoneme code value plus the phoneme position code.

[0123] (5) After the encoding is completed, the first model and the second model are trained respectively. The process of model training will not be described in detail.

[0124] (6) After completing the model training, the model is stored and recorded, and then the target text to be processed is converted into coding features of code values ​​and positions, and first input into the first model for repeated processing to obtain multiple subsequent candidate phoneme sequences;

[0125] (7) inputting the target text and each candidate phoneme sequence into the trained second model for processing, and calculating the similarity between the target text and each candidate phoneme sequence using the input similarity matrix;

[0126] (8) Determine the sequence with the highest similarity to the target text from the candidate phoneme sequences as the final target phoneme sequence, and combine it with the subsequent downstream task system, such as a vocoder, to synthesize the speech content of the text.

[0127] The phoneme sequence determination method provided in the embodiment of the present application processes the target text by presetting a first model to obtain multiple possible candidate phoneme sequences, and then determines the similarity between the target text and different candidate phoneme sequences using the encoding features of the candidate phoneme sequences and the text, and then determines the target phoneme sequence of the target text from the candidate phoneme sequences based on the similarity. Since the preset first model provided in the embodiment of the present application directly establishes a generative relationship from text to phoneme sequence, the effect of the generated candidate phoneme sequence can be guaranteed, and the phoneme sequence is screened by using the similarity, which further guarantees the accuracy of the generated phoneme sequence, thereby improving the effect of subsequent speech synthesis.

[0128] It should be understood that, although each step in the flowcharts involved in the above-mentioned embodiments is displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0129] Based on the same inventive concept, the embodiment of the present application also provides a phoneme sequence determination device for implementing the phoneme sequence determination method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiment of the phoneme sequence determination device provided below can refer to the limitations of the phoneme sequence determination device method above, and the specific limitations will not be repeated here.

[0130] This embodiment also provides a device for determining a phoneme sequence, which can be integrated into a terminal device. Fig.10 As shown, the phoneme sequence determination device may include:

[0131] An input module 1010, configured to input a target text into a preset first model to obtain a candidate phoneme sequence;

[0132] An encoding module 1020, configured to generate a first encoding feature of the target text according to words in the target text, and to generate a second encoding feature of the candidate phoneme sequence according to phonemes in the candidate phoneme sequence;

[0133] A processing module 1030 is used to input the first encoding feature and the second encoding feature into a preset second model to obtain a target similarity between the target text and the candidate phoneme sequence, wherein the target similarity includes a similarity between each word in the target text and each phoneme in the candidate phoneme sequence;

[0134] The screening module 1040 is used to determine the target phoneme sequence of the target text from the candidate phoneme sequences according to the target similarity.

[0135] In some embodiments, the processing module is also used to input the first encoding feature into the first convolution layer in the preset second model to obtain the text matrix of the target text; input the second encoding feature into the second convolution layer in the preset second model to obtain the phoneme matrix of the candidate phoneme sequence; perform matrix operations on the text matrix and the phoneme matrix to obtain a similarity matrix; determine the target similarity between the target text and the candidate phoneme sequence based on the matrix elements in the similarity matrix.

[0136] In some embodiments, the input module is also used to input the target text into a preset first model to obtain a first candidate phoneme sequence; associate the target text and the first candidate phoneme sequence and input them into the preset first model to obtain a second candidate phoneme sequence; and determine the first candidate phoneme sequence and the second candidate phoneme sequence as the candidate phoneme sequence.

[0137] In some embodiments, the phoneme sequence determination device also includes a training module, which is also used to input the sample sequence into the initial first model to obtain a predicted sequence of the sample sequence; wherein the sample sequence is composed of a first sample text and a first sample phoneme sequence; the first model loss value of the initial first model is calculated based on the predicted sequence and the sample sequence; if the first model loss value meets the preset training end condition, the current first model is determined as the preset first model.

[0138] In some embodiments, the training module is also used to divide the prediction sequence to obtain a text prediction sequence and a phoneme prediction sequence; determine a first text loss value between the text prediction sequence and the first sample text, and determine a first phoneme loss value between the phoneme prediction sequence and the first sample phoneme sequence; and determine a first model loss value of the initial first model based on the first text loss value and the first phoneme loss value.

[0139] In some embodiments, the training module is also used to update the initial first model according to the first model loss value to obtain an updated first model if the first model loss value does not meet the preset training end condition; and input the sample sequence into the updated first model to obtain an updated prediction sequence.

[0140] In some embodiments, the training module is also used to input the second sample text and the second sample phoneme sequence into the initial second model to obtain a predicted similarity matrix; determine the second model loss value corresponding to the initial second model based on the matrix elements in the predicted similarity matrix; if the second model loss value meets the preset training end condition, the current second model is determined as the preset second model.

[0141] In some embodiments, the training module is also used to determine the first element loss value based on the diagonal element and a preset first threshold; determine the second element loss value based on the non-diagonal element and a preset second threshold; and determine the second model loss value corresponding to the initial second model based on the first element loss value and the second element loss value.

[0142] In some embodiments, the training module is also used to update the initial second model according to the second model loss value to obtain an updated second model if the second model loss value does not meet the preset training end condition; input the second sample text and the second sample phoneme sequence into the updated second model to obtain an updated prediction similarity matrix.

[0143] In some embodiments, the screening module is further used to input the target phoneme sequence into a preset vocoder to obtain synthesized speech information of the target text.

[0144] Based on the same inventive concept, the embodiment of the present application further provides a computer device, which may be a server or a terminal device, and includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned phoneme sequence determination method when executing the computer program. Thereby, various functions are realized, for example:

[0145] Inputting the target text into a preset first model to obtain a candidate phoneme sequence;

[0146] Generating a first encoding feature of the target text according to words in the target text, and generating a second encoding feature of the candidate phoneme sequence according to phonemes in the candidate phoneme sequence;

[0147] Inputting the first encoding feature and the second encoding feature into a preset second model to obtain a target similarity between the target text and the candidate phoneme sequence, wherein the target similarity includes a similarity between each word in the target text and each phoneme in the candidate phoneme sequence;

[0148] A target phoneme sequence of the target text is determined from the candidate phoneme sequences according to the target similarity.

[0149] The phoneme sequence determination device provided in the embodiment of the present application processes the target text by presetting a first model to obtain multiple possible candidate phoneme sequences, and then determines the similarity between the target text and different candidate phoneme sequences using the encoding features of the candidate phoneme sequences and the text. Subsequently, the target phoneme sequence of the target text can be determined from the candidate phoneme sequences based on the similarity. Since the preset first model provided in the embodiment of the present application directly establishes a generative relationship from text to phoneme sequence, the effect of the generated candidate phoneme sequence can be guaranteed. At the same time, the phoneme sequence is screened using similarity, which further guarantees the accuracy of the generated phoneme sequence, thereby improving the effect of subsequent speech synthesis. The specific implementation of each of the above operations can be found in the previous embodiments, which will not be repeated here.

[0150] In one embodiment, the computer device is a terminal device, for example, its internal structure diagram can be as follows: Fig.11 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for determining a phoneme sequence is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0151] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0152] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0153] Since the computer program stored in the computer-readable storage medium can execute any one of the phoneme sequence determination methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the phoneme sequence determination methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0154] Based on the same inventive concept, the embodiment of the present application further provides a computer program product or a computer program, which includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations in the above embodiments.

[0155] It should be noted that the object data (including but not limited to user device information, user personal information, etc.) and conversation data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions. A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0156] Any reference to the memory, database or other media used in the embodiments provided in the present application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (Re RAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0157] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0158] In the above-mentioned embodiments of the phoneme sequence determination device, computer-readable storage medium, computer device, and computer program product, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process and beneficial effects of the above-mentioned phoneme sequence determination device, computer-readable storage medium, computer program product, computer device and its corresponding units can refer to the description of the phoneme sequence determination method in the above embodiment, and will not be repeated here.

[0159] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0160] The above is a detailed introduction to a phoneme sequence determination method, device, computer equipment, computer-readable storage medium and computer program product provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for determining a phoneme sequence, It is characterized in that include: Inputting the target text into a preset first model to obtain a candidate phoneme sequence; Generating a first encoding feature of the target text according to words in the target text, and generating a second encoding feature of the candidate phoneme sequence according to phonemes in the candidate phoneme sequence; Inputting the first encoding feature and the second encoding feature into a preset second model to obtain a target similarity between the target text and the candidate phoneme sequence, wherein the target similarity includes a similarity between each word in the target text and each phoneme in the candidate phoneme sequence; A target phoneme sequence of the target text is determined from the candidate phoneme sequences according to the target similarity.

2. The method according to claim 1, It is characterized in that The step of inputting the first coding feature and the second coding feature into a preset second model to obtain a target similarity between the target text and the candidate phoneme sequence includes: Inputting the first encoding feature into a first convolutional layer in a preset second model to obtain a text matrix of the target text; Inputting the second encoding feature into a second convolutional layer in a preset second model to obtain a phoneme matrix of the candidate phoneme sequence; Performing matrix operations on the text matrix and the phoneme matrix to obtain a similarity matrix; A target similarity between the target text and the candidate phoneme sequence is determined according to matrix elements in the similarity matrix.

3. The method according to claim 1, It is characterized in that The step of inputting the target text into a preset first model to obtain a candidate phoneme sequence includes: Inputting the target text into a preset first model to obtain a first candidate phoneme sequence; Associatively inputting the target text and the first candidate phoneme sequence into the preset first model to obtain a second candidate phoneme sequence; The first candidate phoneme sequence and the second candidate phoneme sequence are determined as the candidate phoneme sequences.

4. The method according to claim 1, It is characterized in that The preset first model is obtained by training through the following steps: Inputting the sample sequence into the initial first model to obtain a predicted sequence of the sample sequence; wherein the sample sequence consists of a first sample text and a first sample phoneme sequence; Calculate the first model loss value of the initial first model according to the prediction sequence and the sample sequence; If the loss value of the first model meets the preset training end condition, the current first model is determined as the preset first model.

5. The method according to claim 4, It is characterized in that The calculating the first model loss value of the initial first model according to the prediction sequence and the sample sequence includes: Dividing the prediction sequence to obtain a text prediction sequence and a phoneme prediction sequence; Determine a first text loss value between the text prediction sequence and the first sample text, and determine a first phoneme loss value between the phoneme prediction sequence and the first sample phoneme sequence; A first model loss value of the initial first model is determined according to the first text loss value and the first phoneme loss value.

6. The method according to claim 4, It is characterized in that After the step of calculating the first model loss value of the initial first model according to the prediction sequence and the sample sequence, the method further includes: If the first model loss value does not meet the preset training end condition, updating the initial first model according to the first model loss value to obtain an updated first model; The sample sequence is input into the updated first model to obtain an updated prediction sequence.

7. The method according to claim 1, It is characterized in that The preset second model is obtained by training through the following steps: Inputting the second sample text and the second sample phoneme sequence into the initial second model to obtain a prediction similarity matrix; Determine the second model loss value corresponding to the initial second model according to the matrix elements in the prediction similarity matrix; If the loss value of the second model meets the preset training end condition, the current second model is determined as the preset second model.

8. The method according to claim 7, It is characterized in that The matrix elements in the prediction similarity matrix include diagonal elements and off-diagonal elements; Determining the second model loss value corresponding to the initial second model according to the matrix elements in the prediction similarity matrix includes: Determine a first element loss value according to the diagonal element and a preset first threshold; Determine a second element loss value according to the off-diagonal element and a preset second threshold; Determine a second model loss value corresponding to the initial second model according to the first element loss value and the second element loss value.

9. The method according to claim 7, It is characterized in that After the step of determining the second model loss value corresponding to the initial second model according to the matrix elements in the prediction similarity matrix, the method further includes: If the second model loss value does not meet the preset training end condition, updating the initial second model according to the second model loss value to obtain an updated second model; The second sample text and the second sample phoneme sequence are input into the updated second model to obtain an updated prediction similarity matrix.

10. The method according to any one of claims 1 to 9, It is characterized in that After the step of determining the target phoneme sequence of the target text from the candidate phoneme sequences according to the target similarity, the method further includes: The target phoneme sequence is input into a preset vocoder to obtain synthesized speech information of the target text.

11. A device for determining a phoneme sequence, It is characterized in that include: An input module, used for inputting the target text into a preset first model to obtain a candidate phoneme sequence; An encoding module, configured to generate a first encoding feature of the target text according to words in the target text, and to generate a second encoding feature of the candidate phoneme sequence according to phonemes in the candidate phoneme sequence; A processing module, configured to input the first encoding feature and the second encoding feature into a preset second model to obtain a target similarity between the target text and the candidate phoneme sequence, wherein the target similarity includes a similarity between each word in the target text and each phoneme in the candidate phoneme sequence; A screening module is used to determine a target phoneme sequence of the target text from the candidate phoneme sequences according to the target similarity.

12. A computer device, It is characterized in that The method comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps of the phoneme sequence determination method according to any one of claims 1 to 10.

13. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps of the phoneme sequence determination method according to any one of claims 1 to 10.

Images