Text processing method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a text processing method and device, electronic equipment, a storage medium and a program product, and the method comprises the steps: carrying out the feature extraction of a text, obtaining a first text feature of the text, and enabling the first text feature to comprise a text feature of a phoneme dimension, a text feature of a word dimension and a text feature of a word dimension; determining a second text feature of the text according to the first text feature and a position corresponding relationship among the text feature of the phoneme dimension, the text feature of the word dimension and the text feature of the word dimension; and determining an acoustic feature corresponding to the second text feature, and obtaining a voice corresponding to the text according to the acoustic feature. Therefore, the richness and the accuracy of text feature representation can be improved by fusing the multi-dimensional text features, so that the effects of improving the naturalness and the like of the converted voice are achieved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a text processing method, device, electronic device, storage medium, and program product. Background Art

[0002] Text-To-Speech (TTS) technology can convert text data into natural speech output. In related technologies, the text-to-speech method mainly encodes the text, obtains text features and converts them into acoustic features to obtain speech signals. Therefore, the obtained text features are very important for the final generated speech signal and affect the effect of generated speech. Summary of the Invention

[0003] The present disclosure provides a text processing method, device, electronic device, storage medium and program product.

[0004] In a first aspect, the present disclosure provides a text processing method, the text processing method comprising:

[0005] Extracting features from the text to obtain first text features of the text, where the first text features include text features in a phoneme dimension, text features in a character dimension, and text features in a word dimension;

[0006] Determining a second text feature of the text based on the first text feature and a positional correspondence between the text feature of the phoneme dimension, the text feature of the character dimension, and the text feature of the word dimension;

[0007] An acoustic feature corresponding to the second text feature is determined, and a speech corresponding to the text is obtained based on the acoustic feature.

[0008] In a second aspect, the present disclosure provides a text processing device, the text processing device comprising:

[0009] A feature extraction module is used to extract features from a text to obtain first text features of the text, where the first text features include text features in a phoneme dimension, text features in a character dimension, and text features in a word dimension;

[0010] a determination module, configured to determine a second text feature of the text based on the first text feature and a positional correspondence between the text feature of the phoneme dimension, the text feature of the character dimension, and the text feature of the word dimension;

[0011] The conversion module is used to determine the acoustic feature corresponding to the second text feature, and obtain the speech corresponding to the text based on the acoustic feature.

[0012] In a third aspect, the present disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the above-mentioned text processing method.

[0013] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned text processing method when executed by a processor.

[0014] In a fifth aspect, the present disclosure provides a computer program or a computer program product, which includes a computer program stored in a computer-readable storage medium, and the computer program implements the above-mentioned text processing method when executed by a processor.

[0015] The embodiments provided by the present disclosure obtain first text features of multiple dimensions of a text, which may include text features of phoneme dimension, character dimension and word dimension, and determine second text features of the text based on the positional correspondence between the first text features and the text features of the phoneme dimension, the text features of the character dimension and the text features of the word dimension. In this way, text information of multiple dimensions can be integrated so that the obtained second text features can express text information at more different levels, thereby improving the richness and accuracy of the second text features. Based on the second text features that are more accurate and include richer text information, acoustic feature conversion is performed to obtain the speech of the text, which can improve the effect of the obtained speech and make the speech more accurate, more natural, and more human.

[0016] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings. In the accompanying drawings:

[0018] Figure 1 An application scenario diagram of a text processing method and apparatus provided by an embodiment of the present disclosure;

[0019] Figure 2A flowchart of a text processing method provided in an embodiment of the present disclosure;

[0020] Figure 3 A block diagram illustrating the principle of determining text features in a text processing method according to an embodiment of the present disclosure;

[0021] Figure 4 A block diagram of a text processing device provided in an embodiment of the present disclosure;

[0022] Figure 5 A block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] To enable those skilled in the art to better understand the technical solutions of the present disclosure, exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0025] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0026] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprising" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. Similar words such as "connected" or "connected" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0027] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless expressly defined as such herein.

[0028] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution complies with relevant national laws and regulations (for example, the "Information Security Technology Personal Information Security Specification", etc.). For example: corresponding prescribed measures are taken to control access to personal information; the display of personal information is subject to prescribed restrictions; the purpose of using personal information does not exceed the scope of direct or reasonable connection; when using personal information, clear identity reference is eliminated to avoid precise positioning of specific individuals.

[0029] To facilitate understanding of the embodiments of the present disclosure, several concepts involved in the embodiments of the present disclosure are first described below:

[0030] Text-to-speech (TTS) converts text data into natural speech output. TTS technology typically consists of a front-end system, an acoustic model, and a vocoder. The front-end system primarily converts the input text sequence into a phoneme sequence, the acoustic model converts the phoneme sequence into an acoustic feature sequence, and the vocoder converts the acoustic feature sequence into an acoustic signal. TTS technology can be applied in a variety of scenarios, including reading assistance, voice navigation systems, intelligent voice assistants, audiobook generation, language learning tools, and interactive interfaces for various electronic devices.

[0031] Phoneme: The smallest phonetic unit divided according to the natural properties of speech, and the smallest pronunciation unit in a language.

[0032] Bidirectional Encoder Representations from Transformer (BERT): A bidirectional encoding representation learning language model is built based on the Transformer framework, which can be used to encode text and obtain the encoding vector of the text.

[0033] The text-to-speech method in the related art usually first encodes the text features and then obtains the speech signal based on the text feature conversion. Therefore, the representation of the text features is very important for the final generated speech and will affect the effect of the generated speech.

[0034] The text processing method provided in the embodiments of the present disclosure extracts the first text features of the text including multiple dimensions, including text features of the phoneme dimension, text features of the character dimension and text features of the word dimension, and then determines the second text features of the text based on the text features of the multiple dimensions and the positional correspondence between the text features of the phoneme dimension, the text features of the character dimension and the text features of the word dimension, and then obtains the speech corresponding to the text based on the second text features. In this way, the text can be expressed from different dimensions, and information at different levels or different granularities of the text can be obtained, thereby improving the accuracy and comprehensiveness of the second text features of the final text. Therefore, based on the second text features, a more accurate and natural speech can be obtained, thereby improving the speech generation effect.

[0035] Figure 1 The application scenario diagram of the text processing method and device provided by the embodiments of the present disclosure is schematically shown.

[0036] like Figure 1 As shown, an application scenario of an embodiment of the present disclosure may include a terminal device 101, a network 103, and a server 102. The network 103 is used as a medium for providing a communication link between the terminal device 101 and the server 102. The network 103 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0037] The user can use the terminal device 101 to interact with the server 102 via the network 103 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0038] The terminal device 101 may be any electronic device having a display screen and supporting web browsing, including but not limited to a smart phone, a tablet computer, a laptop computer, a desktop computer, and the like.

[0039] The server 102 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the terminal device 101. The background management server may analyze and process received user requests and other data, and feed back the processing results (e.g., web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0040] It should be noted that the text processing method and apparatus provided in the embodiments of the present disclosure can be executed by the server 102. Accordingly, the text processing method and apparatus provided in the embodiments of the present disclosure can be set in the server 102. The text processing method and apparatus provided in the embodiments of the present disclosure can also be executed by a server or server cluster that is different from the server 102 and can communicate with the terminal device 101 and / or the server 102. Accordingly, the text processing method and apparatus provided in the embodiments of the present disclosure can also be set in a server or server cluster that is different from the server 102 and can communicate with the terminal device 101 and / or the server 102.

[0041] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0042] Figure 2 A flowchart of a text processing method provided by an embodiment of the present disclosure. Figure 2 , the method comprising:

[0043] S210: Extract features from the text to obtain first text features of the text. The first text features include text features in a phoneme dimension, text features in a character dimension, and text features in a word dimension.

[0044] In the disclosed embodiment, text features can be extracted from multiple different dimensions to express text information at different levels or granularities. Feature extraction at the phoneme dimension, character dimension, and word dimension can be implemented based on the corresponding encoder model.

[0045] Furthermore, in the embodiments of the present disclosure, it is not limited to the phoneme dimension, character dimension and word dimension, and text features of other dimensions can also be obtained, such as sentence dimension, paragraph dimension, etc., which are not limited in the embodiments of the present disclosure.

[0046] S220: Determine a second text feature of the text according to the first text feature and the positional correspondence between the text feature of the phoneme dimension, the text feature of the character dimension, and the text feature of the word dimension.

[0047] In a possible embodiment, the text features of multiple dimensions may be directly added or weighted averaged to obtain the second text feature of the text.

[0048] Thus, in the disclosed embodiments, by extracting text features from different dimensions, different levels of text information can be perceived. For example, if the text is in Chinese, the phoneme dimension mainly represents the phoneme composition of Chinese Pinyin, and phoneme pronunciation information can be obtained. The character dimension represents the level of a single character, and information such as the association information between characters in the text and semantic information can be obtained. The word dimension represents the word level, and information such as the association information between words in the text and contextual association information at the word level, sentence structure, and grammatical structure can be obtained. Thus, based on the extracted first text feature, a second text feature can be determined. The obtained second text feature can integrate text features of different dimensions, thereby improving the richness and accuracy of the text information expressed by the second text feature.

[0049] S230: Determine the acoustic feature corresponding to the second text feature, and obtain the speech corresponding to the text based on the acoustic feature.

[0050] In the embodiments of the present disclosure, there is no limitation on the specific implementation method of obtaining speech based on the second text feature. For example, in one possible implementation method, determining the acoustic feature corresponding to the second text feature includes: inputting the second text feature into the acoustic model, performing a nonlinear transformation on the second text feature based on the nonlinear mapping relationship between the text feature and the acoustic feature in the acoustic model, and outputting the acoustic feature corresponding to the second text feature.

[0051] Then, based on the acoustic feature decoder, the corresponding speech can be obtained according to the acoustic features.

[0052] Among them, the acoustic model can be obtained through pre-training and can be trained based on a data set containing text and corresponding speech. The acoustic model can learn the nonlinear mapping relationship between text features and acoustic features.

[0053] In an embodiment of the present disclosure, feature extraction is performed on the text to obtain text features in multiple dimensions of the text, and a second text feature of the text is determined based on the text features in multiple dimensions and the positional correspondence between the text features in multiple dimensions. In this way, the second text feature finally obtained may include text features in multiple dimensions, such as text features in phoneme dimension, character dimension, and word dimension, which improves the richness of the text information that can be expressed by the second text feature, such as phoneme pronunciation information, semantic information, inter-character association relationship, inter-word association relationship, sentence structure information, and grammatical structure information, etc., and also improves the accuracy of text representation. Based on the second text feature, text-to-sound conversion is performed to obtain the speech corresponding to the text. Since the second text feature provides more comprehensive language prior information, it is beneficial to obtain more text information at the phoneme level, character level, and word level when converting text to speech, so that the obtained speech is more accurate, natural, and more human.

[0054] The text processing method according to an embodiment of the present disclosure is described below.

[0055] As mentioned above, with respect to the text features of different dimensions in step S210, the present disclosure provides the following possible embodiments:

[0056] In a possible embodiment, for the phoneme dimension, a phoneme sequence corresponding to the text is determined, features are extracted from the phoneme sequence, phoneme features of each phoneme in the phoneme sequence are obtained, and text features of the phoneme dimension are obtained based on the phoneme features of each phoneme.

[0057] Specifically, for example, if the text is in Chinese, a text-to-pinyin tool can be used to determine the phoneme sequence of the text, and further, the phoneme sequence can be converted into a numerical representation, and then based on the phoneme coding model, feature extraction is performed on the phoneme sequence to obtain the phoneme features of each phoneme in the phoneme sequence, and based on the phoneme features of each phoneme, the text features of the phoneme dimension of the text are obtained.

[0058] For example, the text is: "Great Mother". Determine its phoneme sequence and represent it numerically. For example, each pinyin initial consonant is regarded as a token, and each tonal final is regarded as a token. That is, the phoneme sequence can be: "w / ei3 / d / a4 / d / e5 / m / u3 / q / in1". A phoneme sequence of length 10 is obtained. Then, it is input into the phoneme coding model to obtain the text features of the phoneme dimension. For example, if the output dimension size of the phoneme coding model is 768, the obtained text features of the phoneme dimension of the text are 10*768 dimensions.

[0059] In this way, in the embodiment of the present disclosure, text features in the phoneme dimension can be extracted from the phoneme dimension, and the phoneme pronunciation information of the text can be obtained.

[0060] In a possible embodiment, the text is segmented into characters according to the character dimension to obtain a character sequence corresponding to the text, feature extraction is performed on the character sequence to obtain the character features of each character in the character sequence, and based on the character features of each character, the text features of the character dimension of the text are obtained.

[0061] Specifically, a character sequence corresponding to the text is obtained, the character sequence is numerically represented, and based on a character encoding model, features of the character sequence are extracted to obtain the character features of each character in the character sequence, and based on the character features of each character, the text features of the character dimension of the text are obtained.

[0062] For example, the text is: "Great Mother", the character sequence is "Great / Big / Of / Mother / ", with a length of 5. After numerical representation and input into the character encoding model, the text features of the character dimension are obtained. If the output dimension size of the character encoding model is 768, the obtained text features of the character dimension are 5*768 dimensions.

[0063] In the disclosed embodiment, the text features of the character dimension can be obtained from the character dimension, and the character-level association information, semantic information, etc. of the text can be obtained.

[0064] In a possible embodiment, the text is segmented with respect to the word dimension to obtain a word sequence corresponding to the text, feature extraction is performed on the word sequence to obtain the word features of each word in the word sequence, and based on the word features of each word, the text features of the word dimension of the text are obtained.

[0065] Specifically, based on the word segmentation tool, the word sequence of the text is obtained, the word sequence is numerically represented, and based on the word encoding model, the word sequence is feature extracted to obtain the word features of each word in the word sequence, and based on the word features of each word, the text features of the word dimension of the text are obtained.

[0066] For example, the text is "great mother", and the word sequence after word segmentation is ["great", "of", "mother"]. After numerical representation processing, a numerical sequence of length 3 can be obtained. When input into the word encoding model, taking the feature dimension size of each word as 768 as an example, the text feature size of the word dimension obtained is 3*768 dimensions.

[0067] In the disclosed embodiment, text features of the word dimension can be obtained from the word dimension, and word-level association information, grammatical structure, syntactic structure and other information of the text can be obtained.

[0068] With respect to the above step S220, the present disclosure further provides a possible embodiment, which determines a second text feature of the text based on the first text feature and the positional correspondence between the text feature in the phoneme dimension, the text feature in the character dimension, and the text feature in the word dimension, including:

[0069] 1) Determine the first position mapping relationship between phonemes and characters in the text, and align the lengths of the text features in the character dimension and the text features in the phoneme dimension according to the first position mapping relationship to obtain the first character text feature of the text, where the first character text feature has the same length as the text feature in the phoneme dimension.

[0070] For this step, the present disclosure provides a specific implementation manner. According to the first position mapping relationship, determine the first number of phonemes corresponding to each character in the text; for each character, when the first number is greater than 1, copy and splice the character features of the character so that the second number of the spliced character features is the same as the first number; according to the spliced character features of each character, determine the first character text feature of the text.

[0071] For example, one of the characters included in the text is "你 (nǐ)". The phonemes having the first position mapping relationship with this character are "N" and "i3". That is, the first number of phonemes corresponding to the character "你 (nǐ)" is 2. Therefore, to align the length of the character "你 (nǐ)" with the phonemes "N" and "i3", it is necessary to copy the character features of the character "你 (nǐ)" again. The spliced character features still have the first position mapping relationship with the corresponding phonemes. That is, there are two character features of "你 (nǐ)", respectively corresponding to one phoneme "N" and one phoneme "i3".

[0072] 2) Determine the second position mapping relationship between words and characters in the text, and according to the first position mapping relationship and the second position mapping relationship, align the length of the text feature in the word dimension with the text feature in the phoneme dimension to obtain the first word text feature of the text. The length of the first word text feature is the same as the text feature in the phoneme dimension. [[ID=‎7]]

[0073] For this step, the present disclosure provides a specific implementation manner. According to the first position mapping relationship and the second position mapping relationship, determine the third number of phonemes corresponding to each word in the text; for each word, when the third number is greater than 1, copy and splice the word features of the word so that the fourth number of the spliced word features is the same as the third number; according to the spliced word features of each word, determine the first word text feature of the text.

[0074] For example, a certain word included in the text is "你好 (nǐ hǎo)". The characters corresponding to "你好 (nǐ hǎo)" having the second position mapping relationship are "你 (nǐ)" and "好 (hǎo)". The phonemes corresponding to "你 (nǐ)" having the first position mapping relationship are "N" and "i3", and the phonemes corresponding to "好 (hǎo)" having the first position mapping relationship are "h" and "ao3". Therefore, the third number of phonemes corresponding to the word "你好 (nǐ hǎo)" is 4. To align the length of the word "你好 (nǐ hǎo)" with the phonemes "N", "i3", "h", and "ao3", it is necessary to copy 3 word features of "你好 (nǐ hǎo)" again. The spliced word features of "你好 (nǐ hǎo)" are a total of 4, respectively corresponding to these four phonemes "N", "i3", "h", and "ao3".

[0075] 3) Perform a preset operation on the text feature in the phoneme dimension, the first character text feature, and the first word text feature to obtain the second text feature of the text.

[0076] In the embodiment of the present disclosure, there is no limitation on the preset operation, and several possible implementation methods are specifically provided. For example, the text features of the phoneme dimension, the first character text features and the first word text features are spliced ​​together to obtain the second text features of the text.

[0077] For another example, according to the first position mapping relationship and the second position mapping relationship, the text features of the phoneme dimension, the first character text features of the character dimension, and the first word text features of the word dimension are added according to the corresponding positions to obtain the second text features of the text. The addition in this step can be direct addition, or the weights corresponding to the text features of the phoneme dimension, the first character text features, and the first word text features can be set respectively, and weighted addition is performed based on the weights to obtain the second text features. This is not limited in the embodiments of the present disclosure.

[0078] For example, taking the word "hello" included in the text as an example, the length of "hello" is 1, and its word feature is A. The corresponding characters are "you" and "good", with a length of 2, and their character features are B1 and B2 respectively. The corresponding phonemes are "N", "i3", "h" and "ao3", with a length of 4, and their phoneme features are C1, C2, C3 and C4 respectively. According to the second position mapping relationship between words and characters and the first position mapping relationship between characters and phonemes, the lengths are aligned and the corresponding positions are added, and the final second text feature can be [(A+B1+C1), (A+B1+C2), (A+B2+C3), (A+B2+C4)].

[0079] It should also be noted that in the embodiment of the present disclosure, when the first text feature includes phoneme dimension, character dimension and word dimension, the text features of the phoneme dimension, the text features of the character dimension and the text features of the word dimension are length-aligned and then subjected to a preset operation to obtain the second text feature of the text. Of course, if the first text feature includes any two of the phoneme dimension, the character dimension and the word dimension, or also includes text features of other dimensions, it can also be length-aligned and then subjected to a preset operation to obtain the second text feature of the text. This is not limited in the embodiment of the present disclosure.

[0080] In the disclosed embodiment, features of the text can be extracted from the phoneme dimension, word dimension and character dimension respectively, and the text features of multiple dimensions can be integrated to obtain the second text feature of the text. In this way, the text information represented by the obtained second text feature is richer, and can integrate information such as phoneme pronunciation, inter-character association information, semantic information, syntactic information, and grammatical information, and also improve the accuracy of text representation, thereby improving the effect of the final converted speech.

[0081] In some possible embodiments, the present disclosure can perform feature extraction on text based on a phoneme coding model to obtain text features in the phoneme dimension, perform feature extraction on text based on a character coding model to obtain text features in the character dimension, and perform feature extraction on text based on a word coding model to obtain text features in the word dimension. The following describes the training of the phoneme coding model, the character coding model, and the word coding model:

[0082] 1) Phoneme encoding model.

[0083] In the embodiment of the present disclosure, a first training sample set may be obtained, each first training sample in the first training sample set includes a phoneme sequence sample, and a phoneme coding model is obtained by training based on the first training sample set.

[0084] The phoneme coding model may adopt a BERT model, and the BERT model may be trained based on the first training sample set to obtain a phoneme coding model.

[0085] 2) Word encoding model.

[0086] In the embodiment of the present disclosure, a second training sample set may be obtained, each second training sample in the second training sample set includes a word sequence sample, and a word encoding model is obtained by training based on the second training sample set.

[0087] In this case, a BERT model may also be used, and the BERT model is trained based on the second training sample set to obtain a word encoding model.

[0088] 3) Word encoding model.

[0089] In the embodiment of the present disclosure, a word encoding model may be obtained by training based on a training sample set, where each training sample in the training sample set includes a word sequence sample and a part-of-speech tag of each word in the word sequence sample.

[0090] Specifically, regarding the method of obtaining the training sample set, the present disclosure also provides a possible implementation method. In order to further improve the accuracy and training effect, in the embodiment of the present disclosure, word segmentation and word frequency statistics can be performed based on multiple text data, words with higher frequency are extracted from the word database, and the extracted words are digitized to obtain a numerical training sample set. The numerical training sample set can be used for the numerical representation of words during word encoding model training and inference.

[0091] In the embodiment of the present disclosure, a training text sample can be first obtained, and the training text sample can be segmented to obtain a word sequence sample and a part-of-speech tag for each word. The word sequence sample and the part-of-speech tag can also be digitized to obtain a numerical representation of the word sequence sample and a numerical representation of the part-of-speech tag sequence. Then, a word encoding model can be trained based on the numerical representations corresponding to the word sequence sample and the part-of-speech tag sequence, respectively.

[0092] For example, XRC represents an adjective label, ZC represents an auxiliary word label, and MC represents a noun label. The word sequence sample [great, of, mother] is numerically processed, and the resulting numerical representation is [8001, 3, 10992]. The part-of-speech tag sequence corresponding to the word sequence sample is [XRC, ZC, MC], and the numerical processing is performed, and the resulting numerical representation is [4, 5, 1].

[0093] In this way, in the embodiment of the present disclosure, through operations such as word segmentation, part-of-speech tagging, and numerical processing of training text samples, a training sample set ultimately used for word encoding model training can be obtained, which can improve the effectiveness and accuracy of the training sample set.

[0094] The training methods for word encoding models can include:

[0095] 1) Obtain a training sample set, wherein each training sample in the training sample set includes a word sequence sample and a part-of-speech tag of each word in the word sequence sample.

[0096] 2) Randomly mask some words in the word sequence sample to obtain the input sequence sample.

[0097] 3) Inputting the input sequence sample into the word encoding model to obtain the predicted parts of speech of the unmasked words and the predicted words of the masked words in the input sequence sample; and determining the first loss and the second loss of the word encoding model based on the predicted parts of speech of the unmasked words and the predicted words of the masked words.

[0098] 4) Inputting the word sequence sample into the word encoding model, determining the prediction information of the subsequent sentence of the previous word sequence sample input into the word encoding model; and determining the third loss of the word encoding model based on the prediction information.

[0099] 5) Train the word encoding model based on the first loss, the second loss, and the third loss.

[0100] It can be understood that in the embodiments of the present disclosure, in order to improve the accuracy and reliability of word encoding model training, when training the word encoding model, training can be performed based on multiple training tasks. 1) One training task is a part-of-speech prediction task: some words in the word sequence sample can be randomly masked, and then based on the word encoding model, the predicted part of speech of the unmasked words can be predicted. According to the predicted part of speech and the corresponding part-of-speech label, cross entropy calculation is performed to obtain the first loss.

[0101] For example, the second word in the word sequence sample is masked, and its part-of-speech tag can also be masked. The classification result can be obtained through the classification layer of the word encoding model, wherein the classification layer can include a linear mapping layer and a softmax layer. The output dimension of the linear mapping layer is the preset number of part-of-speech tags, and the probability distribution of each part-of-speech tag can be obtained. Then, after passing through the classification layer of the word encoding model, the probability distribution of the unmasked words belonging to each part-of-speech tag can be obtained. The word with the largest probability value is the predicted part-of-speech of the word at that position. Then, according to the probability distribution of the part-of-speech tag and the corresponding predicted part-of-speech, a cross-entropy calculation is performed to obtain the first loss, for example, the first loss = crossEntropy (part-of-speech tag of the word that is not randomly masked, predicted part-of-speech of the word that is not randomly masked), wherein crossEntropy represents the cross-entropy function.

[0102] Among them, the random masking method is not restricted. In this way, by training with random masking, the generalization and accuracy of word encoding model training can be improved. In addition, random masking can also be an option, that is, masking can be omitted, and the first loss is calculated for the predicted part of speech and the corresponding part of speech label of all words in the word sequence sample.

[0103] Thus, in the disclosed embodiment, when training the word encoding model, the part-of-speech prediction task is introduced, which helps the training word encoding model to learn more syntactic information and improve the performance of the word encoding model.

[0104] 2) One training task is a masked word prediction task: based on the word encoding model, the predicted word of the masked word is determined, and the second loss is obtained according to the cross entropy between the predicted word and the corresponding true word (i.e., the corresponding masked word in the word sequence sample).

[0105] 3) A training task is a Boolean prediction task of whether it is a subsequent sentence: In the embodiment of the present disclosure, based on the word encoding model, it is possible to determine whether a word sequence sample is the prediction information of the subsequent sentence of the previous word sequence sample input into the word encoding model, that is, to predict whether a certain word sequence sample is the next sentence of the previous word sequence sample. This can be understood as judging whether it has continuity or context relevance. The obtained prediction information can be a Boolean distribution of yes or no, and then based on the prediction information, the third loss is obtained.

[0106] Of course, in the disclosed embodiments, when training the word encoding model, other training tasks can also be combined, such as the task of predicting the component type of words in a sentence. For example, component types include subject, predicate, object, attributive, adverbial, and complement. During training, the predicted component type of words in the word sequence sample is obtained, and the fourth loss is obtained based on the cross entropy between the predicted component type and the component type label. In this way, through the component type prediction task, the word encoding model's ability to understand syntactic structure can be further enhanced, thereby improving the performance of the word encoding model.

[0107] In the embodiments of the present disclosure, encoding models for feature extraction in different dimensions can be trained, so that feature extraction can be performed on text based on different encoding models to obtain text features in multiple dimensions of the text. When training a word encoding model, training can be performed based on multiple training tasks, thereby improving the accuracy of the word encoding model.

[0108] In addition, in the embodiments of the present disclosure, after feature extraction and corresponding processing based on the trained phoneme coding model, character coding model and word coding model, the final text features of the text are obtained. The acoustic model can also be trained based on the final text features of the text, which can further improve the accuracy of the acoustic model.

[0109] The text processing method in the embodiment of the present disclosure is described below using a specific application scenario. The text is "Great Mother" and the first text feature includes text features of the phoneme dimension, text features of the character dimension and text features of the word dimension as an example. Figure 3 , which is a principle block diagram of determining text features in the text processing method in an embodiment of the present disclosure.

[0110] 1) If Figure 3 As shown, the text is "great mother". After phonemic processing, the phoneme sequence is "w / ei3 / d / a4 / d / e5 / m / u3 / q / in1". The text is segmented into characters, and the character sequence is: "great / big / mother". The text is segmented into words, and the word sequence is: "great / mother".

[0111] 2) Based on the phoneme encoding model, feature extraction is performed on the phoneme sequence to obtain text features in the phoneme dimension. Based on the character encoding model, feature extraction is performed on the character sequence to obtain text features in the character dimension. Based on the word encoding model, feature extraction is performed on the word sequence to obtain text features in the word dimension.

[0112] 3) According to the first position mapping relationship between phonemes and characters in the text, the text features of the character dimension and the text features of the phoneme dimension are aligned in length to obtain the first character text features of the text.

[0113] Determine the second position mapping relationship between words and characters in the text, and align the text features of the word dimension with the text features of the phoneme dimension according to the first position mapping relationship and the second position mapping relationship to obtain the first word text feature of the text.

[0114] 4) Performing a preset operation on the text feature of the phoneme dimension, the first character text feature, and the first word text feature, taking the preset operation as addition as an example, to determine the second text feature of the text.

[0115] In the embodiment of the present disclosure, text features of multiple dimensions of the text are extracted, and based on the text features of multiple dimensions, the second text features of the text are obtained. This can integrate the multi-dimensional text information, making the text information represented by the second text features richer and improving the accuracy. Therefore, based on the second text features, the voice of the text is obtained, and the accuracy, naturalness and anthropomorphism of the voice are also improved.

[0116] It is understood that the above-mentioned various method embodiments mentioned in this disclosure can be combined with each other to form combined embodiments without violating the principle logic. Due to space limitations, this disclosure will not go into details. It is understood by those skilled in the art that in the above-mentioned methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0117] In addition, the present disclosure also provides a text processing device, an electronic device, and a computer-readable storage medium, all of which can be used to implement any text processing method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method section and will not be repeated here.

[0118] Figure 4 A block diagram of a text processing device provided in an embodiment of the present disclosure.

[0119] Reference Figure 4 , an embodiment of the present disclosure provides a text processing device, the text processing device comprising:

[0120] A feature extraction module 41 is configured to extract features from a text to obtain first text features of the text, wherein the first text features include text features in a phoneme dimension, text features in a character dimension, and text features in a word dimension;

[0121] a determination module 42 for determining a second text feature of the text based on the first text feature and a positional correspondence between the text feature of the phoneme dimension, the text feature of the character dimension, and the text feature of the word dimension;

[0122] The conversion module 43 is configured to determine the acoustic feature corresponding to the second text feature, and obtain the speech corresponding to the text based on the acoustic feature.

[0123] In a possible embodiment, when determining the second text feature of the text based on the first text feature and the positional correspondence between the text feature in the phoneme dimension, the text feature in the character dimension, and the text feature in the word dimension, the determination module 42 is configured to:

[0124] Determining a first position mapping relationship between phonemes and characters in the text, and aligning the lengths of the character-dimensional text features and the phoneme-dimensional text features according to the first position mapping relationship to obtain a first character text feature of the text, where the first character text feature has the same length as the phoneme-dimensional text feature;

[0125] Determining a second position mapping relationship between words and characters in the text, and aligning the length of the text feature in the word dimension with the text feature in the phoneme dimension based on the first position mapping relationship and the second position mapping relationship to obtain a first word text feature of the text, where the first word text feature has the same length as the text feature in the phoneme dimension;

[0126] The text feature of the phoneme dimension, the first character text feature, and the first word text feature are subjected to a preset operation to obtain a second text feature of the text.

[0127] In a possible embodiment, when the text features of the character dimension and the text features of the phoneme dimension are aligned in length according to the first position mapping relationship to obtain the text features of the first character of the text, the determination module 42 is configured to:

[0128] Determining a first number of phonemes corresponding to each character in the text according to the first position mapping relationship;

[0129] For each character, when the first number is greater than 1, copying and concatenating the character features of the character so that the second number of the concatenated character features is the same as the first number;

[0130] The text feature of the first character of the text is determined according to the character features of each concatenated character.

[0131] In a possible embodiment, when the text features of the word dimension and the text features of the phoneme dimension are aligned in length according to the first position mapping relationship and the second position mapping relationship to obtain the first word text features of the text, the determination module 42 is configured to:

[0132] Determining a third number of phonemes corresponding to each word in the text according to the first position mapping relationship and the second position mapping relationship;

[0133] For each word, if the third number is greater than 1, copy and concatenate the word features of the word so that the fourth number of the concatenated word features is the same as the third number;

[0134] The first word text feature of the text is determined according to the word feature of each word after being spliced.

[0135] In a possible embodiment, when determining the acoustic feature corresponding to the second text feature, the conversion module 43 is configured to:

[0136] The second text feature is input into the acoustic model, and based on the nonlinear mapping relationship between the text feature and the acoustic feature in the acoustic model, the second text feature is nonlinearly transformed, and the acoustic feature corresponding to the second text feature is output.

[0137] In a possible embodiment, a training module is further included, wherein the text features of the word dimension are obtained through a word encoding model, and the training module is used to train the word encoding model in the following manner:

[0138] Acquire a training sample set, wherein each training sample in the training sample set includes a word sequence sample and a part-of-speech tag of each word in the word sequence sample;

[0139] Randomly masking some words in the word sequence sample to obtain an input sequence sample;

[0140] Inputting the input sequence sample into the word encoding model to obtain predicted parts of speech of unmasked words and predicted words of masked words in the training sequence sample; and determining a first loss and a second loss of the word encoding model based on the predicted parts of speech of the unmasked words and the predicted words of the masked words;

[0141] Inputting the word sequence sample into the word encoding model, determining prediction information of whether the word sequence sample is a subsequent sentence of a previous word sequence sample input into the word encoding model; and determining a third loss of the word encoding model based on the prediction information;

[0142] The word encoding model is trained according to the first loss, the second loss, and the third loss.

[0143] Each module in the above-mentioned text processing device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0144] Figure 5 A block diagram of an electronic device provided in an embodiment of the present disclosure.

[0145] Reference Figure 5 An embodiment of the present disclosure provides an electronic device, which includes: at least one processor 501; at least one memory 502, and one or more I / O interfaces 503 connected between the processor 501 and the memory 502; wherein the memory 502 stores one or more computer programs that can be executed by the at least one processor 501, and the one or more computer programs are executed by the at least one processor 501 to enable the at least one processor 501 to perform the above-mentioned text processing method.

[0146] Each module in the above-mentioned electronic device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0147] An embodiment of the present disclosure further provides a computer-readable storage medium, which may be a volatile or non-volatile computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned text processing method when executed by a processor.

[0148] An embodiment of the present disclosure also provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned text processing method.

[0149] It will be understood by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium).

[0150] As is well known to those skilled in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, as is well known to those skilled in the art, communication media typically contains computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0151] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0152] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0153] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0154] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0155] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0156] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0157] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0158] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for purposes of limitation. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly indicated, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, it will be understood by those skilled in the art that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A text processing method, characterized in that: include: Extracting features from the text to obtain first text features of the text, where the first text features include text features in a phoneme dimension, text features in a character dimension, and text features in a word dimension; Determining a second text feature of the text based on the first text feature and a positional correspondence between the text feature of the phoneme dimension, the text feature of the character dimension, and the text feature of the word dimension; An acoustic feature corresponding to the second text feature is determined, and a speech corresponding to the text is obtained based on the acoustic feature.

2. The text processing method according to claim 1, characterized in that: Determining the second text feature of the text based on the first text feature and the positional correspondence between the text feature of the phoneme dimension, the text feature of the character dimension, and the text feature of the word dimension includes: Determining a first position mapping relationship between phonemes and characters in the text, and aligning the lengths of the character-dimensional text features and the phoneme-dimensional text features according to the first position mapping relationship to obtain a first character text feature of the text, where the first character text feature has the same length as the phoneme-dimensional text feature; Determining a second position mapping relationship between words and characters in the text, and aligning the length of the text feature in the word dimension with the text feature in the phoneme dimension based on the first position mapping relationship and the second position mapping relationship to obtain a first word text feature of the text, where the first word text feature has the same length as the text feature in the phoneme dimension; The text feature of the phoneme dimension, the first character text feature, and the first word text feature are subjected to a preset operation to obtain a second text feature of the text.

3. The text processing method according to claim 2, characterized in that: The step of aligning the lengths of the text features in the character dimension and the text features in the phoneme dimension according to the first position mapping relationship to obtain the first character text features of the text includes: Determining a first number of phonemes corresponding to each character in the text according to the first position mapping relationship; For each character, when the first number is greater than 1, copying and concatenating the character features of the character so that the second number of the concatenated character features is the same as the first number; The text feature of the first character of the text is determined according to the character features of each concatenated character.

4. The text processing method according to claim 2, wherein: The step of aligning the text features of the word dimension with the text features of the phoneme dimension according to the first position mapping relationship and the second position mapping relationship to obtain the first word text features of the text includes: Determining a third number of phonemes corresponding to each word in the text according to the first position mapping relationship and the second position mapping relationship; For each word, if the third number is greater than 1, copy and concatenate the word features of the word so that the fourth number of the concatenated word features is the same as the third number; The first word text feature of the text is determined according to the word feature of each word after being spliced.

5. The text processing method according to any one of claims 1 to 4, characterized in that: The determining of the acoustic feature corresponding to the second text feature includes: The second text feature is input into the acoustic model, and based on the nonlinear mapping relationship between the text feature and the acoustic feature in the acoustic model, the second text feature is nonlinearly transformed, and the acoustic feature corresponding to the second text feature is output.

6. The text processing method according to claim 1, characterized in that: The text features of the word dimension are obtained through a word encoding model, which is trained in the following way: Acquire a training sample set, wherein each training sample in the training sample set includes a word sequence sample and a part-of-speech tag of each word in the word sequence sample; Randomly masking some words in the word sequence sample to obtain an input sequence sample; Inputting the input sequence sample into the word encoding model to obtain predicted parts of speech of unmasked words and predicted words of masked words in the input sequence sample; and determining a first loss and a second loss of the word encoding model based on the predicted parts of speech of the unmasked words and the predicted words of the masked words; Inputting the word sequence sample into the word encoding model, determining prediction information of whether the word sequence sample is a subsequent sentence of a previous word sequence sample input into the word encoding model; and determining a third loss of the word encoding model based on the prediction information; The word encoding model is trained according to the first loss, the second loss, and the third loss.

7. A text processing device, characterized in that: include: A feature extraction module is used to extract features from a text to obtain first text features of the text, where the first text features include text features in a phoneme dimension, text features in a character dimension, and text features in a word dimension; a determination module, configured to determine a second text feature of the text based on the first text feature and a positional correspondence between the text feature of the phoneme dimension, the text feature of the character dimension, and the text feature of the word dimension; The conversion module is used to determine the acoustic feature corresponding to the second text feature, and obtain the speech corresponding to the text based on the acoustic feature.

8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor. The one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the text processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the text processing method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The invention comprises a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the text processing method according to any one of claims 1 to 6.

Images