Phoneme processing method and device, electronic equipment and computer readable storage medium
By comparing and expanding the phoneme sets, the phoneme mapping relationship is established, and the vocal abnormality caused by the difference in phoneme sets in mixed texts in Chinese and English is solved, and the accuracy and coordination of speech conversion are improved.
Patent Information
- Application Number
- CN202310740416.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-20
- Publication Date
- 2024-12-20
AI Technical Summary
In the phoneme processing of mixed Chinese and English texts, the problem of vocal abnormalities due to different phoneme sets, especially in the conversion between continuous Chinese writing and English Latin character languages, the lack of effective front-end modules leads to poor speech conversion quality.
By comparing the phoneme sets of the two languages, the missing phonemes are determined and expanded, and a similar phoneme mapping relationship is established to ensure the pronunciation accuracy of texts in the two languages when speech conversion is performed.
It improves the pronunciation accuracy of Chinese and English mixed text speech conversion, avoids vocal abnormalities, and ensures the coordination and naturalness of speech conversion results.
Smart Images
Figure CN119169991A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a phoneme processing method, device, electronic device and computer-readable storage medium. Background Art
[0002] TTS (Text-to-Speech) technology has been widely used in human-computer interaction systems such as intelligent voice assistants, car navigation, audio e-books, and voice information announcements in major public places, bringing great convenience to people's lives. Phoneme translation, as an important part of TTS text front-end processing, can obtain linguistic features from the input text and input them to the TTS back-end module to predict acoustic features. Therefore, the quality of phoneme translation has a great impact on the quality of the back-end output voice.
[0003] In today's internationalized world, there are more and more Chinese-English mixed texts that need to be processed. Although the end-to-end speech synthesis model can directly input Latin-based languages such as English into the back-end acoustic feature prediction network to predict acoustic features without the need for a front-end module, Chinese is a continuously written language, and the correlation between Chinese characters and their pronunciation is low. Therefore, to achieve a Chinese mixed TTS with good performance, a front-end module is still necessary. However, since Chinese and English have different phoneme sets, for example, there are phonemes in the Chinese phoneme set that do not exist in the English phoneme set, there is a problem of abnormal pronunciation when processing Chinese-English mixed texts. Summary of the invention
[0004] The present application provides a phoneme processing method, which compares the phoneme sets of two languages to obtain the phonemes missing from one phoneme set compared to the other phoneme set, and then expands the phoneme set with the missing phonemes to avoid abnormal pronunciation problems when converting mixed text to speech.
[0005] In a first aspect, the present application provides a phoneme processing method, the method comprising:
[0006] Obtaining a first phoneme set in a first language and a second phoneme set in a second language;
[0007] Determining similar phonemes of each phoneme in the first phoneme set in the second phoneme set;
[0008] If some target phonemes in the first phoneme set cannot be matched to similar phonemes in the second phoneme set, phonemes matching the target phonemes are expanded in the second phoneme set.
[0009] In a second aspect, the present application further provides a phoneme processing device, the device comprising:
[0010] An acquisition module, used for acquiring a first phoneme set of a first language and a second phoneme set of a second language;
[0011] A determination module, configured to determine similar phonemes of each phoneme in the first phoneme set in the second phoneme set;
[0012] The expansion module is used to expand the phonemes matching the target phonemes in the second phoneme set if some target phonemes in the first phoneme set cannot be matched with similar phonemes in the second phoneme set.
[0013] In a third aspect, the present application also provides an electronic device, characterized in that the electronic device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps in any one of the phoneme processing methods.
[0014] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the steps in any one of the phoneme processing methods.
[0015] The phoneme processing method provided in the present application determines the target phonemes missing from one phoneme set compared to the other phoneme set in the phoneme sets between the two languages, and adds matching phonemes to the phoneme set missing the target phonemes, so that when the text including the two languages is subsequently converted into speech, the pronunciation accuracy of the speech conversion results is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0017] Figure 1 is a schematic diagram of a scenario of a phoneme processing system provided in an embodiment of the present application;
[0018] Figure 2 is a flowchart of an embodiment of the phoneme processing method in the embodiment of the present application;
[0019] Figure 3 is a schematic diagram of a functional module of a phoneme processing device in an embodiment of the present application;
[0020] Figure 4 It is a schematic diagram of the structure of the terminal device in the embodiment of the present application. DETAILED DESCRIPTION
[0021] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.
[0022] In the description of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the feature. In the description of the present application, "plurality" means two or more, unless otherwise clearly and specifically defined.
[0023] In this application, the word "exemplary" is used to mean "used as an example, illustration or description". Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. At the same time, it is to be understood that in the specific implementation of this application, when user information, user data and other related data are involved, when the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data shall comply with relevant laws, regulations and standards of relevant countries and regions.
[0024] In order to enable any person skilled in the art to implement and use the present application, the following description is provided. In the following description, details are listed for the purpose of explanation. It should be understood that those of ordinary skill in the art will recognize that the present application can be implemented without using these specific details. In other examples, known structures and processes will not be elaborated in detail to avoid unnecessary details that make the description of the present application obscure. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest range of principles and features disclosed in the present application.
[0025] The present application provides a phoneme processing method, apparatus, device and storage medium, which are described in detail below.
[0026] See also Figure 1 , Figure 1 Schematic diagram of a phoneme processing system provided in an embodiment of the present application. The phoneme processing system may include an electronic device 100 and a storage device 200. The storage device 200 may transmit data to the electronic device 100. Figure 1 The electronic device 100 in the embodiment can obtain the phoneme data stored in the storage device 200 to execute the phoneme processing method in the present application.
[0027] In the embodiment of the present application, the electronic device 100 includes a terminal device, but is not limited to a desktop computer, a portable computer, a network server, a PDA (Personal Digital Assistant), a tablet computer, a wireless electronic device, an embedded device, etc.
[0028] In addition, the electronic device 100 may also include a server, but is not limited to an independent server, and may also be a server network or server cluster composed of servers, specifically including a computer, a network host, a single network server, a plurality of network server sets or a cloud server composed of multiple servers. Among them, the cloud server can be composed of a large number of computers or network servers based on cloud computing (Cloud Computing).
[0029] In an embodiment of the present application, communication between the electronic device 100 and the storage device 200 can be achieved through any communication method, including but not limited to mobile communications based on the 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), and Worldwide Interoperability for Microwave Access (WiMAX), or computer network communications based on the TCP / IP Protocol Suite (TCP / IP) and User Datagram Protocol (UDP).
[0030] It should be noted that Figure 1 The scenario diagram of the phoneme processing system shown is merely an example. The phoneme processing system and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. A person of ordinary skill in the art can appreciate that with the evolution of the phoneme processing system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is equally applicable to similar technical problems.
[0031] like Figure 2 As shown, Figure 2 This is a flow chart of an embodiment of a phoneme processing method in an embodiment of the present application. The phoneme processing method may include the following steps 201 to 203:
[0032] 201. Obtain a first phoneme set in a first language and a second phoneme set in a second language.
[0033] In the embodiment of the present application, the first language and the second language are different types of languages, for example, the first language can be Chinese, and the second language can be English. It should be noted that the first language of the present application can be not only Chinese, but also any type of language, and the second language can be not only English, but also any type of language different from the first language, which is not limited in the specific embodiment of the present application.
[0034] The method of obtaining the first phoneme set may include: obtaining speech data of the first language, and extracting acoustic features of the speech data of the first language to obtain acoustic features of the first language. Then, the acoustic features of the first language are used as phonemes in the first phoneme set.
[0035] Specifically, if the first language is Mandarin Chinese, 32 phonemes can be extracted: 10 vowels a, o, e, ê, i, u, ü, -i (front i) and -i (back i), er; 22 consonants b, p, m, f, z, c, s, d, t, n, l, zh, ch, sh, r, j, q, x, g, k, h, ng. They can also be divided into four categories:
[0036] (1) One letter represents one phoneme: a, o, e, u, b, p, m, f, d, t, n, l, g, k, h, j, q, x, r, z, c, s;
[0037] (2) One letter represents multiple phonemes: i, -i (front i), -i (back i);
[0038] (3) Two letters represent one phoneme: zh, ch, sh, ng, er;
[0039] (4) One letter plus one symbol represents one phoneme: ê, ü.
[0040] Similarly, for the phoneme set of the second language, the process may include acquiring speech data of the second language, and extracting acoustic features of the speech data of the second language to obtain acoustic features of the second language. Afterwards, the acoustic features of the second language may be used as the phonemes in the second phoneme set. Specifically, if the second language is English, the second phoneme set that can be obtained includes 39 basic phonemes.
[0041] 202. Determine similar phonemes in the second phoneme set for each phoneme in the first phoneme set.
[0042] In an embodiment of the present application, after obtaining the first phoneme set and the second phoneme set, each phoneme in the first phoneme set can be sent to the user or staff of manual identification, for example, the first phoneme A in the first phoneme set is sent to the user or staff, so that the user or staff can determine whether there is a phoneme in the second phoneme set that is the same as the sent phoneme by artificial hearing based on the sent phoneme. If the user or staff determines that there is a phoneme in the second phoneme set that is similar to the first phoneme A based on hearing, the user or staff can feedback that the similar phoneme is a similar phoneme of the first phoneme A; or, when the user or staff determines that there is no phoneme similar to the first phoneme A in the second phoneme set, feedback information that there is no similar phoneme can also be fed back. Therefore, based on the information fed back by the user or staff, it can be determined one by one whether each phoneme in the first phoneme set has a corresponding similar phoneme in the second phoneme set.
[0043] 203. If some target phonemes in the first phoneme set cannot be matched with similar phonemes in the second phoneme set, expand the phonemes matching the target phonemes in the second phoneme set.
[0044] According to the above steps, it can be determined manually whether each phoneme in the first phoneme set has a similar phoneme in the second phoneme set. If, based on the manual feedback results, it is determined that there are some phonemes in the first phoneme set that cannot find similar phonemes in the second phoneme set, it is proved that when the phonemes in the second phoneme set are used for voice conversion, it is impossible to produce a sound including the non-existent similar phonemes. Therefore, it is necessary to expand the phonemes of the second phoneme set. Specifically, for example, the Chinese pinyin "q" does not have a similar phoneme in the English phoneme set. Therefore, when expanding the second phoneme set, a phoneme related to "q" can be added to the English phoneme set and represented by "Q". At the same time, the same pronunciation features of the Chinese "q" can be added to the second phoneme set and associated with "Q".
[0045] In addition, since the above steps can also obtain the result that some phonemes in the first phoneme set have similar phonemes in the second phoneme set. Therefore, the relationship between two phonemes being similar phonemes can be mapped and associated. In other words, some phonemes in the first phoneme set are associated with corresponding similar phonemes in the second phoneme set. In this way, the second phoneme set will eventually become a phoneme set containing all phonemes in Chinese and English, so that Chinese and English pronunciations can be mapped in the same phoneme space. Furthermore, if speech conversion is performed on the text including Chinese and English later, the pronunciation accuracy of the converted speech can be made higher.
[0046] The phoneme processing method provided in the present application determines the target phonemes missing from one phoneme set compared to the other phoneme set in the phoneme sets between the two languages, and adds matching phonemes to the phoneme set missing the target phonemes, so that when the text including the two languages is subsequently converted into speech, the pronunciation accuracy of the speech conversion results is improved.
[0047] In order to better implement the embodiment of the present application, in one embodiment of the present application, each phoneme in the first phoneme set includes a corresponding first pronunciation feature, each phoneme in the second phoneme set includes a corresponding second pronunciation feature, and determining a similar phoneme of each phoneme in the first phoneme set in the second phoneme set includes:
[0048] Taking each first pronunciation feature as the target first pronunciation feature, determine the similarity between the target first pronunciation feature and each second pronunciation feature to obtain the highest similarity of the target first pronunciation feature; if the highest similarity is greater than a preset similarity threshold, determine the phoneme of the second pronunciation feature corresponding to the highest similarity as a similar phoneme of the phoneme of the target first pronunciation feature.
[0049] The above embodiment provides a method for manually determining whether each phoneme in the first phoneme set has similar phonemes in the second phoneme set. In order to improve the determination efficiency and save labor costs, this embodiment provides a method for manually determining whether each phoneme in the first phoneme set has similar phonemes in the second phoneme set.
[0050] Specifically, a trained similarity determination model can be set, and the similarity model is obtained by training each phoneme in the second phoneme set. When using the similarity determination model, each phoneme in the first phoneme set can be gradually input into the similarity determination model, and the phonemes of the first phoneme set input into the model are classified. Assuming that the second phoneme set is an English phoneme set, according to the description of the above English phoneme set, the English phoneme set can include 39 phonemes. Therefore, the similarity determination model can be set with 40 classifications. At this time, the specific workflow of the similarity determination model includes, when receiving the phoneme of the input first phoneme set, the input phoneme is calculated with 39 preset phonemes, and the preset 39 phonemes are each phoneme in the second phoneme set. Specifically, the similarity can be determined by calculating the Euclidean distance or cosine similarity between two phoneme features. In this way, 39 similarities can be obtained. At this time, the highest similarity among the 39 similarities can be obtained, and the highest similarity is compared with the similarity threshold. If it exceeds the similarity threshold, the classification result corresponding to the highest similarity is determined and output; if it does not exceed the similarity threshold, the classification result that no similar phonemes exist is output.
[0051] In order to better implement the embodiment of the present application, in one embodiment of the present application, each phoneme in the first phoneme set includes a corresponding first pronunciation feature, each phoneme in the second phoneme set includes a corresponding second pronunciation feature, and the phonemes matching the target phonemes are expanded in the second phoneme set, including:
[0052] According to each second pronunciation feature, a synthesized pronunciation feature similar to the target pronunciation feature is synthesized; and the synthesized pronunciation feature is expanded to the second phoneme set.
[0053] The above embodiment provides a solution of directly adding the phonemes of the first language into the second phoneme set. However, this adding method will cause the pronunciation to be unnatural when the text-to-speech conversion is performed later. For example, when the phonemes in English that lack Chinese pronunciation are directly added to the English phoneme set, when the text-to-speech conversion is performed according to the expanded phoneme set, the English speech produced will have Chinese sounds, resulting in the problem of incoordination.
[0054] Therefore, in order to solve this inconsistency problem, when it is determined that a target phoneme in the first phoneme set does not have a similar phoneme in the second phoneme set, each phoneme in the second phoneme set can be randomly synthesized, and the similarity between the synthesized phoneme and the target phoneme can be calculated. If the similarity threshold is exceeded, the synthesized phoneme is expanded to the second phoneme set and a mapping relationship is established with the target phoneme; or the phonemes in the second phoneme set are analyzed for sound features to determine which phonemes are similar to the target phoneme after synthesis, and the synthesized similar phonemes are added to the second phoneme set and a mapping relationship is established with the target phoneme. The synthesized phonemes obtained by synthesis based on the original phonemes in the second phoneme set will make the pronunciation sound closer to the second language, thus alleviating the inconsistency problem after text is converted to speech.
[0055] It should be noted that if the similarity between the synthesized phonemes obtained after exhaustive random synthesis and the target phonemes cannot exceed the similarity threshold; or if a synthesized phoneme similar to the target phoneme cannot be obtained after phoneme analysis, then you can choose to directly expand the target phoneme into the second phoneme set.
[0056] In order to better implement the embodiment of the present application, in one embodiment of the present application, after expanding the phonemes matching the target phonemes in the second phoneme set, the method further includes:
[0057] A mapping relationship is established between each phoneme in the first phoneme set and a similar phoneme in the second phoneme set, so that each phoneme in the first phoneme set corresponds to a phoneme in the expanded second phoneme set; if a target text including the first language and / or the second language is obtained, the target text is converted into speech according to the phonemes in the expanded second phoneme set and the mapping relationship.
[0058] According to the above embodiment, when the second phoneme set is expanded, a mapping relationship can be established between two similar phonemes. In the embodiment of the present application, a mapping relationship can also be established between two similar phonemes after similar phonemes are determined.
[0059] After the mapping relationship is established, the target text can be converted to speech according to the phonemes in the expanded second phoneme set and the mapping relationship. The conversion process can specifically include: obtaining the target text; performing language detection on the target text. If the target text only includes the first language, it can be determined based on the expanded second phoneme set and the mapping relationship which phonemes in the second phoneme set can be used for the speech conversion of the first language. For example: the target text includes a word X in the first language, and based on the first phoneme set, it can be determined which specific first phonemes constitute the pronunciation of X, and then at this time, it can be determined according to the mapping relationship which specific second phonemes in the second phoneme set can constitute the pronunciation of X, and the speech conversion can be realized. If the target text also includes a second language, the speech conversion of the second language can be directly converted through the expanded second phoneme set, so the converted speech is converted from the same phoneme set, so that the speech obtained after the conversion can be more coordinated.
[0060] In order to better implement the embodiment of the present application, in one embodiment of the present application, before performing speech conversion on the target text according to the phonemes and mapping relationships in the expanded second phoneme set, the method further includes:
[0061] A sample text including a first language and / or a second language is obtained; each character in the sample text is annotated phonemically to obtain a phoneme annotation sequence; the speech category included in the sample text is determined; if the sample text includes the first language and the second language, the characters in the sample text belonging to the first language are annotated with the first language category, and the characters in the sample text belonging to the second language are annotated with the second language category to obtain a language category annotation sequence of the sample text; if the sample text includes the first language, the characters in the sample text belonging to the first language are annotated with the first language category to obtain a language category annotation sequence of the sample text; if the sample text includes the second language, the characters in the sample text belonging to the second language are annotated with the second language category to obtain a language category annotation sequence of the sample text; and a speech conversion model for speech conversion is trained according to the sample text, the phoneme annotation sequence and the language category annotation sequence.
[0062] In the embodiments of the present application, the sample text may be a training text provided manually, or may be a training text obtained from public data, and the specific embodiments of the present application are not limited thereto.
[0063] In addition, since the pronunciation features of Chinese also include tones, namely, the high level tone (first tone), the rising tone (second tone), the falling-rising tone (third tone), and the falling tone (fourth tone), in the present invention, the "light tone" and the "r-coloring" are also regarded as tones. Therefore, during the annotation process, the six different tones can be annotated in sequence with the Arabic numerals "1, 2, 3, 4, 5, 0", that is, the high level tone (1), the rising tone (2), the falling-rising tone (3), the falling tone (4), the light tone (5), and the r-coloring (0). It should be noted that starting from the first phoneme for which the tone needs to be annotated, each subsequent phoneme needs to be annotated with a tone. In particular, the annotation "0" for the r-coloring is only annotated after the last phoneme for which the tone of the Chinese pinyin needs to be indicated, and is combined with the tone annotation of the last phoneme, that is, in the present invention, it is default that the annotation number "0" for the r-coloring is meaningful only when combined with the annotation numbers of other tones. For example, when the sample text includes the first language "Tonghuamen", the pronunciation of this "Tonghuamen" can be "tong hua mener", so based on the preset annotation rules, "Tonghuamen" can be annotated. Based on this, the phoneme annotation sequence obtained for "Tonghuamen" can be "t o 1ng 1h u a 4m e2n 20". Therefore, the embodiments of the present application can annotate the r-coloring in Chinese.
[0064] However, if the sample text also includes English, there are stress tones in English. And the annotation symbols for English stress, Nostress(0), Primary stress(1), Secondary stress(2), are repeated with the tone labels in Chinese, so that when the obtained phoneme sequence is used as the input for the TTS back-end processing, the back-end model cannot distinguish whether "0", "1", "2" are Chinese tones or English stress. Therefore, it is also necessary to perform language category annotation on the text in the text to distinguish different languages. And the embodiments of the present application can use "1" to indicate that the corresponding phoneme or annotation symbol comes from Chinese, and use "2" to indicate that the corresponding phoneme or annotation symbol comes from English. For example: for the phrase "Chinese and English text", after phoneme translation, a phoneme sequence and a language annotation sequence will be obtained. Assuming the phoneme sequence is "ZH UH 1NG 1Y IY 1NG 1W EH 2N 2T EH 1K S T", then the corresponding language annotation sequence is "1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 22", and the lengths of the two sequences are the same, thus obtaining the language category annotation sequence.
[0065] After obtaining the phoneme annotation sequence and the language category annotation sequence, the sample text together with the phoneme annotation sequence and the language category annotation sequence can be input into the speech conversion model for training, so that the speech conversion model can perform speech conversion based on the text. In the embodiment of the present application, a supervised training method can be adopted to train the speech conversion model.
[0066] In order to better implement the embodiment of the present application, in one embodiment of the present application, the first language category is marked for the text in the sample text belonging to the first language, and the second language category is marked for the text in the sample text belonging to the second language, so as to obtain a language category mark sequence of the sample text, including:
[0067] Determine non-text symbols in the sample text; annotate the non-text symbols with symbol labels to obtain labels of the non-text symbols; based on the labels of the non-text symbols, annotate the text belonging to the first language in the sample text with the first language category, and annotate the text belonging to the second language in the sample text with the second language category, to obtain a language category annotation sequence including labels of the non-text symbols.
[0068] Since there are punctuation marks, i.e. non-text symbols, in the text used for training. In order to avoid the influence of non-text symbols on the subsequent speech conversion. Therefore, it is also necessary to mark the non-text symbols in the sample text. For example, "0" is used to indicate that the corresponding sequence content is a symbol that is independent of language, such as punctuation marks. Specifically, the description of the above embodiment is continued. For example: for the short phrase "Chinese and English text." in the sample text, a phoneme sequence and a language tag sequence will be obtained after phoneme translation. Assuming that the phoneme sequence is "ZH UH 1NG 1Y IY 1NG 1W EH 2N 2T EH 1K S T.", the corresponding language tag sequence is "1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 0", and the lengths of the two sequences are the same. It can be seen that the label 0 here corresponds to the "." in "Chinese and English text.". Therefore, the label of non-text symbols can be included in the language category annotation sequence. Furthermore, after the speech conversion model is trained, the influence of non-text symbols on the speech conversion can be avoided.
[0069] In order to better implement the embodiments of the present application, in one embodiment of the present application, obtaining a sample text including a first language and / or a second language includes:
[0070] An initial sample text including a first language and / or a second language is obtained; if the initial sample text includes a first character in the first language, the first character is converted into a target first text format; if the initial sample text includes a second character in the second language, the second character is converted into a target second text format; the initial sample text that has completed the format conversion is regarded as a sample text including the first language and / or the second language.
[0071] When obtaining sample texts input manually or obtained from public data for training, the text format is usually not uniform. Therefore, in order to enable the model to distinguish which words in the sample text are from the first language and which words are from the second language, it is necessary to format the words in the initial sample text, including the following methods:
[0072] When an initial sample text including a first language and / or a second language is obtained, text processing can be performed on the first language and the second language in the sample text. For example, in the embodiment of the present application, if the first text is Chinese, it is necessary to convert the numbers and symbols appearing in the text of the first text into corresponding text formats according to the actual scenario, including converting Arabic numerals in Chinese into Chinese arrays, specifically, in Chinese text, "23:30" is converted into "twenty-three point thirty", "123 students" is converted into "one hundred and twenty-three students", and "the number is 5283" is converted into "the number is five two eight three"; and if the second text is English, Arabic numerals in English can also be converted into English numerals, specifically, in English text, "5 dollars" is converted into "five dollars", and "No.1" is converted into "number one". The purpose of this is that when labeling the text in the text, it is possible to distinguish which text in the text comes from the first language and which comes from the second language, which can facilitate labeling.
[0073] After obtaining each labeled sequence, the two are converted into vectors and used as input to the TTS backend acoustic model to predict acoustic features for speech conversion.
[0074] In order to better implement the phoneme processing method in the embodiment of the present application, in addition to the phoneme processing method, the embodiment of the present application also provides a phoneme processing device, such as Figure 3 As shown, the device 300 includes:
[0075] An acquisition module 301 is used to acquire a first phoneme set in a first language and a second phoneme set in a second language;
[0076] A determination module 302, configured to determine similar phonemes of each phoneme in the first phoneme set in the second phoneme set;
[0077] The expansion module 303 is configured to expand the phonemes matching the target phonemes in the second phoneme set if some of the target phonemes in the first phoneme set cannot be matched to similar phonemes in the second phoneme set.
[0078] The phoneme processing device provided in the present application first obtains the phoneme sets of two languages through the acquisition module 301, and then determines the target phonemes missing in one phoneme set compared to the other phoneme set in the phoneme sets between the two languages through the determination module 302, and adds matching phonemes to the phoneme set missing the target phonemes through the expansion module 303, so that when the text including the two languages is subsequently converted into speech, the pronunciation accuracy of the speech conversion result is improved.
[0079] In some embodiments of the present application, the determination module 302 is specifically used to:
[0080] Taking each first pronunciation feature as a target first pronunciation feature, determining the similarity between the target first pronunciation feature and each second pronunciation feature, and obtaining the highest similarity of the target first pronunciation feature;
[0081] If the highest similarity is greater than a preset similarity threshold, the phoneme of the second pronunciation feature corresponding to the highest similarity is determined to be a similar phoneme of the phoneme of the target first pronunciation feature.
[0082] In some embodiments of the present application, the expansion module 303 is specifically used to:
[0083] synthesizing a synthesized pronunciation feature similar to the target pronunciation feature according to each second pronunciation feature;
[0084] The synthesized pronunciation features are expanded to a second phoneme set.
[0085] In some embodiments of the present application, the phoneme processing device further includes a speech conversion module 304, and the speech conversion module 304 is specifically used to:
[0086] Establishing a mapping relationship between each phoneme in the first phoneme set and a similar phoneme in the second phoneme set, so that each phoneme in the first phoneme set corresponds to a phoneme in the expanded second phoneme set;
[0087] If a target text including the first language and / or the second language is obtained, the target text is converted into phonetic text according to the phonemes in the expanded second phoneme set and the mapping relationship.
[0088] In some embodiments of the present application, the phoneme processing device further includes a training module 305, and the training module 305 is specifically used to:
[0089] Obtaining sample texts including first language and / or second language;
[0090] Perform phoneme annotation on each word in the sample text to obtain a phoneme annotation sequence;
[0091] determining the speech categories included in the sample text;
[0092] If the sample text includes a first language and a second language, the first language category is marked for the characters in the sample text that belong to the first language, and the second language category is marked for the characters in the sample text that belong to the second language, to obtain a language category mark sequence for the sample text;
[0093] If the sample text includes the first language, the first language category is marked for the characters in the sample text that belong to the first language, so as to obtain a language category mark sequence of the sample text;
[0094] If the sample text includes a second language, the second language category is marked for the characters in the sample text that belong to the second language, so as to obtain a language category mark sequence for the sample text;
[0095] A speech conversion model for speech conversion is trained based on sample text, phoneme annotation sequences, and language category annotation sequences.
[0096] In some embodiments of the present application, the training module 305 is further used to:
[0097] Identify non-text symbols in sample text;
[0098] Label the non-text symbols with symbol labels to obtain labels of the non-text symbols;
[0099] Based on the labels of non-text symbols, the characters in the sample text belonging to the first language are annotated with the first language category, and the characters in the sample text belonging to the second language are annotated with the second language category, so as to obtain a language category annotation sequence including the labels of non-text symbols.
[0100] In some embodiments of the present application, the training module 305 is further used to:
[0101] obtaining an initial sample text including a first language and / or a second language;
[0102] If the initial sample text includes a first character in a first language, converting the first character into a target first text format;
[0103] If the initial sample text includes a second text in a second language, converting the second text into a target second text format;
[0104] The initial sample text that has completed the format conversion is regarded as a sample text including the first language and / or the second language.
[0105] The present application also provides an electronic device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in any one of the phoneme processing methods in the present application. The electronic device integrates any one of the phoneme processing methods provided in the present application. Figure 4 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:
[0106] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that Figure 4 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0107] The processor 401 is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, the processor 401 performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 401 may include one or more processing cores; the processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. Preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, the user interface and the application program, etc., and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.
[0108] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.
[0109] The electronic device also includes a power supply 403 for supplying power to each component. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 403 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.
[0110] The electronic device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.
[0111] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby realizing various functions, such as:
[0112] Obtaining a first phoneme set in a first language and a second phoneme set in a second language;
[0113] Determine similar phonemes in the second phoneme set for each phoneme in the first phoneme set;
[0114] If some target phonemes in the first phoneme set cannot be matched to similar phonemes in the second phoneme set, phonemes matching the target phonemes are expanded in the second phoneme set.
[0115] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0116] To this end, the embodiment of the present application provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any of the phoneme processing methods provided in the embodiment of the present application. For example, the computer program is loaded by the processor to execute the following steps:
[0117] Obtaining a first phoneme set in a first language and a second phoneme set in a second language;
[0118] Determine similar phonemes in the second phoneme set for each phoneme in the first phoneme set;
[0119] If some target phonemes in the first phoneme set cannot be matched to similar phonemes in the second phoneme set, phonemes matching the target phonemes are expanded in the second phoneme set.
[0120] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the detailed description of other embodiments above, and will not be repeated here.
[0121] In specific implementation, the above units or structures can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units or structures can refer to the previous method embodiments, which will not be repeated here.
[0122] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0123] The above is a detailed introduction to a phoneme processing method and device provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A phoneme processing method, characterized in that: The method comprises: Obtaining a first phoneme set in a first language and a second phoneme set in a second language; Determining similar phonemes of each phoneme in the first phoneme set in the second phoneme set; If some target phonemes in the first phoneme set cannot be matched to similar phonemes in the second phoneme set, phonemes matching the target phonemes are expanded in the second phoneme set.
2. The phoneme processing method according to claim 1, characterized in that: Each phoneme in the first phoneme set includes a corresponding first pronunciation feature, each phoneme in the second phoneme set includes a corresponding second pronunciation feature, and determining a similar phoneme of each phoneme in the first phoneme set in the second phoneme set includes: Taking each first pronunciation feature as a target first pronunciation feature, determining the similarity between the target first pronunciation feature and each of the second pronunciation features, and obtaining the highest similarity of the target first pronunciation feature; If the highest similarity is greater than a preset similarity threshold, the phoneme of the second pronunciation feature corresponding to the highest similarity is determined to be a similar phoneme of the phoneme of the target first pronunciation feature.
3. The phoneme processing method according to claim 1, characterized in that: Each phoneme in the first phoneme set includes a corresponding first pronunciation feature, each phoneme in the second phoneme set includes a corresponding second pronunciation feature, and the phonemes in the second phoneme set that are expanded to match the target phonemes include: synthesizing a synthesized pronunciation feature similar to the target pronunciation feature according to each of the second pronunciation features; The synthesized pronunciation features are expanded to the second phoneme set.
4. The phoneme processing method according to claim 1, characterized in that: After expanding the second phoneme set with the phonemes matching the target phonemes, the method further includes: Establishing a mapping relationship between each phoneme in the first phoneme set and a similar phoneme in the second phoneme set, so that each phoneme in the first phoneme set corresponds to a phoneme in the expanded second phoneme set; If a target text including the first language and / or the second language is obtained, the target text is converted into phonetic language according to the phonemes in the expanded second phoneme set and the mapping relationship.
5. The phoneme processing method according to claim 4, characterized in that: Before performing speech conversion on the target text according to the phonemes in the expanded second phoneme set and the mapping relationship, the method further includes: obtaining a sample text including the first language and / or the second language; Performing phoneme annotation on each character in the sample text to obtain a phoneme annotation sequence; Determining the speech category included in the sample text; If the sample text includes the first language and the second language, perform a first language category tag on the characters in the sample text that belong to the first language, and perform a second language category tag on the characters in the sample text that belong to the second language, to obtain a language category tag sequence for the sample text; If the sample text includes the first language, marking the characters in the sample text that belong to the first language with the first language category to obtain a language category marking sequence for the sample text; If the sample text includes the second language, marking the second language category of the characters in the sample text that belong to the second language to obtain a language category marking sequence of the sample text; A speech conversion model for speech conversion is trained according to the sample text, the phoneme annotation sequence, and the language category annotation sequence.
6. The phoneme processing method according to claim 5, characterized in that: The step of marking the first language category of the characters in the sample text that belong to the first language, and marking the second language category of the characters in the sample text that belong to the second language, to obtain a language category marking sequence of the sample text, includes: determining non-text symbols in the sample text; Labeling the non-text symbols with symbol labels to obtain labels of the non-text symbols; Based on the labels of the non-text symbols, the characters in the sample text belonging to the first language are annotated with a first language category, and the characters in the sample text belonging to the second language are annotated with a second language category, so as to obtain a language category annotation sequence including the labels of the non-text symbols.
7. The phoneme processing method according to claim 5, characterized in that: The obtaining of the sample text in the first language and / or the second language comprises: Acquiring an initial sample text including the first language and / or the second language; If the initial sample text includes a first character in the first language, converting the first character into a target first text format; If the initial sample text includes a second character in the second language, converting the second character into a target second text format; The initial sample text that has completed format conversion is regarded as the sample text including the first language and / or the second language.
8. A phoneme processing device, characterized in that: The device comprises: An acquisition module, used for acquiring a first phoneme set of a first language and a second phoneme set of a second language; A determination module, configured to determine similar phonemes of each phoneme in the first phoneme set in the second phoneme set; The expansion module is used to expand the phonemes matching the target phonemes in the second phoneme set if some target phonemes in the first phoneme set cannot be matched with similar phonemes in the second phoneme set.
9. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps of the phoneme processing method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in the phoneme processing method according to any one of claims 1 to 7.