Bilingual speech synthesis method and device, computer equipment and storage medium
By acquiring and processing bilingual text features, language features and phonological features, and using feature intensity modulators to generate target bilingual pronunciation, the problem of low intonation accuracy in bilingual pronunciation synthesis is solved, and smoother and more accurate pronunciation synthesis is achieved, improving customer interaction and satisfaction.
Patent Information
- Application Number
- CN202411721130.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-27
- Publication Date
- 2025-03-04
AI Technical Summary
When performing bilingual pronunciation synthesis, the intonation accuracy of simulated second language pronunciation is poor, resulting in the inverse of synthetic pronunciation.
By obtaining the bilingual text features, first language features and first phonological features of the to be processed text, these features are processed using a feature intensity modulator to obtain the target language features and target phonological features, and finally decoding processes are performed to generate the target bilingual voice.
It improves the accuracy of the synthesized voice tones in different scenarios, makes the generated bilingual synthesized voice smoother, improves interactivity with customers, reduces the work burden of customer service personnel and improves customer satisfaction.
Smart Images
Figure CN119559927A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a bilingual speech synthesis method, device, computer equipment and storage medium. Background Art
[0002] In recent years, with the development of natural language processing technology NLP and speech synthesis technology TTS (Text-to-Speech), the use of intelligent voice customer service in online customer service in the financial field has become popular. TTS (Text-to-Speech) is a speech synthesis technology that can convert raw text into synthesized speech. This technology uses artificial intelligence (AI) to convert text into highly anthropomorphic spoken language. For example, through the use of speech synthesis technology for financial customer service, financial market analysis, and finance, it can reduce labor costs and meet the financial field's demand for timely information.
[0003] However, the synthesized speech generated by the current intelligent voice customer service still has the problem of not being humanized enough, especially when there are two or more languages in the original text of the speech to be synthesized. Since speakers often use one language, the pronunciation and intonation of the second language are usually very different from the first language. On the one hand, there are more monolingual corpora used to train speech synthesis models, and mixed language corpora with multiple languages are relatively scarce; on the other hand, it is easy to cause the trained speech synthesis model to have residual stress or inconsistent speech between languages when synthesizing bilingual speech, and it is impossible to avoid mutual interference between the two languages, and it is impossible to accurately simulate the speech and intonation of the second language in different contexts. Therefore, the existing speech synthesis model has the problem of poor accuracy in simulating the intonation of the second language speech when performing bilingual speech synthesis. Summary of the invention
[0004] The embodiments of the present invention provide a bilingual speech synthesis method, apparatus, computer equipment and storage medium, which solve the problem that the existing speech synthesis model has a poor accuracy rate in simulating the intonation of the second language speech when performing bilingual speech synthesis.
[0005] The embodiment of the present invention provides a bilingual speech synthesis method, comprising: Acquire a text to be processed, wherein the text to be processed includes text content corresponding to a first language and text content corresponding to a second language; Encoding the text to be processed to obtain bilingual text features corresponding to the text to be processed; Acquire a first language feature and a first phonetic feature; the first language feature is a language feature between the first language and the second language, and the first phonetic feature is a phonetic feature between the first language and the second language; Using a feature intensity modulator, the bilingual text feature and the first language feature are processed to obtain a target language feature; the bilingual text feature and the first phonological feature are processed to obtain a target phonological feature; The bilingual text features, the target language features and the target phonological features are decoded to obtain the target bilingual speech corresponding to the text to be processed.
[0006] The embodiment of the present invention further provides a bilingual speech synthesis device, comprising: A module for acquiring text to be processed, used for acquiring text to be processed, wherein the text to be processed includes text content corresponding to the first language and text content corresponding to the second language; A bilingual text feature acquisition module, used to encode the text to be processed and acquire bilingual text features corresponding to the text to be processed; A first language feature and a first phonological feature acquisition module, used to acquire a first language feature and a first phonological feature; the first language feature is a language feature between the first language and the second language, and the first phonological feature is a phonological feature between the first language and the second language; A target language feature and target phonological feature acquisition module, used to process the bilingual text feature and the first language feature using a feature intensity modulator to acquire a target language feature; and to process the bilingual text feature and the first phonological feature to acquire a target phonological feature; The target bilingual speech acquisition module is used to decode the bilingual text features, the target language features and the target phonological features to acquire the target bilingual speech corresponding to the text to be processed.
[0007] An embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for synthesizing bilingual speech when executing the computer program.
[0008] The embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the bilingual speech synthesis method is implemented.
[0009] The above-mentioned bilingual speech synthesis method, device, computer equipment and storage medium encode the text to be processed corresponding to the text content of the first speech and the second language to obtain the bilingual text features; according to the first language and the second language, the first language features and the first phonological features of the bilingual are obtained to capture the difference in speech expression features between different language phonological systems; a feature intensity modulator is used to process the bilingual text features, the first language features and the first phonological features respectively to obtain the corresponding target language features and target phonological features, which are used to capture the dynamic intensity of language and speech and adjust the phonological control; finally, decoding is performed according to the bilingual text features, the target language features and the target phonological features to obtain the synthesized target bilingual speech. By adding phonological features and combining with a feature intensity modulator, the expression features between the two languages are captured to adjust the phonological features, reduce the problem of insufficient accuracy of the trained model caused by the small number of mixed corpora, and thus improve the accuracy of the synthesized speech tone in different scenarios. Furthermore, when applied in the financial field, the generated bilingual synthesized speech is more fluent, improves the interactivity with customers, reduces the workload of customer service personnel and improves customer satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0011] Figure 1 is a schematic diagram of an application environment of a bilingual speech synthesis method in one embodiment of the present invention; Figure 2 is a flow chart of a method for synthesizing bilingual speech in one embodiment of the present invention; Figure 3 is a flow chart of step S202 of a method for synthesizing bilingual speech in one embodiment of the present invention; Figure 4 is a flow chart of step S204 of a method for synthesizing bilingual speech in one embodiment of the present invention; Figure 5 is a flow chart of step S204 of a method for synthesizing bilingual speech in one embodiment of the present invention; Figure 6 is a flow chart of step S205 of a method for synthesizing bilingual speech in one embodiment of the present invention; Figure 7 is a flow chart of step S601 of a method for synthesizing bilingual speech in one embodiment of the present invention; Figure 8is a flow chart of step S602 of a method for synthesizing bilingual speech in one embodiment of the present invention; Fig. 9 is a schematic diagram of a bilingual speech synthesis device in one embodiment of the present invention; Fig.10 is a schematic diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0012] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0013] The bilingual speech synthesis method provided by the embodiment of the present invention can be applied in Figure 1 In the application environment shown. Figure 1 As shown, the client (computer device) communicates with the server through the network. The client, also known as the user end, refers to the program corresponding to the server that provides local services to the client. The client (computer device) includes but is not limited to various personal computers, laptops, smart phones, tablets, cameras and portable wearable devices. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.
[0014] The bilingual speech synthesis method provided by the embodiment of the present invention can be applied as follows: Figure 1 Specifically, the bilingual speech synthesis method is applied in a bilingual speech synthesis system, and the bilingual speech synthesis system includes: Figure 1 The client and server shown communicate with each other via a network to implement the bilingual speech synthesis method provided in an embodiment of the present invention.
[0015] It can be understood that the bilingual speech synthesis method provided in the embodiment of the present invention is to input the text to be processed into a trained speech synthesis model, and after being processed by the trained speech synthesis model, synthesize and output the corresponding target bilingual speech. The training of the speech synthesis model is specifically selected according to actual business needs.
[0016] Among them, TTS (Text-to-Speech) is a bilingual text-to-speech synthesis technology that can convert text input into corresponding speech output, realizing the function of the computer automatically converting text into speech. In this example, a multilingual speech synthesis (TTS) model based on Tacotron2 is used. Tacotron2 is a text-to-speech synthesis model based on deep learning. Unlike the traditional Tacotron2 speech synthesis model, the bilingual speech synthesis (TTS) model can support two or more languages at the same time, so that the synthesized speech can generate more fluent synthesized speech when multiple languages appear.
[0017] Furthermore, after combining with the natural language processing model, it is applied to online customer service, online business reminders and other businesses in the financial field. Natural language processing technology is used to process the text content sent by customers or the text content corresponding to the spoken voice, complete the identification of customer needs, and thus generate the corresponding text content, and generate synthetic voice through TTS technology to intelligently complete online customer services.
[0018] The bilingual speech synthesis model can meet the degree of anthropomorphism of the synthesized speech in more scenarios. For example, when "a certain company conducted a certain business through an Initial Public Offering (IPO)", the bilingual speech synthesis model can avoid the interference of Chinese (first language) on the English (second language) intonation, thereby synthesizing speech with more accurate intonation.
[0019] In one embodiment, if Figure 2 As shown, a method for synthesizing bilingual speech is provided, and the method is applied in Figure 1 The server in the example is used as an example to illustrate the process, including the following steps: S201: Acquire a text to be processed, where the text to be processed includes text content corresponding to a first language and text content corresponding to a second language; S202: Encoding the text to be processed to obtain bilingual text features corresponding to the text to be processed; S203: Acquire a first language feature and a first phonological feature; the first language feature is a language feature between the first language and the second language, and the first phonological feature is a phonological feature between the first language and the second language; S204: using a feature intensity modulator, processing the bilingual text feature and the first language feature to obtain a target language feature; processing the bilingual text feature and the first phonological feature to obtain a target phonological feature; S205: Decoding the bilingual text features, the target language features and the target phonological features to obtain the target bilingual speech corresponding to the text to be processed.
[0020] The first language and the second language are not limited, and the number is not limited either, and is limited according to the actual business needs of the text to be processed. For example, if the text to be processed exists in three languages, the first language is Chinese, the second language is English, and there is also a third language, French, then the text to be processed includes the content of the above three languages.
[0021] As an example, in step S201, after receiving the text to be processed input by the natural language processing model, artificial intelligence, or user, the server confirms the language text in the text to be processed, and confirms the corresponding first language and second language according to the different language texts. In this example, there is a first language of Chinese and a second language of English. According to actual business needs, the first language can be used as the main language and the second language can be used as the secondary language to train the speech generation model.
[0022] In this example, the text to be processed may include text content such as speech, questions, and conclusions in two languages in financial customer service. In this example, common professional terms in the financial field include but are not limited to SPAC (Specialpurpose acquisition company), which means a special purpose acquisition company. Usually in financial business communication, it refers to a shell company with no business operations, which acquires other companies after raising funds through listing; YOLO (You only live once) means that you only live once. Usually in financial business communication, it is used to describe the mentality of investors in risky investments.
[0023] As an example, in step S202, after receiving the text to be processed, the server uses an encoder to convert the text to be processed, and uses the attention mechanism in the encoder to capture the deeper semantics of the text to be processed and convert it into bilingual text features.
[0024] In this example, the encoder used in the multilingual speech synthesis model based on Tacotron2 is the Attentional Bridge Encoder, which converts the input text to be processed into high-level semantic bilingual text features, and uses the attention mechanism to capture the relationship and important information between different parts of the input text. The output of the attention bridge encoder is passed to the decoder to help the decoder generate the corresponding target bilingual features.
[0025] In this example, for example, the text to be input is an example conversation in a financial business: "Customer A is in the YOLO state and plans to invest in a SPAC-related project. The customer is in a high-risk state and is advised to consider it carefully." By inputting the text in the above text into the encoder, the encoder is used to capture the features of the text to be processed, and the target bilingual features corresponding to the above conversation are generated.
[0026] As an example, in step S203, the server receives a first language feature and a first phonetic feature, wherein the first language feature is a language feature between the first language and the second language, that is, it has language features of the first language and the second language; the first phonetic feature is a phonetic feature between the first language and the second language, that is, it has phonetic features of the first language and the second language.
[0027] In this example, language features are the conversion of languages (such as English, Chinese, French, etc.) into continuous vector representations, that is, the conversion of languages (such as English, Chinese, French, etc.) into corresponding language features. Language features are used to represent the language information of the text to be processed, and to help the text-to-speech (TTS) model distinguish between different languages, so as to perform targeted multilingual speech synthesis and achieve accurate multilingual speech synthesis.
[0028] For example, the preset language features of Chinese (first language) corresponding to Chinese and English (second language) corresponding to English are used in the speech synthesis model to process the text to be processed including Chinese (first language) corresponding to Chinese and English (second language) corresponding to English. The language features of Chinese (first language) corresponding to Chinese and English (second language) corresponding to English are used to distinguish the pronunciation differences of the two languages in monolingual and embedded languages.
[0029] In this example, a phoneme is the smallest pronunciation unit in speech, which can be a syllable or a sound, and the phonological feature is the vector representation of the converted phoneme information. The phonological feature is used to represent the phoneme information of the text to be processed, that is, the acoustic features of the speech. By using the phonological feature, the TTS model can better capture the acoustic information in the text to generate natural and fluent speech output.
[0030] Furthermore, in multilingual speech synthesis, by introducing multilingual phonological features to capture the pronunciation differences between different phonological systems, better performance can be achieved in multilingual scenarios without the need for a large number of mixed language corpora to train the speech synthesis model. For example, in the example dialogue "Customer A plans to invest in a SPAC-related project with the status of YOLO. This customer is in a high-risk state and is recommended to be carefully considered", the language features of "Customer A plans to invest in a related project with the status of YOLO. This customer is in a high-risk state and is recommended to be carefully considered" in Chinese (first language) and the language features of "YOLO, SPAC" in English (second language) are obtained respectively, and the pre-captured phonological features with Chinese and English pronunciation differences are obtained for subsequent bilingual speech synthesis.
[0031] As an example, in step S204, after receiving the first language feature and the first phonetic feature, the server uses a feature intensity modulator to capture the dynamic intensity of the first language feature and the first phonetic feature in combination with the bilingual text features corresponding to the text to be processed, and obtains the processed target language feature and target phonetic feature.
[0032] In this example, the feature intensity modulator consists of two sub-networks, a multi-head attention network and a feedforward network. Both sub-networks use layer normalization and residual connections to further capture the dynamic components of language and phonology (pronunciation). The static components are mainly used to represent the short-term characteristics of speech, such as spectrum and basic pitch, while the dynamic components are used to represent the temporal changes and dynamic properties of speech, thereby providing a more comprehensive and rich description of speech signals and helping to achieve a more natural and fluent speech synthesis effect.
[0033] As an example, in step S205, the server fuses and decodes the bilingual text features, target language features, and target phonological features according to the target language features and target phonological features output by the feature intensity modulator, thereby determining the final target bilingual speech to be synthesized. In this example, according to the bilingual text features corresponding to the example dialogue "Customer A plans to invest in a SPAC-related project in the state of YOLO. The customer is in a high-risk state and is recommended to be carefully considered", the target language features calculated by the attention mechanism, and the target phonological features with different Chinese and English sounds and optimized in the above example dialogue are fused and decoded to generate a more vivid target bilingual speech of the example dialogue for communication with customers.
[0034] In this example, the decoder corresponding to the speech synthesis model is used to convert the bilingual text features from the attention bridge encoder into acoustic features, and then generate the actual speech waveform. The decoder uses an autoregressive model in the generation process to gradually predict the acoustic features of each time step according to the bilingual text features, target language features, and target phonological features until the entire speech waveform is generated. In this example, when financial customer service needs to communicate with customers through voice, the speech synthesis model in this example is used to generate the corresponding target bilingual speech based on the text to be processed in multiple languages, thereby completing the anthropomorphic communication with customers.
[0035] In this example, the bilingual text features are obtained by encoding the text to be processed corresponding to the text content with the first voice and the second language; the first language features and the first phonological features of the bilingual are obtained according to the first language and the second language, so as to capture the difference in the speech expression features between the different language phonological systems; the bilingual text features, the first language features and the first phonological features are processed by a feature intensity modulator respectively, so as to obtain the corresponding target language features and target phonological features, which are used to capture the dynamic intensity of the language and the voice and adjust the phonological control; finally, the bilingual text features, the target language features and the target phonological features are decoded to obtain the synthesized target bilingual voice. By adding the phonological features and combining the feature intensity modulator, the expression features between the two languages are captured to adjust the phonological features, reduce the problem of insufficient accuracy of the trained model caused by the small number of mixed corpora, and thus improve the accuracy of the synthesized voice tones in different scenarios. Furthermore, when applied in the financial field, the generated bilingual synthesized voice is more fluent, improves the interactivity with customers, reduces the workload of customer service personnel and improves customer satisfaction.
[0036] In one embodiment, if Figure 3 As shown, in step S202, the text to be processed is encoded to obtain bilingual text features corresponding to the text to be processed, including: S301: Encode the text to be processed to obtain a first text feature corresponding to the text to be processed; S302: Separating speaker information from the first text feature to obtain a second text feature corresponding to the text to be processed; S303: Performing gradient reversal processing on the second text feature to obtain bilingual text features corresponding to the text to be processed.
[0037] As an example, in step S301, after receiving the text to be processed, the server converts the text to be processed into a first text feature through an encoder. In this example, the encoder may use a multi-layer Transformer or other deep learning models for encoding.
[0038] As an example, in step S302, after receiving the first text feature, the server uses a speaker classifier to identify the speaker information of the first text feature, and uses the separated first text feature as the second text feature. In this example, a neural network-based speaker classifier is attached to the first text feature output by the encoder to identify the speaker information to which the text to be processed belongs.
[0039] As an example, in step S303, after receiving the second text feature, the server performs gradient reversal processing on the second text feature after speaker information separation to obtain a bilingual text feature with lower noise.
[0040] In this example, gradient reversal is used during the training of the speaker classifier to prevent the encoder from backpropagating the gradient information when updating the parameters of the speaker classifier. By negating the gradient, the encoder is not disturbed by the speaker classifier, allowing the encoder to focus on capturing information about the text without being affected by the speaker information.
[0041] In this example, the speaker information is separated during the encoding of the text to be processed, and the separated second text features are processed to reduce the noise interference of the speaker information on the output bilingual text features during encoding. At the same time, the separation of the speaker information can enable the final synthesized speech to be synthesized into a speech suitable for a specific speaker based on the text to be processed, thereby achieving a personalized speech synthesis effect.
[0042] In one embodiment, if Figure 4 As shown, in step S204, a feature intensity modulator is used to process the bilingual text feature and the first language feature to obtain the target language feature; the bilingual text feature and the first phonological feature are processed to obtain the target phonological feature, including: S401: Using the first sub-network to perform attention calculation on the first language feature and the bilingual text feature, to obtain a first attention output result corresponding to the first language feature; fusing the first attention output result with the first language feature to obtain a second language feature; S402: Use the second sub-network to perform a feedforward operation on the second language feature to obtain a first feedforward output result corresponding to the first language feature; and fuse the first feedforward output result with the second language feature to obtain a target language feature corresponding to the first language feature.
[0043] It can be understood that the feature intensity modulator consists of two sub-networks, namely the multi-head attention network and the feedforward network. Both sub-networks use layer normalization and residual connection. The specific formula is as follows: First subnetwork: Second sub-network: Among them, the output of the encoder that performs position encoding in proportion is Eo (bilingual text feature), LP is the language feature (first language feature) or the phonological feature (first phonological feature), the output of the first sub-network is Mo (the second language feature corresponding to the first language feature, or the second phonological feature corresponding to the first phonological feature), and the output of the second sub-network is Fo (the target language feature corresponding to the first language feature, or the target phonological feature corresponding to the first phonological feature).
[0044] As an example, in step S401, after receiving the bilingual text feature and the first language feature, the server uses a feature intensity modulator to perform attention mechanism calculation processing on the above two features, obtains a first attention output result corresponding to the first language feature, and fuses and splices the first attention output result and the first language feature to obtain a second language feature.
[0045] As an example, in step S402, after receiving the second language feature calculated by the attention mechanism, the server performs a feedforward operation on the second language feature through the second sub-network to obtain a first feedforward output result corresponding to the first language feature, and then fuses and concatenates the first feedforward output result and the second language feature to obtain a target language feature corresponding to the first language feature after the dynamic component is captured.
[0046] In this example, the first language features and bilingual text features are dynamically captured through a feature intensity modulator by a first subnetwork based on a multi-head attention network and a second subnetwork based on a feedforward network for use in subsequent speech synthesis, thereby improving the fluency, intonation accuracy and anthropomorphism of the synthesized speech in a bilingual or multilingual environment.
[0047] In one embodiment, if Figure 5 As shown, in step S204, a feature intensity modulator is used to process the bilingual text feature and the first language feature to obtain the target language feature; the bilingual text feature and the first phonological feature are processed to obtain the target phonological feature, including: S501: using the first sub-network to perform attention calculation on the first phonological feature and the bilingual text feature to obtain a second attention output result corresponding to the first phonological feature; fusing the second attention output result with the first phonological feature to obtain a second phonological feature; S502: Using the second sub-network, perform feedforward operation processing on the second phonological feature to obtain a second feedforward output result corresponding to the first phonological feature; fusing the second feedforward output result and the second phonological feature to obtain a target phonological feature corresponding to the first phonological feature.
[0048] In this example, the accuracy of capturing the dynamic cost of phonology and language can be improved by setting two feature intensity modulators, namely, setting a phonological feature intensity modulator to process phonological features and setting a linguistic feature intensity modulator to process linguistic features.
[0049] As an example, in step S501, after receiving the bilingual text feature and the first phonetic feature, the server uses a feature intensity modulator to perform attention mechanism calculation processing on the above two features, obtains the second attention output result corresponding to the first phonetic feature, and fuses and splices the second attention output result with the first phonetic feature to obtain the second phonetic feature.
[0050] As an example, in step S502, after receiving the second phonological feature calculated by the attention mechanism, the server performs a feedforward operation on the second phonological feature through the second sub-network to obtain a second feedforward output result corresponding to the first phonological feature, and then fuses and concatenates the second feedforward output result and the second phonological feature to obtain a target phonological feature corresponding to the first phonological feature after the dynamic component is captured.
[0051] In this example, the first phonological features and bilingual text features are dynamically captured through a feature intensity modulator by a first subnetwork based on a multi-head attention network and a second subnetwork based on a feedforward network for use in subsequent speech synthesis, thereby improving the fluency, intonation accuracy and anthropomorphism of the synthesized speech in a bilingual or multilingual environment.
[0052] In one embodiment, if Figure 6 As shown, in step S205, the bilingual text features, the target language features and the target phonological features are decoded to obtain the target bilingual speech corresponding to the text to be processed, including: S601: Using the attention mechanism, the bilingual text features, the target language features and the target phonological features are integrated to obtain the target bilingual features; S602: Decode the target bilingual features to obtain the target bilingual speech corresponding to the text to be processed.
[0053] As an example, in step S601, after receiving the bilingual text features, the target language features and the target phonological features, the server uses an attention mechanism to calculate the weights of the bilingual text features, the target language features and the target phonological features, and fuses the bilingual text features, the target language features and the target phonological features according to the calculated weights corresponding to each feature to obtain the final target bilingual features.
[0054] In this example, the attention mechanism used by the decoder is the GMM attention mechanism (Gaussian Mixture Model Attention), which is an attention mechanism based on a Gaussian mixture model that simulates the distribution of target bilingual features by establishing a Gaussian distribution. At each time step, the GMM attention mechanism calculates the weight distribution between the bilingual text features, the target language features, and the target phonological embeddings to determine the target phonological features with adjusted strength to generate speech at the current time step and determine the final target bilingual features.
[0055] As an example, in step S602, after receiving the target bilingual features, the server decodes the target bilingual features, that is, converts the target bilingual features into corresponding Mel spectra, and synthesizes the most popular target bilingual speech according to the Mel spectra.
[0056] In this example, the GMM attention mechanism is used to perform weighted fusion of bilingual text features, target language features, and target phonological embeddings to determine the target bilingual features after phonological adjustment, and synthesize the target bilingual speech through decoding, thereby ensuring the fluency of the generated speech and the accuracy of intonation in a bilingual environment.
[0057] In one embodiment, if Figure 7 As shown, in step S601, the attention mechanism is used to fuse the bilingual text features, the target language features and the target phonological features to obtain the target bilingual features, including: S701: Acquire target speech features according to speaker information corresponding to the text to be processed; S702: fusing the target speech features, the bilingual text features, the target language features and the target phonological features to obtain the target bilingual features.
[0058] As an example, in step S701, after receiving the bilingual text features, target language features and target phonetic features, the server confirms the speaker information corresponding to the text to be processed according to business requirements, so as to confirm the target voice features corresponding to the speaker information, thereby generating the voice required by the business, such as using the voice features of an adult male voice as the target voice features.
[0059] As an example, in step S702, after confirming the target speech features, the server performs weighted fusion on the target speech features, bilingual text features, target language features and target phonological features to obtain the target bilingual features. The fused target bilingual features reflect the speech features of the text to be synthesized at the current time step, including pitch, speech speed, intonation, etc. In this example, the target speech features can be selected according to actual needs. After confirming the target speech features, the GMM attention mechanism is used to perform weighted fusion of the target speech features, bilingual text features, target language features and target phonetic features, so as to synthesize speech suitable for a specific speaker based on the text to be processed, thereby achieving a personalized speech synthesis effect.
[0060] In one embodiment, if Figure 8 As shown, in step S602, decoding processing is performed on the target bilingual features to obtain the target bilingual speech corresponding to the text to be processed, including: S801: Decode the target bilingual features to obtain the target Mel spectrum; S802: converting the target Mel spectrum to obtain a target linear spectrum; S803: Acquire target bilingual speech according to the target linear spectrum.
[0061] As an example, in step S801, after obtaining the target bilingual features, the server decodes the target bilingual features to obtain the target mel spectrum. In this example, when using a trained vocoder, the target bilingual features to be converted are input into the vocoder using acoustic feature extraction technology (such as short-time Fourier transform, mel filter bank, etc.) to obtain the corresponding mel spectrum part.
[0062] As an example, in step S802, after obtaining the target Mel spectrum, the server converts the target Mel spectrum to obtain the target linear spectrum. In this example, the Mel spectrum is converted back to the linear spectrum by applying the Inverse Mel Transform. This step converts the nonlinear characterization of the Mel spectrum into the target linear spectrum for subsequent spectrum reconstruction.
[0063] As an example, in step S803, after acquiring the target linear spectrum, the server uses an inverse spectral transform to convert the linear spectrum back to a waveform signal in the time domain. The inverse spectral transform restores the spectrum information to a waveform signal, thereby generating the final target bilingual speech.
[0064] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.
[0065] In one embodiment, a bilingual speech synthesis device is provided, and the bilingual speech synthesis device corresponds to the bilingual speech synthesis method in the above embodiment. Fig. 9As shown, the bilingual speech synthesis device includes a to-be-processed text acquisition module 901, a bilingual text feature acquisition module 902, a first language feature and a first phonological feature acquisition module 903, a target language feature and a target phonological feature acquisition module 904, and a target bilingual speech acquisition module 905. The functional modules are described in detail as follows: A to-be-processed text acquisition module 901 is used to acquire the to-be-processed text, where the to-be-processed text includes text content corresponding to the first language and text content corresponding to the second language; A bilingual text feature acquisition module 902 is used to encode the text to be processed and obtain the bilingual text features corresponding to the text to be processed; A first language feature and first phonological feature acquisition module 903 is used to acquire a first language feature and a first phonological feature; the first language feature is a language feature between the first language and the second language, and the first phonological feature is a phonological feature between the first language and the second language; The target language feature and target phonological feature acquisition module 904 is used to process the bilingual text feature and the first language feature using a feature intensity modulator to obtain the target language feature; process the bilingual text feature and the first phonological feature to obtain the target phonological feature; The target bilingual speech acquisition module 905 is used to decode the bilingual text features, the target language features and the target phonological features to acquire the target bilingual speech corresponding to the text to be processed.
[0066] In one embodiment, the bilingual text feature acquisition module 902 includes: A first text feature acquisition unit, used to encode the text to be processed and acquire a first text feature corresponding to the text to be processed; A second text feature acquisition unit, used to separate speaker information from the first text feature to acquire a second text feature corresponding to the text to be processed; The bilingual text feature acquisition unit is used to perform gradient reversal processing on the second text feature to obtain the bilingual text feature corresponding to the text to be processed.
[0067] In one embodiment, the target language feature and target phonological feature acquisition module 904 includes: A second language feature acquisition unit is used to use the first sub-network to perform attention calculation on the first language feature and the bilingual text feature to obtain a first attention output result corresponding to the first language feature; and to fuse the first attention output result and the first language feature to obtain a second language feature; The target language feature acquisition unit is used to use the second sub-network to perform a feedforward operation on the second language feature to obtain a first feedforward output result corresponding to the first language feature; and to fuse the first feedforward output result with the second language feature to obtain a target language feature corresponding to the first language feature.
[0068] In one embodiment, the target language feature and target phonological feature acquisition module 904 includes: A second phonological feature acquisition unit is used to use the first sub-network to perform attention calculation on the first phonological feature and the bilingual text feature to obtain a second attention output result corresponding to the first phonological feature; and to perform fusion processing on the second attention output result and the first phonological feature to obtain a second phonological feature; The target phonological feature acquisition unit is used to use the second sub-network to perform a feedforward operation on the second phonological feature to obtain a second feedforward output result corresponding to the first phonological feature; and to fuse the second feedforward output result and the second phonological feature to obtain a target phonological feature corresponding to the first phonological feature.
[0069] In one embodiment, the target bilingual speech acquisition module 905 includes: A target bilingual feature acquisition unit is used to adopt an attention mechanism to fuse bilingual text features, target language features and target phonological features to acquire target bilingual features; The target bilingual speech acquisition unit is used to decode the target bilingual features to acquire the target bilingual speech corresponding to the text to be processed.
[0070] In one embodiment, the target bilingual feature acquisition unit includes: A target speech feature acquisition subunit is used to acquire target speech features according to speaker information corresponding to the text to be processed; The target bilingual feature acquisition subunit is used to fuse the target speech features, bilingual text features, target language features and target phonological features to acquire the target bilingual features.
[0071] In one embodiment, the target bilingual speech acquisition unit includes: A Mel-spectrum-standardized acquisition subunit is used to decode the target bilingual features and acquire the target Mel-spectrum; A target linear spectrum acquisition subunit is used to convert the target Mel spectrum to obtain a target linear spectrum; The target bilingual speech acquisition subunit is used to acquire the target bilingual speech according to the target linear spectrum.
[0072] The specific definition of the bilingual speech synthesis device can be found in the definition of the bilingual speech synthesis method mentioned above, which will not be repeated here. Each module in the above-mentioned bilingual speech synthesis device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0073] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to execute the data used or generated in the process of the synthesis method of bilingual speech. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a synthesis method of bilingual speech is implemented.
[0074] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the bilingual speech synthesis method in the above embodiment is implemented, for example Figure 2 S201-S205 as shown, or Figures 3 to 8 Alternatively, when the processor executes the computer program, the functions of each module / unit in the embodiment of the bilingual speech synthesis device are realized, for example Fig. 9 The functions of the to-be-processed text acquisition module 901, the bilingual text feature acquisition module 902, the first language feature and first phonetic feature acquisition module 903, the target language feature and target phonetic feature acquisition module 904, and the target bilingual speech acquisition module 905 are not described here in detail to avoid repetition.
[0075] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the bilingual speech synthesis method in the above embodiment is implemented, for example Figure 2 S201-S205 as shown, or Figures 3 to 8Alternatively, when the computer program is executed by a processor, the functions of each module / unit in the embodiment of the bilingual speech synthesis device are realized, for example Fig. 9 The functions of the to-be-processed text acquisition module 901, the bilingual text feature acquisition module 902, the first language feature and first phonetic feature acquisition module 903, the target language feature and target phonetic feature acquisition module 904, and the target bilingual speech acquisition module 905 are not described here in detail to avoid repetition.
[0076] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (RambuS) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0077] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0078] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.
Claims
1. A method for synthesizing bilingual speech, characterized in that: include: Acquire a text to be processed, wherein the text to be processed includes text content corresponding to a first language and text content corresponding to a second language; Encoding the text to be processed to obtain bilingual text features corresponding to the text to be processed; Acquire a first language feature and a first phonetic feature; the first language feature is a language feature between the first language and the second language, and the first phonetic feature is a phonetic feature between the first language and the second language; Using a feature intensity modulator, the bilingual text feature and the first language feature are processed to obtain a target language feature; the bilingual text feature and the first phonological feature are processed to obtain a target phonological feature; The bilingual text features, the target language features and the target phonological features are decoded to obtain the target bilingual speech corresponding to the text to be processed.
2. The method for synthesizing bilingual speech according to claim 1, characterized in that: The encoding process of the text to be processed to obtain bilingual text features corresponding to the text to be processed includes: Encoding the text to be processed to obtain a first text feature corresponding to the text to be processed; Separating the speaker information from the first text feature to obtain a second text feature corresponding to the text to be processed; Performing gradient reversal processing on the second text feature to obtain a bilingual text feature corresponding to the text to be processed.
3. The method for synthesizing bilingual speech according to claim 1, characterized in that: The processing of the bilingual text features and the first language features to obtain target language features includes: Using a first sub-network to perform attention calculation on the first language feature and the bilingual text feature to obtain a first attention output result corresponding to the first language feature; fusing the first attention output result with the first language feature to obtain a second language feature; A second sub-network is used to perform a feedforward operation on the second language feature to obtain a first feedforward output result corresponding to the first language feature; and the first feedforward output result and the second language feature are fused to obtain a target language feature corresponding to the first language feature.
4. The method for synthesizing bilingual speech according to claim 1, characterized in that: The step of processing the bilingual text feature and the first phonetic feature to obtain a target phonetic feature includes: Using the first sub-network, performing attention calculation on the first phonological feature and the bilingual text feature to obtain a second attention output result corresponding to the first phonological feature; fusing the second attention output result with the first phonological feature to obtain a second phonological feature; A second sub-network is used to perform a feedforward operation on the second phonological feature to obtain a second feedforward output result corresponding to the first phonological feature; and a fusion process is performed on the second feedforward output result and the second phonological feature to obtain a target phonological feature corresponding to the first phonological feature.
5. The method for synthesizing bilingual speech according to claim 1, characterized in that: The decoding process of the bilingual text features, the target language features and the target phonological features to obtain the target bilingual speech corresponding to the text to be processed includes: Using an attention mechanism, the bilingual text features, the target language features and the target phonological features are fused to obtain target bilingual features; The target bilingual features are decoded to obtain the target bilingual speech corresponding to the text to be processed.
6. The method for synthesizing bilingual speech as claimed in claim 5, characterized in that: The fusing the bilingual text features, the target language features and the target phonological features to obtain target bilingual features includes: Acquiring target speech features according to speaker information corresponding to the text to be processed; The target speech feature, the bilingual text feature, the target language feature and the target phonological feature are fused to obtain a target bilingual feature.
7. The method for synthesizing bilingual speech as claimed in claim 5, characterized in that: The decoding process of the target bilingual features to obtain the target bilingual speech corresponding to the text to be processed includes: Decoding the target bilingual features to obtain a target Mel spectrum; Converting the target Mel spectrum to obtain a target linear spectrum; According to the target linear spectrum, a target bilingual speech is obtained.
8. A bilingual speech synthesis device, characterized in that: include: A module for acquiring text to be processed, used for acquiring text to be processed, wherein the text to be processed includes text content corresponding to the first language and text content corresponding to the second language; A bilingual text feature acquisition module, used to encode the text to be processed and acquire bilingual text features corresponding to the text to be processed; A first language feature and a first phonological feature acquisition module, used to acquire a first language feature and a first phonological feature; the first language feature is a language feature between the first language and the second language, and the first phonological feature is a phonological feature between the first language and the second language; A target language feature and target phonological feature acquisition module, used to process the bilingual text feature and the first language feature using a feature intensity modulator to acquire a target language feature; and to process the bilingual text feature and the first phonological feature to acquire a target phonological feature; The target bilingual speech acquisition module is used to decode the bilingual text features, the target language features and the target phonological features to acquire the target bilingual speech corresponding to the text to be processed.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the bilingual speech synthesis method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the bilingual speech synthesis method according to any one of claims 1 to 7 is implemented.