Speech synthesis method and device, computer equipment and storage medium
Through the self-attention mechanism, the phonemes and rhythms of the initial text and reference audio are embedded and aligned, which solves the problem of emotional position deviations of reference audio in emotional speech synthesis, improves the naturalness and credibility of speech, and enhances the intelligent customer service Communication effect.
Patent Information
- Application Number
- CN202411071543.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-08-05
AI Technical Summary
When the existing emotional voice synthesis method is transferred through style, the emotional position deviation of reference audio is drawn from the reference audio, resulting in a natural difference in the generated voice, affecting the communication experience between users and intelligent customer service.
The self-attention mechanism is used to embed and aligned the phonemes and rhythms of the initial text and reference audio to generate synthetic audio, ensuring that the rhythm appears in the accurate position in the synthetic audio, and improving the naturalness and credibility of the speech.
It improves the naturalness and credibility of voice, enhances the infectiousness and persuasiveness of voice, adapts to different application scenarios and user preferences, and improves the communication effect of intelligent customer service.
Smart Images

Figure CN118918878A_ABST
Abstract
Description
Technical Field
[0001] The present application is applied to the field of speech synthesis technology, and in particular relates to a speech synthesis method, apparatus, computer equipment and storage medium. Background Art
[0002] In the financial field, since the number of online consulting customers that need to be received every day is very huge, if manual customer service is used to receive them one by one, the labor cost will increase dramatically. Therefore, the setting based on intelligent financial customer service can well realize the unification of customer service in the financial system.
[0003] As for the deployment of intelligent financial customer service, speech synthesis of financial customer service is the most critical link. The speech synthesis of intelligent customer service with emotion can improve the naturalness and credibility of speech, enhance the appeal and persuasiveness of speech, and adapt to different application scenarios and user preferences.
[0004] Existing emotional speech synthesis often uses an additional input reference audio to achieve emotional speech synthesis through style transfer. However, in the process of borrowing the emotion of the reference audio, if there is a deviation in the borrowing position, the naturalness of the final generated speech will be very poor, resulting in a poor communication experience between users and intelligent customer service. Summary of the invention
[0005] The present application discloses a speech synthesis method, apparatus, computer equipment and storage medium, which are intended to solve the problem that existing emotional speech synthesis often uses an additional input reference audio to achieve emotional speech synthesis through style transfer, but in the process of borrowing the emotion of the reference audio, once the borrowing position deviates, the naturalness of the final generated speech will be very poor, resulting in a poor communication experience between the user and the intelligent customer service.
[0006] In a first aspect, the present application provides a speech synthesis method, comprising:
[0007] Obtaining the initial text and reference audio to be synthesized;
[0008] Inputting the initial text into a preset phoneme encoder, the phoneme encoder parses the initial text and outputs a plurality of phonemes corresponding to the initial text;
[0009] Inputting the reference audio into a preset rhythm encoder, the rhythm encoder parses the reference audio and outputs a plurality of rhythms corresponding to the reference audio;
[0010] Based on the self-attention mechanism, each phoneme and multiple rhythms are aligned to obtain the embedding information corresponding to each phoneme; the embedding information includes at least one rhythm and the weight corresponding to each rhythm;
[0011] A Mel spectrum is generated based on each phoneme and the embedded information corresponding to the phoneme, and synthetic audio is generated based on the Mel spectrum to complete the rhythm alignment of the initial text and the reference audio.
[0012] In some embodiments, each phoneme and multiple rhythms are aligned based on a self-attention mechanism to obtain embedding information corresponding to each phoneme, including: determining the embedding dimension corresponding to the phoneme and the rhythm according to the number of rhythms; aligning each phoneme with each rhythm according to the embedding dimension based on the self-attention mechanism to obtain the weight corresponding to each rhythm; and constructing embedding information according to multiple rhythms and the weights corresponding to the rhythms.
[0013] Exemplarily, each phoneme is aligned with each rhythm according to the embedding dimension based on the self-attention mechanism to obtain the weight corresponding to each rhythm, including: calculating the weight corresponding to each rhythm according to the flexible maximum activation function, so as to calculate the embedding information according to a preset embedding formula; the expression of the embedding formula includes:
[0014]
[0015] Among them, α is the embedded information, softmax() is the value of the maximum activation function, H is the phoneme, C is the rhythm set composed of multiple rhythms, and d is the embedding dimension.
[0016] In some embodiments, a mel spectrum is generated according to each phoneme and the embedded information corresponding to the phoneme, including: generating an embedded text according to each phoneme information and the corresponding embedded information; normalizing the embedded information corresponding to each phoneme within the range of a standard normal distribution; generating synthetic information according to the embedded text and the normalized phoneme; and inputting the synthetic information into a preset decoder, which parses the synthetic information to generate a mel spectrum.
[0017] Exemplarily, generating synthetic information according to the embedded text and the normalized phonemes includes: inputting the embedded text into a preset linear layer to obtain a linear text corresponding to the embedded text; and outputting the synthetic information according to a preset information synthesis formula, the information synthesis formula including:
[0018]
[0019] Among them, y is the synthetic information, f(W) is the linear text, W is the embedded text, H is the phoneme, and H mean is the average value corresponding to the phoneme, H var is the variance corresponding to the phoneme.
[0020] In some embodiments, obtaining an initial text to be synthesized and a reference audio includes: obtaining the initial text and the scenario type corresponding to the initial text in the financial system; obtaining an intelligent customer service audio library corresponding to the scenario type; the intelligent customer service voice library includes multiple customer service audios and audio content corresponding to the customer service audios; parsing the initial text to obtain text content corresponding to the initial text; matching based on the text content and the audio content, and using the customer service audio corresponding to the successfully matched audio content as the reference audio.
[0021] In some embodiments, generating synthesized audio based on Mel spectrum includes: obtaining voice preference parameters corresponding to target customers in a financial system; the voice preference parameters include at least one or more of gender information, age information, and audio information; inputting the Mel spectrum and the voice preference parameters into a preset vocoder, the vocoder recognizes the Mel spectrum and the voice preference parameters, and generates synthesized audio.
[0022] In a second aspect, the present application provides a speech synthesis device, comprising:
[0023] A first acquisition module, used to acquire an initial text to be synthesized and a reference audio;
[0024] A phoneme output module, used to input the initial text into a preset phoneme encoder, the phoneme encoder parses the initial text, and outputs a plurality of phonemes corresponding to the initial text;
[0025] A rhythm output module, used to input the reference audio into a preset rhythm encoder, the rhythm encoder parses the reference audio, and outputs a plurality of rhythms corresponding to the reference audio;
[0026] The second acquisition module is used to align each phoneme with multiple rhythms based on a self-attention mechanism to obtain embedding information corresponding to each phoneme; the embedding information includes at least one rhythm and a weight corresponding to each rhythm;
[0027] The rhythm alignment module is used to generate a Mel spectrum according to each phoneme and the embedded information corresponding to the phoneme, generate synthetic audio according to the Mel spectrum, and complete the rhythm alignment of the initial text and the reference audio.
[0028] In a third aspect, the present application provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein the memory stores a strategy model, wherein when the computer program is executed by the processor, a speech synthesis method provided in any embodiment of the present application is implemented.
[0029] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the processor enables the processor to implement a speech synthesis method as provided in any embodiment of the present application.
[0030] The embodiments of the present application provide a speech synthesis method, apparatus, computer device and storage medium. The provided method obtains the phonemes corresponding to the initial text and the rhythms corresponding to the reference audio for embedding alignment, and embeds each rhythm according to the corresponding weight, so that the rhythm in the synthesized audio will appear at the correct position, which can improve the naturalness and credibility of the speech, enhance the appeal and persuasiveness of the speech, and adapt to different application scenarios and user preferences.
[0031] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0033] Figure 1 is a schematic flow chart of the steps of a speech synthesis method provided by an embodiment of the present application;
[0034] Figure 2 is a schematic diagram of embedding rhythm and text provided by an embodiment of the present application;
[0035] Figure 3 is a schematic diagram of the principle of a speech synthesis method provided in an embodiment of the present application;
[0036] Figure 4 is a structural schematic diagram of a speech synthesis device provided in an embodiment of the present application;
[0037] Figure 5 It is a schematic block diagram of the structure of a computer device provided in one embodiment of the present application.
[0038] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. DETAILED DESCRIPTION
[0039] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0040] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.
[0041] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0042] It should be understood that, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second", etc. are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art can understand that the words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not necessarily limit the difference.
[0043] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0044] To facilitate understanding of the embodiments of the present application, some words involved in the embodiments of the present application are briefly explained below.
[0045] 1. Emotional speech synthesis: Emotional speech synthesis is a technology that uses artificial intelligence technology to generate speech with corresponding emotional characteristics based on text content and emotional tags. The background of emotional speech synthesis is that with the development of human-computer interaction, traditional speech synthesis technology can no longer meet users' needs for natural, expressive and personalized speech. Therefore, emotional speech synthesis aims to improve the naturalness and credibility of speech, enhance the appeal and persuasiveness of speech, and adapt to different application scenarios and user preferences. The application areas of emotional speech synthesis include intelligent customer service, intelligent education, intelligent entertainment, intelligent medical care, etc.
[0046] 2. Phone: A phone is the smallest unit of speech divided according to the natural properties of speech. Analyzing based on the articulatory movements in a syllable, one movement constitutes one phone. Phones are divided into two major categories: vowels and consonants. For example, the Chinese syllable "ā" has only one phone, "ài" has two phones, and "dài" has three phones, etc.
[0047] The following will describe in detail some embodiments of the present application with reference to the accompanying drawings. Without conflict, the following embodiments and the features in the embodiments can be combined with each other.
[0048] In the financial field, since the number of online consulting customers to be received daily is extremely large, if manual customer service is used to receive them one by one, it will lead to a sharp increase in labor costs. Therefore, the setting of an intelligent financial customer service can well achieve the unification of the customer service in the financial system.
[0049] For the deployment of an intelligent financial customer service, the speech synthesis of the financial customer service is the most crucial link. The speech synthesis of an intelligent customer service with emotion can improve the naturalness and credibility of the speech, enhance the appeal and persuasiveness of the speech, and adapt to different application scenarios and user preferences.
[0050] Existing emotional speech synthesis often uses an additional input reference audio to achieve emotional speech synthesis through style transfer. However, once there is a deviation in the position of borrowing the emotion of the reference audio during the process of borrowing the emotion, it will lead to very poor naturalness of the finally generated speech, resulting in a poor communication experience between the user and the intelligent customer service.
[0051] To solve the above problems, an embodiment of the present application proposes a speech synthesis method. Please refer to Figure 1 , Figure 1 which is a schematic flowchart of the steps of the speech synthesis method provided by an embodiment of the present application.
[0052] It should be noted that the speech synthesis method provided by the present application can be used in scenarios with a large reception demand such as the financial system. Through the deployment of intelligent customer service, the labor cost can be greatly reduced and the unified management of customer service can be achieved. It can also be applied to any other scenarios with a large reception demand during the speech synthesis of intelligent customer service, such as the medical, education, entertainment and other fields. The present application is described by taking the financial scenario as an example.
[0053] As Figure 1 shown, the proposed speech synthesis method includes steps S101 to S105.
[0054] S101. Obtain the initial text to be synthesized and the reference audio.
[0055] Specifically, when performing speech synthesis, the initial text to be synthesized is obtained. For example, when a user to be served in a financial system enters the corresponding intelligent customer service interface, the corresponding dialogue text is first generated as the initial text. At the same time, synthesis is performed based on the reference audio corresponding to the scene, aiming to synthesize the content of the initial text by combining the advantages of the reference audio. This makes the synthesized audio more appealing.
[0056] It should be noted that if Figure 2 As shown, for a sentence (such as Figure 2 In the sentence "I am so happy today", different rhythmic "inflections" can be used for each word. Different alignments between rhythm and text may generate different speech. Figure 2 The alignment below is more reasonable than the one above. The synthesized speech effect is also more expressive. Therefore, this application aims to obtain the most appropriate text and rhythm embedding method for each sentence in each initial text.
[0057] In some embodiments, obtaining an initial text to be synthesized and a reference audio includes: obtaining the initial text and the scenario type corresponding to the initial text in the financial system; obtaining an intelligent customer service audio library corresponding to the scenario type; the intelligent customer service voice library includes multiple customer service audios and audio content corresponding to the customer service audios; parsing the initial text to obtain text content corresponding to the initial text; matching based on the text content and the audio content, and using the customer service audio corresponding to the successfully matched audio content as the reference audio.
[0058] Since the system in which the provided method is applied includes many scenarios, such as financial systems including quotation, renewal, product introduction, daily communication and many other scenarios. The provided method sets the corresponding customer service audio for different scenarios into an intelligent customer service voice library. Then, the corresponding reference audio can be quickly determined according to the content of each initial text. In this way, standardized management of speech synthesis can be achieved, and the most appropriate reference audio can be obtained for each initial text.
[0059] S102. Input the initial text into a preset phoneme encoder, which parses the initial text and outputs a plurality of phonemes corresponding to the initial text.
[0060] Specifically, by inputting the initial text into a preset phoneme encoder, the phoneme encoder can parse the content in the initial text and extract the phonemes corresponding to each word in the initial text, and then can determine the most appropriate rhythm for embedding different phonemes and their contextual relationships.
[0061] S103. Input the reference audio to a preset prosody encoder, which parses the reference audio and outputs a plurality of prosody corresponding to the reference audio.
[0062] Specifically, the prosody encoder extracts prosody embedding from the reference audio. The prosody embedding includes the phonemes of each sound in the reference audio and the corresponding prosody. The prosody includes Figure 2 Before the phonemes and rhythms are aligned for embedding, Figure 3 As shown, the phonemes and rhythms of the initial text and the reference audio are extracted respectively.
[0063] S104. Align each phoneme and multiple rhythms based on the self-attention mechanism to obtain embedding information corresponding to each phoneme; the embedding information includes at least one rhythm and a weight corresponding to each rhythm.
[0064] Specifically, the method provided is to obtain the alignment information of the phoneme rhythm, and use the self-attention mechanism for alignment. The weights of each rhythm corresponding to each phoneme, such as the weight of "cadence", are calculated through the self-attention mechanism. Then the rhythm corresponding to each character of the initial text can be confirmed. At the same time, the method provided can also combine the weights of each rhythm corresponding to each phoneme to fuse and generate the rhythm corresponding to each phoneme. For example, when the rhythm weight corresponding to a phoneme is: 0.9 cadence + 0.05 rise + 0.04 intonation + 0.01 intonation. Then the cadence can be used as the corresponding rhythm, or the weights corresponding to the four rhythms of cadence, intonation and intonation can be combined for fusion to generate the target rhythm corresponding to the phoneme. Then the method provided can flexibly determine the rhythm corresponding to the phoneme.
[0065] In some embodiments, each phoneme and multiple rhythms are aligned based on a self-attention mechanism to obtain embedding information corresponding to each phoneme, including: determining the embedding dimension corresponding to the phoneme and the rhythm according to the number of rhythms; aligning each phoneme with each rhythm according to the embedding dimension based on the self-attention mechanism to obtain the weight corresponding to each rhythm; and constructing embedding information according to multiple rhythms and the weights corresponding to the rhythms.
[0066] The provided method determines its embedding dimension according to the number of rhythms, such as when the rhythm includes intonation, the embedding dimension is 4. Then, based on the self-attention mechanism, the weights of each rhythm corresponding to each phoneme are calculated to obtain the embedding information of the rhythm corresponding to each phoneme.
[0067] Exemplarily, each phoneme is aligned with each rhythm according to the embedding dimension based on the self-attention mechanism to obtain the weight corresponding to each rhythm, including: calculating the weight corresponding to each rhythm according to the flexible maximum activation function, so as to calculate the embedding information according to a preset embedding formula; the expression of the embedding formula includes:
[0068]
[0069] Among them, α is the embedding information, softmax() is the value of the maximum activation function, H is the phoneme, C is the rhythm set composed of multiple rhythms, and d is the embedding dimension. The provided method can quickly calculate the rhythm corresponding to each phoneme of the initial text combined with the reference audio.
[0070] S105. Generate a Mel spectrum according to each phoneme and the embedded information corresponding to the phoneme, generate a synthetic audio according to the Mel spectrum, and complete the rhythm alignment of the initial text and the reference audio.
[0071] Specifically, after obtaining the embedding information corresponding to each phoneme, the corresponding inference operation can be performed to generate a Mel spectrum to generate the final synthesized audio, and then the phonemes corresponding to the initial text and the rhythms corresponding to the reference audio are obtained for embedding alignment, and each rhythm is embedded according to the corresponding weight. The rhythm in the synthesized audio will appear at the correct position, which can improve the naturalness and credibility of the speech, enhance the appeal and persuasiveness of the speech, and adapt to different application scenarios and user preferences.
[0072] In some embodiments, a mel spectrum is generated according to each phoneme and the embedded information corresponding to the phoneme, including: generating an embedded text according to each phoneme information and the corresponding embedded information; normalizing the embedded information corresponding to each phoneme within the range of a standard normal distribution; generating synthetic information according to the embedded text and the normalized phoneme; and inputting the synthetic information into a preset decoder, which parses the synthetic information to generate a mel spectrum.
[0073] Traditional speech synthesis methods directly connect text and prosody as input. Figure 3 As shown, the present application sends the embedded text generated by the aligned rhythm embedding information and the normalized phonemes into the decoder. The present application uses the conditional layer normalization (CLN) algorithm to change the distribution of the phoneme embedding latent variable space with the embedded text as a parameter to combine the two.
[0074] Exemplarily, generating synthetic information according to the embedded text and the normalized phonemes includes: inputting the embedded text into a preset linear layer to obtain a linear text corresponding to the embedded text; and outputting the synthetic information according to a preset information synthesis formula, the information synthesis formula including:
[0075]
[0076] Among them, y is the synthetic information, f(W) is the linear text, W is the embedded text, H is the phoneme, and H mean is the average value corresponding to the phoneme, H varis the variance corresponding to the phoneme. The mean and variance of the data H are changed by inputting the aligned prosodic embedding W. The decoder input at this time is y, which is the text embedding H after the mean and variance are changed according to the prosodic embedding W.
[0077] In some embodiments, generating synthesized audio based on Mel spectrum includes: obtaining voice preference parameters corresponding to target customers in a financial system; the voice preference parameters include at least one or more of gender information, age information, and audio information; inputting the Mel spectrum and the voice preference parameters into a preset vocoder, the vocoder recognizes the Mel spectrum and the voice preference parameters, and generates synthesized audio.
[0078] The provided method can obtain the voice preference parameters of the target customer when they enter, such as gender, age (confirming whether the voice is deep or light) and audio information (confirming the customer service voice volume, speaking speed, etc.) of the synthesized voice. It can also include information such as dialect type. Then, a more humanized audio can be generated through a preset vocoder to improve the user experience.
[0079] Through the method provided in this application, when deploying intelligent customer service in a financial system, for example, the phonemes corresponding to the initial text and the rhythms corresponding to the reference audio of the specified standard customer service are obtained for embedding and alignment, and each rhythm is embedded according to the corresponding weight. Then, the rhythm in the synthesized audio will appear at the correct position, which can improve the naturalness and credibility of the speech, enhance the appeal and persuasiveness of the speech, and adapt to different application scenarios and user preferences.
[0080] like Figure 4 As shown, Figure 4 1 is a schematic diagram of the structure of a speech synthesis device provided in an embodiment of the present application, wherein the speech synthesis device is used to execute the aforementioned speech synthesis method, wherein the speech synthesis device can be configured in a terminal or a server.
[0081] like Figure 4 As shown, the speech synthesis device 200 includes a first acquisition module 201 , a phoneme output module 202 , a prosody output module 203 , a second acquisition module 204 and a prosody alignment module 205 .
[0082] The first acquisition module 201 is used to acquire the initial text to be synthesized and the reference audio.
[0083] The phoneme output module 202 is used to input the initial text into a preset phoneme encoder, and the phoneme encoder parses the initial text and outputs a plurality of phonemes corresponding to the initial text.
[0084] The prosody output module 203 is used to input the reference audio into a preset prosody encoder, and the prosody encoder analyzes the reference audio and outputs a plurality of prosodys corresponding to the reference audio.
[0085] The second acquisition module 204 is used to align each phoneme with multiple rhythms based on the self-attention mechanism to obtain the embedding information corresponding to each phoneme; the embedding information includes at least one rhythm and the weight corresponding to each rhythm.
[0086] The prosody alignment module 205 is used to generate a Mel spectrum according to each phoneme and the embedded information corresponding to the phoneme, generate a synthetic audio according to the Mel spectrum, and complete the prosody alignment of the initial text and the reference audio.
[0087] In some embodiments, the second acquisition module 204 is further used to: determine the embedding dimension corresponding to the phoneme and the rhythm according to the number of rhythms; align each phoneme with each rhythm according to the embedding dimension based on the self-attention mechanism to obtain the weight corresponding to each rhythm; and construct embedding information according to multiple rhythms and the weights corresponding to the rhythms.
[0088] Exemplarily, the second acquisition module 204 is further used to: calculate the weight corresponding to each rhythm according to the flexible maximum activation function, so as to calculate the embedding information according to a preset embedding formula; the expression of the embedding formula includes:
[0089]
[0090] Among them, α is the embedded information, softmax() is the value of the maximum activation function, H is the phoneme, C is the rhythm set composed of multiple rhythms, and d is the embedding dimension.
[0091] In some embodiments, the prosody alignment module 205 is further used to: generate embedded text based on each phoneme information and the corresponding embedded information; normalize the embedded information corresponding to each phoneme within the range of a standard normal distribution; generate synthetic information based on the embedded text and the normalized phoneme; input the synthetic information into a preset decoder, and the decoder parses the synthetic information to generate a Mel spectrum.
[0092] Exemplarily, the prosody alignment module 205 is further used to: input the embedded text into a preset linear layer to obtain a linear text corresponding to the embedded text; and output the synthesized information according to a preset information synthesis formula, where the information synthesis formula includes:
[0093]
[0094] Among them, y is the synthetic information, f(W) is the linear text, W is the embedded text, H is the phoneme, and H mean is the average value corresponding to the phoneme, H var is the variance corresponding to the phoneme.
[0095] In some embodiments, the first acquisition module 201 is also used to: obtain the initial text and the scenario type corresponding to the initial text in the financial system; obtain the intelligent customer service audio library corresponding to the scenario type; the intelligent customer service voice library includes multiple customer service audios and audio content corresponding to the customer service audio; parse the initial text and obtain the text content corresponding to the initial text; match according to the text content and the audio content, and use the customer service audio corresponding to the successfully matched audio content as the reference audio.
[0096] In some embodiments, the prosody alignment module 205 is further used to: obtain voice preference parameters corresponding to target customers in the financial system; the voice preference parameters include at least one or more of gender information, age information and audio information; input the Mel spectrum and voice preference parameters into a preset vocoder, and the vocoder recognizes the Mel spectrum and voice preference parameters to generate synthetic audio.
[0097] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0098] The above-mentioned device can be implemented in the form of a computer program. Figure 5 Runs on the computer device shown.
[0099] See also Figure 5 , Figure 5 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server. Figure 5 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.
[0100] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any speech synthesis method.
[0101] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.
[0102] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any speech synthesis method.
[0103] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 5The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0104] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0105] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:
[0106] Obtaining the initial text and reference audio to be synthesized;
[0107] Inputting the initial text into a preset phoneme encoder, the phoneme encoder parses the initial text and outputs a plurality of phonemes corresponding to the initial text;
[0108] Inputting the reference audio into a preset rhythm encoder, the rhythm encoder parses the reference audio and outputs a plurality of rhythms corresponding to the reference audio;
[0109] Based on the self-attention mechanism, each phoneme and multiple rhythms are aligned to obtain the embedding information corresponding to each phoneme; the embedding information includes at least one rhythm and the weight corresponding to each rhythm;
[0110] A Mel spectrum is generated based on each phoneme and the embedded information corresponding to the phoneme, and synthetic audio is generated based on the Mel spectrum to complete the rhythm alignment of the initial text and the reference audio.
[0111] In some embodiments, each phoneme and multiple rhythms are aligned based on a self-attention mechanism to obtain embedding information corresponding to each phoneme, including: determining the embedding dimension corresponding to the phoneme and the rhythm according to the number of rhythms; aligning each phoneme with each rhythm according to the embedding dimension based on the self-attention mechanism to obtain the weight corresponding to each rhythm; and constructing embedding information according to multiple rhythms and the weights corresponding to the rhythms.
[0112] Exemplarily, each phoneme is aligned with each rhythm according to the embedding dimension based on the self-attention mechanism to obtain the weight corresponding to each rhythm, including: calculating the weight corresponding to each rhythm according to the flexible maximum activation function, so as to calculate the embedding information according to a preset embedding formula; the expression of the embedding formula includes:
[0113]
[0114] Among them, α is the embedded information, softmax() is the value of the maximum activation function, H is the phoneme, C is the rhythm set composed of multiple rhythms, and d is the embedding dimension.
[0115] In some embodiments, a mel spectrum is generated according to each phoneme and the embedded information corresponding to the phoneme, including: generating an embedded text according to each phoneme information and the corresponding embedded information; normalizing the embedded information corresponding to each phoneme within the range of a standard normal distribution; generating synthetic information according to the embedded text and the normalized phoneme; and inputting the synthetic information into a preset decoder, which parses the synthetic information to generate a mel spectrum.
[0116] Exemplarily, generating synthetic information according to the embedded text and the normalized phonemes includes: inputting the embedded text into a preset linear layer to obtain a linear text corresponding to the embedded text; and outputting the synthetic information according to a preset information synthesis formula, the information synthesis formula including:
[0117]
[0118] Among them, y is the synthetic information, f(W) is the linear text, W is the embedded text, H is the phoneme, and H mean is the average value corresponding to the phoneme, H var is the variance corresponding to the phoneme.
[0119] In some embodiments, obtaining an initial text to be synthesized and a reference audio includes: obtaining the initial text and the scenario type corresponding to the initial text in the financial system; obtaining an intelligent customer service audio library corresponding to the scenario type; the intelligent customer service voice library includes multiple customer service audios and audio content corresponding to the customer service audios; parsing the initial text to obtain text content corresponding to the initial text; matching based on the text content and the audio content, and using the customer service audio corresponding to the successfully matched audio content as the reference audio.
[0120] In some embodiments, generating synthesized audio based on Mel spectrum includes: obtaining voice preference parameters corresponding to target customers in a financial system; the voice preference parameters include at least one or more of gender information, age information, and audio information; inputting the Mel spectrum and the voice preference parameters into a preset vocoder, the vocoder recognizes the Mel spectrum and the voice preference parameters, and generates synthesized audio.
[0121] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a computer program. The computer program includes program instructions. The processor executes the program instructions to implement any speech synthesis method provided in the embodiment of the present application.
[0122] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.
[0123] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A speech synthesis method, characterized in that: include: Obtaining the initial text and reference audio to be synthesized; Inputting the initial text into a preset phoneme encoder, the phoneme encoder parses the initial text and outputs a plurality of phonemes corresponding to the initial text; Inputting the reference audio into a preset prosody encoder, the prosody encoder parses the reference audio and outputs a plurality of prosody corresponding to the reference audio; Acquire each of the phonemes and the multiple rhythms for alignment based on the self-attention mechanism, and acquire the embedding information corresponding to each of the phonemes; The embedded information includes at least one of the rhythms and a weight corresponding to each of the rhythms; A Mel spectrum is generated according to each of the phonemes and the embedded information corresponding to the phonemes, and a synthetic audio is generated according to the Mel spectrum, so as to complete the rhythm alignment of the initial text and the reference audio.
2. The method according to claim 1, characterized in that The step of obtaining each of the phonemes and aligning the multiple prosody based on the self-attention mechanism to obtain the embedding information corresponding to each of the phonemes includes: Determining the embedding dimension corresponding to the phoneme and the rhythm according to the number of the rhythm; Based on the self-attention mechanism, each of the phonemes is aligned with each of the rhythms according to the embedding dimension, and a weight corresponding to each of the rhythms is obtained; The embedded information is constructed according to a plurality of the rhythms and weights corresponding to the rhythms.
3. The method according to claim 2, characterized in that The step of aligning each of the phonemes with each of the rhythms based on the embedding dimension based on the self-attention mechanism to obtain a weight corresponding to each of the rhythms includes: Calculating the weight corresponding to each rhythm according to the flexible maximum activation function, so as to calculate the embedding information according to a preset embedding formula; The expression of the embedded formula includes: Among them, α is the embedding information, softmax() is the value of the maximum activation function, H is the phoneme, C is a rhythm set composed of multiple rhythms, and d is the embedding dimension.
4. The method according to claim 1, characterized in that: Generating a Mel spectrum according to each of the phonemes and the embedded information corresponding to the phonemes includes: Generate embedded text according to each of the phoneme information and the corresponding embedded information; Normalizing the embedded information corresponding to each of the phonemes within the range of a standard normal distribution; generating synthesis information according to the embedded text and the normalized phonemes; The synthesized information is input into a preset decoder, and the decoder parses the synthesized information to generate the Mel spectrum.
5. The method according to claim 4, characterized in that The generating synthetic information according to the embedded text and the normalized phonemes includes: Inputting the embedded text into a preset linear layer to obtain a linear text corresponding to the embedded text; The synthesized information is output according to a preset information synthesis formula, wherein the information synthesis formula includes: Where y is the synthesis information, f(W) is the linear text, W is the embedded text, H is the phoneme, and H mean is the average value corresponding to the phoneme, H var is the variance corresponding to the phoneme.
6. The method according to claim 1, characterized in that The step of obtaining the initial text to be synthesized and the reference audio comprises: Acquire the initial text and the scenario type corresponding to the initial text in the financial system; Obtain an intelligent customer service audio library corresponding to the scenario type; the intelligent customer service voice library includes multiple customer service audios and audio content corresponding to the customer service audios; Parsing the initial text to obtain text content corresponding to the initial text; Matching is performed based on the text content and the audio content, and the customer service audio corresponding to the successfully matched audio content is used as the reference audio.
7. The method according to claim 1, characterized in that The step of generating a synthesized audio according to the Mel spectrum comprises: Acquire voice preference parameters corresponding to target customers in the financial system; the voice preference parameters include at least one or more of gender information, age information and audio information; The Mel spectrum and the voice preference parameter are input into a preset vocoder, and the vocoder recognizes the Mel spectrum and the voice preference parameter to generate the synthesized audio.
8. A speech synthesis device, characterized in that: include: A first acquisition module, used to acquire an initial text to be synthesized and a reference audio; A phoneme output module, used for inputting the initial text into a preset phoneme encoder, wherein the phoneme encoder parses the initial text and outputs a plurality of phonemes corresponding to the initial text; A rhythm output module, used for inputting the reference audio into a preset rhythm encoder, wherein the rhythm encoder parses the reference audio and outputs a plurality of rhythms corresponding to the reference audio; A second acquisition module is used to acquire each of the phonemes and the plurality of prosody based on a self-attention mechanism for alignment, and acquire embedding information corresponding to each of the phonemes; The embedded information includes at least one of the rhythms and a weight corresponding to each of the rhythms; The prosody alignment module is used to generate a Mel spectrum according to each of the phonemes and the embedded information corresponding to the phonemes, generate a synthetic audio according to the Mel spectrum, and complete the prosody alignment of the initial text and the reference audio.
9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the speech synthesis method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the speech synthesis method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Information processing method and device
CN113096641A
Speech synthesis method and device, equipment and storage medium
CN113178188A
Speech synthesis method and device, medium, computer equipment and program product
CN114360493A
Neural network-based multi-scale fusion pronunciation evaluation model optimization method
CN115240710A
Voice cloning method and system based on cross-domain consistency loss
CN116229932A
Cited By
Audio synthesis method, training method of audio synthesis model and related device
CN119763540A
