Speech synthesis methods, devices, computer equipment and storage media

Abstract

This application provides a speech synthesis method, apparatus, computer device, and storage medium. The method relates to speech synthesis technology and is applied in the financial field. It includes: acquiring initial text to be synthesized and reference audio; inputting the initial text into a preset phoneme encoder, outputting multiple phonemes corresponding to the initial text; inputting the reference audio into a preset prosodic encoder, outputting multiple prosodices corresponding to the reference audio; aligning each phoneme and multiple prosodices based on a self-attention mechanism, acquiring embedding information corresponding to each phoneme; the embedding information includes at least one prosodic and a weight corresponding to each prosodic; generating a Mel spectrum based on each phoneme and its corresponding embedding information; generating synthesized audio based on the Mel spectrum, thus completing the prosodic alignment of the initial text and the reference audio. After synthesis, the prosodices will appear in accurate positions in the audio, improving the naturalness, credibility, persuasiveness, and appeal of the speech, and adapting to different application scenarios and user preferences.

Description

Technical Field

[0001] This application relates to the field of speech synthesis technology, and in particular to a speech synthesis method, apparatus, computer equipment, and storage medium. Background Technology

[0002] In the financial sector, the sheer number of online inquiries received daily would drastically increase labor costs if handled manually. Therefore, intelligent financial customer service systems can effectively unify customer service across the entire financial system.

[0003] For the deployment of intelligent financial customer service, voice synthesis is the most crucial component. Voice synthesis with emotional intelligence can improve the naturalness and credibility of the voice, enhance its persuasiveness and appeal, and adapt to different application scenarios and user preferences.

[0004] Existing emotional speech synthesis methods often use an additional input reference audio to achieve emotional speech synthesis through style transfer. However, if the location of the reference audio is misaligned during the process of borrowing emotions, the resulting speech will have poor naturalness, leading to a poor communication experience between users and intelligent customer service. Summary of the Invention

[0005] This application discloses a speech synthesis method, apparatus, computer device, and storage medium, which aims to solve the problem that existing emotional speech synthesis often uses an additional input reference audio to achieve emotional speech synthesis through style transfer. However, if the position of the reference audio is deviated during the process of borrowing the emotion, the final generated speech will have poor naturalness, resulting in a poor communication experience for users and intelligent customer service.

[0006] Firstly, this application provides a speech synthesis method, including:

[0007] Obtain the initial text to be synthesized and the reference audio;

[0008] The initial text is input into the preset phoneme encoder, which parses the initial text and outputs multiple phonemes corresponding to the initial text.

[0009] The reference audio is input into a preset prosodic encoder, which parses the reference audio and outputs multiple prosodices corresponding to the reference audio.

[0010] Each phoneme is aligned with multiple prosody based on a self-attention mechanism to obtain the embedding information corresponding to each phoneme; the embedding information includes at least one prosody and the weight corresponding to each prosody.

[0011] A Mel spectrum is generated based on each phoneme and its corresponding embedding information. Synthesized audio is then generated based on the Mel spectrum, completing the prosodic alignment between the initial text and the reference audio.

[0012] In some embodiments, each phoneme is aligned with multiple prosody based on a self-attention mechanism to obtain embedding information corresponding to each phoneme, including: determining the embedding dimension corresponding to the phoneme and the prosody based on the number of prosody; aligning each phoneme with each prosody according to the embedding dimension based on the self-attention mechanism to obtain the weight corresponding to each prosody; and constructing embedding information based on multiple prosody and the weights corresponding to the prosody.

[0013] For example, based on a self-attention mechanism, each phoneme is aligned with each prosody according to the embedding dimension to obtain the weight corresponding to each prosody. This includes: calculating the weight corresponding to each prosody based on a flexible maximum activation function, and then calculating the embedding information according to a preset embedding formula; the expression of the embedding formula includes:

[0014]

[0015] Where α is the embedding information, softmax() is the value of the maximum activation function, H is the phoneme, C is the prosodic set composed of multiple prosodices, and d is the embedding dimension.

[0016] In some embodiments, generating a Mel spectrum based on each phoneme and its corresponding embedding information includes: generating embedded text based on each phoneme information and its corresponding embedding information; normalizing the embedding information corresponding to each phoneme within a standard normal distribution range; generating synthesis information based on the embedded text and the normalized phoneme; and inputting the synthesis information to a preset decoder, which parses the synthesis information to generate a Mel spectrum.

[0017] For example, generating synthesized information based on embedded text and normalized phonemes includes: inputting the embedded text into a preset linear layer to obtain the linear text corresponding to the embedded text; and outputting synthesized information according to a preset information synthesis formula, the information synthesis formula including:

[0018]

[0019] Where y represents the synthesized information, f(W) represents the linear text, W represents the embedded text, H represents the phoneme, and H represents the... mean H is the average value corresponding to the phonemes. var denoted as the variance of the phonemes.

[0020] In some embodiments, obtaining the initial text to be synthesized and the reference audio includes: obtaining the initial text and the scenario type corresponding to the initial text in the financial system; obtaining the intelligent customer service audio library corresponding to the scenario type; the intelligent customer service audio library includes multiple customer service audios and the audio content corresponding to the customer service audios; parsing the initial text and obtaining the text content corresponding to the initial text; matching the text content and the audio content, and using the customer service audio corresponding to the successfully matched audio content as the reference audio.

[0021] In some embodiments, generating synthesized audio based on the Mel spectrum includes: obtaining voice preference parameters corresponding to a target customer in a financial system; the voice preference parameters include at least one or more of gender information, age information, and audio information; inputting the Mel spectrum and voice preference parameters into a preset vocoder, the vocoder recognizing the Mel spectrum and voice preference parameters to generate synthesized audio.

[0022] Secondly, this application provides a speech synthesis device, comprising:

[0023] The first acquisition module is used to acquire the initial text to be synthesized and the reference audio.

[0024] The phoneme output module is used to input the initial text into the preset phoneme encoder. The phoneme encoder parses the initial text and outputs multiple phonemes corresponding to the initial text.

[0025] The prosody output module is used to input reference audio to a preset prosody encoder. The prosody encoder parses the reference audio and outputs multiple prosody corresponding to the reference audio.

[0026] The second acquisition module is used to acquire each phoneme and multiple prosody based on a self-attention mechanism and align them, and to acquire the embedding information corresponding to each phoneme; the embedding information includes at least one prosody and the weight corresponding to each prosody.

[0027] The prosody alignment module is used to generate a Mel spectrum based on each phoneme and its corresponding embedding information, generate synthesized audio based on the Mel spectrum, and complete the prosody alignment of the initial text and the reference audio.

[0028] Thirdly, this application provides a computer device including a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein the memory stores a policy model, and wherein the computer program, when executed by the processor, implements the speech synthesis method provided in any embodiment of this application.

[0029] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the speech synthesis method provided in any embodiment of this application.

[0030] This application provides a speech synthesis method, apparatus, computer device, and storage medium. The provided method embeds and aligns the phonemes corresponding to the initial text and the prosody corresponding to the reference audio, embedding each prosody according to its individual weight. As a result, the prosody appears in the accurate position in the synthesized audio, improving the naturalness and credibility of the speech, enhancing its persuasiveness and appeal, and adapting to different application scenarios and user preferences.

[0031] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a schematic flowchart illustrating the steps of a speech synthesis method provided in an embodiment of this application;

[0034] Figure 2 This is a schematic diagram illustrating the embedding of rhythm and text according to an embodiment of this application;

[0035] Figure 3 This is a schematic diagram illustrating the principle of a speech synthesis method provided in an embodiment of this application;

[0036] Figure 4 This is a schematic diagram of the structure of a speech synthesis device provided in an embodiment of this application;

[0037] Figure 5 This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application.

[0038] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Detailed Implementation

[0039] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0040] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0041] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0042] It should be understood that, in order to clearly describe the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish the same or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0043] It should also be understood that the term "and / or" as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0044] To facilitate understanding of the embodiments of this application, some terms involved in the embodiments of this application will be briefly explained below.

[0045] 1. Emotional Speech Synthesis: Emotional speech synthesis is a technology that uses artificial intelligence to generate speech with corresponding emotional characteristics based on text content and emotional tags. The background of emotional speech synthesis is that, with the development of human-computer interaction, traditional speech synthesis technology can no longer meet users' needs for natural, expressive, and personalized voices. Therefore, emotional speech synthesis aims to improve the naturalness and credibility of speech, enhance its persuasiveness and appeal, and adapt to different application scenarios and user preferences. Application areas of emotional speech synthesis include intelligent customer service, intelligent education, intelligent entertainment, and intelligent healthcare.

[0046] 2. Phone: A phone is the smallest speech unit divided according to the natural attributes of speech. Analyzing based on the pronunciation actions in a syllable, one action constitutes one phone. Phones are divided into two major categories: vowels and consonants. For example, the Chinese syllable "ā" has only one phone, "ài" has two phones, and "dài" has three phones, etc.

[0047] The following will结合附图,对本申请的一些实施方式作详细说明。在不冲突的情况下,下述的实施例及实施例中的特征可以相互组合。 Hereinafter, some embodiments of the present application will be described in detail with reference to the accompanying drawings. Without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0048] In the financial field, since the number of online consulting customers to be received daily is extremely large, if manual customer service is used to receive them one by one, it will lead to a sharp increase in labor costs. Therefore, the setting of an intelligent financial customer service can well achieve the unification of the customer service in the financial system.

[0049] For the deployment of an intelligent financial customer service, the speech synthesis of the financial customer service is the most crucial link. The speech synthesis of an intelligent customer service with emotion can improve the naturalness and credibility of the speech, enhance the appeal and persuasion of the speech, and adapt to different application scenarios and user preferences.

[0050] Existing emotional speech synthesis often uses an additional input reference audio to achieve emotional speech synthesis through style transfer. However, once there is a deviation in the borrowing position during the process of borrowing the emotion of the reference audio, it will lead to very poor naturalness of the finally generated speech, resulting in a poor communication experience between the user and the intelligent customer service.

[0051] To solve the above problems, an embodiment of the present application proposes a speech synthesis method. Please refer to Figure 1 , Figure 1 is a schematic flowchart of the steps of the speech synthesis method provided by an embodiment of the present application.

[0052] It should be noted that the speech synthesis method provided by the present application can be used in a scenario with a large reception demand such as a financial system. Through the deployment of an intelligent customer service, the labor cost can be greatly reduced and the unified management of the customer service can be achieved. It can also be applied to any other scenario with a large reception demand during the speech synthesis of an intelligent customer service, such as the medical, educational, entertainment, etc. fields. The present application is described by taking the financial scenario as an example.

[0053] 如 Figure 1 所示,所提的语音合成方法包括步骤S101至S105。 As shown in

[0054] S101. Obtain the initial text to be synthesized and the reference audio.

[0055] Specifically, during speech synthesis, the initial text to be synthesized is obtained. For example, when a user in a financial system enters the corresponding intelligent customer service interface, the corresponding dialogue text is generated first as the initial text. Simultaneously, reference audio for that scenario is used for synthesis, aiming to combine the advantages of the reference audio to synthesize the content of the initial text. This results in a synthesized audio with strong appeal.

[0056] It should be noted that, as Figure 2 As shown, for a sentence (such as...) Figure 2 The phrase "I'm so happy today" can be synthesized by applying different intonations to each character. Different alignments between the intonation and the text can generate different sounds, obviously... Figure 2 The alignment below is more reasonable than the alignment above. The synthesized speech effect will also be more expressive. Therefore, this application aims to obtain the most suitable text and prosody embedding method for each sentence in each initial text.

[0057] In some embodiments, obtaining the initial text to be synthesized and the reference audio includes: obtaining the initial text and the scenario type corresponding to the initial text in the financial system; obtaining the intelligent customer service audio library corresponding to the scenario type; the intelligent customer service audio library includes multiple customer service audios and the audio content corresponding to the customer service audios; parsing the initial text and obtaining the text content corresponding to the initial text; matching the text content and the audio content, and using the customer service audio corresponding to the successfully matched audio content as the reference audio.

[0058] Since the provided method is applied to systems with numerous scenarios, such as financial systems including quotations, policy renewals, product introductions, and daily communication, the method creates an intelligent customer service voice library by setting corresponding customer service audio sets for different scenarios. This allows for the rapid determination of the corresponding reference audio based on the content of each initial text. This enables standardized management of speech synthesis and ensures that each initial text obtains the most suitable reference audio.

[0059] S102. Input the initial text into the preset phoneme encoder. The phoneme encoder parses the initial text and outputs multiple phonemes corresponding to the initial text.

[0060] Specifically, by inputting the initial text into a preset phoneme encoder, the phoneme encoder can parse the content of the initial text and extract the phoneme corresponding to each character in the initial text. Then, it can selectively embed the most suitable prosody for different phonemes and their contextual relationships.

[0061] S103. Input the reference audio to the preset prosody encoder. The prosody encoder parses the reference audio and outputs multiple prosody corresponding to the reference audio.

[0062] Specifically, the prosodic encoder extracts prosodic embeddings from the reference audio. The prosodic embeddings include the phoneme of each note in the reference audio and its corresponding prosodic pattern. The prosodic pattern includes, for example,... Figure 2 The "inflection" shown. Furthermore, before embedding and aligning phonemes and prosody, as... Figure 3 The phonemes and prosody of the initial text and the reference audio are extracted first, as shown.

[0063] S104. Align each phoneme with multiple prosody based on a self-attention mechanism, and obtain the embedding information corresponding to each phoneme; the embedding information includes at least one prosody and the weight corresponding to each prosody.

[0064] Specifically, the provided method obtains phoneme prosody alignment information using a self-attention mechanism. The self-attention mechanism calculates the weights of each prosody corresponding to each phoneme, such as the weights of "intonation" or "pause." This allows the prosody corresponding to each character in the initial text to be determined. The method can also combine the weights of each phoneme's prosody to generate the prosody for each phoneme. For example, if a phoneme's prosody weights are: 0.9 for depression + 0.05 for rise + 0.04 for pause + 0.01 for pause, then depression can be used as the corresponding prosody, or the weights of the four prosody weights (intonation, rise, pause, and pause) can be combined to generate the target prosody for the phoneme. Therefore, the provided method can flexibly determine the prosody corresponding to a phoneme.

[0065] In some embodiments, each phoneme is aligned with multiple prosody based on a self-attention mechanism to obtain embedding information corresponding to each phoneme, including: determining the embedding dimension corresponding to the phoneme and the prosody based on the number of prosody; aligning each phoneme with each prosody according to the embedding dimension based on the self-attention mechanism to obtain the weight corresponding to each prosody; and constructing embedding information based on multiple prosody and the weights corresponding to the prosody.

[0066] The proposed method determines the embedding dimension based on the number of prosody, such as 4 when the prosody includes intonation. Then, it calculates the weight of each prosody corresponding to each phoneme based on a self-attention mechanism to obtain the embedding information of the prosody corresponding to each phoneme.

[0067] For example, based on a self-attention mechanism, each phoneme is aligned with each prosody according to the embedding dimension to obtain the weight corresponding to each prosody. This includes: calculating the weight corresponding to each prosody based on a flexible maximum activation function, and then calculating the embedding information according to a preset embedding formula; the expression of the embedding formula includes:

[0068]

[0069] Where α represents the embedding information, softmax() is the value of the maximum activation function, H represents the phoneme, C is the prosodic set composed of multiple prosodices, and d is the embedding dimension. The provided method can then quickly calculate the prosodices corresponding to each phoneme of the initial text combined with the reference audio.

[0070] S105. Generate a Mel spectrum based on each phoneme and its corresponding embedding information, generate synthesized audio based on the Mel spectrum, and complete the prosodic alignment of the initial text and the reference audio.

[0071] Specifically, after obtaining the embedding information corresponding to each phoneme, corresponding inference operations can be performed to generate the Mel spectrum to generate the final synthesized audio. Then, by obtaining the phonemes corresponding to the initial text and the prosody corresponding to the reference audio, embedding alignment is performed. Each prosody is embedded according to its individual corresponding weight, so that the prosody will appear in the accurate position in the synthesized audio. This can improve the naturalness and credibility of the speech, enhance the persuasiveness and appeal of the speech, and adapt to different application scenarios and user preferences.

[0072] In some embodiments, generating a Mel spectrum based on each phoneme and its corresponding embedding information includes: generating embedded text based on each phoneme information and its corresponding embedding information; normalizing the embedding information corresponding to each phoneme within a standard normal distribution range; generating synthesis information based on the embedded text and the normalized phoneme; and inputting the synthesis information to a preset decoder, which parses the synthesis information to generate a Mel spectrum.

[0073] Traditional speech synthesis methods directly concatenate text and prosody as input. For example... Figure 3 As shown, this application feeds the embedded text generated from the aligned prosodic embedding information and the normalized phonemes into the decoder together. This application uses the Conditional Layer Normalization (CLN) algorithm to combine the two by changing the distribution of the phoneme embedding latent variable space with the embedded text as a parameter.

[0074] For example, generating synthesized information based on embedded text and normalized phonemes includes: inputting the embedded text into a preset linear layer to obtain the linear text corresponding to the embedded text; and outputting synthesized information according to a preset information synthesis formula, the information synthesis formula including:

[0075]

[0076] Where y represents the synthesized information, f(W) represents the linear text, W represents the embedded text, H represents the phoneme, and H represents the... mean H is the average value corresponding to the phonemes. vary represents the variance corresponding to the phonemes. The mean and variance of the data H are changed by inputting the aligned prosodic embedding W. The decoder input at this time is y, which is the text embedding H after changing the mean and variance according to the prosodic embedding W.

[0077] In some embodiments, generating synthesized audio based on the Mel spectrum includes: obtaining voice preference parameters corresponding to a target customer in a financial system; the voice preference parameters include at least one or more of gender information, age information, and audio information; inputting the Mel spectrum and voice preference parameters into a preset vocoder, the vocoder recognizing the Mel spectrum and voice preference parameters to generate synthesized audio.

[0078] The provided method acquires customer voice preference parameters upon entry, such as gender and age (to confirm whether the voice is deep or brisk) and audio information (to confirm the customer service representative's voice volume and speaking speed). It can also include information such as dialect type. Furthermore, a preset vocoder can generate more human-like audio, improving the user experience.

[0079] When deploying intelligent customer service in systems such as financial systems using the method provided in this application, the phonemes corresponding to the initial text and the prosody corresponding to the reference audio of a specified standard customer service are embedded and aligned. Each prosody is embedded according to its own corresponding weight, so that the prosody will appear in the accurate position in the synthesized audio. This can improve the naturalness and credibility of the speech, enhance the persuasiveness and appeal of the speech, and adapt to different application scenarios and user preferences.

[0080] like Figure 4 As shown, Figure 4 This is a schematic diagram of a speech synthesis device provided in an embodiment of this application. The speech synthesis device is used to perform the aforementioned speech synthesis method. The speech synthesis device can be configured on a terminal or a server.

[0081] like Figure 4 As shown, the speech synthesis device 200 includes a first acquisition module 201, a phoneme output module 202, a prosody output module 203, a second acquisition module 204, and a prosody alignment module 205.

[0082] The first acquisition module 201 is used to acquire the initial text to be synthesized and the reference audio.

[0083] The phoneme output module 202 is used to input the initial text to the preset phoneme encoder, the phoneme encoder parses the initial text and outputs multiple phonemes corresponding to the initial text.

[0084] The prosody output module 203 is used to input reference audio to a preset prosody encoder. The prosody encoder parses the reference audio and outputs multiple prosody corresponding to the reference audio.

[0085] The second acquisition module 204 is used to acquire each phoneme and multiple prosody based on a self-attention mechanism and align them, and acquire the embedding information corresponding to each phoneme; the embedding information includes at least one prosody and the weight corresponding to each prosody.

[0086] The prosody alignment module 205 is used to generate a Mel spectrum based on each phoneme and its corresponding embedding information, generate synthesized audio based on the Mel spectrum, and complete the prosody alignment of the initial text and the reference audio.

[0087] In some embodiments, the second acquisition module 204 is further configured to: determine the embedding dimension corresponding to the phoneme and the prosody based on the number of prosody; align each phoneme with each prosody according to the embedding dimension based on the self-attention mechanism, and obtain the weight corresponding to each prosody; and construct embedding information based on multiple prosody and the weights corresponding to the prosody.

[0088] For example, the second acquisition module 204 is further configured to: calculate the weight corresponding to each prosody based on the flexible maximum activation function, so as to calculate the embedding information according to a preset embedding formula; the expression of the embedding formula includes:

[0089]

[0090] Where α is the embedding information, softmax() is the value of the maximum activation function, H is the phoneme, C is the prosodic set composed of multiple prosodices, and d is the embedding dimension.

[0091] In some embodiments, the prosody alignment module 205 is further configured to: generate embedded text based on each phoneme information and the corresponding embedded information; normalize the embedded information corresponding to each phoneme within a standard normal distribution range; generate synthesis information based on the embedded text and the normalized phoneme; and input the synthesis information to a preset decoder, which parses the synthesis information to generate a Mel spectrum.

[0092] For example, the prosody alignment module 205 is further configured to: input embedded text into a preset linear layer to obtain the linear text corresponding to the embedded text; and output synthesized information according to a preset information synthesis formula, the information synthesis formula including:

[0093]

[0094] Where y represents the synthesized information, f(W) represents the linear text, W represents the embedded text, H represents the phoneme, and H represents the... mean H is the average value corresponding to the phonemes. var denoted as the variance of the phonemes.

[0095] In some embodiments, the first acquisition module 201 is further configured to: acquire the initial text and the scenario type corresponding to the initial text in the financial system; acquire the intelligent customer service audio library corresponding to the scenario type; the intelligent customer service audio library includes multiple customer service audios and audio content corresponding to the customer service audios; parse the initial text and acquire the text content corresponding to the initial text; match the text content and the audio content, and use the customer service audio corresponding to the successfully matched audio content as the reference audio.

[0096] In some embodiments, the prosody alignment module 205 is further configured to: obtain voice preference parameters corresponding to target customers in the financial system; the voice preference parameters include at least one or more of gender information, age information, and audio information; input the Mel spectrum and voice preference parameters to a preset vocoder, the vocoder recognizes the Mel spectrum and voice preference parameters, and generates synthesized audio.

[0097] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus and modules can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0098] The aforementioned device can be implemented as a computer program, which can be used in, for example... Figure 5 It runs on the computer device shown.

[0099] Please see Figure 5 , Figure 5 This is a schematic block diagram illustrating the structure of a computer device according to an embodiment of this application. The computer device may be a server. See also... Figure 5 The computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0100] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any speech synthesis method.

[0101] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0102] Internal memory provides an environment for the execution of computer programs stored in non-volatile storage media. When these computer programs are executed by a processor, the processor can perform any speech synthesis method.

[0103] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 5The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0104] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0105] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps:

[0106] Obtain the initial text to be synthesized and the reference audio;

[0107] The initial text is input into the preset phoneme encoder, which parses the initial text and outputs multiple phonemes corresponding to the initial text.

[0108] The reference audio is input into a preset prosodic encoder, which parses the reference audio and outputs multiple prosodices corresponding to the reference audio.

[0109] Each phoneme is aligned with multiple prosody based on a self-attention mechanism to obtain the embedding information corresponding to each phoneme; the embedding information includes at least one prosody and the weight corresponding to each prosody.

[0110] A Mel spectrum is generated based on each phoneme and its corresponding embedding information. Synthesized audio is then generated based on the Mel spectrum, completing the prosodic alignment between the initial text and the reference audio.

[0111] In some embodiments, each phoneme is aligned with multiple prosody based on a self-attention mechanism to obtain embedding information corresponding to each phoneme, including: determining the embedding dimension corresponding to the phoneme and the prosody based on the number of prosody; aligning each phoneme with each prosody according to the embedding dimension based on the self-attention mechanism to obtain the weight corresponding to each prosody; and constructing embedding information based on multiple prosody and the weights corresponding to the prosody.

[0112] For example, based on a self-attention mechanism, each phoneme is aligned with each prosody according to the embedding dimension to obtain the weight corresponding to each prosody. This includes: calculating the weight corresponding to each prosody based on a flexible maximum activation function, and then calculating the embedding information according to a preset embedding formula; the expression of the embedding formula includes:

[0113]

[0114] Where α is the embedding information, softmax() is the value of the maximum activation function, H is the phoneme, C is the prosodic set composed of multiple prosodices, and d is the embedding dimension.

[0115] In some embodiments, generating a Mel spectrum based on each phoneme and its corresponding embedding information includes: generating embedded text based on each phoneme information and its corresponding embedding information; normalizing the embedding information corresponding to each phoneme within a standard normal distribution range; generating synthesis information based on the embedded text and the normalized phoneme; and inputting the synthesis information to a preset decoder, which parses the synthesis information to generate a Mel spectrum.

[0116] For example, generating synthesized information based on embedded text and normalized phonemes includes: inputting the embedded text into a preset linear layer to obtain the linear text corresponding to the embedded text; and outputting synthesized information according to a preset information synthesis formula, the information synthesis formula including:

[0117]

[0118] Where y represents the synthesized information, f(W) represents the linear text, W represents the embedded text, H represents the phoneme, and H represents the... mean H is the average value corresponding to the phonemes. var denoted as the variance of the phonemes.

[0119] In some embodiments, obtaining the initial text to be synthesized and the reference audio includes: obtaining the initial text and the scenario type corresponding to the initial text in the financial system; obtaining the intelligent customer service audio library corresponding to the scenario type; the intelligent customer service audio library includes multiple customer service audios and the audio content corresponding to the customer service audios; parsing the initial text and obtaining the text content corresponding to the initial text; matching the text content and the audio content, and using the customer service audio corresponding to the successfully matched audio content as the reference audio.

[0120] In some embodiments, generating synthesized audio based on the Mel spectrum includes: obtaining voice preference parameters corresponding to a target customer in a financial system; the voice preference parameters include at least one or more of gender information, age information, and audio information; inputting the Mel spectrum and voice preference parameters into a preset vocoder, the vocoder recognizing the Mel spectrum and voice preference parameters to generate synthesized audio.

[0121] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the speech synthesis methods provided in the embodiments of this application.

[0122] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.

[0123] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A speech synthesis method characterized by, The method comprises the steps of: obtaining initial text and reference audio to be synthesized; inputting the initial text into a preset phoneme encoder, the phoneme encoder analyzing the initial text and outputting a plurality of phonemes corresponding to the initial text; inputting the reference audio into a preset prosody encoder, the prosody encoder analyzing the reference audio and outputting a plurality of prosodies corresponding to the reference audio; aligning each phoneme with a plurality of prosodies based on a self-attention mechanism to obtain embedding information corresponding to each phoneme; the embedding information comprises at least one prosody and a weight corresponding to each prosody; generating a mel spectrum according to each phoneme and the embedding information corresponding to the phoneme, comprising: generating an embedding text according to each phoneme information and the corresponding embedding information; normalizing the embedding information corresponding to each phoneme within a standard normal distribution range; generating synthesis information according to the embedding text and the normalized phoneme, comprising: inputting the embedding text into a preset linear layer to obtain linear text corresponding to the embedding text; outputting the synthesis information according to a preset information synthesis formula; inputting the synthesis information into a preset decoder, the decoder analyzing the synthesis information to generate the mel spectrum, the embedding text generated by the embedding information of the aligned prosody and the normalized phoneme are sent into the decoder together, the conditional layer normalization algorithm is used to change the distribution of the phoneme embedding hidden variable space with the embedding text as the parameter to combine the embedding text and the phoneme, and a synthesized audio is generated according to the mel spectrum, completing the prosody alignment of the initial text and the reference audio.

2. The method of claim 1, wherein, The method comprises the steps of: determining the embedding dimension corresponding to the phoneme and the prosody according to the number of prosodies; aligning each phoneme with each prosody based on a self-attention mechanism according to the embedding dimension to obtain a weight corresponding to each prosody; constructing the embedding information according to a plurality of prosodies and the weight corresponding to the prosody.

3. The method of claim 2, wherein, The method comprises the steps of: calculating the weight corresponding to each prosody according to a flexible maximum activation function to calculate the embedding information according to a preset embedding formula; the expression of the embedding formula comprises: ； wherein, is the embedding information, is the value of the maximum activation function, H is the phoneme, C is a prosodic set of a plurality of prosodic components, and d is the embedding dimension.

4. The method of claim 1, wherein, the information synthesis formula comprises: ； wherein y is the synthesis information, f(W) is the linear text, W is the embedded text, H is the phonemes, is the average value corresponding to the phonemes, is the variance corresponding to the phonemes.

5. The method of claim 1, wherein, The method comprises the steps of: obtaining the initial text and the scene type corresponding to the initial text in the financial system; obtaining an intelligent customer service audio library corresponding to the scene type; the intelligent customer service audio library comprises a plurality of customer service audios and audio content corresponding to the customer service audios; analyzing the initial text to obtain text content corresponding to the initial text; matching the text content with the audio content, and taking the customer service audio corresponding to the audio content that matches successfully as the reference audio.

6. The method of claim 1, wherein, The method comprises the steps of: Acquire the voice preference parameters corresponding to the target customer in the financial system; The voice preference parameters at least include one or more of gender information, age information and audio information; Input the mel spectrum and the voice preference parameters into a preset vocoder, and the vocoder identifies the mel spectrum and the voice preference parameters to generate the synthesized audio.

7. A speech synthesis apparatus characterized by comprising: Comprise: The first acquisition module is used for acquiring the initial text to be synthesized and the reference audio; The phoneme output module is used for inputting the initial text into a preset phoneme encoder, and the phoneme encoder analyzes the initial text and outputs a plurality of phonemes corresponding to the initial text; The prosody output module is used for inputting the reference audio into a preset prosody encoder, and the prosody encoder analyzes the reference audio and outputs a plurality of prosodies corresponding to the reference audio; The second acquisition module is used for acquiring the alignment of each phoneme and a plurality of prosodies based on a self-attention mechanism, and acquiring embedding information corresponding to each phoneme; The embedding information comprises at least one prosody and a weight corresponding to each prosody; The prosody alignment module is used for generating a mel spectrum according to each phoneme and the embedding information corresponding to the phoneme, comprising: generating an embedding text according to each phoneme information and the corresponding embedding information; normalizing the embedding information corresponding to each phoneme in the range of standard normal distribution; generating synthesis information according to the embedding text and the normalized phoneme, comprising: inputting the embedding text into a preset linear layer to acquire a linear text corresponding to the embedding text; outputting the synthesis information according to a preset information synthesis formula; inputting the synthesis information into a preset decoder, and the decoder analyzes the synthesis information to generate the mel spectrum, the embedding text generated by the embedding information of the aligned prosody and the normalized phoneme are sent into the decoder together, the conditional layer normalization algorithm is used to change the distribution of phoneme embedding hidden variable space with the embedding text as the parameter to combine the embedding text and the phoneme, the synthesis audio is generated according to the mel spectrum, and the prosody alignment of the initial text and the reference audio is completed.

8. A computer device, comprising: The computer device comprises a memory and a processor; The memory is used for storing a computer program; The processor is used for executing the computer program and realizing the voice synthesis method in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to make the processor realize the voice synthesis method in any one of claims 1 to 6.

Images