Speech synthesis method and device, training method and device, equipment and storage medium
By extracting and fusing the emotional feature vectors in the speech synthesis method, voices that meet the emotional categories and intensity are generated, which solves the problem of unnatural speech synthesis effects in the prior art, and improves the naturalness and user experience of speech synthesis.
Patent Information
- Application Number
- CN202411406224.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-09
- Publication Date
- 2025-01-10
AI Technical Summary
Existing speech synthesis methods cannot generate synthetic speech that meets the current emotional categories and emotional intensity, resulting in unnatural speech synthesis effects.
By obtaining the ranking scores of the emotional category and emotional intensity of the text, feature extraction and fusion processing are performed, and a preset speech synthesis model is used to generate synthetic speech that matches the emotional category and emotional intensity.
It improves the naturalness and user experience of speech synthesis, enhances the emotional transmission effect, and makes the speech more vivid and natural.
Smart Images
Figure CN119296509A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of artificial intelligence and financial technology, and in particular to a speech synthesis method, training method, device, equipment and storage medium. Background Art
[0002] Text-to-Speech (TTS) is an advanced technology that can automatically convert input text into human speech. Speech synthesis has promoted the development of smart devices, made human-computer interaction more natural and smooth, and improved work efficiency and quality of life.
[0003] However, existing speech synthesis methods cannot generate synthesized speech that conforms to the current emotion category and current emotion intensity, which is not conducive to improving the speech synthesis effect of the current text. The reason is that the existing speech synthesis model only learns the reference audio or reference Mel spectrum, which does not have the emotion intensity, resulting in the speech synthesized by the speech synthesis model often appearing stiff and unnatural, which is not conducive to improving the speech synthesis effect of the current text. Summary of the invention
[0004] The present invention provides a speech synthesis method, a training method, a computer device and a storage medium to solve the technical problem that the existing speech synthesis method cannot generate synthesized speech that conforms to the current emotion category and the current emotion intensity, which is not conducive to improving the speech synthesis effect of the current text.
[0005] In a first aspect, a speech synthesis method is provided, comprising:
[0006] Acquire the current text, and input the current text into the trained sentiment intensity model;
[0007] Based on the trained sentiment intensity model, obtaining recognition information of the current text;
[0008] Obtaining a ranking score of a current emotion category and a current emotion intensity in the recognition information;
[0009] Performing feature extraction on the current emotion category to obtain a first feature vector, performing feature extraction on the ranking score of the current emotion intensity to obtain a second feature vector, and performing feature extraction on the current text to obtain a third feature vector;
[0010] Based on a predefined fusion method, the first feature vector, the second feature vector and the third feature vector are fused to determine a fourth feature vector;
[0011] The fourth feature vector is processed by speech synthesis using a preset speech synthesis model and a preset processing method to generate synthesized speech that conforms to the current emotion category and the current emotion intensity.
[0012] Furthermore, the obtaining of the recognition information of the current text based on the trained sentiment intensity model includes:
[0013] Obtaining an output file of the sentiment intensity model;
[0014] In the output file, the recognition information output by the trained sentiment intensity model based on the current text is obtained.
[0015] Further, the obtaining of the ranking score of the current emotion category and the current emotion intensity in the recognition information includes:
[0016] Obtain a first read instruction and a second read instruction;
[0017] The current emotion category in the identification information is read through the first reading instruction, and the ranking score of the current emotion intensity in the identification information is read through the second reading instruction.
[0018] Further, the feature extraction of the current emotion category to obtain a first feature vector, the feature extraction of the ranking score of the current emotion intensity to obtain a second feature vector, and the feature extraction of the current text to obtain a third feature vector include:
[0019] Input the current emotion category into a preset word vector model to obtain a first feature vector output by the word vector model based on the current emotion category;
[0020] Input the ranking score of the current sentiment intensity into the word vector model, and obtain the second feature vector output by the word vector model based on the ranking score of the current sentiment intensity; input the current text into the word vector model, and obtain the third feature vector output by the word vector model based on the current text.
[0021] Further, the fusing the first feature vector, the second feature vector and the third feature vector based on a predefined fusion method to determine a fourth feature vector includes:
[0022] Inputting the first feature vector, the second feature vector and the third feature vector into an encoder;
[0023] The first feature vector, the second feature vector and the third feature vector are fused through the attention mechanism of the encoder to obtain a fourth feature vector.
[0024] Further, the performing speech synthesis processing on the fourth feature vector by using a preset speech synthesis model and a preset processing method to generate a synthesized speech that conforms to the current emotion category and the current emotion intensity includes:
[0025] Obtaining a Mel-spectrogram of the fourth eigenvector, and inputting the Mel-spectrogram into a vocoder of the speech synthesis model;
[0026] The mel-spectrogram is processed by the vocoder for speech synthesis to generate synthesized speech that conforms to the current emotion category and the current emotion intensity.
[0027] In a second aspect, a training method for an analysis model is provided, wherein the analysis model is applied to the above-mentioned speech synthesis method, and the training method comprises:
[0028] Obtaining a preset text, obtaining annotation information of the preset text, obtaining a ranking score of an actual emotion category and an actual emotion intensity in the annotation information, obtaining prediction information output by the emotion intensity model based on the preset text, and obtaining a ranking score of a predicted emotion category and a predicted emotion intensity in the prediction information;
[0029] Obtain a first loss value between the predicted emotion category and the actual emotion category, and obtain a second loss value between the predicted emotion intensity ranking score and the actual emotion intensity ranking score; add the first loss value and the second loss value to obtain a third loss value, and train the emotion intensity model based on the third loss value. When the third loss value is less than the preset loss value, stop training the emotion intensity model and save the trained emotion intensity model.
[0030] In a third aspect, a speech synthesis device is provided, comprising:
[0031] A first acquisition module is used to acquire the current text and input the current text into the trained emotion intensity model;
[0032] A second acquisition module is used to acquire recognition information of the current text based on the trained sentiment intensity model;
[0033] A third acquisition module is used to obtain the ranking score of the current emotion category and the current emotion intensity in the recognition information;
[0034] An extraction module, used to perform feature extraction on the current emotion category to obtain a first feature vector, perform feature extraction on the ranking score of the current emotion intensity to obtain a second feature vector, and perform feature extraction on the current text to obtain a third feature vector;
[0035] A processing module, configured to fuse the first feature vector, the second feature vector and the third feature vector based on a predefined fusion method to determine a fourth feature vector;
[0036] The synthesis module is used to perform speech synthesis processing on the fourth feature vector through a preset speech synthesis model and a preset processing method to generate a synthesized speech that conforms to the current emotion category and the current emotion intensity.
[0037] In a fourth aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned speech synthesis method or the above-mentioned training method when executing the computer program.
[0038] In a fifth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned speech synthesis method or the above-mentioned training method are implemented.
[0039] The present application provides a speech synthesis method, training method, computer device and storage medium, which obtain a current text, input the current text into a trained emotion intensity model; based on the trained emotion intensity model, obtain recognition information of the current text; obtain the current emotion category and the ranking score of the current emotion intensity in the recognition information; perform feature extraction on the current emotion category to obtain a first feature vector, perform feature extraction on the ranking score of the current emotion intensity to obtain a second feature vector, perform feature extraction on the current text to obtain a third feature vector; perform fusion processing on the first feature vector, the second feature vector and the third feature vector based on a predefined fusion method to determine a fourth feature vector; and perform fusion processing on the first feature vector, the second feature vector and the third feature vector based on a preset speech synthesis model. The fourth eigenvector is processed by speech synthesis according to the preset speech synthesis model and the preset processing method to generate a synthesized speech that conforms to the current emotion category and the current emotion intensity. The beneficial effects are in two aspects. On the one hand, the ranking score of the current emotion intensity is subjected to feature extraction to obtain a second eigenvector, so that the speech synthesis model can learn the current emotion intensity through the second eigenvector; on the other hand, the first eigenvector, the second eigenvector and the third eigenvector are fused to determine the fourth eigenvector. The fourth eigenvector is processed by speech synthesis according to the preset speech synthesis model and the preset processing method to generate a synthesized speech that conforms to the current emotion category and the current emotion intensity, which is beneficial to improving the speech synthesis effect of the current text. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0041] Figure 1 is a schematic diagram of an application environment of a speech synthesis method according to an embodiment of the present invention;
[0042] Figure 2 A flowchart of a speech synthesis method provided by an embodiment of the present invention;
[0043] Figure 3 yes Figure 1 A schematic flow chart of a specific implementation of step S23;
[0044] Figure 4 yes Figure 1 A schematic flow chart of a specific implementation of step S25;
[0045] Figure 5 A schematic diagram of a flow chart of a training method provided by an embodiment of the present invention;
[0046] Figure 6 is a structural schematic diagram of a speech synthesis device in one embodiment of the present invention;
[0047] Figure 7 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;
[0048] Figure 8 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION
[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0050] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram of an application environment of a speech synthesis method according to an embodiment of the present invention. The speech synthesis method provided by the embodiment of the present invention can be applied in Figure 1 In an application environment, a client communicates with a server through a network.
[0051] The server obtains the current text through the client, and inputs the current text into the trained emotion intensity model;
[0052] Based on the trained sentiment intensity model, obtaining recognition information of the current text;
[0053] Obtaining a ranking score of a current emotion category and a current emotion intensity in the recognition information;
[0054] Performing feature extraction on the current emotion category to obtain a first feature vector, performing feature extraction on the ranking score of the current emotion intensity to obtain a second feature vector, and performing feature extraction on the current text to obtain a third feature vector;
[0055] Based on a predefined fusion method, the first feature vector, the second feature vector and the third feature vector are fused to determine a fourth feature vector;
[0056] The fourth feature vector is processed by speech synthesis using a preset speech synthesis model and a preset processing method to generate synthesized speech that conforms to the current emotion category and the current emotion intensity.
[0057] In the solution implemented by the above-mentioned speech synthesis method, training method, device and medium, the beneficial effects are as follows:
[0058] 1. Improve user experience: By strengthening emotional expression in speech synthesis, the synthesized speech can be made more natural and vivid, enhancing the user experience. For example, when using AI customer service, you can choose to express different emotional tones according to the usage scenario. For example, use a solemn and serious voice when broadcasting formal notices, and use a more friendly and enthusiastic voice when introducing products and communicating.
[0059] 2. Enhance emotional transmission: In some situations, speech synthesis is needed to convey important emotions. For example, in advertising, passion needs to be expressed, and the emotional expression needs to be stronger. In the field of education, patience and encouragement need to be expressed, and the emotions need to be delicate and soothing. The speech synthesis model can help select appropriate emotional expressions and enhance the effect of emotional transmission.
[0060] The device running the client is referred to as a client device.
[0061] The device running the server is referred to as the server device.
[0062] The client devices may include but are not limited to various personal computers, laptops, smart phones, tablet computers, and portable wearable devices.
[0063] The server device can be implemented with an independent task database or a task database cluster composed of multiple task databases. The present invention is described in detail below through specific embodiments.
[0064] See also Figure 2 , Figure 2 A flowchart of a speech synthesis method provided by an embodiment of the present invention includes the following steps:
[0065] S21, obtaining a current text, and inputting the current text into a trained emotion intensity model;
[0066] Exemplarily, obtaining a current text and inputting the current text into a trained sentiment intensity model includes:
[0067] Connect to a target platform, obtain a current text sent by the target platform, and input the current text into a trained emotion intensity model, wherein the target platform includes one of a financial platform and an insurance platform or a combination thereof.
[0068] The current text is the text to be speech synthesized.
[0069] S22, based on the trained sentiment intensity model, obtaining recognition information of the current text;
[0070] Wherein, obtaining the recognition information of the current text based on the trained sentiment intensity model includes:
[0071] Obtaining an output file of the sentiment intensity model;
[0072] In the output file, the recognition information output by the trained sentiment intensity model based on the current text is obtained.
[0073] S23, obtaining the ranking score of the current emotion category and the current emotion intensity in the recognition information;
[0074] Among them, the current sentiment category is the sentiment category of the current text.
[0075] Among them, the current sentiment intensity is the sentiment intensity of the current text.
[0076] S24, performing feature extraction on the current emotion category to obtain a first feature vector, performing feature extraction on the ranking score of the current emotion intensity to obtain a second feature vector, and performing feature extraction on the current text to obtain a third feature vector;
[0077] The step of extracting features from the current emotion category to obtain a first feature vector, extracting features from the ranking score of the current emotion intensity to obtain a second feature vector, and extracting features from the current text to obtain a third feature vector includes:
[0078] Input the current emotion category into a preset word vector model to obtain a first feature vector output by the word vector model based on the current emotion category;
[0079] Inputting the ranking score of the current sentiment intensity into the word vector model, and obtaining a second feature vector output by the word vector model based on the ranking score of the current sentiment intensity;
[0080] The current text is input into the word vector model, and a third feature vector output by the word vector model based on the current text is obtained.
[0081] S25, based on a predefined fusion method, fusing the first feature vector, the second feature vector and the third feature vector to determine a fourth feature vector;
[0082] S26, performing speech synthesis processing on the fourth feature vector through a preset speech synthesis model and a preset processing method to generate a synthesized speech that conforms to the current emotion category and the current emotion intensity.
[0083] Among them, the speech synthesis model (Text-to-Speech, TTS) is an important branch in the field of artificial intelligence, which aims to convert text into speech signals that can be understood by humans.
[0084] The speech synthesis models include but are not limited to the FastSpeech2 model and the Tacotron model.
[0085] Preferably, the speech synthesis model adopts the FastSpeech2 model.
[0086] Among them, the FastSpeech2 model is an improved version of FastSpeech, which aims to further improve the efficiency and quality of text-to-speech generation. By introducing more optimization strategies and technical improvements, the FastSpeech2 model improves the naturalness and accuracy of speech while maintaining its original efficiency, making it more suitable for various practical application scenarios.
[0087] Among them, the Tacotron model is based on an end-to-end model that converts text into speech waveforms through an encoder-decoder structure.
[0088] The specific implementation of step S26 is described in detail as follows:
[0089] Obtaining a Mel-spectrogram of the fourth eigenvector, and inputting the Mel-spectrogram into a vocoder of the speech synthesis model;
[0090] The mel-spectrogram is processed by the vocoder for speech synthesis to generate synthesized speech that conforms to the current emotion category and the current emotion intensity.
[0091] In an embodiment of the present invention, the beneficial effects lie in two aspects. On the one hand, feature extraction is performed on the ranking score of the current emotion intensity to obtain a second feature vector, so that the speech synthesis model can learn the current emotion intensity through the second feature vector; on the other hand, the first feature vector, the second feature vector and the third feature vector are fused to determine a fourth feature vector, and speech synthesis processing is performed on the fourth feature vector through a preset speech synthesis model and a preset processing method to generate a synthesized speech that conforms to the current emotion category and the current emotion intensity, which is beneficial to improving the speech synthesis effect of the current text.
[0092] See also Figure 3 , Figure 3 yes Figure 1 A specific implementation flow diagram of step S23 is described in detail as follows:
[0093] S31, obtaining a first read instruction and a second read instruction;
[0094] S32, reading the current emotion category in the identification information through the first reading instruction, and reading the ranking score of the current emotion intensity in the identification information through the second reading instruction.
[0095] Among them, the ranking score directly reflects the strength of the current emotion.
[0096] Among them, the higher the ranking score, the stronger the current emotion intensity; the lower the ranking score, the lower the current emotion intensity.
[0097] For the sake of explanation, the following are examples:
[0098] For example, when the emotion category is joy, in a five-point evaluation system, one point represents extremely low emotion intensity, which is extremely low joy; while five points represent extremely high emotion intensity, which is extreme joy.
[0099] For example, when the emotion category is sadness, in a five-point evaluation system, one point represents extremely low emotion intensity, which is extremely low sadness, while five points represent extremely high emotion intensity, which is extreme sadness.
[0100] In the embodiment of the present invention, the ranking score of the current emotion intensity in the identification information is read, and the ranking score provides a clear quantitative standard for evaluating the current emotion intensity.
[0101] See also Figure 4 , Figure 4 yes Figure 1 A specific implementation flow diagram of step S25 is described in detail as follows:
[0102] S41, inputting the first feature vector, the second feature vector and the third feature vector into an encoder;
[0103] S42: fusing the first feature vector, the second feature vector and the third feature vector through the attention mechanism of the encoder to obtain a fourth feature vector.
[0104] In an embodiment of the present invention, when the attention mechanism fuses the first feature vector, the second feature vector and the third feature vector, different weights are automatically assigned according to the importance of the first feature vector, the second feature vector and the third feature vector to the current task. This dynamic adjustment method enables the fused fourth feature vector to more accurately reflect the important features of the current text.
[0105] See also Figure 5 , Figure 5 A flowchart of a training method provided by an embodiment of the present invention is described in detail as follows:
[0106] S51, obtaining a preset text, obtaining annotation information of the preset text, obtaining the ranking score of the actual emotion category and the actual emotion intensity in the annotation information, obtaining the prediction information output by the emotion intensity model based on the preset text, and obtaining the ranking score of the predicted emotion category and the predicted emotion intensity in the prediction information;
[0107] The preset text is the text that has been speech synthesized.
[0108] S52, obtaining a first loss value between the predicted emotion category and the actual emotion category, and obtaining a second loss value between the ranking score of the predicted emotion intensity and the ranking score of the actual emotion intensity;
[0109] S53, adding the first loss value and the second loss value to obtain a third loss value, training the emotion intensity model based on the third loss value, and when the third loss value is less than the preset loss value, stopping the training of the emotion intensity model and saving the trained emotion intensity model.
[0110] Exemplarily, obtaining a first loss value between the predicted emotion category and the actual emotion category, and obtaining a second loss value between the ranking score of the predicted emotion intensity and the ranking score of the actual emotion intensity includes:
[0111] Obtaining a first loss value between the predicted emotion category and the actual emotion category through a preset cross entropy loss function;
[0112] A second loss value between the ranking score of the predicted sentiment intensity and the ranking score of the actual sentiment intensity is obtained through a preset ranking loss function.
[0113] Among them, the sorting loss function includes but is not limited to Pairwise loss function, Pointwise loss function, and Listwise loss function.
[0114] The Chinese name of the Pairwise loss function is: Pairwise loss function; the English name of the Pairwise loss function is: Pairwise Loss. The Pairwise loss function calculates the loss by comparing the ranking score of the predicted sentiment intensity with the ranking score of the actual sentiment intensity.
[0115] Among them, the Chinese name of the Pointwise loss function: point-by-point loss function; the English name of the Pointwise loss function: Pointwise Loss; The Pointwise loss function adopts an intuitive and relatively simple logic, treating the sorting task as a series of independent classification or regression problems.
[0116] Among them, the Chinese name of the Listwise loss function is: list loss function, and the English name of the Listwise loss function is: Listwise Loss. The Listwise loss function directly calculates the overall gap between the predicted sorted list and the actual sorted list.
[0117] In the embodiment of the present invention, the trained emotion intensity model is saved, and the trained emotion intensity model is reloaded and used when needed without retraining, which is conducive to reducing training costs.
[0118] See also Figure 6 , Figure 6 FIG. 1 is a schematic diagram of a structure of a speech synthesis device according to an embodiment of the present invention. Figure 6 As shown, the speech synthesis device includes a first acquisition module 101, a second acquisition module 102, a third acquisition module 103, an extraction module 104, a processing module 105, and a synthesis module 106. The functional modules are described in detail as follows:
[0119] A first acquisition module 101 is used to acquire a current text and input the current text into a trained emotion intensity model;
[0120] A second acquisition module 102 is used to acquire recognition information of the current text based on the trained sentiment intensity model;
[0121] The third acquisition module 103 is used to obtain the ranking score of the current emotion category and the current emotion intensity in the recognition information;
[0122] An extraction module 104 is used to perform feature extraction on the current emotion category to obtain a first feature vector, perform feature extraction on the ranking score of the current emotion intensity to obtain a second feature vector, and perform feature extraction on the current text to obtain a third feature vector;
[0123] A processing module 105 is used to fuse the first feature vector, the second feature vector and the third feature vector based on a predefined fusion method to determine a fourth feature vector;
[0124] The synthesis module 106 is used to perform speech synthesis processing on the fourth feature vector through a preset speech synthesis model and a preset processing method to generate a synthesized speech that meets the current emotion category and the current emotion intensity.
[0125] In one embodiment, the second acquisition module 102 includes:
[0126] A first acquisition subunit is used to acquire an output file of the emotion intensity model;
[0127] The second acquisition subunit is used to acquire, in the output file, recognition information output by the trained sentiment intensity model based on the current text.
[0128] In one embodiment, the third acquisition module 103 includes:
[0129] A third acquisition subunit, used to acquire a first read instruction and a second read instruction;
[0130] The reading subunit is used to read the current emotion category in the identification information through the first reading instruction, and read the ranking score of the current emotion intensity in the identification information through the second reading instruction.
[0131] In one embodiment, the extraction module 104 includes:
[0132] A first input subunit, used to input the current emotion category into a preset word vector model, and obtain a first feature vector output by the word vector model based on the current emotion category;
[0133] A second input subunit is used to input the ranking score of the current sentiment intensity into the word vector model, and obtain a second feature vector output by the word vector model based on the ranking score of the current sentiment intensity;
[0134] The third input subunit is used to input the current text into the word vector model to obtain a third feature vector output by the word vector model based on the current text.
[0135] In one embodiment, the processing module 105 includes:
[0136] a fourth input subunit, configured to input the first feature vector, the second feature vector and the third feature vector into an encoder;
[0137] The processing subunit is used to fuse the first feature vector, the second feature vector and the third feature vector through the attention mechanism of the encoder to obtain a fourth feature vector.
[0138] In one embodiment, the synthesis module 106 includes:
[0139] a fifth input subunit, configured to obtain a Mel-spectrogram of the fourth feature vector, and input the Mel-spectrogram into a vocoder of the speech synthesis model;
[0140] The merging subunit is used to perform speech synthesis processing on the Mel-spectrogram through the vocoder to generate synthesized speech that conforms to the current emotion category and the current emotion intensity.
[0141] In an embodiment of the present invention, the beneficial effects lie in two aspects. On the one hand, feature extraction is performed on the ranking score of the current emotion intensity to obtain a second feature vector, so that the speech synthesis model can learn the current emotion intensity through the second feature vector; on the other hand, the first feature vector, the second feature vector and the third feature vector are fused to determine a fourth feature vector, and speech synthesis processing is performed on the fourth feature vector through a preset speech synthesis model and a preset processing method to generate a synthesized speech that conforms to the current emotion category and the current emotion intensity, which is beneficial to improving the speech synthesis effect of the current text.
[0142] For the specific definition of the speech synthesis device, please refer to the definition of the speech synthesis method above, which will not be repeated here.
[0143] Each module in the above-mentioned speech synthesis device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.
[0144] See also Figure 7 , Figure 7 is a schematic diagram of a computer device in one embodiment of the present invention. In one embodiment, a computer device is provided. The computer device may be a server device, and its internal structure diagram may be as shown in FIG. Figure 7 The computer device includes a processor, a memory, a network interface and a database connected through a system bus.
[0145] The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, a function or step of a speech synthesis method can be implemented on the server device.
[0146] See also Figure 8 , Figure 8 is another structural diagram of a computer device in one embodiment of the present invention. In one embodiment, a computer device is provided. The computer device may be a client device, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external task database via a network connection. When the computer program is executed by the processor, the functions or steps of a speech synthesis method can be implemented on the client device.
[0147] In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor.
[0148] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant description in the aforementioned method embodiment. In order to avoid repetition, they will not be described one by one here.
[0149] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.
[0150] The above description and accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent possible changes only. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. Parts and subsamples of some embodiments may be included in or replace parts and subsamples of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of a stated sub-sample, whole, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other sub-samples, wholes, steps, operations, elements, components and / or groups of these. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.
[0151] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. Technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure. Technicians can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0152] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some sub-samples can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.
[0153] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.
Claims
1. A speech synthesis method, characterized in that: include: Acquire the current text, and input the current text into the trained sentiment intensity model; Based on the trained sentiment intensity model, obtaining recognition information of the current text; Obtaining a ranking score of a current emotion category and a current emotion intensity in the recognition information; Performing feature extraction on the current emotion category to obtain a first feature vector, performing feature extraction on the ranking score of the current emotion intensity to obtain a second feature vector, and performing feature extraction on the current text to obtain a third feature vector; Based on a predefined fusion method, the first feature vector, the second feature vector and the third feature vector are fused to determine a fourth feature vector; The fourth feature vector is processed by speech synthesis using a preset speech synthesis model and a preset processing method to generate synthesized speech that conforms to the current emotion category and the current emotion intensity.
2. The speech synthesis method according to claim 1, characterized in that: The step of obtaining the recognition information of the current text based on the trained sentiment intensity model includes: Obtaining an output file of the sentiment intensity model; In the output file, the recognition information output by the trained sentiment intensity model based on the current text is obtained.
3. The speech synthesis method according to claim 1, characterized in that: The step of obtaining the ranking score of the current emotion category and the current emotion intensity in the recognition information includes: Obtain a first read instruction and a second read instruction; The current emotion category in the identification information is read through the first reading instruction, and the ranking score of the current emotion intensity in the identification information is read through the second reading instruction.
4. The speech synthesis method according to claim 1, characterized in that: The step of extracting features from the current emotion category to obtain a first feature vector, extracting features from the ranking score of the current emotion intensity to obtain a second feature vector, and extracting features from the current text to obtain a third feature vector includes: Input the current emotion category into a preset word vector model to obtain a first feature vector output by the word vector model based on the current emotion category; Inputting the ranking score of the current sentiment intensity into the word vector model, and obtaining a second feature vector output by the word vector model based on the ranking score of the current sentiment intensity; The current text is input into the word vector model, and a third feature vector output by the word vector model based on the current text is obtained.
5. The speech synthesis method according to claim 1, characterized in that: The fusing the first feature vector, the second feature vector and the third feature vector based on a predefined fusing method to determine a fourth feature vector includes: Inputting the first feature vector, the second feature vector and the third feature vector into an encoder; The first feature vector, the second feature vector and the third feature vector are fused through the attention mechanism of the encoder to obtain a fourth feature vector.
6. The speech synthesis method according to claim 1, characterized in that: The method of performing speech synthesis processing on the fourth feature vector by using a preset speech synthesis model and a preset processing method to generate a synthesized speech that conforms to the current emotion category and the current emotion intensity includes: Obtaining a Mel-spectrogram of the fourth eigenvector, and inputting the Mel-spectrogram into a vocoder of the speech synthesis model; The mel-spectrogram is processed by the vocoder for speech synthesis to generate synthesized speech that conforms to the current emotion category and the current emotion intensity.
7. A training method for an analysis model, characterized in that: The analysis model is applied to the speech synthesis method according to any one of claims 1 to 6, and the training method comprises: Obtaining a preset text, obtaining annotation information of the preset text, obtaining a ranking score of an actual emotion category and an actual emotion intensity in the annotation information, obtaining prediction information output by the emotion intensity model based on the preset text, and obtaining a ranking score of a predicted emotion category and a predicted emotion intensity in the prediction information; Obtaining a first loss value between the predicted emotion category and the actual emotion category, and obtaining a second loss value between the ranking score of the predicted emotion intensity and the ranking score of the actual emotion intensity; The first loss value and the second loss value are added to obtain a third loss value, and the emotion intensity model is trained based on the third loss value. When the third loss value is less than the preset loss value, the training of the emotion intensity model is stopped, and the trained emotion intensity model is saved.
8. A speech synthesis device, characterized in that: include: A first acquisition module is used to acquire the current text and input the current text into the trained emotion intensity model; A second acquisition module is used to acquire recognition information of the current text based on the trained sentiment intensity model; A third acquisition module is used to obtain the ranking score of the current emotion category and the current emotion intensity in the recognition information; An extraction module, configured to perform feature extraction on the current emotion category to obtain a first feature vector, perform feature extraction on the ranking score of the current emotion intensity to obtain a second feature vector, and perform feature extraction on the current text to obtain a third feature vector; A processing module, configured to fuse the first feature vector, the second feature vector and the third feature vector based on a predefined fusion method to determine a fourth feature vector; The synthesis module is used to perform speech synthesis processing on the fourth feature vector through a preset speech synthesis model and a preset processing method to generate a synthesized speech that conforms to the current emotion category and the current emotion intensity.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the speech synthesis method according to any one of claims 1 to 6 or the training method according to claim 7 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the speech synthesis method according to any one of claims 1 to 6 or the training method according to claim 7 are implemented.