Speech synthesis method and device, readable medium and electronic equipment

Abstract

The invention relates to a speech synthesis method and device, a readable medium and electronic equipment. The method comprises the steps of obtaining a to-be-processed target text; determining a target emotion representation vector used for reflecting continuous features of emotions; and generating a synthetic audio corresponding to the target text according to the target text and the target emotion representation vector. Wherein compared with a single emotion category (or emotion label), the target emotion representation vector has richer information related to emotion, and particularly can reflect the difference between different emotion representation modes under the same emotion category, so that the synthesized speech has richer expressive force in emotion, and the speech quality is improved. And the emotion speech synthesis effect can be improved. On the basis, richer emotional expressions can be provided for the user in an audio reading scene, and the book listening experience of the user is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular, to a speech synthesis method, device, readable medium and electronic device. Background Art

[0002] In the scenario of audio reading (or audiobooks, audio novels), the text content in the book can be converted into audio to help users switch from reading to listening to the book. Due to the differences in characters, scenes, etc. in the book, different emotions are usually required in audio reading in order to distinguish them. This requires the use of emotional speech synthesis. Emotional speech synthesis is an important part of human-computer interaction. Its purpose is to reproduce the rich emotional expressions in speech to enhance the experience of human-computer interaction. At present, for emotional modeling in emotional speech synthesis, the common practice is to emotionally annotate the sound library with rich emotional expressions, or to customize appropriate audio for emotional categories. However, the above modeling methods have problems such as high difficulty, long cycle, and reliance on the subjective feelings of the annotator and difficulty in ensuring consistency. Summary of the invention

[0003] This section is provided to introduce the concepts in a brief form, which will be described in detail in the detailed implementation section below. This section is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to be used to limit the scope of the technical solution claimed for protection.

[0004] In a first aspect, the present disclosure provides a speech synthesis method, the method comprising:

[0005] Get the target text to be processed;

[0006] Determine a target emotion representation vector for reflecting the continuous characteristics of emotion;

[0007] Based on the target text and the target emotion representation vector, synthetic audio corresponding to the target text is generated.

[0008] In a second aspect, the present disclosure provides a speech synthesis device, the device comprising:

[0009] A first acquisition module is used to acquire a target text to be processed;

[0010] A first determination module is used to determine a target emotion representation vector for reflecting the continuity characteristics of emotion;

[0011] A generation module is used to generate a synthetic audio corresponding to the target text based on the target text and the target emotion representation vector.

[0012] In a third aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first aspect of the present disclosure.

[0013] In a fourth aspect, the present disclosure provides an electronic device, including:

[0014] a storage device having at least one computer program stored thereon;

[0015] At least one processing device is used to execute the at least one computer program in the storage device to implement the steps of the method described in the first aspect of the present disclosure.

[0016] Through the above technical solution, the target text to be processed is obtained, and the target emotion representation vector used to reflect the continuity characteristics of the emotion is determined, and then the synthetic audio corresponding to the target text is generated according to the target text and the target emotion representation vector. Among them, compared with a single emotion category (or emotion label), the target emotion representation vector has richer emotion-related information, especially it can reflect the difference between different emotion expressions under the same emotion category, and thus can make the synthesized speech more emotionally expressive, which is conducive to improving the effect of emotional speech synthesis. Based on this, in the audio reading scene, it can provide users with richer emotional expressions and enhance the user's listening experience.

[0017] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings:

[0019] Figure 1 is a flowchart of a speech synthesis method provided according to an embodiment of the present disclosure;

[0020] Figure 2 is a block diagram of a speech synthesis device provided according to an embodiment of the present disclosure;

[0021] Figure 3 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0022] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0023] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0024] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0025] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0026] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0027] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0028] All actions of acquiring signals, information or data in the present disclosure are performed in compliance with the relevant data protection laws and policies of the location and with the authorization given by the owner of the corresponding device.

[0029] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0030] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0031] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0032] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0033] At the same time, it is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0034] As described in the background technology, in the emotional speech synthesis task, a high-quality emotional sound library is generally required as a basis, and the sound library should contain emotional audio with rich emotional expressions and a certain amount of emotional category labels. Based on the above emotional sound library, a sequence-to-sequence model can be combined, and the emotional category labels can be introduced into the model modeling through a table lookup to train the emotional speech synthesis model. The model obtained based on this method can make the emotion of the speech synthesis result conform to the corresponding emotional category by specifying the emotional category label, and can usually reproduce the emotional category already in the emotional sound library.

[0035] In actual application scenarios, in order to obtain the above-mentioned emotional sound library, it is generally necessary to emotionally annotate the sound library with rich emotional expressions, or customize appropriate audio for emotional categories. However, the emotional annotation method has the problems of high difficulty, long cycle, and difficulty in ensuring consistency due to reliance on the subjective feelings of the annotator. This will result in a reduction in emotional categories if consistency is to be guaranteed, and insufficient consistency if the richness of emotional categories is to be guaranteed. In other words, the richness of emotional categories and the consistency of annotation cannot be achieved at the same time.

[0036] In addition, the current emotion modeling uses the same label for emotions of the same category, which will inevitably ignore the differences between similar data. That is, the rich data space is mapped to several discrete points in the emotion space, resulting in the loss of intra-class information (that is, the differences between emotions of the same category). If such data is used as the input of the model, the constructed model will inevitably lack emotional expressiveness, thus resulting in insufficient emotional expressiveness in the results of emotional speech synthesis.

[0037] Therefore, in order to solve the above technical problems, the present disclosure provides a speech synthesis method, device, readable medium and electronic device.

[0038] Figure 1 FIG. 1 is a flow chart of a speech synthesis method provided according to an embodiment of the present disclosure. Figure 1 As shown, the method provided by the present disclosure may include steps 11 to 13.

[0039] In step 11, the target text to be processed is obtained.

[0040] In step 12, a target emotion representation vector for reflecting the continuous characteristics of emotion is determined.

[0041] In a possible implementation, step 12 may include the following steps:

[0042] Get the target audio;

[0043] According to the target audio, a pre-generated first emotion representation determination model is used to generate a target emotion representation vector corresponding to the target audio.

[0044] The target audio can be obtained according to actual needs. For example, if it is desired to synthesize a new audio with similar emotions to an existing audio, the existing audio can be used as the target audio.

[0045] Among them, the first emotion representation determination model is used to determine the emotion representation vector based on the audio.

[0046] Optionally, the first emotion representation determination model may be generated in the following manner:

[0047] Acquire first sample data, each first sample data includes an audio sample and an emotion category corresponding to the audio sample;

[0048] The model is trained by taking the audio sample as the input of the model and taking the emotion category corresponding to the audio sample as the target output of the model to obtain a trained emotion category determination model, wherein the emotion category determination model includes a feature extraction module, a convolution layer, a fully connected layer and an activation function;

[0049] The feature extraction module, convolution layer and full connection layer in the emotion category determination model are determined as the first emotion representation determination model.

[0050] For example, the first sample data can be taken from open source sentiment data. The open source sentiment data has rich samples and does not require additional data preparation, which can effectively improve the efficiency of model generation and further improve the efficiency of determining the target sentiment representation vector.

[0051] The emotion category determination model is used to determine the emotion category of the audio according to the audio. It can include a feature extraction module, a convolution layer, a fully connected layer and an activation function. For example, the feature extraction module can adopt a Wav2vec model.

[0052] The purpose of training the emotion category determination model is to make the emotion category output by the model infinitely close to the emotion category corresponding to the audio sample. Therefore, the loss value of the model can be calculated based on the emotion category corresponding to the audio sample and the emotion category output by the model for the audio sample during training, and the internal parameters of the current model can be adjusted using the loss value. After that, the adjusted model is used in the next training, and the cycle is repeated until the conditions for stopping training are met, and a trained emotion category determination model can be obtained. For example, model training can be implemented based on a neural network.

[0053] In this way, the trained emotion category determination model has an excellent ability to determine emotion categories based on audio, and the middle layer of the model must contain rich emotion-related information, especially the output of the fully connected layer, which is a feature that can reflect the continuity of emotion information (i.e., emotion representation vector). Therefore, the part of the emotion category determination model from the model input to the fully connected layer can be extracted as the first emotion representation determination model, so that the first emotion representation determination model also has an excellent ability to determine the emotion representation vector based on audio.

[0054] Therefore, based on the generated first emotion representation determination model, the target audio is input into the first emotion representation determination model (for example, input frame level data), and the first emotion representation determination model will output an emotion representation vector corresponding to the target audio, that is, the target emotion representation vector.

[0055] In another possible implementation, step 12 may include the following steps:

[0056] Determine the target emotion category;

[0057] Obtaining the correspondence between the stored emotion categories and emotion representation vectors;

[0058] According to the corresponding relationship, the emotion representation vector corresponding to the target emotion category is determined as the target emotion representation vector.

[0059] The target emotion category can be determined according to actual needs. For example, if it is desired to synthesize audio that conforms to an existing emotion category, the existing audio can be used as the target emotion category.

[0060] The target emotion category may be one of a plurality of emotion categories that are pre-set. That is, a plurality of emotion categories may be pre-defined, for example, the emotion categories may be defined based on the emotion categories included in the open source emotion data. Furthermore, for these pre-set emotion categories, the emotion representation vectors corresponding to each emotion category may be determined respectively, and further stored as the above correspondence. Among them, the emotion representation vector corresponding to one emotion category may be one or more than one.

[0061] By way of example, the emotion representation vector corresponding to the emotion category can be determined by the above-mentioned first emotion representation determination model, that is, the audio corresponding to the emotion category is obtained, the audio is input into the first emotion representation determination model, and the emotion representation vector output by the first emotion representation determination model is obtained. Thus, the emotion representation vector is the emotion representation vector corresponding to the emotion category.

[0062] In another possible implementation, step 12 may include the following steps:

[0063] According to the target text, a pre-generated second sentiment representation determination model is used to generate a target sentiment representation vector corresponding to the target text.

[0064] The second emotion representation determination model is used to determine the emotion representation vector based on the text.

[0065] Optionally, the second emotion representation determination model may be generated in the following manner:

[0066] Acquire second sample data, each second sample data including a first text and a sentiment representation vector corresponding to the first text;

[0067] The model is trained by taking the first text as the input of the model and taking the sentiment representation vector corresponding to the first text as the target output of the model to obtain a trained second sentiment representation determination model.

[0068] By way of example, the emotion representation vector corresponding to the first text in the second sample data can be determined by the above-mentioned first emotion representation determination model, that is, the audio corresponding to the first text is obtained, the audio is input into the first emotion representation determination model, and the emotion representation vector output by the first emotion representation determination model is obtained. Thus, the emotion representation vector is the emotion representation vector corresponding to the first text.

[0069] The purpose of training the second emotion representation determination model is to make the emotion representation vector output by the model infinitely close to the emotion representation vector corresponding to the first text. Therefore, the loss value of the model can be calculated based on the emotion representation vector corresponding to the first text and the emotion representation vector output by the model for the first text during training, and the internal parameters of the current model can be adjusted using the loss value. After that, the adjusted model is used in the next training, and the cycle is repeated until the conditions for stopping training are met, and the trained second emotion representation determination model can be obtained. By way of example, model training can be implemented based on a neural network.

[0070] In this way, the trained second sentiment representation determination model has an excellent ability to determine the sentiment representation vector based on the text.

[0071] In step 13, a synthetic audio corresponding to the target text is generated according to the target text and the target emotion representation vector.

[0072] In a possible implementation, step 13 may include the following steps:

[0073] Generate a target input sequence based on the target text and the target sentiment representation vector;

[0074] Inputting the target input sequence into a pre-generated speech synthesis model to obtain acoustic feature information corresponding to the target input sequence output by the speech synthesis model;

[0075] Generate synthetic audio based on acoustic feature information.

[0076] Among them, the target input sequence is a concatenated vector generated based on the target text and the target sentiment representation vector.

[0077] Optionally, the speech synthesis model can be generated by:

[0078] Acquire third sample data, each of which includes the second text, a sentiment representation vector corresponding to the second text, and acoustic feature information corresponding to the second text;

[0079] The model is trained by taking the second text and the emotion representation vector corresponding to the second text as the input of the model, and taking the acoustic feature information corresponding to the second text as the target output of the model to obtain a trained speech synthesis model.

[0080] The purpose of training the speech synthesis model is to make the acoustic feature information output by the model infinitely close to the acoustic feature information corresponding to the second text. Therefore, the loss value of the model can be calculated based on the acoustic feature information corresponding to the second text and the acoustic feature information output by the model for the second text during training, and the internal parameters of the current model can be adjusted using the loss value. After that, the adjusted model is used in the next training, and the cycle is repeated until the conditions for stopping training are met, and a trained speech synthesis model can be obtained. For example, model training can be implemented based on a neural network, such as TTS (Text-to-Speech Synthesis).

[0081] In this way, the trained speech synthesis model has an excellent ability to determine acoustic feature information based on text and emotion representation vectors.

[0082] For example, the acoustic feature information may be a Mel spectrum (Mel spectrum), a linear spectrum, or the like.

[0083] For example, generating synthetic audio based on acoustic feature information can be achieved through a neural network vocoder.

[0084] Through the above technical solution, the target text to be processed is obtained, and the target emotion representation vector used to reflect the continuity characteristics of the emotion is determined, and then the synthetic audio corresponding to the target text is generated according to the target text and the target emotion representation vector. Among them, compared with a single emotion category (or emotion label), the target emotion representation vector has richer emotion-related information, especially it can reflect the difference between different emotion expressions under the same emotion category, and thus can make the synthesized speech more emotionally expressive, which is conducive to improving the effect of emotional speech synthesis. Based on this, in the audio reading scene, it can provide users with richer emotional expressions and enhance the user's listening experience.

[0085] Figure 2 is a block diagram of a speech synthesis device provided according to an embodiment of the present disclosure. Figure 2 As shown, the device 20 comprises:

[0086] A first acquisition module 21, used to acquire a target text to be processed;

[0087] A first determination module 22, used to determine a target emotion representation vector for reflecting the continuity characteristics of emotion;

[0088] The generating module 23 is used to generate a synthetic audio corresponding to the target text according to the target text and the target emotion representation vector.

[0089] Optionally, the first determining module 22 includes:

[0090] A first acquisition submodule, used to acquire target audio;

[0091] The first generation submodule is used to generate a target emotion representation vector corresponding to the target audio based on the target audio using a pre-generated first emotion representation determination model, wherein the first emotion representation determination model is used to determine the emotion representation vector based on the audio.

[0092] Optionally, the first emotion representation determination model is generated by the following modules:

[0093] A second acquisition module is used to acquire first sample data, each of which includes an audio sample and an emotion category corresponding to the audio sample;

[0094] A first training module is used to train the model by taking the audio sample as the input of the model and taking the emotion category corresponding to the audio sample as the target output of the model, so as to obtain a trained emotion category determination model, wherein the emotion category determination model includes a feature extraction module, a convolution layer, a fully connected layer and an activation function;

[0095] The second determination module is used to determine the feature extraction module, the convolution layer and the fully connected layer in the emotion category determination model as the first emotion representation determination model.

[0096] Optionally, the first determining module 22 includes:

[0097] The first determination submodule is used to determine the target emotion category;

[0098] The second acquisition submodule is used to acquire the corresponding relationship between the stored emotion categories and the emotion representation vectors;

[0099] The second determination submodule is used to determine, according to the corresponding relationship, an emotion representation vector corresponding to the target emotion category as the target emotion representation vector.

[0100] Optionally, the first determining module 22 includes:

[0101] The second generation submodule is used to generate a target emotion representation vector corresponding to the target text based on the target text using a pre-generated second emotion representation determination model, wherein the second emotion representation determination model is used to determine the emotion representation vector based on the text.

[0102] Optionally, the second emotion representation determination model is generated by the following modules:

[0103] A third acquisition module is used to acquire second sample data, each of which includes a first text and a sentiment representation vector corresponding to the first text;

[0104] The second training module is used to perform model training by taking the first text as the input of the model and taking the sentiment representation vector corresponding to the first text as the target output of the model to obtain a trained second sentiment representation determination model.

[0105] Optionally, the generating module 23 includes:

[0106] A third generation submodule is used to generate a target input sequence according to the target text and the target sentiment representation vector;

[0107] A processing submodule, configured to input the target input sequence into a pre-generated speech synthesis model, and obtain acoustic feature information corresponding to the target input sequence output by the speech synthesis model;

[0108] The fourth generating submodule is used to generate the synthesized audio according to the acoustic feature information.

[0109] Optionally, the speech synthesis model is generated by the following modules:

[0110] A fourth acquisition module, used to acquire third sample data, each of which includes a second text, a sentiment representation vector corresponding to the second text, and acoustic feature information corresponding to the second text;

[0111] The third training module is used to perform model training by taking the second text and the emotion representation vector corresponding to the second text as the input of the model, and taking the acoustic feature information corresponding to the second text as the target output of the model, so as to obtain a trained speech synthesis model.

[0112] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0113] Reference below Figure 3 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0114] like Figure 3 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0115] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0116] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0117] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0118] In some embodiments, the server can communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0119] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0120] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains the target text to be processed; determines a target emotion representation vector for reflecting the continuity characteristics of emotions; and generates a synthetic audio corresponding to the target text based on the target text and the target emotion representation vector.

[0121] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0122] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0123] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a module does not limit the module itself in some cases. For example, the first acquisition module may also be described as a "module for acquiring a target text to be processed".

[0124] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0125] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0126] According to one or more embodiments of the present disclosure, a speech synthesis method is provided, the method comprising:

[0127] Get the target text to be processed;

[0128] Determine a target emotion representation vector for reflecting the continuous characteristics of emotion;

[0129] Based on the target text and the target emotion representation vector, synthetic audio corresponding to the target text is generated.

[0130] According to one or more embodiments of the present disclosure, a speech synthesis method is provided, wherein determining a target emotion representation vector for reflecting the continuity characteristics of an emotion comprises:

[0131] Get the target audio;

[0132] According to the target audio, a target emotion representation vector corresponding to the target audio is generated using a pre-generated first emotion representation determination model, wherein the first emotion representation determination model is used to determine the emotion representation vector according to the audio.

[0133] According to one or more embodiments of the present disclosure, a speech synthesis method is provided, wherein the first emotion representation determination model is generated in the following manner:

[0134] Acquire first sample data, each of the first sample data includes an audio sample and an emotion category corresponding to the audio sample;

[0135] The model is trained by taking the audio sample as the input of the model and taking the emotion category corresponding to the audio sample as the target output of the model to obtain a trained emotion category determination model, wherein the emotion category determination model includes a feature extraction module, a convolution layer, a fully connected layer and an activation function;

[0136] The feature extraction module, the convolutional layer and the fully connected layer in the emotion category determination model are determined as the first emotion representation determination model.

[0137] According to one or more embodiments of the present disclosure, a speech synthesis method is provided, wherein determining a target emotion representation vector for reflecting the continuity characteristics of an emotion comprises:

[0138] Determine the target emotion category;

[0139] Obtaining the correspondence between the stored emotion categories and emotion representation vectors;

[0140] According to the corresponding relationship, an emotion representation vector corresponding to the target emotion category is determined as the target emotion representation vector.

[0141] According to one or more embodiments of the present disclosure, a speech synthesis method is provided, wherein determining a target emotion representation vector for reflecting the continuity characteristics of an emotion comprises:

[0142] According to the target text, a target emotion representation vector corresponding to the target text is generated using a pre-generated second emotion representation determination model, wherein the second emotion representation determination model is used to determine the emotion representation vector according to the text.

[0143] According to one or more embodiments of the present disclosure, a speech synthesis method is provided, wherein the second emotion representation determination model is generated in the following manner:

[0144] Acquire second sample data, each of the second sample data includes a first text and a sentiment representation vector corresponding to the first text;

[0145] The model is trained by taking the first text as the input of the model and taking the sentiment representation vector corresponding to the first text as the target output of the model to obtain a trained second sentiment representation determination model.

[0146] According to one or more embodiments of the present disclosure, a speech synthesis method is provided, wherein generating a synthesized audio corresponding to the target text according to the target text and the target emotion representation vector comprises:

[0147] Generate a target input sequence according to the target text and the target sentiment representation vector;

[0148] Inputting the target input sequence into a pre-generated speech synthesis model to obtain acoustic feature information corresponding to the target input sequence output by the speech synthesis model;

[0149] The synthesized audio is generated according to the acoustic feature information.

[0150] According to one or more embodiments of the present disclosure, a speech synthesis method is provided, wherein the speech synthesis model is generated in the following manner:

[0151] Acquire third sample data, each of the third sample data includes a second text, a sentiment representation vector corresponding to the second text, and acoustic feature information corresponding to the second text;

[0152] The model is trained by taking the second text and the emotion representation vector corresponding to the second text as the input of the model, and taking the acoustic feature information corresponding to the second text as the target output of the model to obtain a trained speech synthesis model.

[0153] According to one or more embodiments of the present disclosure, a speech synthesis device is provided, the device comprising:

[0154] A first acquisition module is used to acquire a target text to be processed;

[0155] A first determination module is used to determine a target emotion representation vector for reflecting the continuity characteristics of emotion;

[0156] A generation module is used to generate a synthetic audio corresponding to the target text based on the target text and the target emotion representation vector.

[0157] According to one or more embodiments of the present disclosure, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processing device, the steps of the speech synthesis method described in any embodiment of the present disclosure are implemented.

[0158] According to one or more embodiments of the present disclosure, there is provided an electronic device, including:

[0159] a storage device having at least one computer program stored thereon;

[0160] At least one processing device is used to execute the at least one computer program in the storage device to implement the steps of the speech synthesis method described in any embodiment of the present disclosure.

[0161] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0162] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0163] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. A speech synthesis method, characterized in that: The method comprises: Get the target text to be processed; Determine a target emotion representation vector for reflecting the continuous characteristics of emotion; Based on the target text and the target emotion representation vector, synthetic audio corresponding to the target text is generated.

2. The method according to claim 1, characterized in that The step of determining a target emotion representation vector for reflecting the continuity characteristics of emotion includes: Get the target audio; According to the target audio, a target emotion representation vector corresponding to the target audio is generated using a pre-generated first emotion representation determination model, wherein the first emotion representation determination model is used to determine the emotion representation vector according to the audio.

3. The method according to claim 2, characterized in that The first emotion representation determination model is generated in the following manner: Acquire first sample data, each of the first sample data includes an audio sample and an emotion category corresponding to the audio sample; The model is trained by taking the audio sample as the input of the model and taking the emotion category corresponding to the audio sample as the target output of the model to obtain a trained emotion category determination model, wherein the emotion category determination model includes a feature extraction module, a convolution layer, a fully connected layer and an activation function; The feature extraction module, the convolutional layer and the fully connected layer in the emotion category determination model are determined as the first emotion representation determination model.

4. The method according to claim 1, characterized in that The step of determining a target emotion representation vector for reflecting the continuity characteristics of emotion includes: Determine the target emotion category; Obtaining the correspondence between the stored emotion categories and emotion representation vectors; According to the corresponding relationship, an emotion representation vector corresponding to the target emotion category is determined as the target emotion representation vector.

5. The method according to claim 1, characterized in that The step of determining a target emotion representation vector for reflecting the continuity characteristics of emotion includes: According to the target text, a target emotion representation vector corresponding to the target text is generated using a pre-generated second emotion representation determination model, wherein the second emotion representation determination model is used to determine the emotion representation vector according to the text.

6. The method according to claim 5, characterized in that The second emotion representation determination model is generated in the following manner: Acquire second sample data, each of the second sample data includes a first text and a sentiment representation vector corresponding to the first text; The model is trained by taking the first text as the input of the model and taking the sentiment representation vector corresponding to the first text as the target output of the model to obtain a trained second sentiment representation determination model.

7. The method according to claim 1, characterized in that The step of generating a synthetic audio corresponding to the target text according to the target text and the target emotion representation vector comprises: Generate a target input sequence according to the target text and the target sentiment representation vector; Inputting the target input sequence into a pre-generated speech synthesis model to obtain acoustic feature information corresponding to the target input sequence output by the speech synthesis model; The synthesized audio is generated according to the acoustic feature information.

8. The method according to claim 7, characterized in that The speech synthesis model is generated in the following way: Acquire third sample data, each of the third sample data includes a second text, a sentiment representation vector corresponding to the second text, and acoustic feature information corresponding to the second text; The model is trained by taking the second text and the emotion representation vector corresponding to the second text as the input of the model, and taking the acoustic feature information corresponding to the second text as the target output of the model to obtain a trained speech synthesis model.

9. A speech synthesis device, characterized in that: The device comprises: A first acquisition module, used to acquire a target text to be processed; A first determination module is used to determine a target emotion representation vector for reflecting the continuity characteristics of emotion; A generation module is used to generate a synthetic audio corresponding to the target text based on the target text and the target emotion representation vector.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processing device, the steps of the method described in any one of claims 1 to 8 are implemented.

11. An electronic device, characterized in that: include: a storage device having at least one computer program stored thereon; At least one processing device, configured to execute the at least one computer program in the storage device to implement the steps of the method according to any one of claims 1 to 8.

Images