Speech synthesis method and device based on multi-scale emotion, equipment, storage medium

Abstract

The embodiment of the application provides a speech synthesis method and device based on multi-scale emotion, equipment, storage medium, belongs to artificial intelligence technical field. The method comprises the following steps: obtaining a target sentence composed of a plurality of sentence texts, determining global text emotion information through a global emotion module; converting the target sentence into a corresponding phoneme sequence, adjusting the reference mel spectrum to obtain intonation information after the phoneme sequence is subjected to speech emotion prediction through a sentence emotion module to obtain speech emotion information; syllable intensity information is predicted through a local emotion module; and synthesizing target speech according to the global text emotion information, the intonation information, the syllable intensity information and the target sentence. According to the technical scheme of the embodiment, the synthesized speech has more rich emotion information through multi-scale emotion, can reflect the prosody of each character, improves the authenticity of speech broadcast, and thus improves the user experience.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a speech synthesis method, apparatus, device, and storage medium based on multi-scale emotion. Background Technology

[0002] Currently, speech synthesis technology can convert any text information into speech and read it aloud in real time, and it has been widely used in various fields. While ensuring the accuracy of speech synthesis, improving user experience also requires synthesized speech to have emotional expression. A common approach is to perform semantic recognition on the text information to classify the overall emotion of the sentence, and then broadcast the speech according to the pre-defined intonation for each category. Although this adds an emotional dimension to the speech, current emotional interpretation is usually applied to the entire sentence and does not consider multi-scale attributes such as prosody. A significant difference still exists between synthesized speech and real speech, resulting in insufficient emotional depth in the speech and a poor user experience. Summary of the Invention

[0003] The main objective of this application is to propose a speech synthesis method, apparatus, device, and storage medium based on multi-scale emotion, aiming to improve the emotional level of synthesized speech and enhance user experience.

[0004] To achieve the above objectives, a first aspect of this application proposes a speech synthesis method based on multi-scale emotion, the method comprising:

[0005] A target sentence composed of multiple sentence texts is obtained, and the target sentence is input into a trained speech synthesis model, which includes a global sentiment module, a sentence sentiment module, and a local sentiment module.

[0006] The global sentiment module is used to predict the sentiment of the target statement and determine the global sentiment information of the target statement.

[0007] The target sentence is converted into a phoneme sequence, and the phoneme sequence is input into the sentence emotion module and the local emotion module respectively. Each phoneme in the phoneme sequence corresponds one-to-one with the sentence text.

[0008] The speech emotion module performs speech emotion prediction on the phoneme sequence to obtain speech emotion information, which represents the emotional changes of the phoneme sequence in the time dimension.

[0009] Obtain a preset reference Mel spectrogram, and adjust the reference Mel spectrogram according to the speech emotion information to obtain intonation information;

[0010] The local emotion module divides the phoneme sequence into multiple phoneme groups, and performs emotion intensity prediction on each phoneme group to obtain syllable intensity information.

[0011] The target speech is synthesized based on the global text sentiment information, the intonation information, the syllable intensity information, and the target sentence.

[0012] In some embodiments, the step of performing sentiment prediction on the target statement through the global sentiment module to determine the global text sentiment information of the target statement includes:

[0013] Obtain a preset global sentiment lookup table, which includes multiple selectable sentiment categories;

[0014] The global sentiment module performs sentiment prediction on the target statement to determine the category prediction probability of the target statement for each of the optional sentiment categories;

[0015] The optional sentiment categories are weighted and summed based on the predicted probabilities of the categories, and the optional sentiment category that is closest to the result of the weighted sum is determined as the global text sentiment information.

[0016] In some embodiments, adjusting the reference Mel spectrogram based on the speech emotion information to obtain intonation information includes:

[0017] The reference Mel spectrum is processed by convolution, normalization, dropout, and mean pooling through the sentiment module to obtain the reference output vector.

[0018] The reference output vector is adjusted to the target output vector based on the voice emotion information;

[0019] The target output vector is determined as the intonation information.

[0020] In some embodiments, adjusting the reference output vector to a target output vector based on the speech emotion information includes:

[0021] Align the speech emotion information with the reference output vector in time;

[0022] The numerical difference between the reference output vector and the speech emotion information is determined as the target output vector.

[0023] In some embodiments, dividing the phoneme sequence into multiple phoneme groups through the local emotion module includes:

[0024] Semantic recognition is performed on the target statement, and the associated statement texts are combined into statement phrases;

[0025] Based on the mapping relationship between the literal text of the statement and the phonemes, determine the phoneme phrase corresponding to the literal text of the statement.

[0026] In some embodiments, the step of predicting the emotional intensity of each phoneme group to obtain syllable intensity information includes:

[0027] Emotion intensity prediction is performed on each phoneme group to obtain phoneme emotion intensity information;

[0028] For each of the aforementioned sentence phrases, sentiment intensity prediction is performed to obtain phrase sentiment intensity information;

[0029] The numerical difference between the phoneme emotional intensity information and the phrase emotional intensity information is determined as the syllable intensity information.

[0030] In some embodiments, the speech synthesis model further includes a speech synthesis encoder, wherein synthesizing target speech based on the global text sentiment information, the intonation information, the syllable intensity information, and the target sentence includes:

[0031] The global text sentiment information, the intonation information, and the syllable intensity information are input into the speech synthesis encoder, and the vector output by the speech synthesis encoder is determined as the target Mel spectrogram.

[0032] The target Mel spectrogram and the target sentence are aligned in time and then converted into speech to obtain the target speech.

[0033] To achieve the above objectives, a second aspect of this application provides a speech synthesis device based on multi-scale emotion, the device comprising:

[0034] The sentence acquisition module is used to acquire a target sentence composed of multiple sentence texts and input the target sentence into a trained speech synthesis model. The speech synthesis model includes a global sentiment module, a sentence sentiment module, and a local sentiment module.

[0035] A global prediction module is used to perform text sentiment prediction on the target statement through the global sentiment module, and determine the global text sentiment information of the target statement;

[0036] The sentence conversion module is used to convert the target sentence into a phoneme sequence, and input the phoneme sequence into the sentence emotion module and the local emotion module respectively. Each phoneme in the phoneme sequence corresponds one-to-one with the sentence text.

[0037] The sentence prediction module is used to predict the speech emotion of the phoneme sequence through the sentence emotion module to obtain speech emotion information, wherein the speech emotion information represents the emotion change of the phoneme sequence in the time dimension;

[0038] The intonation determination module is used to obtain a preset reference Mel spectrogram and adjust the reference Mel spectrogram according to the speech emotion information to obtain intonation information;

[0039] The syllable prediction module is used to divide the phoneme sequence into multiple phoneme groups through the local emotion module, and perform emotion intensity prediction on each phoneme group to obtain syllable intensity information;

[0040] The speech synthesis module is used to synthesize target speech based on the global text sentiment information, the intonation information, the syllable intensity information, and the target sentence.

[0041] To achieve the above objectives, a third aspect of the present application provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first aspect.

[0042] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0043] This application proposes a speech synthesis method, apparatus, device, and storage medium based on multi-scale emotion. The method includes: acquiring a target sentence composed of multiple sentence texts; inputting the target sentence into a trained speech synthesis model, the speech synthesis model including a global emotion module, a sentence emotion module, and a local emotion module; performing text emotion prediction on the target sentence through the global emotion module to determine the global text emotion information of the target sentence; converting the target sentence into a phoneme sequence, and inputting the phoneme sequence into the sentence emotion module and the local emotion module respectively, each phoneme in the phoneme sequence corresponding one-to-one with the sentence text; performing speech emotion prediction on the phoneme sequence through the sentence emotion module to obtain speech emotion information, the speech emotion information representing the emotion change of the phoneme sequence in the time dimension; acquiring a preset reference Mel spectrogram, adjusting the reference Mel spectrogram according to the speech emotion information to obtain intonation information; dividing the phoneme sequence into multiple phoneme groups through the local emotion module, and performing emotion intensity prediction on each phoneme group to obtain syllable intensity information; and synthesizing target speech based on the global text emotion information, the intonation information, the syllable intensity information, and the target sentence. According to the technical solution of this embodiment, global emotion, sentence emotion and syllable emotion can be comprehensively considered when synthesizing speech. Through multi-scale emotion, the synthesized speech has richer emotional information, can reflect the rhythm of each word, improve the realism of speech broadcasting, and thus improve the user experience. Attached Figure Description

[0044] Figure 1 This is a flowchart of a speech synthesis method based on multi-scale emotion provided in one embodiment of this application;

[0045] Figure 2 This is a structural diagram of a speech synthesis model in another embodiment of this application;

[0046] Figure 3 yes Figure 1 The flowchart of step S102 in the document;

[0047] Figure 4 yes Figure 1 The flowchart of step S105 in the process;

[0048] Figure 5 yes Figure 4 The flowchart of step S402 in the document;

[0049] Figure 6 yes Figure 1 The flowchart of step S106 in the process;

[0050] Figure 7 yes Figure 1 The flowchart of step S106 in the process;

[0051] Figure 8 yes Figure 1 The flowchart of step S107 in the process;

[0052] Figure 9 This is a schematic diagram of the structure of the speech synthesis device based on multi-scale emotion provided in the embodiments of this application;

[0053] Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0057] First, let's analyze some of the terms used in this application:

[0058] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0059] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as the target language, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0060] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0061] Information extraction is a text processing technique that extracts factual information such as entities, relationships, and events from natural language text and outputs it as structured data. Information extraction is a technique for extracting specific information from text data. Text data is composed of specific units, such as sentences, paragraphs, and chapters. Text information is composed of smaller, specific units, such as characters, words, phrases, sentences, paragraphs, or combinations of these units. Extracting noun phrases, names of people, and place names from text data is an example of text information extraction. Of course, text information extraction techniques can extract information of various types.

[0062] BERT model: Released by Google in 2018, this language model trains deep bidirectional representations by jointly using bidirectional transformers across all layers. BERT combines the advantages of numerous natural language processing models, achieving superior performance in various natural language processing tasks. In related technologies, the input vector of the BERT model is the sum of token embeddings, position embeddings, and segment embeddings. Token embeddings represent the vectorized representation of characters, position embeddings represent the position of characters in the text, and segment embeddings represent the order of sentences in the text.

[0063] Based on this, embodiments of this application provide a speech synthesis method, apparatus, device, and storage medium based on multi-scale emotion, aiming to improve the emotional level of synthesized speech and enhance user experience.

[0064] The speech synthesis method, apparatus, device, and storage medium based on multi-scale emotion provided in this application are specifically described through the following embodiments. First, the speech synthesis method based on multi-scale emotion in this application is described.

[0065] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0066] The speech synthesis method based on multi-scale emotion provided in this application relates to the field of artificial intelligence technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the speech synthesis method based on multi-scale emotion, but is not limited to the above forms.

[0067] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0068] Figure 1 This is an optional flowchart of the speech synthesis method based on multi-scale emotion provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S105.

[0069] Step S101: Obtain the target sentence composed of multiple sentence texts, and input the target sentence into the trained speech synthesis model. The speech synthesis model includes a global sentiment module, a sentence sentiment module, and a local sentiment module.

[0070] Step S102: Perform text sentiment prediction on the target sentence through the global sentiment module to determine the global text sentiment information of the target sentence;

[0071] Step S103: Convert the target sentence into a phoneme sequence, and input the phoneme sequence into the sentence emotion module and the local emotion module respectively. Each phoneme in the phoneme sequence corresponds one-to-one with the sentence text.

[0072] Step S104: The phoneme sequence is predicted for speech emotion through the speech emotion module to obtain speech emotion information, which represents the emotional changes of the phoneme sequence in the time dimension.

[0073] Step S105: Obtain a preset reference Mel spectrogram, and adjust the reference Mel spectrogram according to the speech emotion information to obtain intonation information;

[0074] Step S106: Divide the phoneme sequence into multiple phoneme groups through the local emotion module, and predict the emotion intensity of each phoneme group to obtain syllable intensity information;

[0075] Step S107: Synthesize the target speech based on global text sentiment information, intonation information, syllable intensity information and target sentence.

[0076] It should be noted that the target statement can be a sentence composed of multiple statement texts. Of course, the target statement can also include substatements. This embodiment does not limit the length of the target statement.

[0077] It should be noted that, referring to Figure 2 The speech synthesis model includes a global sentiment module, a sentence sentiment module, and a local sentiment module. When training the speech synthesis model, the training data may include training text sentences, sample Mel spectrograms corresponding to the training text sentences, and sentiment categories. Each module of the speech synthesis model is trained collaboratively using labeled training data. The specific training method is a technique well known to those skilled in the art and will not be elaborated here.

[0078] It should be noted that, for speech, the emotion of each sentence is consistent, such as expressing happiness or care. Therefore, the emotional basis of the target sentence can be determined through the global emotion module. Since global emotion is usually emotion in a broad sense, emotion recognition can be performed directly on the text of the target sentence. Emotion recognition can be determined by simple classification through semantic recognition. This embodiment does not impose too many restrictions on the specific process.

[0079] It should be noted that in order to determine the phonetic prosody of the target sentence, it is necessary to superimpose emotional features on the existing prosody of each text. Therefore, the target sentence can be converted into a phoneme sequence by a speech synthesis encoder, and each sentence text can be converted into a phoneme in the phoneme sequence to achieve phoneme embedding.

[0080] It should be noted that the sentence sentiment module can further predict the sentiment of a sentence based on the global sentiment. For example, the sentence sentiment module can predict sentence changes based on phoneme sequences, thereby determining the change in tone over time when broadcasting the target sentence.

[0081] It should be noted that the phoneme sequence is a sequence arranged in chronological order. The sentiment prediction module can perform temporal-dimensional prediction on the phoneme sequence. Based on this, the sentiment prediction results are adjusted using the reference Mel spectrogram, and the adjusted result of the reference Mel spectrogram is determined as the intonation information. The principle and generation method of Mel spectrogram are well known to those skilled in the art and will not be elaborated here.

[0082] It should be noted that after obtaining the global sentiment and sentence sentiment scales, a local scale can be introduced to enhance the realism of the speech playback. For example, for each sentence or phrase, the phoneme sequence can be divided into multiple phoneme groups through the local sentiment module. Each phoneme group can include several related phrases. By extracting the sentiment intensity of the phoneme groups, syllable intensity information can be obtained, which can reflect the rhythm of different phrases during speech playback and improve its realism.

[0083] It should be noted that after obtaining the global text sentiment information, intonation information, and syllable intensity information, the target sentiment information of the target sentence can be obtained through AND operation. The target sentence can then be synthesized using the target sentiment information to obtain the target speech. The synthesis of speech with the given sentiment information and sentence is a technique well known to those skilled in the art, and will not be elaborated upon here.

[0084] Steps S101 to S107 as shown in the embodiments of this application can comprehensively consider global emotion, sentence emotion and syllable emotion when synthesizing speech. Through multi-scale emotion, the synthesized speech has richer emotional information, can reflect the rhythm of each word, improve the realism of speech broadcasting, and thus improve the user experience.

[0085] In some embodiments, please refer to Figure 3 Step S102 may also include, but is not limited to, the following steps:

[0086] Step S301: Obtain a preset global sentiment lookup table, which includes multiple selectable sentiment categories;

[0087] Step S302: Perform sentiment prediction on the target statement through the global sentiment module to determine the category prediction probability of the target statement for each optional sentiment category;

[0088] Step S303: The optional sentiment categories are weighted and summed according to the predicted probability of the categories, and the optional sentiment category that is closest to the result of the weighted sum is determined as the global text sentiment information.

[0089] It should be noted that since the global sentiment module predicts sentiment based on the textual information of the target sentence, and since the relationship between text and sentiment is not unique (for example, some texts may correspond to different sentiments in different contexts), in order to improve prediction efficiency, a trainable global sentiment lookup table can be set in the global sentiment module. The global sentiment lookup table can have multiple selectable sentiment categories set in it, and the specific sentiment categories can be selected according to actual needs, without further restrictions here.

[0090] It should be noted that the global sentiment module can be configured with a pre-trained sentiment classifier. For example, a linear layer and an activation function can be added to the pre-trained BERT-Base model as a sentiment classifier. Those skilled in the art will be motivated to adjust the structure of the sentiment classifier according to actual needs, and no further restrictions will be imposed here.

[0091] It should be noted that since sentiment prediction results are usually category prediction probabilities, global text sentiment information can be determined by weighted summation, for example, using the following formula: Among them, h global p represents global text sentiment information. i f1(e) represents the predicted category probability corresponding to the i-th optional sentiment category. i Let be the weight of the i-th optional sentiment category, and M be the number of optional sentiment categories. After obtaining h... global Then, you can select the weight values ​​and h. global The closest available sentiment category was determined to be the global text sentiment information.

[0092] In some embodiments, please refer to Figure 4 Step S105 may also include, but is not limited to, the following steps:

[0093] Step S401: The reference Mel spectrogram is processed by convolution, normalization, dropout, and mean pooling through the sentiment module to obtain the reference output vector;

[0094] Step S402: Adjust the reference output vector to the target output vector based on the speech emotion information;

[0095] Step S403: Determine the target output vector as intonation information.

[0096] It should be noted that a sentence variation encoder can be set in the sentence sentiment module. This encoder consists of two 1D convolutional layers. The reference Mel spectrogram generated by the speech synthesis decoder is input into the encoder and subjected to two convolutions, followed by normalization and dropout. Mean pooling on the time axis is applied to the output to obtain a reference output vector of the speech sentiment variation. The specific parameters of the sentence variation encoder only affect the value of the reference output vector. Those skilled in the art can select the parameters of the neural network according to actual needs, and no further restrictions are imposed here.

[0097] It should be noted that the obtained reference output vector can be used as a benchmark. The speech emotion module performs sentence change prediction on the phoneme sequence to determine the emotion prediction result of the sentence. The reference output vector is adjusted according to the emotion prediction result. For example, if the emotion is sadness, the reference output vector is adjusted according to the gain corresponding to sadness so that the obtained target output vector can reflect the emotion, and the obtained intonation can match the actual emotion.

[0098] In some embodiments, please refer to Figure 5 Step S402 may also include, but is not limited to, the following steps:

[0099] Step S501: Align the speech emotion information with the reference output vector in time;

[0100] Step S502: The numerical difference between the reference output vector and the speech emotion information is determined as the target output vector.

[0101] It should be noted that in order to obtain the target output vector from the reference output vector and the speech emotion information, the two need to be aligned in time, and the target output vector is calculated using the following formula: Among them, L utt h is the target output vector. utt As a reference output vector, To obtain emotional information from speech, the reference Mel spectrogram is adjusted through interpolation calculation. During the training of the speech emotion module, its output can be minimized. The h output of the variational encoder for the sum and statement utt The L1 loss function provides supervision, and the L1 loss function can be set according to the convergence requirements of the model. No further restrictions are imposed here.

[0102] In some embodiments, please refer to Figure 6 Step S106 may also include, but is not limited to, the following steps:

[0103] Step S601: Perform semantic recognition on the target statement and combine related statement texts into statement phrases;

[0104] Step S602: Determine the phoneme phrase corresponding to the sentence phrase based on the mapping relationship between sentence characters and phonemes.

[0105] It should be noted that, in order to determine the sentiment of different word groups, the target sentence can first be segmented into multiple sentence word groups based on simple semantic recognition. Then, the phonemes in the phoneme sequence corresponding to the sentence word groups are identified as phoneme word groups. Through phoneme grouping, the sentiment intensity can be extracted based on the phoneme word groups. This allows the introduction of sentiment intensity at the word group level during speech synthesis, thereby improving the realism of the target speech.

[0106] It should be noted that during the training of the local emotion module, labeled syllable segments can be input for training, so that the local emotion module can extract the emotion intensity. No further restrictions are imposed here.

[0107] In some embodiments, please refer to Figure 7 Step S106 may also include, but is not limited to, the following steps:

[0108] Step S701: Perform emotion intensity prediction for each phoneme group to obtain phoneme emotion intensity information;

[0109] Step S702: Perform sentiment intensity prediction on each sentence phrase to obtain phrase sentiment intensity information;

[0110] Step S703: The numerical difference between the phoneme emotional intensity information and the phrase emotional intensity information is determined as the syllable intensity information.

[0111] It should be noted that, in order to determine the emotional intensity of each phrase, two modules can be set up in the local emotion module: text local emotion intensity prediction and speech local emotion intensity extraction. The emotional intensity of the phrase is predicted by the text local emotion intensity prediction, and the obtained phrase emotional intensity information can be used as the basic information of emotion. Then, the phoneme emotional intensity information is used as the adjustment information, and the syllable intensity information is obtained by difference calculation, so that the syllable intensity information can reflect the emotional intensity of the phrase.

[0112] In some embodiments, the speech synthesis model further includes a speech synthesis encoder; see [link to relevant documentation]. Figure 8 Step S107 may also include, but is not limited to, the following steps:

[0113] Step S801: Input global text sentiment information, intonation information and syllable intensity information into the speech synthesis encoder, and determine the vector output by the speech synthesis encoder as the target Mel spectrogram;

[0114] Step S802: Align the target Mel spectrogram and the target sentence in time and then perform speech conversion to obtain the target speech.

[0115] It should be noted that after obtaining the global text sentiment information, intonation information, and syllable intensity information, a bitwise AND operation can be performed on the above information. The result carries multi-scale sentiment information, which is then input into the speech synthesis decoder to obtain the target Mel spectrogram. Given the target Mel spectrogram, those skilled in the art are familiar with how to perform speech synthesis, and will not elaborate further here. Through the technical solution of this embodiment, the target speech can possess multi-scale sentiment, improving realism.

[0116] Please see Figure 9 This application also provides a speech synthesis device based on multi-scale emotion, which can implement the above-mentioned speech synthesis method based on multi-scale emotion. The speech synthesis device 900 based on multi-scale emotion includes:

[0117] The sentence acquisition module 901 is used to acquire a target sentence composed of multiple sentence texts and input the target sentence into the trained speech synthesis model. The speech synthesis model includes a global sentiment module, a sentence sentiment module and a local sentiment module.

[0118] The global prediction module 902 is used to predict the text sentiment of the target sentence through the global sentiment module and determine the global text sentiment information of the target sentence.

[0119] The sentence conversion module 903 is used to convert the target sentence into a phoneme sequence, and input the phoneme sequence into the sentence emotion module and the local emotion module respectively. Each phoneme in the phoneme sequence corresponds one-to-one with the sentence text.

[0120] The sentence prediction module 904 is used to predict the speech emotion of the phoneme sequence through the sentence emotion module to obtain speech emotion information, which represents the emotional changes of the phoneme sequence in the time dimension.

[0121] The intonation determination module 905 is used to obtain a preset reference Mel spectrogram and adjust the reference Mel spectrogram according to the speech emotion information to obtain intonation information;

[0122] Syllable prediction module 906 is used to divide the phoneme sequence into multiple phoneme groups through the local emotion module, and to predict the emotion intensity of each phoneme group to obtain syllable intensity information;

[0123] The speech synthesis module 907 is used to synthesize target speech based on global text sentiment information, intonation information, syllable intensity information and target sentence.

[0124] The specific implementation of this speech synthesis device based on multi-scale emotion is basically the same as the specific implementation of the speech synthesis method based on multi-scale emotion described above, and will not be repeated here.

[0125] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned speech synthesis method based on multi-scale emotion. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0126] Please see Figure 10 , Figure 10 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0127] The processor 1001 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0128] The memory 1002 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1002 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001 to implement the speech synthesis method based on multi-scale emotion of the embodiments of this application.

[0129] Input / output interface 1003 is used to implement information input and output;

[0130] The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0131] Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004);

[0132] The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0133] This application embodiment also provides a storage medium, which is a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the above-described speech synthesis method based on multi-scale emotion.

[0134] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0135] This application provides a speech synthesis method, apparatus, device, and storage medium based on multi-scale emotion. The method includes: acquiring a target sentence composed of multiple sentence texts; inputting the target sentence into a trained speech synthesis model, the speech synthesis model including a global emotion module, a sentence emotion module, and a local emotion module; performing text emotion prediction on the target sentence through the global emotion module to determine the global text emotion information of the target sentence; converting the target sentence into a phoneme sequence, and inputting the phoneme sequence into the sentence emotion module and the local emotion module respectively, each phoneme in the phoneme sequence corresponding one-to-one with the sentence text; performing speech emotion prediction on the phoneme sequence through the sentence emotion module to obtain speech emotion information, the speech emotion information representing the emotion change of the phoneme sequence in the time dimension; acquiring a preset reference Mel spectrogram, adjusting the reference Mel spectrogram according to the speech emotion information to obtain intonation information; dividing the phoneme sequence into multiple phoneme groups through the local emotion module, and performing emotion intensity prediction on each phoneme group to obtain syllable intensity information; and synthesizing target speech based on the global text emotion information, the intonation information, the syllable intensity information, and the target sentence. According to the technical solution of this embodiment, global emotion, sentence emotion and syllable emotion can be comprehensively considered when synthesizing speech. Through multi-scale emotion, the synthesized speech has richer emotional information, can reflect the rhythm of each word, improve the realism of speech broadcasting, and thus improve the user experience.

[0136] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0137] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0138] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0139] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0140] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0141] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0142] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0143] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0144] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0145] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0146] This embodiment can be used in numerous general-purpose or special-purpose computer device environments or configurations. Examples include: personal computers, server computers, handheld or portable electronic devices, tablet-type electronic devices, multiprocessor devices, microprocessor-based devices, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above devices or electronic devices, etc. This application can be described in the general context of a computer program executed by a computer, such as a program module. Generally, a program module includes routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing electronic devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage electronic devices.

[0147] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0148] It should be noted that although several modules or units for the electronic device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0149] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing electronic device (such as a personal computer, server, touch terminal, or network electronic device, etc.) to execute the method according to the embodiments of this application.

[0150] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0151] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

[0152] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A speech synthesis method based on multi-scale emotion, characterized in that, The method includes: A target sentence composed of multiple sentence texts is obtained, and the target sentence is input into a trained speech synthesis model, which includes a global sentiment module, a sentence sentiment module, and a local sentiment module. The global sentiment module is used to predict the sentiment of the target statement and determine the global sentiment information of the target statement. The target sentence is converted into a phoneme sequence, and the phoneme sequence is input into the sentence emotion module and the local emotion module respectively. Each phoneme in the phoneme sequence corresponds one-to-one with the sentence text. The speech emotion module performs speech emotion prediction on the phoneme sequence to obtain speech emotion information, which represents the emotional changes of the phoneme sequence in the time dimension. Obtain a preset reference Mel spectrogram, and adjust the reference Mel spectrogram according to the speech emotion information to obtain intonation information; The local emotion module divides the phoneme sequence into multiple phoneme groups, and performs emotion intensity prediction on each phoneme group to obtain syllable intensity information. The target speech is synthesized based on the global text sentiment information, the intonation information, the syllable intensity information, and the target sentence.

2. The speech synthesis method based on multi-scale emotion according to claim 1, characterized in that, The step of performing sentiment prediction on the target statement through the global sentiment module to determine the global text sentiment information of the target statement includes: Obtain a preset global sentiment lookup table, which includes multiple selectable sentiment categories; The global sentiment module performs sentiment prediction on the target statement to determine the category prediction probability of the target statement for each of the optional sentiment categories; The optional sentiment categories are weighted and summed based on the predicted probabilities of the categories, and the optional sentiment category that is closest to the result of the weighted sum is determined as the global text sentiment information.

3. The speech synthesis method based on multi-scale emotion according to claim 1, characterized in that, The step of adjusting the reference Mel spectrogram based on the speech emotion information to obtain intonation information includes: The reference Mel spectrum is processed by convolution, normalization, dropout, and mean pooling through the sentiment module to obtain the reference output vector. The reference output vector is adjusted to the target output vector based on the voice emotion information; The target output vector is determined as the intonation information.

4. The speech synthesis method based on multi-scale emotion according to claim 3, characterized in that, The step of adjusting the reference output vector to the target output vector based on the voice emotion information includes: Align the speech emotion information with the reference output vector in time; The numerical difference between the reference output vector and the speech emotion information is determined as the target output vector.

5. The speech synthesis method based on multi-scale emotion according to claim 1, characterized in that, The process of dividing the phoneme sequence into multiple phoneme groups using the local emotion module includes: Semantic recognition is performed on the target statement, and the associated statement texts are combined into statement phrases; Based on the mapping relationship between the literal text of the statement and the phonemes, determine the phoneme phrase corresponding to the literal text of the statement.

6. The speech synthesis method based on multi-scale emotion according to claim 5, characterized in that, The process of predicting the emotional intensity of each phoneme group to obtain syllable intensity information includes: Emotion intensity prediction is performed on each phoneme group to obtain phoneme emotion intensity information; For each of the aforementioned sentence phrases, sentiment intensity prediction is performed to obtain phrase sentiment intensity information; The numerical difference between the phoneme emotional intensity information and the phrase emotional intensity information is determined as the syllable intensity information.

7. The speech synthesis method based on multi-scale emotion according to claim 1, characterized in that, The speech synthesis model further includes a speech synthesis encoder, wherein synthesizing target speech based on the global text sentiment information, the intonation information, the syllable intensity information, and the target sentence includes: The global text sentiment information, the intonation information, and the syllable intensity information are input into the speech synthesis encoder, and the vector output by the speech synthesis encoder is determined as the target Mel spectrogram. The target Mel spectrogram and the target sentence are aligned in time and then converted into speech to obtain the target speech.

8. A speech synthesis device based on multi-scale emotion, characterized in that, The device includes: The sentence acquisition module is used to acquire a target sentence composed of multiple sentence texts and input the target sentence into a trained speech synthesis model. The speech synthesis model includes a global sentiment module, a sentence sentiment module, and a local sentiment module. A global prediction module is used to perform text sentiment prediction on the target statement through the global sentiment module, and determine the global text sentiment information of the target statement; The sentence conversion module is used to convert the target sentence into a phoneme sequence, and input the phoneme sequence into the sentence emotion module and the local emotion module respectively. Each phoneme in the phoneme sequence corresponds one-to-one with the sentence text. The sentence prediction module is used to predict the speech emotion of the phoneme sequence through the sentence emotion module to obtain speech emotion information, wherein the speech emotion information represents the emotion change of the phoneme sequence in the time dimension; The intonation determination module is used to obtain a preset reference Mel spectrogram and adjust the reference Mel spectrogram according to the speech emotion information to obtain intonation information; The syllable prediction module is used to divide the phoneme sequence into multiple phoneme groups through the local emotion module, and perform emotion intensity prediction on each phoneme group to obtain syllable intensity information; The speech synthesis module is used to synthesize target speech based on the global text sentiment information, the intonation information, the syllable intensity information, and the target sentence.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the speech synthesis method based on multi-scale emotion as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the speech synthesis method based on multi-scale emotion as described in any one of claims 1 to 7.

Images