Deep learning-based emotional speech synthesis method and device

Abstract

A deep learning-based emotional speech synthesis method and device. The method at least comprising the following steps: extracting text information to be processed and preamble information of the process-pending text information (S1); using a pre-constructed first model to generate emotion feature information by assigning the process-pending text information and the preamble information as an input, (S2); and using a pre-trained second model to synthesize emotional speech by assigning the emotion feature information and the process-pending text information as an input (S3). In the described method, only the text information is acquired, wherein the emotional speech can be synthesized based on deep learning without manually labeled emotion of each acoustic pronunciation in advance. Moreover, labeling errors and labor costs are further reduced, with improving the suitability of emotion information, diversifying conversational speech emotions, enhancing the naturalness and smoothness of synthesized speech, promoting the experience of human-machine communication, and widening the applicability.

Description

DEEP LEARNING-BASED EMOTIONAL SPEECH SYNTHESIS METHOD AND DEVICE Technical field The present invention relates to the field of speech synthesis technologies, in particular to a deep learning-based emotional speech synthesis method and a device. Background With the development of the society, people wish to develop robots to substitute some simple and redundant manual work, such as announcing and simple customer service tasks, wherein humans should smoothly and naturally communicate with the robots. Speech, as an important communication tool for human society, can contribute to natural and harmonic man- machine communication. Therefore, speech synthesis plays an important role in emotional computation and signal processing. In the meanwhile, delicate emotional expression can greatly improve the synthesized speech quality. The current methods generally rely on labelled information, wherein each acoustic pronunciation in a speech is labelled with texts and emotions manually. On the other hand, the average and variance of the emotion's basic frequency, speech power, time variance and other parameters are set manually. In other words, by setting all specifications, segments are acquired and combined for synthesis. The described traditional approaches are completed manually, wherein the labelling workers are trained with the associated labelling process, wherein different works tend to have different understanding for standard labels. As a result, the mood in a sentence is understood differently, leading to non-standardized labels and large labelling errors. While the emotion matching is poor, the applicable scenarios of labelling contents are inflexible and not diversified. The speech emotions out of the applicable range will be very mechanical and strange. In the meanwhile, the data labelling requires a great amount of manual labor. Summary Aiming at the aforementioned technical problems, a deep learning-based emotional speech synthesis method and a device are provided in the present invention, permitting emotional speech synthesis without requiring manual labelling for text emotions. The technical proposals provided in the present invention includes: From the first perspective, a deep learning-based emotional speech synthesis method is provided, at least comprising: extracting text information to be processed and preamble information of the process- pending text information, wherein the preamble information includes the preamble text information; using a pre-constructed first model to generate emotion feature information by assigning the process-pending text information and the preamble information as an input; and using a pre-trained second model to synthesize emotional speech by assigning the emotion feature information and the process-pending text information as an input. In some preferred embodiments, the described first model is composed of a first sub model, a second sub model, and a third sub model, wherein the described use of a pre-constructed first model to generate emotion feature information by assigning the described process-pending text information and the preamble information as an input includes: assigning the described process-pending text information and the preamble information as an input then extracting features by the pre-trained first sub model to obtain a first intermediate output; assigning the described process-pending text information and the first intermediate output as an input, then extracting features by the pre-trained second sub model to obtain an emotion type and a second intermediate output; and assigning the described second intermediate output, process-pending text information, and the emotion type or a user-desired emotion type as an input, then extracting features by the pre- trained third sub model to obtain the emotion feature information. In some preferred embodiments, when the described preamble information further includes preamble speech information, the described first model further includes a fourth sub model, a fifth sub model, and a sixth sub model connected in series, wherein the described use of a pre- constructed first model to generate emotion feature information by assigning the described process-pending text information and the preamble information as an input includes: assigning the described process-pending text information and the preamble information as an input then extracting features by the pre-trained fourth sub model to obtain a fourth intermediate output; assigning the described process-pending text information and the fourth intermediate output as an input, then extracting features by the pre-trained fifth sub model to obtain an emotion type and a fifth intermediate output; and assigning the described fifth intermediate output, process-pending text information, and the emotion type or a user-desired emotion type as an input, then extracting features by the pre- trained sixth sub model to obtain the emotion feature information. In some preferred embodiments, the pre-training process of the described second model includes: extracting a video image sample, a text information sample, and a conversation information sample of a video sample; labelling the described video image sample according to the pre-set emotional type to obtain an emotion labelling information sample; training to obtain the third model by assigning the described video image sample as an input and the described emotion labelling information sample as an output, then extracting the third intermediate output of the described third model as the emotion information of the described video image sample; and training to obtain the second model by assigning the described emotion information and the text information sample as an input, and the conversation information sample as an output. In some preferred embodiments, the pre-training process of the described first model includes: extracting the current text information sample and the preamble information sample of the video sample, wherein the described preamble information sample includes a preamble text information sample; training to obtain the described first sub model by an input of the described current text information sample and the preamble information sample and an output of that if the current text information sample has changed compared to the preamble information sample, then extracting the first intermediate output by the described first sub model; training to obtain the described second sub model by assigning the first intermediate output and the current text information sample as an input and the emotion type as an output, then extracting the second intermediate output by the second sub model; and training to obtain the described third sub model by an input of the second intermediate output, the current text information sample, and the emotion type or a user-desired emotion type as an input, and an output of the emotion information by the third model. In some preferred embodiments, the pre-training process of the described first model includes: extracting the current text information sample and the preamble information sample of the video sample, wherein the described preamble information sample includes a preamble text information sample and a preamble speech information sample; training to obtain the described fourth sub model by an input of the described current text information sample and the preamble information sample and an output of that if the current text information sample has changed compared to the preamble information sample, then extracting the fourth intermediate output by the described fourth sub model; training to obtain the described fifth sub model by assigning the first intermediate output and the current text information sample as an input and the emotion type as an output, then extracting the fifth intermediate output by the fifth sub model; and training to obtain the described sixth sub model by an input of the second intermediate output, the current text information sample, and the emotion type or a user-desired emotion type as an input, and an output of the emotion information by the third model. In some preferred embodiments, the described pre-training process of the described second model further includes the pre-processing of the video sample, at least comprising: dividing the described video image sample into several segmented video image samples according to a pre-set time interval, then defining the texts within any time interval as the current time information sample, and defining the texts before the mentioned time interval as the preamble text information sample. From the other perspective, a deep learning-based emotional speech synthesis device on the basis of any described methods is further provided, at least comprising: an extraction module, configured to extract text information to be processed and preamble information of the process-pending text information, wherein the preamble information includes the preamble text information; an emotion feature information generation module, configured to use a pre-constructed first model for generating emotion feature information by assigning the process-pending text information and the preamble information as an input; and an emotional speech synthesis module, configured to use a pre-trained second model for synthesizing emotional speech by assigning the emotion feature information and the process- pending text information as an input. In some preferred embodiments, wherein the described first model is composed of a first sub model, a second sub model, and a third sub model, the described emotion feature information generation module at least includes: a first feature extraction unit, configured to assign the described process-pending text information and the preamble information as an input then extract features by the pre-trained first sub model to obtain a first intermediate output; a second feature extraction unit, configured to assign the described process-pending text information and the first intermediate output as an input, then extract features by the pre-trained second sub model to obtain an emotion type and a second intermediate output; and a third feature extraction unit, configured to assign the described second intermediate output, process-pending text information, and the emotion type or a user-desired emotion type as an input, then extract features by the pre-trained third sub model to obtain the emotion feature information. In some preferred embodiments, when the described preamble information further includes preamble speech information, the described first model further includes a fourth sub model, a fifth sub model, and a sixth sub model connected in series, wherein the described emotion feature information generation module at least includes: a fourth feature extraction module, configured to assign the described process-pending text information and the preamble information as an input then extract features by the pre-trained fourth sub model to obtain a fourth intermediate output; a fifth feature extraction module, configured to assign the described process-pending text information and the fourth intermediate output as an input, then extract features by the pre-trained fifth sub model to obtain an emotion type and a fifth intermediate output; and a sixth feature extraction module, configured to assign the described fifth intermediate output, process-pending text information, and the emotion type or a user-desired emotion type as an input, then extract features by the pre-trained sixth sub model to obtain the emotion feature information. In some preferred embodiments, the described device further includes a model training module. The described model training module is at least composed of the second model training unit to train the described second model. The described second model training unit at least includes: a first extraction subunit, configured to extract a video image sample, a text information sample, and a conversation information sample of a video sample; an emotion labeling subunit, configured to label the described video image sample according to the pre-set emotional type to obtain an emotion labelling information sample; a first training subunit, configured to train and obtain the third model by assigning the described video image sample as an input and the described emotion labelling information sample as an output, then extract the third intermediate output of the described third model as the emotion information of the described video image sample; and the first training subunit, further configured to train and obtain the second model by assigning the described emotion information and the text information sample as an input, and the conversation information sample as an output. In some preferred embodiments, the described model training module is further composed of a first model training unit, configured to train the described first model, comprising: a second extraction subunit, configured to extract the current text information sample and the preamble information sample of the video sample, wherein the described preamble information sample includes a preamble text information sample; a second training subunit, configured to train and obtain the described first sub model by an input of the described current text information sample and the preamble information sample and an output of that if the current text information sample has changed compared to the preamble information sample, then extract the first intermediate output by the described first sub model; the second training subunit, further configured to train and obtain the described second sub- model by assigning the first intermediate output and the current text information sample as an input and the emotion type as an output, then extract the second intermediate output by the second sub- model; and the second training subunit, further configured to train and obtain the described third sub- model by an input of the second intermediate output, the current text information sample, and the emotion type or a user-desired emotion type as an input, and an output of the emotion information by the third model. In some preferred embodiments, the described model training module is further composed of a third model training unit, configured to train another first model, comprising: a third extraction subunit, configured to extract the current text information sample and the preamble information sample of the video sample, wherein the described preamble information sample includes a preamble text information sample and a preamble speech information sample; a third training subunit, configured to train and obtain the described fourth sub model by an input of the described current text information sample and the preamble information sample and an output of that if the current text information sample has changed compared to the preamble information sample, then extract the fourth intermediate output by the described fourth sub model; the third training subunit, further configured to train and obtain the described fifth sub- model by assigning the first intermediate output and the current text information sample as an input and the emotion type as an output, then extracting the fifth intermediate output by the fifth sub model; and the third training subunit, further configured to train and obtain the described sixth sub- model by an input of the second intermediate output, the current text information sample, and the emotion type or a user-desired emotion type as an input, and an output of the emotion information by the third model. In some preferred embodiments, the described second model training unit further includes: a pre-processing subunit, configured to divide the described video image sample into several segmented video image samples according to a pre-set time interval, then define the texts within any time interval as the current time information sample, and defined the texts before the mentioned any time interval as the preamble text information sample. The benefits provided in the present invention include that: the deep learning-based emotional speech synthesis method disclosed in the present invention generates emotion feature information by a pre-constructed first model based on extracted process-pending text information and the preamble information of the described process- pending text information; synthesizes emotional speech according to the emotion feature information and the process-pending text information based on a second model pre-trained by video samples. In the described method, only the text information is acquired, wherein the emotional speech can be synthesized based on deep learning without manually labeled emotion of each acoustic pronunciation in advance. Moreover, labeling errors and labor costs are further reduced, with improving the suitability of emotion information, diversifying conversational speech emotions, enhancing the naturalness and smoothness of synthesized speech, promoting the experience of human-machine communication, and widening the applicability. Besides, during the model training in the present invention, based on the video image information, text information and the speech information in a video, the emotion information is first obtained by the video image to construct a video image based speech synthesis module. Then, based on the text information, an emotional speech synthesis module targeting the emotion information is constructed, to achieve the emotional speech generation based on text information. The described method is applicable to scenarios of video communications, voice communications, and communications with text information. The widened applicability of the present method can further promote man-machine communication experience. In addition, in the deep learning-based emotional speech synthesis method, the speech synthesis model (the second model) is constructed according to video image samples, corresponding text information samples, and conversation information samples. Therefore, the acquired emotion is more delicate, and the synthesized speech has more accurate and natural emotion and tone. To clarify, the proposals in the present invention can only achieve one of technical benefits. Brief descriptions of the drawings For a better explanation of the technical proposal of embodiments in the present invention, the accompanying drawings are briefly introduced in the following. Obviously, the following drawings represent only a portion of embodiments of the present invention. Those skilled in the art are able to create other drawings according to the accompanying drawings without making creative efforts. Fig. 1 is a flow diagram of a deep learning-based emotional speech synthesis method in embodiment one of the present invention; Fig. 2 is a logical diagram of the deep learning-based emotional speech synthesis method in embodiment one of the present invention; Fig. 3 is a logical diagram of training the second model in embodiment one of the present invention; Fig. 4 is a logical diagram of training the first model in embodiment one of the present invention; and Fig. 5 is a structure diagram of the deep learning-based emotional speech synthesis device in embodiment two of the present invention. Detailed descriptions In order to make the objective, the technical protocol, and the advantages of the present invention clearer, the present invention will be explained further in detail precisely below with references to the accompanying drawings. Obviously, the embodiments described below are only a portion of embodiments of the present invention and cannot represent all possible embodiments. Based on the embodiments in the present invention, the other applications by those skilled in the art without any creative works are falling within the scope of the present invention. Embodiment one As shown in Fig. 1, a deep learning-based emotional speech synthesis method is provided in the present embodiment, related to the field of voice synthesis. Implementing the present method can permit emotional speech synthesis without requiring manually labelled emotions, and effectively improve the quality of emotion in the synthesized speech. Referring to Fig. 1 and 2, the present method comprises: S1, extracting text information to be processed and preamble information of the process- pending text information. In particular, where if the processing target is a text object, the preamble information includes preamble text information; and where if the processing target is a voice object or a video object, the preamble information includes the preamble text information and the preamble speech information. To clarify, in the present step, different extraction tools can be used to extract the text information from a text object, the text information and the voice information from a voice information, and the text information and the voice information from a video information. The detailed adoption methods are general techniques in the field, and are not explained in further detail. S2, using a pre-constructed first model to generate emotion feature information by assigning the process-pending text information and the preamble information as an input. In particular, where if the processing target is a text object, the step S2 comprises the following sub-steps: S211, assigning the described process-pending text information and the preamble information as an input then extracting features by the pre-trained first sub model to obtain a first intermediate output; S212, assigning the described process-pending text information and the first intermediate output as an input, then extracting features by the pre-trained second sub model to obtain an emotion type and a second intermediate output; and S213, assigning the described second intermediate output, process-pending text information, and the emotion type or a user-desired emotion type as an input, then extracting features by the pre-trained third sub model to obtain the emotion feature information. In particular, one of the input ports of the third sub model is an emotion controlling port, wherein the emotion controlling port can input the emotion type exported by the second sub model, or input an emotion type set manually by a user. Therefore, the acquisition of emotion types can be fully based on models. When model data has poor reliability, emotion types can be manually adjusted, to further improve the accuracy and reliability of the acquired emotion information. In particular, the first intermediate output is the output feature vector from the layer before the logical judgement layer of the first sub model, including the current conversation tone extracted by the first sub model and the emotion characteristics of the current texts. The second intermediate output is the output feature vector from the layer before the classification layer of the second sub model, including the emotion characteristics of the current text extracted by the second sub model according to the first intermediate output. As another preferred embodiment, when the process target is a voice object or a video object, the step S2 includes the following sub-steps: S221, assigning the described process-pending text information and the preamble information as an input then extracting features by the pre-trained fourth sub model to obtain a fourth intermediate output; S222, assigning the described process-pending text information and the fourth intermediate output as an input, then extracting features by the pre-trained fifth sub model to obtain an emotion type and a fifth intermediate output; and S223, assigning the described fifth intermediate output, process-pending text information, and the emotion type or a user-desired emotion type as an input, then extracting features by the pre-trained sixth sub model to obtain the emotion feature information. In particular, the fourth intermediate output is the output feature vector from the layer before the logical judgement layer of the fourth sub model, including the current conversation tone and the emotion characteristics of the current texts extracted by the fourth sub model according to the input conversation speech or the input video. The fifth intermediate output is the output feature vector from the layer before the classification layer of the fifth sub model, including the emotion characteristics of the current text extracted by the second sub model according to the fourth intermediate output. Being verified, the current text information includes the preamble text information and the preamble speech information. The acquired emotion speech characteristics information is more reliable. S3, using a pre-trained second model to synthesize emotional speech by assigning the emotion feature information and the process-pending text information as an input. Based on the aforementioned step S2, the natural-tone emotional speech can be generated on the basis of the text information. Therefore, the deep learning-based emotional speech synthesis method disclosed in the present invention generates emotion feature information by a pre-constructed first model based on extracted process-pending text information and the preamble information of the described process- pending text information; synthesizes emotional speech according to the emotion feature information and the process-pending text information based on a second model pre-trained by video samples. In the described method, only the text information is acquired, wherein the emotional speech can be synthesized based on deep learning without manually labeled emotion of each acoustic pronunciation in advance. Moreover, labeling errors and labor costs are further reduced, with improving the suitability of emotion information, diversifying conversational speech emotions, enhancing the naturalness and smoothness of synthesized speech, and promoting the experience of human-machine communication. Besides, when the present method is implemented to synthesize emotional speeches, the processing target can be text only, or the combination of the texts and voices. Therefore, the present method can be applicable to any of texts, voices, and videos for emotional speech synthesis, with wide applicability. Furthermore, the present method further includes model training steps, to pre-train the first model and the second model. First of all, the pre-training process of the described second model includes: Sal, extracting a video image sample, a text information sample, and a conversation information sample of a video sample; Sa2, labelling the described video image sample according to the pre-set emotional type to obtain an emotion labelling information sample; and Sa3, training to obtain the third model by assigning the described video image sample as an input and the described emotion labelling information sample as an output, then extracting the third intermediate output of the described third model as the emotion information of the described video image sample; and training to obtain the second model by assigning the described emotion information and the text information sample as an input, and the conversation information sample as an output. Illustratively, the third model is constructed based on ResNet-50, carrying the cross- entropy loss function. The second model is constructed based on Tacotron2, carrying the mean square error loss function and L2 distance loss function. In particular, as shown in Fig. 3, the third model is connected with the second model in series and two models are trained together. After extracting the video image samples, text information samples, and conversation information samples, the video image samples are sent to the third model input port <semantics>(I3)<annotation encoding="application / x-tex">(I_3)< / annotation>< / semantics> and the third intermediate output <semantics>(O3)<annotation encoding="application / x-tex">(O_3)< / annotation>< / semantics> is sent to an input port of the second model (I51). The second model uses the text information samples as inputs (I52). The third model and the second model take the emotion labelling information <semantics>(O32)<annotation encoding="application / x-tex">(O_{32})< / annotation>< / semantics> and the conversation information samples <semantics>(O5)<annotation encoding="application / x-tex">(O_5)< / annotation>< / semantics> as targets, respectively, to train both models together. The third intermediate output <semantics>(O31)<annotation encoding="application / x-tex">(O_{31})< / annotation>< / semantics> is intercepted as the emotion information. After completion of the second model, the first model is trained. The first model can be constructed according to applied objects. For example, the model applied to text-based contents is different from the model for voice or video-based contents. After receiving the process-pending objects, the system automatically determines the object types and select an appropriate first model. When the first model only applicable to text-based contents, the training process of the first model includes: Sb1, extracting the current text information sample and the preamble information sample of the video sample, wherein the described preamble information sample includes a preamble text information sample; Sb2, training to obtain the described first sub model by an input of the described current text information sample and the preamble information sample and an output of that if the current text information sample has changed compared to the preamble information sample, then extracting the first intermediate output by the described first sub model; Sb3, training to obtain the described second sub model by assigning the first intermediate output and the current text information sample as an input and the emotion type as an output, then extracting the second intermediate output by the second sub model; and Sb4, training to obtain the described third sub model by an input of the second intermediate output, the current text information sample, and the emotion type or a user-desired emotion type as an input, and an output of the emotion information by the third model. In detail, as shown in Fig. 4, the first sub model, the second sub model, and the third sub model are connected in series. After the preamble information samples and the preamble text information samples are acquired, the described three sub models are trained together. Illustratively, the first sub model is constructed based on Trasformer-x1, wherein the LSTM+CNN structure replaces the Decoder part, and defined as the logical judgement of the first sub model to generate outputs. The outputs carry the cross-entropy loss function. The second sub model is constructed based on Transformer, wherein the LSTM+CNN structure replaces the Decoder part, and defined as the logical judgement of the second sub model to generate outputs. The outputs carry the cross-entropy loss function. The third sub model is constructed based on StarGAN, wherein the ConvID network layer replaces the Conv2D in the structure. The output carries mean square error loss function and L2 distance loss function. The preamble information samples and the current text information samples are defined as two inputs for the first model (I11, I12). In particular, the current text information samples are assigned as an input for each sub model (<semantics>I11<annotation encoding="application / x-tex">I_{11}< / annotation>< / semantics>, <semantics>I21<annotation encoding="application / x-tex">I_{21}< / annotation>< / semantics>, <semantics>I42<annotation encoding="application / x-tex">I_{42}< / annotation>< / semantics>). the current information samples are the other input <semantics>(I12)<annotation encoding="application / x-tex">(I_{12})< / annotation>< / semantics> for the first sub model, and the output <semantics>(O12)<annotation encoding="application / x-tex">(O_{12})< / annotation>< / semantics> is that whether the emotion of the current text information sample is changed compared to the emotion of the preamble information sample. The first intermediate output <semantics>(O11)<annotation encoding="application / x-tex">(O_{11})< / annotation>< / semantics> is assigned as the other input for the second sub model <semantics>(I21)<annotation encoding="application / x-tex">(I_{21})< / annotation>< / semantics>, and the output of the second sub model is the emotion types <semantics>(O22)<annotation encoding="application / x-tex">(O_{22})< / annotation>< / semantics>. The second intermediate output is assigned as the other input for the third sub model (I41), and the emotion information acquired by the third model is assigned as the output (O4). And the aforementioned three sub models are trained together. When a first model applicable to voice or video contents is trained, the training process includes: Sc1, extracting the current text information sample and the preamble information sample of the video sample, wherein the described preamble information sample includes a preamble text information sample and a preamble speech information sample; Sc2, training to obtain the described fourth sub model by an input of the described current text information sample and the preamble information sample and an output of that if the current text information sample has changed compared to the preamble information sample, then extracting the fourth intermediate output by the described fourth sub model; Sc3, training to obtain the described fifth sub model by assigning the first intermediate output and the current text information sample as an input and the emotion type as an output, then extracting the fifth intermediate output by the fifth sub model; and Sc4, training to obtain the described sixth sub model by an input of the second intermediate output, the current text information sample, and the emotion type or a user-desired emotion type as an input, and an output of the emotion information by the third model. Illustratively, the fourth sub model is constructed based on ResNet~50 and Transformer- x1, wherein the dense layer of ResNet~50 is discarded and the ConvLstm2D structural network layer replaces the Conv2D in ResNet-50. The pooling layer output of the ResNet-50 is merged to the output of Encoder in Transformer-x1. The LSTM+CNN structure replaces the decoder part of the, and defined as the logical judgement of the fourth sub model to generate outputs, carrying cross-entropy loss function. The fifth sub model is constructed based on Transformer, wherein the LSTM+CNN structure replaces the Decoder part, and defined as the logical judgement of the fifth sub model to generate outputs. The outputs carry the cross-entropy loss function. The sixth model is constructed based on StarGAN, wherein the Conv1D structure network layer replaces the Conv2D structure network layer. The output carries mean square error loss function and L2 distance loss function. The two training methods of the first model are the same, wherein the relevant input and output relationships can refer to the first training model. The only difference is that the preamble speech information samples are added to the fourth sub model in the second training method. Therefore, during the model training in the present invention, based on the video image information, text information and the speech information in a video, the emotion information is first obtained by the video image to construct a video image based speech synthesis module. Then, based on the text information, an emotional speech synthesis module targeting on the emotion information is constructed, to achieve the emotional speech generation based on text information. The described method is applicable to scenarios of video communications, voice communications, and communications with text information. The widened applicability of the present method can further promote man-machine communication experience. In addition, in the deep learning-based emotional speech synthesis method, the speech synthesis model (the second model) is constructed according to video image samples, corresponding text information samples, and conversation information samples. Therefore, the acquired emotion is more delicate, and the synthesized speech has more accurate and natural emotion and tone. Embodiment two To implement the deep learning-based emotional speech synthesis method in embodiment one, a deep learning-based emotional speech synthesis device 100 is provided in the present embodiment. Fig. 5 is a structure diagram of the deep learning-based emotional speech synthesis device. As shown in Fig. 5, the device 100 at least comprises: an extraction module 1, configured to extract text information to be processed and preamble information of the process-pending text information, wherein the preamble information includes the preamble text information; an emotion feature information generation module 2, configured to use a pre-constructed first model for generating emotion feature information by assigning the process-pending text information and the preamble information as an input; and an emotional speech synthesis module 3, configured to use a pre-trained second model for synthesizing emotional speech by assigning the emotion feature information and the process- pending text information as an input. In some preferred embodiments, wherein the described first model is composed of a first sub model, a second sub model, and a third sub model, the described emotion feature information generation module 2 at least includes: a first feature extraction unit 21, configured to assign the described process-pending text information and the preamble information as an input then extract features by the pre-trained first sub model to obtain a first intermediate output; a second feature extraction unit 22, configured to assign the described process-pending text information and the first intermediate output as an input, then extract features by the pre-trained second sub model to obtain an emotion type and a second intermediate output; and a third feature extraction unit 23, configured to assign the described second intermediate output, process-pending text information, and the emotion type or a user-desired emotion type as an input, then extract features by the pre-trained third sub model to obtain the emotion feature information. In some preferred embodiments, when the described preamble information further includes preamble speech information, the described first model further includes a fourth sub model, a fifth sub model, and a sixth sub model connected in series, wherein the described emotion feature information generation module 2 at least includes: a fourth feature extraction module 21', configured to assign the described process-pending text information and the preamble information as an input then extract features by the pre-trained fourth sub model to obtain a fourth intermediate output; a fifth feature extraction module 22', configured to assign the described process-pending text information and the fourth intermediate output as an input, then extract features by the pre- trained fifth sub model to obtain an emotion type and a fifth intermediate output; and a sixth feature extraction module 23', configured to assign the described fifth intermediate output, process-pending text information, and the emotion type or a user-desired emotion type as an input, then extract features by the pre-trained sixth sub model to obtain the emotion feature information. In some preferred embodiments, the described device further includes a model training module 4. The described model training module 4 is at least composed of the second model training unit 41to train the described second model. The described second model training unit 41at least includes: a first extraction subunit 411, configured to extract a video image sample, a text information sample, and a conversation information sample of a video sample; an emotion labeling subunit 412, configured to label the described video image sample according to the pre-set emotional type to obtain an emotion labelling information sample; and a first training subunit 413, configured to train and obtain the third model by assigning the described video image sample as an input and the described emotion labelling information sample as an output, then extract the third intermediate output of the described third model as the emotion information of the described video image sample; and further configured to train and obtain the second model by assigning the described emotion information and the text information sample as an input, and the conversation information sample as an output. In some preferred embodiments, the described model training module is further composed of a first model training unit 42, configured to train the described first model 42, comprising: a second extraction subunit 421, configured to extract the current text information sample and the preamble information sample of the video sample, wherein the described preamble information sample includes a preamble text information sample; a second training subunit 422, configured to train and obtain the described first sub model by an input of the described current text information sample and the preamble information sample and an output of that if the current text information sample has changed compared to the preamble information sample, then extract the first intermediate output by the described first sub model; the second training subunit, further configured to train and obtain the described second sub- model by assigning the first intermediate output and the current text information sample as an input and the emotion type as an output, then extract the second intermediate output by the second sub- model; and the second training subunit, further configured to train and obtain the described third sub- model by an input of the second intermediate output, the current text information sample, and the emotion type or a user-desired emotion type as an input, and an output of the emotion information by the third model. In some preferred embodiments, the described model training module is further composed of a third model training unit 43, configured to train another first model, comprising: a third extraction subunit 431, configured to extract the current text information sample and the preamble information sample of the video sample, wherein the described preamble information sample includes a preamble text information sample and a preamble speech information sample; a third training subunit 432, configured to train and obtain the described fourth sub model by an input of the described current text information sample and the preamble information sample and an output of that if the current text information sample has changed compared to the preamble information sample, then extract the fourth intermediate output by the described fourth sub model; the third training subunit, further configured to train and obtain the described fifth sub- model by assigning the first intermediate output and the current text information sample as an input and the emotion type as an output, then extracting the fifth intermediate output by the fifth sub model; and the third training subunit, further configured to train and obtain the described sixth sub- model by an input of the second intermediate output, the current text information sample, and the emotion type or a user-desired emotion type as an input, and an output of the emotion information by the third model. In some preferred embodiments, the described second model training unit 41 further includes: a pre-processing subunit 414, configured to divide the described video image sample into several segmented video image samples according to a pre-set time interval, then define the texts within any time interval as the current time information sample, and defined the texts before the mentioned any time interval as the preamble text information sample. To clarify, when the deep learning-based emotional speech synthesis service is invoked in the deep learning-based emotional speech synthesis device in the aforementioned embodiments, the described functional module configurations are used for illustration only. In practical applications, the described functions can be assigned to different functional modules according to practical demands, wherein the internal structural configuration of the device is divided into different functional modules to perform all or a portion of the described functions. Besides, the aforementioned deep learning-based emotional speech synthesis device in the embodiment adopts the same concepts in the described deep learning-based emotional speech synthesis method embodiments. The described device is based on the implementation of the deep learning-based emotional speech synthesis method, whereas the detailed procedures can be referred to the method embodiments and are not explained in further detail. Those skills in that art cam understand that all or a portion of the aforementioned embodiments can be achieved by hardware, or by hardware driven by programs, stored on a readable computer storage medium. The aforementioned storage medium can be but not limited to memory, diskettes, or discs. The aforementioned technical proposals can be achieved by any combinations of the embodiments in the present invention. In other words, the embodiments can be combined to meet requirements of different application scenarios, wherein all possible combinations are falling in the scope of the present invention, and are not explained in further detail.

Claims

2. The device of claim 1, wherein the device further comprises: a first feature extraction unit, configured to assign the described process-pending text information and the preamble information as an input then extract features by the pre-trained first sub model to obtain a first intermediate output.

3. The device of claim 1, wherein the device further comprises: a second feature extraction unit, configured to assign the described process-pending text information and the first intermediate output as an input, then extract features by the pre-trained second sub model to obtain an emotion type and a second intermediate output.

4. The device of claim 1, wherein the device further comprises: a third feature extraction unit, configured to assign the described second intermediate output, process-pending text information, and the emotion type or a user-desired emotion type as an input, then extract features by the pre-trained third sub model to obtain the emotion feature information.

5. The device of claim 1, wherein the device further comprises: a fourth feature extraction module, configured to assign the described process-pending text information and the preamble information as an input then extract features by the pre-trained fourth sub model to obtain a fourth intermediate output.

6. The device of claim 1, wherein the device further comprises: a fifth feature extraction module, configured to assign the described process-pending text information and the fourth intermediate output as an input, then extract features by the pre-trained fifth sub model to obtain an emotion type and a fifth intermediate output.

7. The device of claim 1, wherein the device further comprises: a sixth feature extraction module, configured to assign the described fifth intermediate output, process-pending text information, and the emotion type or a user-desired emotion type as an input, then extract features by the pre-trained sixth sub model to obtain the emotion feature information.

8. The device of the claim 1, wherein the first model comprises a first sub model, a second sub model, and a third sub model.

9. The device of any one of the claims 1 to 2, wherein the device further comprises: assigning the described process-pending text information and the preamble information as an input then extracting features by the pre-trained first sub model to obtain a first intermediate output.

10. The device of any one of the claims 1 to 3, wherein the device further comprises: assigning the described process-pending text information and the first intermediate output as an input, then extracting features by the pre-trained second sub model to obtain an emotion type and a second intermediate output.

11. The device of any one of the claims 1 to 4, wherein the device further comprises: assigning the described second intermediate output, process-pending text information, and the emotion type or a user-desired emotion type as an input, then extracting features by the pre-trained third sub model to obtain the emotion feature information.

12. The device of any one of the claims 1 to 5, wherein when the described preamble information further includes preamble speech information, the described first model further includes a fourth sub model, a fifth sub model, and a sixth sub model connected in series.

13. The device of any one of the claims 1 to 6, wherein the device further comprises: assigning the described process-pending text information and the preamble information as an input then extracting features by the pre-trained fourth sub model to obtain a fourth intermediate output.

14. The device of any one of the claims 1 to 7, wherein the device further comprises: assigning the described process-pending text information and the fourth intermediate output as an input, then extracting features by the pre-trained fifth sub model to obtain an emotion type and a fifth intermediate output.

15. The device of any one of the claims 1 to 8, wherein the device further comprises: assigning the described fifth intermediate output, process-pending text information, and the emotion type or a user-desired emotion type as an input, then extracting features by the pre-trained sixth sub- model to obtain the emotion feature information.

16. The device of any one of the claims 1 to 8, wherein the device further comprises: extracting a video image sample, a text information sample, and a conversation information sample of a video sample.

17. The device of any one of the claims 1 to 9, wherein the device further comprises: labelling the described video image sample according to the pre-set emotional type to obtain an emotion labelling information sample.

18. The device of any one of the claims 1 to 10, wherein the device further comprises: training to obtain the third model by assigning the described video image sample as an input and the described emotion labelling information sample as an output, then extracting the third intermediate output of the described third model as the emotion information of the described video image sample.

19. The device of any one of the claims 1 to 12, wherein the device further comprises: training to obtain the second model by assigning the described emotion information and the text information sample as an input, and the conversation information sample as an output.

20. The device of any one of the claims 1 to 13, wherein the device further comprises: extracting the current text information sample and the preamble information sample of the video sample, wherein the described preamble information sample includes a preamble text information sample.

21. The device of any one of the claims 1 to 14, wherein the device further comprises: training to obtain the described first sub model by an input of the described current text information sample and the preamble information sample and an output of that if the current text information sample has changed compared to the preamble information sample, then extracting the first intermediate output by the described first sub model.

22. The device of any one of the claims 1 to 14, wherein the device further comprises: training to obtain the described second sub model by assigning the first intermediate output and the current text information sample as an input and the emotion type as an output, then extracting the second intermediate output by the second sub model.

23. The device of any one of the claims 1 to 16, wherein the device further comprises: training to obtain the described third sub model by an input of the second intermediate output, the current text information sample, and the emotion type or a user-desired emotion type as an input, and an output of the emotion information by the third model.

24. The device of any one of the claims 1 to 17, wherein the device further comprises: extracting the current text information sample and the preamble information sample of the video sample, wherein the described preamble information sample includes a preamble text information sample and a preamble speech information sample.

25. The device of any one of the claims 1 to 18, wherein the device further comprises: training to obtain the described fourth sub model by an input of the described current text information sample and the preamble information sample and an output of that if the current text information sample has changed compared to the preamble information sample, then extracting the fourth intermediate output by the described fourth sub model.

26. The device of any one of the claims 1 to 19, wherein the device further comprises: training to obtain the described fifth sub model by assigning the first intermediate output and the current text information sample as an input and the emotion type as an output, then extracting the fifth intermediate output by the fifth sub model.

27. The device of any one of the claims 1 to 20, wherein the device further comprises: training to obtain the described sixth sub model by an input of the second intermediate output, the current text information sample, and the emotion type or a user-desired emotion type as an input, and an output of the emotion information by the third model.

28. The device of any one of the claims 1 to 21, wherein the device further comprises: dividing the described video image sample into several segmented video image samples according to a pre- set time interval, then defining the texts within any time interval as the current time information sample, and defining the texts before the mentioned any time interval as the preamble text information sample.

29. The device of any one of the claims 1 to 22, wherein the device further comprises: extracting text information to be processed and preamble information of the process-pending text information.

30. The device of any one of the claims 1 to 23, wherein when the processing target is a text object, the preamble information includes preamble text information.

31. The device of any one of the claims 1 to 23, wherein when the processing target is a voice object or a video object, the preamble information includes the preamble text information and the preamble speech information.

32. The device of any one of the claims 1 to 25, wherein different extraction tools are used to extract the text information from a text object, the text information and the voice information from a voice information, and the text information and the voice information from a video information.

33. The device of any one of the claims 1 to 26, wherein the device further comprises: using a pre- constructed first model to generate emotion feature information by assigning the process- pending text information and the preamble information as an input.

34. The device of any one of the claims 1 to 26, wherein the device further comprises: assigning the described process-pending text information and the preamble information as an input then extracting features by the pre-trained first sub model to obtain a first intermediate output.

35. The device of any one of the claims 1 to 28, wherein the device further comprises: assigning the described process-pending text information and the first intermediate output as an input, then extracting features by the pre-trained second sub model to obtain an emotion type and a second intermediate output.

36. The device of any one of the claims 1 to 29, wherein the device further comprises: assigning the described second intermediate output, process-pending text information, and the emotion type or a user-desired emotion type as an input, then extracting features by the pre-trained third sub model to obtain the emotion feature information.

37. The device of any one of the claims 1 to 30, wherein the input ports of the third sub model is an emotion controlling port, wherein the emotion controlling port can input the emotion type exported by the second sub model, or input an emotion type set manually by a user.

38. The device of any one of the claims 1 to 31, wherein the first intermediate output is the output feature vector from the layer before the logical judgement layer of the first sub model, including the current conversation tone extracted by the first sub model and the emotion characteristics of the current texts.

39. The device of any one of the claims 1 to 32, wherein the second intermediate output is the output feature vector from the layer before the classification layer of the second sub model, including the emotion characteristics of the current text extracted by the second sub model according to the first intermediate output.

40. The device of any one of the claims 1 to 33, wherein the device further comprises: assigning the described process-pending text information and the preamble information as an input then extracting features by the pre-trained fourth sub model to obtain a fourth intermediate output.

41. The device of any one of the claims 1 to 34, wherein the device further comprises: assigning the described process-pending text information and the fourth intermediate output as an input, then extracting features by the pre-trained fifth sub model to obtain an emotion type and a fifth intermediate output.

42. The device of any one of the claims 1 to 35, wherein the device further comprises: assigning the described fifth intermediate output, process-pending text information, and the emotion type or a user-desired emotion type as an input, then extracting features by the pre-trained sixth sub model to obtain the emotion feature information.

43. The device of any one of the claims 1 to 36, wherein the fourth intermediate output is the output feature vector from the layer before the logical judgement layer of the fourth sub model, including the current conversation tone and the emotion characteristics of the current texts extracted by the fourth sub model according to the input conversation speech or the input video.

44. The device of any one of the claims 1 to 36, wherein the fifth intermediate output is the output feature vector from the layer before the classification layer of the fifth sub model, including the emotion characteristics of the current text extracted by the second sub model according to the fourth intermediate output.

45. The device of any one of the claims 1 to 37, wherein after verification, the current text information includes the preamble text information and the preamble speech information.

46. The device of any one of the claims 1 to 38, wherein the device further comprises: using a pre- trained second model to synthesize emotional speech by assigning the emotion feature information and the process-pending text information as an input.

47. The device of any one of the claims 1 to 40, wherein the natural-tone emotional speech is generated on the basis of the text information.

48. The device of any one of the claims 1 to 41, wherein the device further comprises: extracting a video image sample, a text information sample, and a conversation information sample of a video sample.

49. The device of any one of the claims 1 to 42, wherein the device further comprises: labelling the described video image sample according to the pre-set emotional type to obtain an emotion labelling information sample.

50. The device of any one of the claims 1 to 43, wherein the device further comprises: training to obtain the third model by assigning the described video image sample as an input and the described emotion labelling information sample as an output, then extracting the third intermediate output of the described third model as the emotion information of the described video image sample.

51. The device of any one of the claims 1 to 44, wherein the device further comprises: training to obtain the second model by assigning the described emotion information and the text information sample as an input, and the conversation information sample as an output.

52. The device of any one of the claims 1 to 45, wherein the third model is constructed based on ResNet-50, carrying the cross-entropy loss function.

53. The device of any one of the claims 1 to 46, wherein the second model is constructed based on Tacotron2, carrying the mean square error loss function and L2 distance loss function.

54. The device of any one of the claims 1 to 47, wherein the third model is connected with the second model in series and two models are trained together.

55. The device of any one of the claims 1 to 48, wherein after extracting the video image samples, text information samples, and conversation information samples, the video image samples are sent to the third model input port and the third intermediate output is sent to an input port of the second model.

56. The device of any one of the claims 1 to 49, wherein the device further comprises: extracting the current text information sample and the preamble information sample of the video sample, wherein the described preamble information sample includes a preamble text information sample.

57. The device of any one of the claims 1 to 50, wherein the device further comprises: training to obtain the described first sub model by an input of the described current text information sample and the preamble information sample and an output of that if the current text information sample has changed compared to the preamble information sample, then extracting the first intermediate output by the described first sub model.

58. The device of any one of the claims 1 to 51, wherein the device further comprises: training to obtain the described second sub model by assigning the first intermediate output and the current text information sample as an input and the emotion type as an output, then extracting the second intermediate output by the second sub model.

59. The device of any one of the claims 1 to 52, wherein the device further comprises: training to obtain the described third sub model by an input of the second intermediate output, the current text information sample, and the emotion type or a user-desired emotion type as an input, and an output of the emotion information by the third model.

60. The device of any one of the claims 1 to 53, wherein the device further comprises: extracting the current text information sample and the preamble information sample of the video sample, wherein the described preamble information sample includes a preamble text information sample and a preamble speech information sample.

61. The device of any one of the claims 1 to 54, wherein the device further comprises: training to obtain the described fourth sub model by an input of the described current text information sample and the preamble information sample and an output of that if the current text information sample has changed compared to the preamble information sample, then extracting the fourth intermediate output by the described fourth sub model.

62. The device of any one of the claims 1 to 55, wherein the device further comprises: training to obtain the described fifth sub model by assigning the first intermediate output and the current text information sample as an input and the emotion type as an output, then extracting the fifth intermediate output by the fifth sub model.

63. The device of any one of the claims 1 to 56, wherein the device further comprises: training to obtain the described sixth sub model by an input of the second intermediate output, the current text information sample, and the emotion type or a user-desired emotion type as an input, and an output of the emotion information by the third model.

64. A computer readable physical memory having stored thereon a computer program executed by a computer configured to: extract text information to be processed and preamble information of the process- pending text information wherein the preamble information includes the preamble text information; use a pre-constructed first model to generate emotion feature information by assigning the process-pending text information and the preamble information as an input; and use a pre-trained second model to synthesize emotional speech by assigning the emotion feature information and the process-pending text information as an input.

65. The memory of the claim 64, wherein the first model comprises a first sub model, a second sub model, and a third sub model.

66. The memory of any one of the claims 64 to 65, wherein the memory further comprises: assigning the described process-pending text information and the preamble information as an input then extracting features by the pre-trained first sub model to obtain a first intermediate output.

67. The memory of any one of the claims 64 to 66, wherein the memory further comprises: assigning the described process-pending text information and the first intermediate output as an input, then extracting features by the pre-trained second sub model to obtain an emotion type and a second intermediate output.

68. The memory of any one of the claims 64 to 67, wherein the memory further comprises: assigning the described second intermediate output, process-pending text information, and the emotion type or a user-desired emotion type as an input, then extracting features by the pre- trained third sub model to obtain the emotion feature information.

69. The memory of any one of the claims 64 to 68, wherein when the described preamble information further includes preamble speech information, the described first model further includes a fourth sub model, a fifth sub model, and a sixth sub model connected in series.

70. The memory of any one of the claims 64 to 69, wherein the memory further comprises: assigning the described process-pending text information and the preamble information as an input then extracting features by the pre-trained fourth sub model to obtain a fourth intermediate output.

71. The memory of any one of the claims 64 to 70, wherein the memory further comprises: assigning the described process-pending text information and the fourth intermediate output as an input, then extracting features by the pre-trained fifth sub model to obtain an emotion type and a fifth intermediate output.

72. The memory of any one of the claims 64 to 71, wherein the memory further comprises: assigning the described fifth intermediate output, process-pending text information, and the emotion type or a user-desired emotion type as an input, then extracting features by the pre- trained sixth sub model to obtain the emotion feature information.

73. The memory of any one of the claims 64 to 71, wherein the memory further comprises: extracting a video image sample, a text information sample, and a conversation information sample of a video sample.

74. The memory of any one of the claims 64 to 72, wherein the memory further comprises: labelling the described video image sample according to the pre-set emotional type to obtain an emotion labelling information sample.

75. The memory of any one of the claims 64 to 73, wherein the memory further comprises: training to obtain the third model by assigning the described video image sample as an input and the described emotion labelling information sample as an output, then extracting the third intermediate output of the described third model as the emotion information of the described video image sample.

76. The memory of any one of the claims 64 to 75, wherein the memory further comprises: training to obtain the second model by assigning the described emotion information and the text information sample as an input, and the conversation information sample as an output.

77. The memory of any one of the claims 64 to 76, wherein the memory further comprises: extracting the current text information sample and the preamble information sample of the video sample, wherein the described preamble information sample includes a preamble text information sample.

78. The memory of any one of the claims 64 to 77, wherein the memory further comprises: training to obtain the described first sub model by an input of the described current text information sample and the preamble information sample and an output of that if the current text information sample has changed compared to the preamble information sample, then extracting the first intermediate output by the described first sub model.

79. The memory of any one of the claims 64 to 77, wherein the memory further comprises: training to obtain the described second sub model by assigning the first intermediate output and the current text information sample as an input and the emotion type as an output, then extracting the second intermediate output by the second sub model.

80. The memory of any one of the claims 64 to 79, wherein the memory further comprises: training to obtain the described third sub model by an input of the second intermediate output, the current text information sample, and the emotion type or a user-desired emotion type as an input, and an output of the emotion information by the third model.

81. The memory of any one of the claims 64 to 80, wherein the memory further comprises: extracting the current text information sample and the preamble information sample of the video sample, wherein the described preamble information sample includes a preamble text information sample and a preamble speech information sample.

82. The memory of any one of the claims 64 to 81, wherein the memory further comprises: training to obtain the described fourth sub model by an input of the described current text information sample and the preamble information sample and an output of that if the current text information sample has changed compared to the preamble information sample, then extracting the fourth intermediate output by the described fourth sub model.

83. The memory of any one of the claims 64 to 82, wherein the memory further comprises: training to obtain the described fifth sub model by assigning the first intermediate output and the current text information sample as an input and the emotion type as an output, then extracting the fifth intermediate output by the fifth sub model.

84. The memory of any one of the claims 64 to 83, wherein the memory further comprises: training to obtain the described sixth sub model by an input of the second intermediate output, the current text information sample, and the emotion type or a user-desired emotion type as an input, and an output of the emotion information by the third model.

85. The memory of any one of the claims 64 to 84, wherein the memory further comprises: dividing the described video image sample into several segmented video image samples according to a pre-set time interval, then defining the texts within any time interval as the current time information sample, and defining the texts before the mentioned any time interval as the preamble text information sample.

86. The memory of any one of the claims 64 to 85, wherein the memory further comprises: extracting text information to be processed and preamble information of the process-pending text information.

87. The memory of any one of the claims 64 to 86, wherein when the processing target is a text object, the preamble information includes preamble text information.

88. The memory of any one of the claims 64 to 86, wherein when the processing target is a voice object or a video object, the preamble information includes the preamble text information and the preamble speech information.

89. The memory of any one of the claims 64 to 88, wherein different extraction tools are used to extract the text information from a text object, the text information and the voice information from a voice information, and the text information and the voice information from a video information.

90. The memory of any one of the claims 64 to 89, wherein the memory further comprises: using a pre-constructed first model to generate emotion feature information by assigning the process- pending text information and the preamble information as an input.

91. The memory of any one of the claims 64 to 89, wherein the memory further comprises: assigning the described process-pending text information and the preamble information as an input then extracting features by the pre-trained first sub model to obtain a first intermediate output.

92. The memory of any one of the claims 64 to 91, wherein the memory further comprises: assigning the described process-pending text information and the first intermediate output as an input, then extracting features by the pre-trained second sub model to obtain an emotion type and a second intermediate output.

93. The memory of any one of the claims 64 to 92, wherein the memory further comprises: assigning the described second intermediate output, process-pending text information, and the emotion type or a user-desired emotion type as an input, then extracting features by the pre- trained third sub model to obtain the emotion feature information.

94. The memory of any one of the claims 64 to 93, wherein the input ports of the third sub model is an emotion controlling port, wherein the emotion controlling port can input the emotion type exported by the second sub model, or input an emotion type set manually by a user.

95. The memory of any one of the claims 64 to 94, wherein the first intermediate output is the output feature vector from the layer before the logical judgement layer of the first sub model, including the current conversation tone extracted by the first sub model and the emotion characteristics of the current texts.

96. The memory of any one of the claims 64 to 95, wherein the second intermediate output is the output feature vector from the layer before the classification layer of the second sub model, including the emotion characteristics of the current text extracted by the second sub model according to the first intermediate output.

97. The memory of any one of the claims 64 to 96, wherein the memory further comprises: assigning the described process-pending text information and the preamble information as an input then extracting features by the pre-trained fourth sub model to obtain a fourth intermediate output.

98. The memory of any one of the claims 64 to 97, wherein the memory further comprises: assigning the described process-pending text information and the fourth intermediate output as an input, then extracting features by the pre-trained fifth sub model to obtain an emotion type and a fifth intermediate output.

99. The memory of any one of the claims 64 to 98, wherein the memory further comprises: assigning the described fifth intermediate output, process-pending text information, and the emotion type or a user-desired emotion type as an input, then extracting features by the pre- trained sixth sub model to obtain the emotion feature information.

100. The memory of any one of the claims 64 to 99, wherein the fourth intermediate output is the output feature vector from the layer before the logical judgement layer of the fourth sub model, including the current conversation tone and the emotion characteristics of the current texts extracted by the fourth sub model according to the input conversation speech or the input video.

101. The memory of any one of the claims 64 to 99, wherein the fifth intermediate output is the output feature vector from the layer before the classification layer of the fifth sub model, including the emotion characteristics of the current text extracted by the second sub model according to the fourth intermediate output.

102. The memory of any one of the claims 64 to 100, wherein after verification, the current text information includes the preamble text information and the preamble speech information.

103. The memory of any one of the claims 64 to 101, wherein the memory further comprises: using a pre-trained second model to synthesize emotional speech by assigning the emotion feature information and the process-pending text information as an input.

104. The memory of any one of the claims 64 to 103, wherein the natural-tone emotional speech is generated on the basis of the text information.

105. The memory of any one of the claims 64 to 104, wherein the memory further comprises: extracting a video image sample, a text information sample, and a conversation information sample of a video sample.

106. The memory of any one of the claims 64 to 105, wherein the memory further comprises: labelling the described video image sample according to the pre-set emotional type to obtain an emotion labelling information sample.

107. The memory of any one of the claims 64 to 106, wherein the memory further comprises: training to obtain the third model by assigning the described video image sample as an input and the described emotion labelling information sample as an output, then extracting the third intermediate output of the described third model as the emotion information of the described video image sample.

108. The memory of any one of the claims 64 to 107, wherein the memory further comprises: training to obtain the second model by assigning the described emotion information and the text information sample as an input, and the conversation information sample as an output.

109. The memory of any one of the claims 64 to 108, wherein the third model is constructed based on ResNet-50, carrying the cross-entropy loss function.

110. The memory of any one of the claims 64 to 109, wherein the second model is constructed based on Tacotron2, carrying the mean square error loss function and L2 distance loss function.

111. The memory of any one of the claims 64 to 110, wherein the third model is connected with the second model in series and two models are trained together.

112. The memory of any one of the claims 64 to 111, wherein after extracting the video image samples, text information samples, and conversation information samples, the video image samples are sent to the third model input port and the third intermediate output is sent to an input port of the second model.

113. The memory of any one of the claims 64 to 112, wherein the memory further comprises: extracting the current text information sample and the preamble information sample of the video sample, wherein the described preamble information sample includes a preamble text information sample.

114. The memory of any one of the claims 64 to 113, wherein the memory further comprises: training to obtain the described first sub model by an input of the described current text information sample and the preamble information sample and an output of that if the current text information sample has changed compared to the preamble information sample, then extracting the first intermediate output by the described first sub model.

115. The memory of any one of the claims 64 to 114, wherein the memory further comprises: training to obtain the described second sub model by assigning the first intermediate output and the current text information sample as an input and the emotion type as an output, then extracting the second intermediate output by the second sub model.

116. The memory of any one of the claims 64 to 115, wherein the memory further comprises: training to obtain the described third sub model by an input of the second intermediate output, the current text information sample, and the emotion type or a user-desired emotion type as an input, and an output of the emotion information by the third model.

117. The memory of any one of the claims 64 to 116, wherein the memory further comprises: extracting the current text information sample and the preamble information sample of the video sample, wherein the described preamble information sample includes a preamble text information sample and a preamble speech information sample.

118. The memory of any one of the claims 64 to 117, wherein the memory further comprises: training to obtain the described fourth sub model by an input of the described current text information sample and the preamble information sample and an output of that if the current text information sample has changed compared to the preamble information sample, then extracting the fourth intermediate output by the described fourth sub model.

119. The memory of any one of the claims 64 to 118, wherein the memory further comprises: training to obtain the described fifth sub model by assigning the first intermediate output and the current text information sample as an input and the emotion type as an output, then extracting the fifth intermediate output by the fifth sub model.

120. The memory of any one of the claims 64 to 119, wherein the memory further comprises: training to obtain the described sixth sub model by an input of the second intermediate output, the current text information sample, and the emotion type or a user-desired emotion type as an input, and an output of the emotion information by the third model.

121. A deep learning-based emotional speech synthesis method, the method comprising: extracting, by an extraction module, text information to be processed and preamble information of the process-pending text information wherein the preamble information includes the preamble text information; using a pre-constructed first model to generate emotion feature information by assigning the process-pending text information and the preamble information as an input; and using a pre-trained second model to synthesize emotional speech by assigning the emotion feature information and the process-pending text information as an input.

122. The method of the claim 121, wherein the first model comprises a first sub model, a second sub model, and a third sub model.

123. The method of any one of the claims 121 to 122, wherein the method further comprises: assigning the described process-pending text information and the preamble information as an input then extracting features by the pre-trained first sub model to obtain a first intermediate output.

124. The method of any one of the claims 121 to 123, wherein the method further comprises: assigning the described process-pending text information and the first intermediate output as an input, then extracting features by the pre-trained second sub model to obtain an emotion type and a second intermediate output.

125. The method of any one of the claims 121 to 124, wherein the method further comprises: assigning the described second intermediate output, process-pending text information, and the emotion type or a user-desired emotion type as an input, then extracting features by the pre- trained third sub model to obtain the emotion feature information.

126. The method of any one of the claims 121 to 125, wherein when the described preamble information further includes preamble speech information, the described first model further includes a fourth sub model, a fifth sub model, and a sixth sub model connected in series.

127. The method of any one of the claims 121 to 126, wherein the method further comprises: assigning the described process-pending text information and the preamble information as an input then extracting features by the pre-trained fourth sub model to obtain a fourth intermediate output.

128. The method of any one of the claims 121 to 127, wherein the method further comprises: assigning the described process-pending text information and the fourth intermediate output as an input, then extracting features by the pre-trained fifth sub model to obtain an emotion type and a fifth intermediate output.

129. The method of any one of the claims 121 to 128, wherein the method further comprises: assigning the described fifth intermediate output, process-pending text information, and the emotion type or a user-desired emotion type as an input, then extracting features by the pre- trained sixth sub model to obtain the emotion feature information.

130. The method of any one of the claims 121 to 128, wherein the method further comprises: extracting a video image sample, a text information sample, and a conversation information sample of a video sample.

131. The method of any one of the claims 121 to 129, wherein the method further comprises: labelling the described video image sample according to the pre-set emotional type to obtain an emotion labelling information sample.

132. The method of any one of the claims 121 to 130, wherein the method further comprises: training to obtain the third model by assigning the described video image sample as an input and the described emotion labelling information sample as an output, then extracting the third intermediate output of the described third model as the emotion information of the described video image sample.

133. The method of any one of the claims 121 to 132, wherein the method further comprises: training to obtain the second model by assigning the described emotion information and the text information sample as an input, and the conversation information sample as an output.

134. The method of any one of the claims 121 to 133, wherein the method further comprises: extracting the current text information sample and the preamble information sample of the video sample, wherein the described preamble information sample includes a preamble text information sample.

135. The method of any one of the claims 121 to 134, wherein the method further comprises: training to obtain the described first sub model by an input of the described current text information sample and the preamble information sample and an output of that if the current text information sample has changed compared to the preamble information sample, then extracting the first intermediate output by the described first sub model.

136. The method of any one of the claims 121 to 134, wherein the method further comprises: training to obtain the described second sub model by assigning the first intermediate output and the current text information sample as an input and the emotion type as an output, then extracting the second intermediate output by the second sub model.

137. The method of any one of the claims 121 to 136, wherein the method further comprises: training to obtain the described third sub model by an input of the second intermediate output, the current text information sample, and the emotion type or a user-desired emotion type as an input, and an output of the emotion information by the third model.

138. The method of any one of the claims 121 to 137, wherein the method further comprises: extracting the current text information sample and the preamble information sample of the video sample, wherein the described preamble information sample includes a preamble text information sample and a preamble speech information sample.

139. The method of any one of the claims 121 to 138, wherein the method further comprises: training to obtain the described fourth sub model by an input of the described current text information sample and the preamble information sample and an output of that if the current text information sample has changed compared to the preamble information sample, then extracting the fourth intermediate output by the described fourth sub model.

140. The method of any one of the claims 121 to 139, wherein the method further comprises: training to obtain the described fifth sub model by assigning the first intermediate output and the current text information sample as an input and the emotion type as an output, then extracting the fifth intermediate output by the fifth sub model.

141. The method of any one of the claims 121 to 140, wherein the method further comprises: training to obtain the described sixth sub model by an input of the second intermediate output, the current text information sample, and the emotion type or a user-desired emotion type as an input, and an output of the emotion information by the third model.

142. The method of any one of the claims 121 to 141, wherein the method further comprises: dividing the described video image sample into several segmented video image samples according to a pre-set time interval, then defining the texts within any time interval as the current time information sample, and defining the texts before the mentioned any time interval as the preamble text information sample.

143. The method of any one of the claims 121 to 142, wherein the method further comprises: extracting text information to be processed and preamble information of the process-pending text information.

144. The method of any one of the claims 121 to 143, wherein when the processing target is a text object, the preamble information includes preamble text information.

145. The method of any one of the claims 121 to 143, wherein when the processing target is a voice object or a video object, the preamble information includes the preamble text information and the preamble speech information.

146. The method of any one of the claims 121 to 145, wherein different extraction tools are used to extract the text information from a text object, the text information and the voice information from a voice information, and the text information and the voice information from a video information.

147. The method of any one of the claims 121 to 146, wherein the method further comprises: using a pre-constructed first model to generate emotion feature information by assigning the process-pending text information and the preamble information as an input.

148. The method of any one of the claims 121 to 146, wherein the method further comprises: assigning the described process-pending text information and the preamble information as an input then extracting features by the pre-trained first sub model to obtain a first intermediate output.

149. The method of any one of the claims 121 to 148, wherein the method further comprises: assigning the described process-pending text information and the first intermediate output as an input, then extracting features by the pre-trained second sub model to obtain an emotion type and a second intermediate output.

150. The method of any one of the claims 121 to 149, wherein the method further comprises: assigning the described second intermediate output, process-pending text information, and the emotion type or a user-desired emotion type as an input, then extracting features by the pre- trained third sub model to obtain the emotion feature information.

151. The method of any one of the claims 121 to 150, wherein the input ports of the third sub model is an emotion controlling port, wherein the emotion controlling port can input the emotion type exported by the second sub model, or input an emotion type set manually by a user.

152. The method of any one of the claims 121 to 151, wherein the first intermediate output is the output feature vector from the layer before the logical judgement layer of the first sub- model, including the current conversation tone extracted by the first sub model and the emotion characteristics of the current texts.

153. The method of any one of the claims 121 to 152, wherein the second intermediate output is the output feature vector from the layer before the classification layer of the second sub- model, including the emotion characteristics of the current text extracted by the second sub- model according to the first intermediate output.

154. The method of any one of the claims 121 to 153, wherein the method further comprises: assigning the described process-pending text information and the preamble information as an input then extracting features by the pre-trained fourth sub model to obtain a fourth intermediate output.

155. The method of any one of the claims 121 to 154, wherein the method further comprises: assigning the described process-pending text information and the fourth intermediate output as an input, then extracting features by the pre-trained fifth sub model to obtain an emotion type and a fifth intermediate output.

156. The method of any one of the claims 121 to 155, wherein the method further comprises: assigning the described fifth intermediate output, process-pending text information, and the emotion type or a user-desired emotion type as an input, then extracting features by the pre- trained sixth sub model to obtain the emotion feature information.

157. The method of any one of the claims 121 to 156, wherein the fourth intermediate output is the output feature vector from the layer before the logical judgement layer of the fourth sub model, including the current conversation tone and the emotion characteristics of the current texts extracted by the fourth sub model according to the input conversation speech or the input video.

158. The method of any one of the claims 121 to 156, wherein the fifth intermediate output is the output feature vector from the layer before the classification layer of the fifth sub model, including the emotion characteristics of the current text extracted by the second sub model according to the fourth intermediate output.

159. The method of any one of the claims 121 to 157, wherein after verification, the current text information includes the preamble text information and the preamble speech information.

160. The method of any one of the claims 121 to 158, wherein the method further comprises: using a pre-trained second model to synthesize emotional speech by assigning the emotion feature information and the process-pending text information as an input.

161. The method of any one of the claims 121 to 160, wherein the natural-tone emotional speech is generated on the basis of the text information.

162. The method of any one of the claims 121 to 161, wherein the method further comprises: extracting a video image sample, a text information sample, and a conversation information sample of a video sample.

163. The method of any one of the claims 121 to 162, wherein the method further comprises: labelling the described video image sample according to the pre-set emotional type to obtain an emotion labelling information sample.

164. The method of any one of the claims 121 to 163, wherein the method further comprises: training to obtain the third model by assigning the described video image sample as an input and the described emotion labelling information sample as an output, then extracting the third intermediate output of the described third model as the emotion information of the described video image sample.

165. The method of any one of the claims 121 to 164, wherein the method further comprises: training to obtain the second model by assigning the described emotion information and the text information sample as an input, and the conversation information sample as an output.

166. The method of any one of the claims 121 to 165, wherein the third model is constructed based on ResNet-50, carrying the cross-entropy loss function.

167. The method of any one of the claims 121 to 166, wherein the second model is constructed based on Tacotron2, carrying the mean square error loss function and L2 distance loss function.

168. The method of any one of the claims 121 to 167, wherein the third model is connected with the second model in series and two models are trained together.

169. The method of any one of the claims 121 to 168, wherein after extracting the video image samples, text information samples, and conversation information samples, the video image samples are sent to the third model input port and the third intermediate output is sent to an input port of the second model.

170. The method of any one of the claims 121 to 169, wherein the method further comprises: extracting the current text information sample and the preamble information sample of the video sample, wherein the described preamble information sample includes a preamble text information sample.

171. The method of any one of the claims 121 to 170, wherein the method further comprises: training to obtain the described first sub model by an input of the described current text information sample and the preamble information sample and an output of that if the current text information sample has changed compared to the preamble information sample, then extracting the first intermediate output by the described first sub model.

172. The method of any one of the claims 121 to 171, wherein the method further comprises: training to obtain the described second sub model by assigning the first intermediate output and the current text information sample as an input and the emotion type as an output, then extracting the second intermediate output by the second sub model.

173. The method of any one of the claims 121 to 172, wherein the method further comprises: training to obtain the described third sub model by an input of the second intermediate output, the current text information sample, and the emotion type or a user-desired emotion type as an input, and an output of the emotion information by the third model.

174. The method of any one of the claims 121 to 173, wherein the method further comprises: extracting the current text information sample and the preamble information sample of the video sample, wherein the described preamble information sample includes a preamble text information sample and a preamble speech information sample.

175. The method of any one of the claims 121 to 174, wherein the method further comprises: training to obtain the described fourth sub model by an input of the described current text information sample and the preamble information sample and an output of that if the current text information sample has changed compared to the preamble information sample, then extracting the fourth intermediate output by the described fourth sub model.

176. The method of any one of the claims 121 to 175, wherein the method further comprises: training to obtain the described fifth sub model by assigning the first intermediate output and the current text information sample as an input and the emotion type as an output, then extracting the fifth intermediate output by the fifth sub model.

177. The method of any one of the claims 121 to 176, wherein the method further comprises: training to obtain the described sixth sub model by an input of the second intermediate output, the current text information sample, and the emotion type or a user-desired emotion type as an input, and an output of the emotion information by the third model.