Speech synthesis method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the present application provide a speech synthesis method and device, electronic equipment and storage medium, belonging to the field of artificial intelligence technology, and suitable for the field of financial technology. The method comprises: obtaining multi-modal reference data; extracting visual reference embedding features, audio reference embedding features, text reference embedding features and sentiment description embedding features from the multi-modal reference data; performing feature projection on the visual reference embedding features, the audio reference embedding features and the text reference embedding features based on a multi-modal projection matrix to obtain multi-modal implicit sentiment embedding features; performing feature projection on the sentiment description embedding features based on the multi-modal projection matrix to obtain sentiment prompt projection embedding features; performing feature calculation based on the multi-modal implicit sentiment embedding features and the sentiment prompt projection embedding features to obtain sentiment reference embedding features; and performing speech synthesis based on the sentiment reference embedding features and speech synthesis text embedding features. The embodiments of the present application can improve the naturalness of speech synthesis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and is suitable for the field of financial technology, in particular to a speech synthesis method and device, an electronic device and a storage medium. BACKGROUND

[0002] Speech synthesis is a technology for converting text content into natural speech, which can be applied to various scenarios. For example, in the financial field, it can be used to synthesize natural speech in scenarios such as intelligent customer service, intelligent assistants, and intelligent outbound calls.

[0003] Currently, in the process of giving synthesized speech emotional characteristics, the main methods are to learn the emotional characteristics of the speaker in a specific speech segment or to adjust the speech according to preset emotional parameters. However, in actual application scenarios, the above methods often fail to accurately capture and reproduce emotional characteristics, resulting in unnatural emotional expression of synthesized speech and affecting the naturalness of speech synthesis.

[0004] Therefore, how to improve the naturalness of speech synthesis has become a technical problem to be solved. SUMMARY

[0005] The main purpose of the embodiments of the present application is to propose a speech synthesis method and device, an electronic device and a storage medium, which aims to improve the naturalness of speech synthesis.

[0006] To achieve the above purpose, the first aspect of the embodiments of the present application proposes a speech synthesis method, which comprises:

[0007] Obtaining multi-modal reference data; wherein the multi-modal reference data comprises visual reference data, audio reference data, text reference data and emotional description data;

[0008] Extracting features from the multi-modal reference data to obtain visual reference embedding features of the visual reference data, audio reference embedding features of the audio reference data, text reference embedding features of the text reference data, and emotional description embedding features of the emotional description data;

[0009] Projecting the visual reference embedding features, the audio reference embedding features and the text reference embedding features based on a preset multi-modal projection matrix to obtain multi-modal implicit emotional embedding features;

[0010] Projecting the emotional description embedding features based on the multi-modal projection matrix to obtain emotional prompt projection embedding features;

[0011] Performing feature calculation based on the multi-modal implicit emotional embedding features and the emotional prompt projection embedding features to obtain emotional reference embedding features;

[0012] text preprocessing is performed on the pre-acquired speech synthesis text to obtain speech synthesis text embedding features;

[0013] speech synthesis is performed based on the emotion reference embedding features and the speech synthesis text embedding features.

[0014] In some embodiments, the multi-modal projection matrix includes a visual projection matrix, an audio projection matrix, and a text projection matrix; the feature projection is performed on the visual reference embedding features, the audio reference embedding features, and the text reference embedding features based on the preset multi-modal projection matrix to obtain multi-modal implicit emotion embedding features, including:

[0015] The feature projection is performed on the visual reference embedding features based on the visual projection matrix to obtain visual projection embedding features;

[0016] The feature projection is performed on the audio reference embedding features based on the audio projection matrix to obtain audio projection embedding features;

[0017] The feature projection is performed on the text reference embedding features based on the text projection matrix to obtain text projection embedding features;

[0018] The emotion extraction is performed on the visual projection embedding features, the audio projection embedding features, and the text projection embedding features to obtain the multi-modal implicit emotion embedding features.

[0019] In some embodiments, the emotion prompt projection embedding features include visual prompt projection embedding features, audio prompt projection embedding features, and text prompt projection embedding features, the visual prompt projection embedding features are obtained by performing the feature projection on the emotion description embedding features based on the visual projection matrix; the audio prompt projection embedding features are obtained by performing the feature projection on the emotion description embedding features based on the audio projection matrix; the text prompt projection embedding features are obtained by performing the feature projection on the emotion description embedding features based on the text projection matrix; the feature calculation is performed based on the multi-modal implicit emotion embedding features and the emotion prompt projection embedding features to obtain emotion reference embedding features, including:

[0020] The emotion extraction is performed on the visual prompt projection embedding features to obtain visual explicit emotion embedding features;

[0021] The emotion extraction is performed on the audio prompt projection embedding features to obtain audio explicit emotion embedding features;

[0022] The emotion extraction is performed on the text prompt projection embedding features to obtain text explicit emotion embedding features;

[0023] The visual explicit sentiment embedding feature, the audio explicit sentiment embedding feature and the text explicit sentiment embedding feature are filtered based on the multi-modal implicit sentiment embedding feature, to obtain the sentiment reference embedding feature.

[0024] In some embodiments, the filtering of the visual explicit sentiment embedding feature, the audio explicit sentiment embedding feature and the text explicit sentiment embedding feature based on the multi-modal implicit sentiment embedding feature to obtain the sentiment reference embedding feature comprises:

[0025] A first sentiment similarity data is obtained by calculating the similarity between the multi-modal implicit sentiment embedding feature and the visual explicit sentiment embedding feature;

[0026] A second sentiment similarity data is obtained by calculating the similarity between the multi-modal implicit sentiment embedding feature and the audio explicit sentiment embedding feature;

[0027] A third sentiment similarity data is obtained by calculating the similarity between the multi-modal implicit sentiment embedding feature and the text explicit sentiment embedding feature;

[0028] A target sentiment similarity data is obtained by extracting the maximum value from the first sentiment similarity data, the second sentiment similarity data and the third sentiment similarity data;

[0029] A target explicit sentiment embedding feature is obtained by filtering the visual explicit sentiment embedding feature, the audio explicit sentiment embedding feature and the text explicit sentiment embedding feature based on the target sentiment similarity data, and the target explicit sentiment embedding feature is taken as the sentiment reference embedding feature.

[0030] In some embodiments, the speech synthesis based on the sentiment reference embedding feature and the speech synthesis text embedding feature comprises:

[0031] An original phoneme sequence is obtained by converting the speech synthesis text embedding feature into phonemes;

[0032] An initial phoneme sequence is obtained by adjusting the original phoneme sequence based on the sentiment reference embedding feature;

[0033] A target phoneme sequence is obtained by normalizing the initial phoneme sequence;

[0034] The speech synthesis is performed based on the target phoneme sequence.

[0035] In some embodiments, the speech synthesis based on the sentiment reference embedding feature and the speech synthesis text embedding feature comprises:

[0036] perform feature combination on the affective reference embedding feature and the speech synthesis text embedding feature to obtain an original text embedding feature;

[0037] perform attention calculation on the original text embedding feature to obtain an initial text embedding feature;

[0038] perform context learning on the initial text embedding feature to obtain a target text embedding feature;

[0039] perform decoding on the target text embedding feature to obtain target speech spectrum data;

[0040] perform speech synthesis based on the target speech spectrum data.

[0041] In some embodiments, the speech synthesis based on the affective reference embedding feature and the speech synthesis text embedding feature comprises:

[0042] perform phoneme conversion on the speech synthesis text embedding feature to obtain an original speech phoneme sequence;

[0043] perform feature splicing on the original speech phoneme sequence and the affective reference embedding feature to obtain an initial speech phoneme sequence;

[0044] perform attention calculation on the initial speech phoneme sequence to obtain an attention embedding feature;

[0045] perform duration adjustment on the attention embedding feature to obtain a speech synthesis embedding feature;

[0046] perform spectrum conversion on the speech synthesis embedding feature to obtain synthesis speech spectrum data;

[0047] perform speech conversion on the synthesis speech spectrum data.

[0048] To achieve the above object, a second aspect of an embodiment of the present application proposes a speech synthesis device, which comprises:

[0049] a multi-modal data acquisition module configured to acquire multi-modal reference data; wherein the multi-modal reference data comprises visual reference data, audio reference data, text reference data and affective description data;

[0050] a feature extraction module configured to perform feature extraction on the multi-modal reference data to obtain a visual reference embedding feature of the visual reference data, an audio reference embedding feature of the audio reference data, a text reference embedding feature of the text reference data and an affective description embedding feature of the affective description data;

[0051] The multi-modal feature projection module is configured to project the visual reference embedding feature, the audio reference embedding feature and the text reference embedding feature based on a preset multi-modal projection matrix to obtain multi-modal implicit emotion embedding features.

[0052] The prompt feature projection module is configured to project the emotion description embedding feature based on the multi-modal projection matrix to obtain an emotion prompt projection embedding feature.

[0053] The reference feature calculation module is configured to calculate features based on the multi-modal implicit emotion embedding features and the emotion prompt projection embedding feature to obtain an emotion reference embedding feature.

[0054] The text preprocessing module is configured to perform text preprocessing on the pre-acquired speech synthesis text to obtain speech synthesis text embedding features.

[0055] The speech synthesis module is configured to perform speech synthesis based on the emotion reference embedding feature and the speech synthesis text embedding feature.

[0056] To achieve the above object, a third aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method of the first aspect when executing the computer program.

[0057] To achieve the above object, a fourth aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect.

[0058] The speech synthesis method and device, electronic device and storage medium provided by the present application can obtain multi-modal reference data, extract features from the multi-modal reference data to obtain visual reference embedding features of visual reference data, audio reference embedding features of audio reference data, text reference embedding features of text reference data and emotion description embedding features of emotion description data, project the visual reference embedding features, the audio reference embedding features and the text reference embedding features based on a multi-modal projection matrix to obtain multi-modal implicit emotion embedding features, project the emotion description embedding features based on the multi-modal projection matrix to obtain emotion prompt projection embedding features, calculate features based on the multi-modal implicit emotion embedding features and the emotion prompt projection embedding features to obtain emotion reference embedding features, realize the fusion of multi-modal information, and finally perform speech synthesis based on the emotion reference embedding features and speech synthesis text, which can improve the richness and naturalness of the emotional expression of the speech synthesis, and make the synthesized speech closer to the real emotional expression. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 is a flowchart of the speech synthesis method provided by the embodiment of the present application;

[0060] Figure 2 is Figure 1 is a flowchart of step S103 in

[0061] Figure 3 is Figure 1 is a flowchart of step S105 in

[0062] Figure 4 is Figure 3 is a flowchart of step S304 in

[0063] Figure 5 is Figure 1 is a flowchart of step S107 in

[0064] Figure 6 is Figure 1 is another flowchart of step S107 in

[0065] Figure 7 is another flowchart of step S107 in Figure 1

[0066] Figure 8 is a structural schematic diagram of the speech synthesis apparatus provided by the embodiment of the present application;

[0067] Figure 9 is a hardware structural schematic diagram of the electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0069] It should be noted that although the functional modules are divided in the apparatus schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the apparatus or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application. ​

[0071] First, the terms involved in the present application are analyzed:

[0072] Artificial intelligence (AI): It is a new technical science that studies, develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; artificial intelligence is a branch of computer science, and artificial intelligence tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0073] Natural language processing (NLP): NLP uses computers to process, understand and use human language (such as Chinese, English, etc.). NLP is a branch of artificial intelligence and is a cross-disciplinary subject of computer science and linguistics, also commonly known as computational linguistics. Natural language processing includes syntax analysis, semantic analysis, discourse understanding, etc. Natural language processing is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining, etc. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research related to language processing, and language computing related linguistic research.

[0074] Text to Speech (TTS) is an artificial intelligence technology that can convert text content into natural speech. Speech synthesis technology involves acoustics, linguistics, digital signal processing, computer science and other disciplines, and is a frontier technology in the field of information processing. Speech synthesis technology is widely used in various scenarios, such as voice assistants, intelligent navigation, voice broadcasting, news broadcasting, weather forecasting, etc., greatly enriching human life experience.

[0075] In addition, speech synthesis technology can also be applied in the financial field, used in intelligent customer service, intelligent assistants, intelligent outbound calls and other scenarios to synthesize natural speech.

[0076] At present, in the process of giving the synthesized voice emotional characteristics, the way of learning the emotional characteristics of the speaker in a specific voice segment or adjusting the voice according to the preset emotional parameters is mainly adopted, and in the actual application scene, the above-mentioned way is often difficult to accurately capture and reproduce the emotional characteristics, resulting in unnatural emotional expression of the synthesized voice, which affects the naturalness of voice synthesis.

[0077] Based on this, the embodiment of the application provides a voice synthesis method and device, electronic equipment and storage medium, aiming to improve the naturalness of voice synthesis.

[0078] The voice synthesis method and device, electronic equipment and storage medium provided by the embodiment of the application are specifically explained by the following embodiment, first describe the voice synthesis method in the embodiment of the application.

[0079] The embodiment of the application can acquire and process related data based on artificial intelligence technology. Among them, artificial intelligence (Artificial Intelligence, AI) is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results. Theory, method, technology and application system.

[0080] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc. Several major directions.

[0081] The voice synthesis method provided by the embodiment of the application relates to the field of artificial intelligence technology and is suitable for the field of financial technology. The voice synthesis method provided by the embodiment of the application can be applied in a terminal, can also be applied in a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and basic cloud computing services such as big data and artificial intelligence platforms; the software can be an application that implements the voice synthesis method, but is not limited to the above forms.

[0082] The application is operable in a multitude of generic or specific computer system environments or configurations. Examples of well known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media including memory storage devices.

[0083] It should be noted that in each specific embodiment of the present application, when it is necessary to process relevant data related to the identity or characteristics of the user according to user information, user behavior data, user history data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or a jump to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for enabling the embodiments of the present application to normally operate will be obtained.

[0084] Figure 1 is an optional flowchart of the voice synthesis method provided by the embodiments of the present application, Figure 1 The method in can include but is not limited to including steps S101 to S107.

[0085] Step S101, obtaining multi-modal reference data; wherein the multi-modal reference data includes visual reference data, audio reference data, text reference data, and emotion description data;

[0086] Step S102, performing feature extraction on the multi-modal reference data to obtain visual reference embedding features of the visual reference data, audio reference embedding features of the audio reference data, text reference embedding features of the text reference data, and emotion description embedding features of the emotion description data;

[0087] Step S103, performing feature projection on the visual reference embedding features, the audio reference embedding features, and the text reference embedding features based on a preset multi-modal projection matrix to obtain multi-modal implicit emotion embedding features;

[0088] Step S104, performing feature projection on the emotional description embedding feature based on the multi-modal projection matrix to obtain an emotional prompt projection embedding feature;

[0089] Step S105, performing feature calculation based on the multi-modal implicit emotional embedding feature and the emotional prompt projection embedding feature to obtain an emotional reference embedding feature;

[0090] Step S106, performing text preprocessing on the pre-acquired speech synthesis text to obtain a speech synthesis text embedding feature;

[0091] Step S107, performing speech synthesis based on the emotional reference embedding feature and the speech synthesis text embedding feature.

[0092] The steps S101 to S107 shown in the embodiments of the present application, by acquiring multi-modal reference data, and performing feature extraction on the multi-modal reference data, obtaining visual reference embedding features of visual reference data, audio reference embedding features of audio reference data, text reference embedding features of text reference data, and emotional description embedding features of emotional description data; further performing feature projection on the visual reference embedding features, the audio reference embedding features, and the text reference embedding features based on a multi-modal projection matrix to obtain multi-modal implicit emotional embedding features, and performing feature projection on the emotional description embedding features based on the multi-modal projection matrix to obtain emotional prompt projection embedding features; then performing feature calculation based on the multi-modal implicit emotional embedding features and the emotional prompt projection embedding features to obtain emotional reference embedding features, realizing the fusion of multi-modal information; finally, performing speech synthesis based on the emotional reference embedding features and the speech synthesis text, which can improve the richness and naturalness of the emotional expression of speech synthesis, making the synthesized speech closer to the real emotional expression.

[0093] In step S101 of some embodiments, multi-modal reference data containing emotional information can be collected from various sources, including but not limited to: visual reference data (such as facial expression images or videos), audio reference data (such as speech clips with emotions), text reference data (such as text with emotions), and emotional description data (such as emotional labels or emotional scores), rich and accurate emotional information can be extracted from multi-modal reference data, which helps to enhance the emotional performance ability of speech synthesis and improve the naturalness of speech synthesis.

[0094] In step S102 of some embodiments, the multi-modal reference data is extracted by a pre-trained multi-modal encoder;

[0095] Specifically, the multi-modal encoder comprises a visual encoder, an audio encoder, a text encoder, and a prompt encoder; the visual encoder is configured to perform feature extraction on the visual reference data to obtain visual reference embedding features; the audio encoder is configured to perform feature extraction on the audio reference data to obtain audio reference embedding features; the text encoder is configured to perform feature extraction on the text reference data to obtain text reference embedding features; and the prompt encoder is configured to perform feature extraction on the sentiment description data to obtain sentiment description embedding features.

[0096] It should be noted that the feature dimensions of the visual reference embedding features, the audio reference embedding features, the text reference embedding features, and the sentiment description embedding features are the same, for example, all are 512 dimensions.

[0097] In an embodiment, the model architecture of the visual encoder, the text encoder, and the prompt encoder can adopt a CLIP model, and the model architecture of the audio encoder can adopt a Wav2Vec2.0.

[0098] In another embodiment, the visual encoder, the audio encoder, the text encoder, and the prompt encoder can adopt a model structure of a convolutional neural network (CNN), a recurrent neural network (RNN), or a Transformer, and the specific selection is not limited thereto and can be selected according to actual application scenarios.

[0099] In some embodiments, in order to align the sentiment features of different modalities, a prompt-based anchoring scheme is adopted. First, the visual reference embedding features, the audio reference embedding features, and the text reference embedding features are projected by using a preset multi-modal projection matrix to map them into the same sentiment space, thereby facilitating subsequent alignment; the multi-modal projection matrix comprises a visual projection matrix, an audio projection matrix, and a text projection matrix.

[0100] Please refer to Figure 2 In some embodiments, step S103 can include, but is not limited to, steps S201 to S204.

[0101] In step S201, the visual reference embedding features are projected based on the visual projection matrix to obtain visual projection embedding features.

[0102] In step S202, the audio reference embedding features are projected based on the audio projection matrix to obtain audio projection embedding features.

[0103] In step S203, the text reference embedding features are projected based on the text projection matrix to obtain text projection embedding features.

[0104] Step S204, the visual projection embedding feature, the audio projection embedding feature and the text projection embedding feature are subjected to emotion extraction to obtain a multi-modal implicit emotion embedding feature.

[0105] The steps S201 to S204 shown in the embodiments of the present application are as follows: the visual reference embedding feature is subjected to feature projection based on a visual projection matrix to obtain a visual projection embedding feature; the audio reference embedding feature is subjected to feature projection based on an audio projection matrix to obtain an audio projection embedding feature; the text reference embedding feature is subjected to feature projection based on a text projection matrix to obtain a text projection embedding feature; then, the visual projection embedding feature, the audio projection embedding feature and the text projection embedding feature are subjected to emotion extraction to obtain a multi-modal implicit emotion embedding feature which fuses multi-modal information, which can effectively unify emotion information from different modalities (vision, audio and text) into the same emotion feature space, thereby realizing more accurate and comprehensive emotion capture and expression, providing more accurate emotion guidance for subsequent speech synthesis and being beneficial to improving the naturalness of speech synthesis.

[0106] In some embodiments, the preset multi-modal projection matrix is obtained in the training process of the vision encoder, the audio encoder and the text encoder and can reflect the emotion information extraction capability of the encoders. The multi-modal projection matrix is used to project the visual / audio / text reference embedding feature into a preset feature space, which is convenient for subsequent emotion extraction.

[0107] In some embodiments, the visual projection matrix can be denoted as W vis , the audio projection matrix can be denoted as W audio , and the text projection matrix can be denoted as W text .

[0108] The visual reference embedding feature can be denoted as f vis , the audio reference embedding feature can be denoted as f audio , and the text reference embedding feature can be denoted as f text .

[0109] The process of projecting the visual reference embedding feature based on the visual projection matrix is shown in formula (1):

[0110] u vis = W vis · f vis (1).

[0111] Wherein, u vis is the visual projection embedding feature.

[0112] The process of projecting the audio reference embedding feature based on the audio projection matrix is shown in formula (2):

[0113] uaudio = W audio · f audio (2);

[0114] wherein u audio is an audio projection embedding feature.

[0115] The process of projecting the text reference embedding feature based on the text projection matrix is shown in formula (3):

[0116] u text = W text · f text (3);

[0117] wherein u text is a text projection embedding feature.

[0118] In step S204 of some embodiments, the visual projection embedding feature, the audio projection embedding feature and the text projection embedding feature can be extracted by feature splicing to obtain implicit emotional features, i.e. multi-modal implicit emotional embedding features.

[0119] It can be understood that the implicit emotional features are uncertain in emotional information, and therefore explicit emotional features and multi-modal emotional information (visual projection matrix, audio projection matrix, text projection matrix) need to be used in subsequent steps to obtain more accurate emotional features.

[0120] In step S104 of some embodiments, the emotional description embedding feature is projected by the visual projection matrix, the audio projection matrix and the text projection matrix respectively to obtain the visual prompt projection embedding feature, the audio prompt projection embedding feature and the text prompt projection embedding feature, so that the emotional description embedding feature can be combined with multi-modal emotional information, which helps to improve the accuracy of emotional feature extraction.

[0121] Specifically, the emotional description embedding feature is denoted as f prop .

[0122] The emotional description embedding feature is projected by the visual projection matrix to obtain the visual prompt projection embedding feature, and the specific calculation process is shown in formula (4):

[0123]

[0124] wherein v is the visual prompt projection embedding feature.

[0125] The emotional description embedding feature is projected by the audio projection matrix to obtain the audio prompt projection embedding feature, and the specific calculation process is shown in formula (5):

[0126]

[0127] wherein, is the audio prompt projection embedding feature.

[0128] The emotion description embedding feature is projected by a text projection matrix to obtain a text prompt projection embedding feature. The specific calculation process is shown in formula (6):

[0129]

[0130] wherein, is the text prompt projection embedding feature.

[0131] Referring to Figure 3 In some embodiments, step S105 can include but is not limited to steps S301 to S304:

[0132] Step S301, extracting emotions from the visual prompt projection embedding feature to obtain a visual explicit emotion embedding feature;

[0133] Step S302, extracting emotions from the audio prompt projection embedding feature to obtain an audio explicit emotion embedding feature;

[0134] Step S303, extracting emotions from the text prompt projection embedding feature to obtain a text explicit emotion embedding feature;

[0135] Step S304, performing feature screening on the visual explicit emotion embedding feature, the audio explicit emotion embedding feature and the text explicit emotion embedding feature based on the multi-modal implicit emotion embedding feature to obtain an emotion reference embedding feature.

[0136] The steps S301 to S304 shown in the embodiments of the present application extract emotions from the visual prompt projection embedding feature, the audio prompt projection embedding feature and the text prompt projection embedding feature respectively to obtain the visual explicit emotion embedding feature, the audio explicit emotion embedding feature and the text explicit emotion embedding feature. Then, the visual explicit emotion embedding feature, the audio explicit emotion embedding feature and the text explicit emotion embedding feature are screened based on the multi-modal implicit emotion embedding feature to obtain the emotion reference embedding feature, which realizes the alignment between the multi-modal features and the prompt features, can obtain more accurate and comprehensive emotion reference embedding features, provides more accurate emotion guidance for subsequent speech synthesis, and is beneficial to improve the naturalness of speech synthesis.

[0137] It can be understood that since the emotion description data (such as emotion label or emotion score) belongs to explicit emotion feature, its emotion information is determined. Therefore, the explicit emotion embedding feature with determined emotion is extracted from the multi-modal prompt projection embedding feature.

[0138] In step S301 of some embodiments, embedding features is performed by projecting the visual cue Perform feature transformation to achieve emotion extraction and obtain visual explicit emotion embedding features, where the visual explicit emotion embedding features can be recorded as

[0139] In step S302 of some embodiments, the embedded feature is projected onto the audio prompt. Perform feature transformation to achieve emotion extraction and obtain audio explicit emotion embedding features, where the audio explicit emotion embedding features can be recorded as

[0140] In step S303 of some embodiments, the text prompt is projected with an embedded feature. Perform feature transformation to achieve sentiment extraction and obtain text explicit sentiment embedding features, where the text explicit sentiment embedding features can be recorded as

[0141] See also Figure 4 In some embodiments, step S304 may include but is not limited to steps S401 to S405:

[0142] Step S401, performing similarity calculation on the multimodal implicit emotion embedding feature and the visual explicit emotion embedding feature to obtain first emotion similarity data;

[0143] Step S402: performing similarity calculation on the multimodal implicit emotion embedding feature and the audio explicit emotion embedding feature to obtain second emotion similarity data;

[0144] Step S403, performing similarity calculation on the multimodal implicit emotion embedding feature and the text explicit emotion embedding feature to obtain third emotion similarity data;

[0145] Step S404, extracting the emotion similarity data with the largest value from the first emotion similarity data, the second emotion similarity data, and the third emotion similarity data to obtain target emotion similarity data;

[0146] Step S405 , based on the target emotion similarity data, the visual explicit emotion embedding features, the audio explicit emotion embedding features, and the text explicit emotion embedding features are screened to obtain the target explicit emotion embedding features, and the target explicit emotion embedding features are used as the emotion reference embedding features.

[0147] The steps S401 to S405 shown in the embodiments of the present application obtain first, second and third emotion similarity data by performing similarity calculation on the multi-modal implicit emotion embedding feature and the visual explicit emotion embedding feature, the audio explicit emotion embedding feature and the text explicit emotion embedding feature respectively; then, the maximum value emotion similarity data is extracted from the first, second and third emotion similarity data to obtain target emotion similarity data; the visual explicit emotion embedding feature, the audio explicit emotion embedding feature and the text explicit emotion embedding feature are filtered based on the target emotion similarity data to obtain target explicit emotion embedding features, and the target explicit emotion embedding features are used as emotion reference embedding features, which can ensure that the selected explicit emotion embedding features are effective and representative, provide more accurate emotion guidance for subsequent speech synthesis, and help to improve the naturalness of speech synthesis.

[0148] In some embodiments, the similarity calculation can adopt cosine similarity calculation method, Pearson similarity calculation method, Euclidean distance calculation method, Manhattan distance calculation method, etc., and the specific selection needs to be combined with the actual application scene, which is not limited thereto.

[0149] In step S401 of some embodiments, the first emotion similarity data is obtained by performing cosine similarity calculation on the multi-modal implicit emotion embedding feature and the visual explicit emotion embedding feature, and the specific calculation process is shown in formula (7):

[0150]

[0151] Wherein, logits1 is the first emotion similarity data, t is a preset temperature parameter, u imp is the multi-modal implicit emotion embedding feature, and σ is a normalization operation.

[0152] In step S402 of some embodiments, the second emotion similarity data is obtained by performing cosine similarity calculation on the multi-modal implicit emotion embedding feature and the audio explicit emotion embedding feature, and the specific calculation process is shown in formula (8):

[0153]

[0154] Wherein, logits2 is the second emotion similarity data, t is a preset temperature parameter, u imp is the multi-modal implicit emotion embedding feature, and σ is a normalization operation.

[0155] In step S403 of some embodiments, the third emotion similarity data is obtained by performing cosine similarity calculation on the multi-modal implicit emotion embedding feature and the text explicit emotion embedding feature, and the specific calculation process is shown in formula (9):

[0156]

[0157] where logits3 is the third sentiment similarity data, t is a preset temperature parameter, u imp is a multi-modal implicit sentiment embedding feature, and σ is a normalization operation.

[0158] It should be noted that the preset temperature parameter t is a hyperparameter used to scale the value of the cosine similarity to avoid the situation that the feature gap is amplified.

[0159] In some embodiments, the maximum value of the sentiment similarity data is extracted from the first sentiment similarity data logits1, the second sentiment similarity data logits2, and the third sentiment similarity data logits3 to obtain target sentiment similarity data, and the corresponding explicit sentiment feature is filtered from the visual explicit sentiment embedding feature, the audio explicit sentiment embedding feature, and the text explicit sentiment embedding feature according to the target sentiment similarity data to obtain the target explicit sentiment embedding feature, and the target explicit sentiment embedding feature is used as the sentiment reference embedding feature.

[0160] In step S106 of some embodiments, a regular expression or an NLP library can be used to implement text preprocessing on the speech synthesis text, specifically including but not limited to removing punctuation, special symbols, stop words, word segmentation, etc.

[0161] Further, the preprocessed speech synthesis text is encoded by a pre-trained text encoder to obtain speech synthesis text embedding features;

[0162] It should be noted that the speech synthesis text is the content that needs to be synthesized, which is a script, a dialogue, etc. text content set in combination with actual application scenarios, and can be used in intelligent customer service, intelligent assistants, intelligent outbound calls, intelligent dubbing, etc. to synthesize natural speech.

[0163] In step S107 of some embodiments, the sentiment reference embedding feature and the speech synthesis text embedding feature are synthesized by a preset speech synthesis component.

[0164] Specifically, the speech synthesis component includes a plurality of pre-trained speech synthesis models, including but not limited to Tacotron2 model, VITS model, and FastSpeech2 model, which can be used for sentiment speech synthesis in different scenarios and requirements, and can adapt to various applications, whether it is entertainment content that requires high emotional expression or customized voice services that require precise control of speaker characteristics.

[0165] In one embodiment, the VITS model is selected as a speech synthesis model, and speech synthesis is performed on the emotion reference embedding features and the speech synthesis text embedding features.

[0166] See also Figure 5 In some embodiments, step S107 may also include but is not limited to steps S501 to S504:

[0167] Step S501, performing phoneme conversion on the speech synthesis text embedding features to obtain an original phoneme sequence;

[0168] Step S502, emotion adjustment is performed on the original phoneme sequence based on the emotion reference embedding feature to obtain an initial phoneme sequence;

[0169] Step S503, normalizing the initial phoneme sequence to obtain a target phoneme sequence;

[0170] Step S504: Perform speech synthesis based on the target phoneme sequence.

[0171] In steps S501 to S504 shown in the embodiment of the present application, an original phoneme sequence is obtained by performing phoneme conversion on the speech synthesis text embedding features, and the original phoneme sequence is emotionally adjusted based on the emotional reference embedding features to obtain an initial phoneme sequence, and the reference emotional information is integrated into the phoneme sequence to make the synthesized speech closer to natural expression; the initial phoneme sequence is then normalized to obtain a target phoneme sequence, which can eliminate the differences between phonemes and improve the stability and consistency of speech synthesis; finally, speech synthesis is performed based on the target phoneme sequence to improve the naturalness and emotional expression ability of speech synthesis.

[0172] In step S501 of some embodiments, each word in the speech synthesis text embedding feature can be converted into a corresponding phoneme representation through a Seq2Seq model, a convolutional neural network (CNN) model, or other models set in the VITS model, thereby providing a phoneme-level representation for subsequent emotion adjustment and speech synthesis.

[0173] In step S502 of some embodiments, the duration of the original phoneme sequence is predicted by the length adjuster of the VITS model in combination with the emotional reference embedding feature, thereby achieving emotional adjustment and obtaining an initial phoneme sequence with emotional information; by adjusting the duration of each phoneme, the phoneme can be given emotional characteristics, which helps to make the synthesized speech closer to natural expression.

[0174] In step S503 of some embodiments, the initial phoneme sequence can be normalized by a reversible neural network, an autoregressive flow, etc., so as to eliminate the difference between phonemes and improve the stability and consistency of speech synthesis. Specifically, the normalization processing can include phoneme duration normalization, phoneme volume normalization, etc., without being limited thereto.

[0175] In step S504 of some embodiments, the target phoneme sequence can be synthesized by the vocoder of the VITS model, so as to obtain more natural synthesized speech.

[0176] In one embodiment, the Tacotron2 model is selected as the speech synthesis model to synthesize the emotional reference embedding feature and the speech synthesis text embedding feature.

[0177] Please refer to Figure 6 In some embodiments, step S107 further includes but is not limited to steps S601 to S605:

[0178] In step S601, the emotional reference embedding feature and the speech synthesis text embedding feature are combined to obtain original text embedding features;

[0179] In step S602, attention calculation is performed on the original text embedding features to obtain initial text embedding features;

[0180] In step S603, context learning is performed on the initial text embedding features to obtain target text embedding features;

[0181] In step S604, the target text embedding features are decoded to obtain target speech spectrum data;

[0182] In step S605, speech synthesis is performed based on the target speech spectrum data.

[0183] The steps S601 to S605 shown in the embodiments of the present application combine the emotional reference embedding feature and the speech synthesis text embedding feature, and then perform attention calculation, context learning, and decoding to obtain target speech spectrum data, so as to realize the fusion of the reference emotional information into the spectrum data, so that the synthesized speech is closer to natural expression. Finally, speech synthesis is performed based on the target speech spectrum data, which improves the naturalness and expressiveness of the speech synthesis.

[0184] In step S601 of some embodiments, the emotional reference embedding feature and the speech synthesis text embedding feature are combined to obtain original text embedding features with emotional information,

[0185] Further, the original text embedding features are calculated by attention mechanism to obtain initial text embedding features. The attention mechanism can enable the model to focus on important words or phrases when processing long text, thereby generating more accurate speech. Through attention calculation, the initial text embedding features are obtained, which focus more on the key information in the text and integrate emotional information.

[0186] In step S603 of some embodiments, the initial text embedding features are learned by a convolutional neural network (CNN) or a long short-term memory network (LSTM) set in the Tacotron2 model to obtain target text embedding features. Context learning can capture the context relationship in the text, making the generated speech more coherent and fluent.

[0187] In step S604 of some embodiments, the target text embedding features are decoded by the decoder in the Tacotron2 model to realize the conversion of the target text embedding features to the spectrum data, and obtain the target speech spectrum data. The spectrum data can be mel cepstrum data, mel spectrum data, etc., but is not limited thereto.

[0188] In step S605 of some embodiments, the WaveNet sub-model in the Tacotron2 model is used to synthesize speech from the target speech spectrum data to obtain real and natural speech.

[0189] In one embodiment, the FastSpeech2 model is selected as the speech synthesis model to synthesize speech from the emotional reference embedding features and the speech synthesis text embedding features.

[0190] Please refer to Figure 7 In some embodiments, step S107 can further include, but is not limited to, steps S701 to S706:

[0191] Step S701, converting the speech synthesis text embedding features into phonemes to obtain an original speech phoneme sequence;

[0192] Step S702, splicing the original speech phoneme sequence with the emotional reference embedding features to obtain an initial speech phoneme sequence;

[0193] Step S703, calculating the attention of the initial speech phoneme sequence to obtain attention embedding features;

[0194] Step S704, adjusting the duration of the attention embedding features to obtain speech synthesis embedding features;

[0195] Step S705, performing spectrum conversion on the speech synthesis embedding features to obtain synthesized speech spectrum data;

[0196] Step S706: Perform voice conversion on the synthesized speech spectrum data.

[0197] In steps S701 to S706 shown in the embodiment of the present application, an original speech phoneme sequence is obtained by performing phoneme conversion on the speech synthesis text embedding features, and the original speech phoneme sequence is feature-concatenated with the emotion reference embedding features to obtain an initial speech phoneme sequence; attention calculation, duration adjustment, and spectrum conversion are then performed on the initial speech phoneme sequence to obtain synthesized speech spectrum data, and finally, speech conversion is performed on the synthesized speech spectrum data, thereby improving the naturalness and richness of the synthesized speech in terms of emotional expression.

[0198] In step S701 of some embodiments, each word in the speech synthesis text embedding feature is converted into a corresponding phoneme representation through a Seq2Seq model, a convolutional neural network (CNN) model, or other models set in the FastSpeech2 model, thereby providing a phoneme-level representation for subsequent emotion adjustment and speech synthesis.

[0199] In step S702 of some embodiments, the original speech phoneme sequence is fused with the emotion reference embedded feature by feature concatenation to obtain an initial speech phoneme sequence.

[0200] In step S703 of some embodiments, attention calculation is performed on the initial speech phoneme sequence through the cross-attention mechanism in the FastSpeech2 model, which enhances the ability to capture key information in the speech content and helps to better adjust the duration and rhythm of the speech in subsequent steps to meet the needs of emotional expression.

[0201] In step S704 of some embodiments, the duration of the attention embedding feature is adjusted by the speech duration adjuster in the FastSpeech2 model so that the rhythm of each phoneme is more consistent with the requirements of emotional expression.

[0202] In addition, the energy regulator and pitch regulator in the FsatSpeech2 model can be used to adjust the energy and pitch of the attention embedding features, so that each phoneme can better meet the requirements of emotional expression in terms of energy and pitch.

[0203] In step S705 of some embodiments, the spectrum decoder in the FastSpeech2 model performs spectrum conversion on the speech synthesis embedded features to obtain synthesized speech spectrum data. The spectrum data may be mel-cepstrum data, mel-spectrogram data, etc., but is not limited thereto.

[0204] In step S706 of some embodiments, speech synthesis is performed on the synthesized speech spectrum data using the WaveNet sub-model in the FastSpeech2 model to obtain real and natural speech.

[0205] In some embodiments, the speech synthesis method provided in the embodiments of the present application needs to select an appropriate speech synthesis model from the speech synthesis component in combination with the actual application scenario. For example:

[0206] The Tacotron2 model is used for intelligent customer service scenarios in the financial sector, as intelligent customer service requires high-quality speech synthesis for natural user interactions. The Tacotron2 model produces natural, high-quality speech and supports multiple languages, making it well-suited for intelligent customer service scenarios. While the Tacotron2 model has drawbacks such as time-consuming training and a lack of fine-grained control, these drawbacks are acceptable in scenarios like intelligent customer service, where high-quality speech output is crucial.

[0207] The VITS model is used for intelligent assistants in the financial sector because these assistants typically handle a variety of tasks, including speech synthesis, speech recognition, and natural language processing. While the training process for the VITS model is complex, once trained, it generates high-quality speech, which is crucial for intelligent assistants. Furthermore, the VITS model uses a pre-trained speech encoder, which provides flexibility in handling diverse speech synthesis tasks.

[0208] The FastSpeech2 model is used for intelligent outbound call scenarios in the financial sector because these systems require fast and accurate speech synthesis and robustness to handle diverse speech synthesis tasks. By individually modeling key speech attributes such as duration, fundamental frequency, and energy, the FastSpeech2 model generates high-quality speech with high robustness. Furthermore, the FastSpeech2 model can directly utilize duration information provided by external alignment tools, enabling it to perform better when processing long or complex texts.

[0209] See also Figure 8 The present invention also provides a speech synthesis device that can implement the above-mentioned speech synthesis method. The device includes:

[0210] The multimodal data acquisition module 801 is used to acquire multimodal reference data; wherein the multimodal reference data includes visual reference data, audio reference data, text reference data and emotional description data;

[0211] The feature extraction module 802 is configured to perform feature extraction on the multi-modal reference data to obtain visual reference embedding features of the visual reference data, audio reference embedding features of the audio reference data, text reference embedding features of the text reference data, and sentiment description embedding features of the sentiment description data.

[0212] The multi-modal feature projection module 803 is configured to perform feature projection on the visual reference embedding features, the audio reference embedding features, and the text reference embedding features based on a preset multi-modal projection matrix to obtain multi-modal implicit sentiment embedding features.

[0213] The prompt feature projection module 804 is configured to perform feature projection on the sentiment description embedding features based on the multi-modal projection matrix to obtain sentiment prompt projection embedding features.

[0214] The reference feature calculation module 805 is configured to perform feature calculation based on the multi-modal implicit sentiment embedding features and the sentiment prompt projection embedding features to obtain sentiment reference embedding features.

[0215] The text preprocessing module 806 is configured to perform text preprocessing on the pre-acquired speech synthesis text to obtain speech synthesis text embedding features.

[0216] The speech synthesis module 807 is configured to perform speech synthesis based on the sentiment reference embedding features and the speech synthesis text embedding features.

[0217] The specific implementation of the speech synthesis device is basically the same as that of the above-described specific embodiments of the speech synthesis method, and thus will not be described herein again.

[0218] The embodiments of the present application further provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above-described speech synthesis method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0219] Please refer to Figure 9 , Figure 9 The hardware structure of the electronic device of another embodiment is illustrated, which includes:

[0220] The processor 901 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0221] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 902 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 902 and are invoked and executed by the processor 901 to perform the voice synthesis method of the embodiments of the present application.

[0222] The input / output interface 903 is configured to realize information input and output.

[0223] The communication interface 904 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0224] The bus 905 is configured to transmit information between various components (for example, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) of the device.

[0225] The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are connected to each other through the bus 905 to realize the communication connection between the device.

[0226] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the voice synthesis method.

[0227] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory that is remotely arranged relative to the processor. These remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0228] The voice synthesis method and device, the electronic device and the storage medium provided by the embodiments of the present application obtain multi-modal reference data, perform feature extraction on the multi-modal reference data, obtain visual reference embedding features of visual reference data, audio reference embedding features of audio reference data, text reference embedding features of text reference data, and emotion description embedding features of emotion description data, perform feature projection on the visual reference embedding features, the audio reference embedding features and the text reference embedding features based on a multi-modal projection matrix, obtain multi-modal implicit emotion embedding features, perform feature projection on the emotion description embedding features based on the multi-modal projection matrix, obtain emotion prompt projection embedding features, perform feature calculation based on the multi-modal implicit emotion embedding features and the emotion prompt projection embedding features, obtain emotion reference embedding features, realize fusion of multi-modal information, and finally perform voice synthesis based on the emotion reference embedding features and voice synthesis text, so that the richness and naturalness of emotion expression of voice synthesis are improved, and the synthesized voice is closer to real emotion expression.

[0229] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0230] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than those shown in the figures, or combine certain steps or different steps.

[0231] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0232] Those skilled in the art can understand that all or some steps in the above disclosed method, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0233] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0234] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0235] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0236] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0237] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0238] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various other media that can store programs.

[0239] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not intended to limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A speech synthesis method characterized by, The method comprises: acquiring multi-modal reference data; wherein the multi-modal reference data comprises visual reference data, audio reference data, text reference data and emotional description data; performing feature extraction on the multi-modal reference data to obtain visual reference embedding features of the visual reference data, audio reference embedding features of the audio reference data, text reference embedding features of the text reference data, and emotional description embedding features of the emotional description data; performing feature projection on the visual reference embedding features, the audio reference embedding features and the text reference embedding features based on a preset multi-modal projection matrix to obtain multi-modal implicit emotional embedding features; performing feature projection on the emotional description embedding features based on the multi-modal projection matrix to obtain emotional prompt projection embedding features; performing feature calculation based on the multi-modal implicit emotional embedding features and the emotional prompt projection embedding features to obtain emotional reference embedding features; performing text preprocessing on the pre-acquired speech synthesis text to obtain speech synthesis text embedding features; performing speech synthesis based on the emotional reference embedding features and the speech synthesis text embedding features; wherein the multi-modal projection matrix comprises a visual projection matrix, an audio projection matrix and a text projection matrix; the performing feature projection on the visual reference embedding features, the audio reference embedding features and the text reference embedding features based on the preset multi-modal projection matrix to obtain multi-modal implicit emotional embedding features comprises: performing feature projection on the visual reference embedding features based on the visual projection matrix to obtain visual projection embedding features; performing feature projection on the audio reference embedding features based on the audio projection matrix to obtain audio projection embedding features; performing feature projection on the text reference embedding features based on the text projection matrix to obtain text projection embedding features; performing emotional extraction on the visual projection embedding features, the audio projection embedding features and the text projection embedding features to obtain the multi-modal implicit emotional embedding features; the emotional prompt projection embedding features comprise visual prompt projection embedding features, audio prompt projection embedding features and text prompt projection embedding features, the visual prompt projection embedding features being obtained by performing feature projection on the emotional description embedding features based on the visual projection matrix, the audio prompt projection embedding features being obtained by performing feature projection on the emotional description embedding features based on the audio projection matrix, and the text prompt projection embedding features being obtained by performing feature projection on the emotional description embedding features based on the text projection matrix; the performing feature calculation based on the multi-modal implicit emotional embedding features and the emotional prompt projection embedding features to obtain emotional reference embedding features comprises: performing emotional extraction on the visual prompt projection embedding features to obtain visual explicit emotional embedding features; performing emotional extraction on the audio prompt projection embedding features to obtain audio explicit emotional embedding features; performing emotional extraction on the text prompt projection embedding features to obtain text explicit emotional embedding features; The visual explicit sentiment embedding feature, the audio explicit sentiment embedding feature and the text explicit sentiment embedding feature are filtered based on the multi-modal implicit sentiment embedding feature, to obtain the sentiment reference embedding feature.

2. The method of claim 1, wherein, The filtering of the visual explicit sentiment embedding feature, the audio explicit sentiment embedding feature and the text explicit sentiment embedding feature based on the multi-modal implicit sentiment embedding feature to obtain the sentiment reference embedding feature comprises: Similarity calculation is performed on the multi-modal implicit sentiment embedding feature and the visual explicit sentiment embedding feature to obtain first sentiment similarity data; Similarity calculation is performed on the multi-modal implicit sentiment embedding feature and the audio explicit sentiment embedding feature to obtain second sentiment similarity data; Similarity calculation is performed on the multi-modal implicit sentiment embedding feature and the text explicit sentiment embedding feature to obtain third sentiment similarity data; The maximum value of the first sentiment similarity data, the second sentiment similarity data and the third sentiment similarity data is extracted to obtain target sentiment similarity data; The visual explicit sentiment embedding feature, the audio explicit sentiment embedding feature and the text explicit sentiment embedding feature are filtered based on the target sentiment similarity data to obtain target explicit sentiment embedding features, and the target explicit sentiment embedding features are taken as the sentiment reference embedding feature.

3. The method according to any one of claims 1 to 2, characterized in that, The voice synthesis based on the sentiment reference embedding feature and the voice synthesis text embedding feature comprises: Phoneme conversion is performed on the voice synthesis text embedding feature to obtain an original phoneme sequence; Sentiment adjustment is performed on the original phoneme sequence based on the sentiment reference embedding feature to obtain an initial phoneme sequence; Normalization processing is performed on the initial phoneme sequence to obtain a target phoneme sequence; Voice synthesis is performed based on the target phoneme sequence.

4. The method according to any one of claims 1 to 2, characterized in that, The voice synthesis based on the sentiment reference embedding feature and the voice synthesis text embedding feature comprises: Feature combination is performed on the sentiment reference embedding feature and the voice synthesis text embedding feature to obtain an original text embedding feature; Attention calculation is performed on the original text embedding feature to obtain an initial text embedding feature; Context learning is performed on the initial text embedding feature to obtain a target text embedding feature; Decoding is performed on the target text embedding feature to obtain target voice spectrum data; Voice synthesis is performed based on the target voice spectrum data.

5. The method according to any one of claims 1 to 2, characterized in that, The voice synthesis based on the sentiment reference embedding feature and the voice synthesis text embedding feature comprises: Phoneme conversion is performed on the voice synthesis text embedding feature to obtain an original voice phoneme sequence; Feature splicing is performed on the original voice phoneme sequence and the sentiment reference embedding feature to obtain an initial voice phoneme sequence; Attention calculation is performed on the initial voice phoneme sequence to obtain an attention embedding feature; Duration adjustment is performed on the attention embedding feature to obtain a voice synthesis embedding feature; Spectrum conversion is performed on the voice synthesis embedding feature to obtain synthesis voice spectrum data; Voice conversion is performed on the synthesis voice spectrum data.

6. A speech synthesis apparatus characterized by comprising: The device comprises: a multi-modal data acquisition module for acquiring multi-modal reference data; wherein the multi-modal reference data comprises visual reference data, audio reference data, text reference data and emotional description data; a feature extraction module for performing feature extraction on the multi-modal reference data to obtain visual reference embedding features of the visual reference data, audio reference embedding features of the audio reference data, text reference embedding features of the text reference data, and emotional description embedding features of the emotional description data; a multi-modal feature projection module for performing feature projection on the visual reference embedding features, the audio reference embedding features and the text reference embedding features based on a preset multi-modal projection matrix to obtain multi-modal implicit emotional embedding features; a prompt feature projection module for performing feature projection on the emotional description embedding features based on the multi-modal projection matrix to obtain emotional prompt projection embedding features; a reference feature calculation module for performing feature calculation based on the multi-modal implicit emotional embedding features and the emotional prompt projection embedding features to obtain emotional reference embedding features; a text preprocessing module for performing text preprocessing on pre-acquired speech synthesis text to obtain speech synthesis text embedding features; a speech synthesis module for performing speech synthesis based on the emotional reference embedding features and the speech synthesis text embedding features; wherein the multi-modal projection matrix comprises a visual projection matrix, an audio projection matrix and a text projection matrix; the feature projection on the visual reference embedding features, the audio reference embedding features and the text reference embedding features based on the preset multi-modal projection matrix to obtain multi-modal implicit emotional embedding features comprises: performing feature projection on the visual reference embedding features based on the visual projection matrix to obtain visual projection embedding features; performing feature projection on the audio reference embedding features based on the audio projection matrix to obtain audio projection embedding features; performing feature projection on the text reference embedding features based on the text projection matrix to obtain text projection embedding features; performing emotional extraction on the visual projection embedding features, the audio projection embedding features and the text projection embedding features to obtain the multi-modal implicit emotional embedding features; the emotional prompt projection embedding features comprise visual prompt projection embedding features, audio prompt projection embedding features and text prompt projection embedding features, the visual prompt projection embedding features being obtained by performing feature projection on the emotional description embedding features based on the visual projection matrix; the audio prompt projection embedding features being obtained by performing feature projection on the emotional description embedding features based on the audio projection matrix; the text prompt projection embedding features being obtained by performing feature projection on the emotional description embedding features based on the text projection matrix; the feature calculation based on the multi-modal implicit emotional embedding features and the emotional prompt projection embedding features to obtain emotional reference embedding features comprises: performing emotional extraction on the visual prompt projection embedding features to obtain visual explicit emotional embedding features; extracting emotions from the audio prompt projection embedding features to obtain audio explicit emotion embedding features; extracting emotions from the text prompt projection embedding features to obtain text explicit emotion embedding features; performing feature screening on the visual explicit emotion embedding features, the audio explicit emotion embedding features and the text explicit emotion embedding features based on the multi-modal implicit emotion embedding features to obtain the emotion reference embedding features.

7. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to implement the method in any one of claims 1 to 5.

Images