Multimodal-based speech synthesis method and device, equipment and storage medium

Abstract

The application relates to the technical field of artificial intelligence, and discloses a speech synthesis method and device based on multiple modes, equipment and a storage medium, which comprises the following steps: preprocessing a text to be synthesized to obtain character sequence information, character-level graph sequence information and word-level graph sequence information as input sequence information; encoding the character sequence information to obtain a time domain coding vector; encoding the character-level graph sequence information and the word-level graph sequence information to obtain a first space domain coding vector and a second space domain coding vector; performing first cross-modal attention calculation on the time domain coding vector and the first space domain coding vector to obtain a first decoding vector; performing second cross-modal attention calculation on the first decoding vector and the second space domain coding vector to obtain a second decoding vector; and obtaining a speech spectrum graph according to the second decoding vector to generate synthesized speech. The application guarantees the prosody and accuracy of the synthesized speech and effectively improves the level of financial services.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multimodal speech synthesis method, apparatus, device, and storage medium. Background Technology

[0002] Speech synthesis is a text-to-speech (TTS) technology that includes multiple steps such as text analysis, acoustic modeling, and voice synthesis modules. In the financial sector, financial institutions widely adopt speech synthesis technology in business scenarios such as greetings, business consultations, promotional announcements, and Q&A sessions. To simplify the speech synthesis process, reduce human intervention, and lower the requirements for linguistic background knowledge, end-to-end speech synthesis systems achieve speech synthesis by directly inputting text or phonetic characters at the input end and outputting audio waveforms at the output end. However, existing end-to-end speech synthesis methods neglect the importance of visual information, utilizing only the features of single-modal text information, and thus cannot accurately and comprehensively achieve speech synthesis.

[0003] Prosody represents information such as rhythm, emphasis, and intonation in speech. Prosodic information determines the naturalness and fluency of synthesized speech and plays a crucial role in speech synthesis. Existing end-to-end speech synthesis methods use standard databases with one-to-one text-to-speech correspondences to train speech synthesis models. However, due to the limited capacity of these databases, the speech synthesis models cannot learn the prosodic rules of semantic connections, resulting in synthesized speech lacking rhythm and impacting the service level of financial services. For example, a harsh tone in customer service voice messages can reduce customer satisfaction. Summary of the Invention

[0004] Therefore, it is necessary to provide a multimodal speech synthesis method, apparatus, device, and storage medium to address the aforementioned technical problems, thereby solving the issues of single modal features and poor prosody in speech synthesis.

[0005] A multimodal speech synthesis method includes:

[0006] The text to be synthesized is preprocessed to obtain input sequence information; the input sequence information includes character sequence information, character-level graph sequence information, and word-level graph sequence information.

[0007] The character sequence information is encoded to obtain a time-domain encoding vector; the character-level graph sequence information is encoded to obtain a first spatial domain encoding vector; and the word-level graph sequence information is encoded to obtain a second spatial domain encoding vector.

[0008] A first cross-modal attention calculation is performed on the temporal domain encoding vector and the first spatial domain encoding vector to obtain a first decoding vector;

[0009] A second cross-modal attention calculation is performed on the first decoding vector and the second spatial domain encoding vector to obtain the second decoding vector;

[0010] The speech spectrogram is obtained based on the second decoding vector to generate the synthesized speech of the text to be synthesized.

[0011] A multimodal speech synthesis device, comprising:

[0012] The preprocessing module is used to preprocess the text to be synthesized to obtain input sequence information; the input sequence information includes character sequence information, character-level graph sequence information, and word-level graph sequence information.

[0013] The encoding processing module is used to encode the character sequence information to obtain a time-domain encoding vector; to encode the character-level graph sequence information to obtain a first spatial domain encoding vector; and to encode the word-level graph sequence information to obtain a second spatial domain encoding vector.

[0014] The first attention calculation module is used to perform a first cross-modal attention calculation on the temporal domain encoding vector and the first spatial domain encoding vector to obtain a first decoding vector;

[0015] The second attention calculation module is used to perform a second cross-modal attention calculation on the first decoding vector and the second spatial domain encoding vector to obtain the second decoding vector.

[0016] The synthesized speech generation module is used to obtain a speech spectrogram based on the second decoding vector in order to generate synthesized speech of the text to be synthesized.

[0017] A computer device includes a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor implements the above-described multimodal speech synthesis method when executing the computer-readable instructions.

[0018] One or more readable storage media storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the multimodal speech synthesis method described above.

[0019] The aforementioned multimodal speech synthesis method, apparatus, device, and storage medium preprocess the text to be synthesized to obtain input sequence information. This input sequence information includes character sequence information, character-level graph sequence information, and word-level graph sequence information. The character sequence information is encoded to obtain a temporal coding vector. The character-level graph sequence information is encoded to obtain a first spatial coding vector. The word-level graph sequence information is encoded to obtain a second spatial coding vector. A first cross-modal attention calculation is performed on the temporal coding vector and the first spatial coding vector to obtain a first decoding vector. A second cross-modal attention calculation is performed on the first decoding vector and the second spatial coding vector to obtain a second decoding vector. A speech spectrogram is obtained based on the second decoding vector to generate synthesized speech from the text to be synthesized. This invention utilizes character-level graph embedding encoding to calculate the hidden states of character-level graph embeddings and word-level graph embedding encoding to calculate the hidden states of word-level graph embeddings. This enables the extraction of semantic information from text at multiple levels, improving the prosody of synthesized speech. Simultaneously, it employs a cross-modal attention mechanism for multimodal feature fusion. By calculating attention weights, the temporal modality encoding vector receives information from two different spatial modality encoding vectors and performs feature selection, ensuring the prosody and accuracy of synthesized speech and effectively improving the level of financial services. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a multimodal speech synthesis method according to an embodiment of the present invention.

[0022] Figure 2 This is a flowchart illustrating a multimodal speech synthesis method according to an embodiment of the present invention.

[0023] Figure 3 This is a schematic diagram of a multimodal speech synthesis device according to an embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] This application's embodiments can acquire and process voice data based on artificial intelligence (AI) technology. AI is the theory, methods, technology, and application system that uses digital computers or computers-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. Basic AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, robotics, biometrics, voice processing, natural language processing, and machine learning / deep learning.

[0027] Speech synthesis technology, also known as text-to-speech (TTS), is a technology that converts text information, whether generated by a computer or input from an external source, into fluent, understandable speech output. A TTS system is essentially an artificial intelligence system. To synthesize high-quality language, it relies not only on various rules, including semantic, lexical, and phonetic rules, but also on a strong understanding of the semantic content of the text. This embodiment applies a multimodal speech synthesis method to financial service scenarios to improve the level of financial services. Specifically, banks integrate intelligent voice interaction robots into their self-service systems, guiding customers through card opening, fund transfers, and remittances using high-quality synthesized speech; insurance companies employ intelligent human customer service in claims processing, helping customers understand the claims process and resolve claims issues through high-quality synthesized speech.

[0028] In one embodiment, such as Figure 1 As shown, a multimodal speech synthesis method is provided, including the following steps S10-S50.

[0029] S10. Preprocess the text to be synthesized to obtain input sequence information; the input sequence information includes character sequence information, character-level graph sequence information and word-level graph sequence information.

[0030] Understandably, preprocessing for speech synthesis includes language processing, which plays a crucial role in text-to-speech conversion. Language processing involves analyzing the text to be synthesized, performing text normalization, word segmentation, syntactic analysis, and semantic analysis by simulating human understanding of natural language, enabling the computer to fully understand the text and generate the input sequence information. Preprocessing for speech synthesis also includes prosodic processing, which plans segmental features for the synthesized speech, such as pitch, duration, and intensity, ensuring the synthesized speech accurately expresses meaning and sounds more natural. Graph-to-Sequence (GST) embeds the input text sequence into a graph structure to represent text content, grammatical relationships, and semantic connections between texts, preserving prosodic information. In one embodiment, character embedding is performed on the input text to be synthesized to obtain character sequence information; character-level graph embedding and word-level graph embedding are performed on the text to be synthesized to obtain character-level graph sequence information and word-level graph sequence information, respectively.

[0031] S20. Encode the character sequence information to obtain a time-domain encoding vector; encode the character-level graph sequence information to obtain a first spatial domain encoding vector; encode the word-level graph sequence information to obtain a second spatial domain encoding vector.

[0032] Understandably, the temporal domain encoding vector is an encoding vector obtained from character sequence information, used to represent the temporal step order between characters in the text to be synthesized; the spatial domain encoding vector is an encoding vector obtained from graph sequence information, used to represent the semantic relationships between characters in the text to be synthesized. End-to-end speech synthesis is a sequence-to-sequence (Seq2Seq) model, including an encoder and a decoder. The encoder is a recurrent neural network for text understanding, used to encode the input sequence information into a hidden state vector; the decoder is a recurrent neural network for text generation, used to decode and translate the hidden state vector output by the encoder at each time step. The encoding vector is specifically a One-Hot encoding vector, also known as a one-bit effective encoding, which uses an N-bit state register to encode N states, each state having an independent register bit. A One-Hot encoding vector is a vector with exactly one element that is 1, and all other elements are 0. The encoder maps each character sequence data of the character sequence information to a discrete One-Hot encoding vector, and then encodes it into a low-dimensional continuous embedding form to obtain a time-domain encoding vector; the encoder maps each graph sequence data of the character-level graph sequence information and the word-level graph sequence information to a discrete One-Hot encoding vector, and then encodes it into a low-dimensional continuous embedding form to obtain a first spatial domain encoding vector and a second spatial domain encoding vector.

[0033] S30. Perform a first cross-modal attention calculation on the temporal coding vector and the first spatial coding vector to obtain a first decoding vector.

[0034] Understandably, in end-to-end speech synthesis, the decoder decodes word by word. If too much information is received during each decoding process, it may lead to internal confusion and decoding errors. For example, the encoder encodes "Today the weather is fine," then passes the encoded vector to the decoder to decode "Today the weather is fine." The attention mechanism in neural networks is a resource allocation scheme that, when computational power is limited, allocates computational resources to more important tasks while solving the problem of information overload. Introducing an attention mechanism into the decoder can avoid content confusion. For example, when decoding "today," it is more related to "day" than to "weather," so the attention mechanism focuses more attention on "today" and less on "weather." Feature vectors represent entity data, which can be an image, a single word, or a sentence. Multimodal feature vectors come from multiple aspects of entity representation information and have properties such as smoothness, temporal and spatial consistency, sparsity, and natural clustering. When constructing multimodal features using neural networks, each modality's data passes through a separate neural network layer, and then through one or more hidden layers to map multiple modalities into a joint space, resulting in cross-modal joint features. In one embodiment, character sequence information is encoded by an encoder to obtain a temporal encoded vector, and character-level graph sequence information is encoded by an encoder to obtain a first spatial encoded vector. A first cross-modal attention calculation is performed on the temporal encoded vector and the first spatial encoded vector, and the features of the character sequence information and the character-level graph sequence information are jointly combined to obtain a first decoded vector.

[0035] S40. Perform a second cross-modal attention calculation on the first decoding vector and the second spatial domain encoding vector to obtain the second decoding vector.

[0036] Understandably, the text to be synthesized contains semantic connections not only between characters but also between words. The first decoding vector is a vector representation of the combined features of character sequence information and character-level graph sequence information, and the second decoding vector is a vector representation of the combined features of the first decoding vector and word-level graph sequence information. In one embodiment, the word-level graph sequence information is encoded by an encoder to obtain a second spatial domain encoding vector. A second cross-modal attention calculation is performed on the first decoding vector and the second spatial domain encoding vector to combine the features of the character sequence information, the character-level graph sequence information, and the word-level graph sequence information to obtain the second decoding vector.

[0037] S50. Obtain the speech spectrogram based on the second decoding vector to generate the synthesized speech of the text to be synthesized.

[0038] Understandably, the speech synthesis process also includes adding a post-processing network after the decoder. The decoder obtains multiple second-decoding vectors after attention calculations at multiple time steps. The post-processing network optimizes these second-decoding vectors to obtain speech mel spectrum features. Based on these features, a speech mel spectrum is generated. This mel spectrum undergoes spectral transformation, and the inverse mel spectrum is transformed into waveform samples, thus yielding the synthesized speech.

[0039] This embodiment obtains input sequence information by preprocessing the text to be synthesized. The input sequence information includes character sequence information, character-level graph sequence information, and word-level graph sequence information. The character sequence information is encoded to obtain a temporal coding vector. The character-level graph sequence information is encoded to obtain a first spatial coding vector. The word-level graph sequence information is encoded to obtain a second spatial coding vector. A first cross-modal attention calculation is performed on the temporal coding vector and the first spatial coding vector to obtain a first decoding vector. A second cross-modal attention calculation is performed on the first decoding vector and the second spatial coding vector to obtain a second decoding vector. A speech spectrogram is obtained based on the second decoding vector to generate synthesized speech of the text to be synthesized. This invention utilizes character-level graph embedding encoding to calculate the hidden states of character-level graph embeddings and word-level graph embedding encoding to calculate the hidden states of word-level graph embeddings. This enables the extraction of semantic information from text at multiple levels, improving the prosodic quality of synthesized speech. Simultaneously, it employs a cross-modal attention mechanism for multimodal feature fusion. By calculating attention weights, the temporal modality encoding vector receives information from two different spatial modality encoding vectors and performs feature selection, thereby improving the accuracy of synthesized speech.

[0040] Optionally, step S10, namely, preprocessing the text to be synthesized to obtain input sequence information, includes:

[0041] S101. Extract the character feature data of the text to be synthesized;

[0042] S102. Phoneme embedding is performed on the character feature data to generate character sequence information.

[0043] Understandably, a character is a glyph-like unit or symbol in the text to be synthesized, including letters, numbers, operators, punctuation marks, and functional symbols. A phoneme is the smallest unit of speech defined based on the natural attributes of speech, analyzed according to the articulation actions within a syllable; one action constitutes one phoneme. Embedding is a data representation pattern that, by adding hidden layers to a neural network, preserves the relationships between data and controls the output dimensionality of the hidden layers, achieving dimensionality reduction for high-cardinality classification data. This embodiment extracts character feature data from the text to be synthesized, performs phoneme embedding on the character feature data, and obtains character sequence information.

[0044] In this embodiment, the character feature data of the text to be synthesized is converted into character sequence information so that the encoder can encode the character sequence information to obtain the latent state vector in the time domain.

[0045] Optionally, step S10, namely, preprocessing the text to be synthesized to obtain input sequence information, further includes:

[0046] S103. Extract the character feature data of the text to be synthesized;

[0047] S104. Perform graph embedding on the character feature data to obtain character-level graph node information and character-level graph boundary information;

[0048] S105. Generate the character-level graph sequence information based on the character-level graph node information and the character-level graph boundary information.

[0049] Understandably, this embodiment extracts character feature data from the text to be synthesized; performs graph embedding on the character feature data: represents the character feature data (each letter, symbol, etc.) with nodes of a graph to obtain character-level graph node information; models the adjacency relationship between character feature data using graph boundaries, that is, adjacent graph nodes are connected by directed edges to obtain character-level graph boundary information; and concatenates the character-level graph node information and the character-level graph boundary information to generate character-level graph sequence information.

[0050] This embodiment transforms the character feature data of the text to be synthesized into a graph structure for embedding. Node embedding is used to represent the text content, enriching the modal features; boundary embedding is used to represent grammatical relations and semantic connections between text characters, preserving prosodic information; at the same time, it facilitates the encoder to encode the character-level graph sequence information to obtain the character-level hidden state vector in the spatial domain.

[0051] Optionally, step S10, namely, preprocessing the text to be synthesized to obtain input sequence information, further includes:

[0052] S106. Extract the word feature data of the text to be synthesized;

[0053] S107. Perform graph embedding on the word feature data to obtain word-level graph node information and word-level graph boundary information;

[0054] S108. Generate the word-level graph sequence information based on the word-level graph node information and the word-level graph boundary information.

[0055] Understandably, a syntactic analysis tree is a tree-like data structure used to determine whether the syntactic structure of a word sequence conforms to a given grammar. This embodiment uses a pre-defined syntactic analysis tree to perform syntactic processing on the text to be synthesized, extracting word feature data; it then performs graph embedding on the word feature data: representing each word feature data (each word) with graph nodes to obtain word-level graph node information; it models the adjacency relationships between word feature data using graph boundaries, i.e., adjacent graph nodes are connected by directed edges to obtain word-level graph boundary information; finally, it concatenates the word-level graph node information and the word-level graph boundary information to generate word-level graph sequence information.

[0056] This embodiment transforms the word feature data of the text to be synthesized into a graph structure for embedding. Node embedding is used to represent the text content, enriching the modal features; boundary embedding is used to represent grammatical relations and semantic connections between text words, preserving prosodic information; at the same time, it facilitates the encoder to encode the word-level graph sequence information to obtain the word-level hidden state vector in the spatial domain.

[0057] Optionally, step S30, namely, performing a first cross-modal attention calculation on the temporal-domain encoding vector and the first spatial-domain encoding vector to obtain a first decoding vector, includes:

[0058] S301. Obtain the first query space vector corresponding to the first spatial domain encoding vector;

[0059] S302. Obtain the first key space vector and the first value space vector corresponding to the time-domain coding vector;

[0060] S303. Calculate the first attention weight based on the first query space vector and the first key space vector;

[0061] S304. The first value space vector is weighted according to the first attention weight to obtain the first decoding vector.

[0062] Understandably, self-attention mechanisms often employ a query-key-value pattern. In a unimodal self-attention mechanism, the query space vector (Query, Q), key space vector (Key, K), and value space vector (Value, V) are obtained by multiplying the same encoded vector by three corresponding trainable parameter matrices. First, Q and K are multiplied to calculate similarity. The resulting similarity weight matrix is ​​normalized and then multiplied by V to calculate a weighted sum, yielding the decoded vector. This embodiment employs a cross-modal attention mechanism. Q, K, and V do not originate from the same encoded vector; Q comes from the spatial domain encoded vector, while K and V come from the temporal domain encoded vector. Modal features are jointly expressed by calculating the similarity between the spatial and temporal domain encoded vectors. The cross-modal attention decoder has two different attention layers. In the first attention layer, the first query space vector Q1 corresponding to the first spatial domain encoding vector is obtained; the first key space vector K1 and the first value space vector V1 corresponding to the temporal domain encoding vector are obtained; the first attention weight a1 is calculated based on the first query space vector Q1 and the first key space vector K1; and the first value space vector V1 is weighted based on the first attention weight a1 to obtain the first decoding vector context1.

[0063] This embodiment performs cross-modal attention computation between the first spatial domain encoding vector and the temporal domain encoding vector, which enriches the modal features while preserving the prosodic information features between the character sequence and the character-level graph sequence.

[0064] Optionally, step S40, namely, performing a second cross-modal attention calculation on the first decoding vector and the second spatial domain encoding vector to obtain the second decoding vector, includes:

[0065] S401. Obtain the second query space vector corresponding to the second spatial domain encoding vector;

[0066] S402, Obtain the second key space vector and the second value space vector corresponding to the first decoding vector;

[0067] S403. Calculate the second attention weight based on the second query space vector and the second key space vector;

[0068] S404. The second value space vector is weighted according to the second attention weight to obtain the second decoding vector.

[0069] Understandably, the cross-modal attention decoder in this embodiment has two different attention layers. In the second attention layer, the second query space vector Q2 corresponding to the second spatial domain encoding vector is obtained; the second key space vector K2 and the second value space vector V2 corresponding to the first decoding vector are obtained; the second attention weight a2 is calculated based on the second query space vector Q2 and the second key space vector K2; and the second value space vector V2 is weighted based on the second attention weight a2 to obtain the second decoding vector context2.

[0070] This embodiment performs cross-modal attention calculation between the second spatial domain encoding vector and the first decoding vector, which enriches the modal features and preserves the prosodic information features between character sequences, character-level graph sequences and word-level graph sequences, thereby improving the accuracy of speech synthesis.

[0071] Optionally, step S50, namely obtaining the speech spectrogram sequence based on the second decoding vector to generate the synthesized speech of the text to be synthesized, includes:

[0072] S501. Generate a speech Mel spectrum based on the second decoding vector;

[0073] S502. Perform spectral conversion on the speech Mel spectrum to obtain the synthesized speech of the text to be synthesized.

[0074] Understandably, a speech Mel spectrum contains time-domain and frequency-domain information, perception-related amplitude information, and perception-related frequency-domain information. Based on the human ear's sensitivity to low-frequency sounds and insensitivity to high-frequency sounds, the decoding vector is optimized to generate the speech Mel spectrum. In one embodiment, the decoder obtains multiple second decoding vectors after cross-modal attention calculations at multiple time steps. These second decoding vectors are then optimized using a post-processing network to obtain speech Mel spectrum features. The speech Mel spectrum is then generated based on these features. After generating the speech Mel spectrum corresponding to the text to be synthesized, WaveNet is used to perform a spectral transformation on the speech Mel spectrum, inversely transforming the speech Mel spectrum feature representation into waveform samples to generate the synthesized speech corresponding to the text to be synthesized.

[0075] This embodiment generates synthesized speech corresponding to the text to be synthesized by performing spectral conversion on the speech Mel spectrum, thus ensuring the naturalness and accuracy of the synthesized speech.

[0076] like Figure 2The flowchart shown illustrates a multimodal speech synthesis method. The input text is then processed by three separate encoder recurrent neural networks (character embedding-character encoder, character-level graph embedding-character-level graph encoder, and word-level graph embedding-word-level graph encoder) to obtain character-level spatial domain encoded vectors, word-level spatial domain encoded vectors, and temporal domain encoded vectors. These are then decoded through two attention layers of a cross-modal attention decoder to obtain a Mel spectrogram. Synthetic speech is then generated based on the Mel spectrogram. Notably, the processing in the two spatial domains is auxiliary; that is, in the cross-modal attention decoder, the two spatial domain modalities are used to assist the attention calculation of the temporal modality. The temporal encoded vector can receive prosodic information from the two spatial domain encoded vectors and perform spatial feature joint processing.

[0077] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0078] In one embodiment, a multimodal speech synthesis device is provided, which corresponds one-to-one with the multimodal speech synthesis methods described in the above embodiments. For example... Figure 3 As shown, the multimodal speech synthesis device includes a preprocessing module 10, an encoding processing module 20, a first attention calculation module 30, a second attention calculation module 40, and a synthesized speech generation module 50. Detailed descriptions of each functional module are as follows:

[0079] Preprocessing module 10 is used to preprocess the text to be synthesized to obtain input sequence information; the input sequence information includes character sequence information, character-level graph sequence information, and word-level graph sequence information;

[0080] The encoding processing module 20 is used to encode the character sequence information to obtain a time-domain encoding vector; to encode the character-level graph sequence information to obtain a first spatial domain encoding vector; and to encode the word-level graph sequence information to obtain a second spatial domain encoding vector.

[0081] The first attention calculation module 30 is used to perform a first cross-modal attention calculation on the temporal domain coding vector and the first spatial domain coding vector to obtain a first decoding vector.

[0082] The second attention calculation module 40 is used to perform a second cross-modal attention calculation on the first decoding vector and the second spatial domain encoding vector to obtain a second decoding vector.

[0083] The synthesized speech generation module 50 is used to obtain a speech spectrogram based on the second decoding vector in order to generate synthesized speech of the text to be synthesized.

[0084] Optionally, the preprocessing module 10 includes:

[0085] The first character feature data extraction unit is used to extract the character feature data of the text to be synthesized;

[0086] The character sequence information generation unit is used to perform phoneme embedding on the character feature data to generate character sequence information.

[0087] Optionally, the preprocessing module 10 also includes:

[0088] The second character feature data extraction unit is used to extract the character feature data of the text to be synthesized;

[0089] A character feature data graph embedding unit is used to perform graph embedding on the character feature data to obtain character-level graph node information and character-level graph boundary information;

[0090] A character-level graph sequence information generation unit is used to generate the character-level graph sequence information based on the character-level graph node information and the character-level graph boundary information.

[0091] Optionally, the preprocessing module 10 also includes:

[0092] A word feature data extraction unit is used to extract word feature data from the text to be synthesized;

[0093] A word feature data graph embedding unit is used to perform graph embedding on the word feature data to obtain word-level graph node information and word-level graph boundary information;

[0094] A word-level graph sequence information generation unit is used to generate word-level graph sequence information based on word-level graph node information and word-level graph boundary information.

[0095] Optionally, the first attention calculation module 30 includes:

[0096] The first spatial domain encoding vector processing unit is used to obtain the first query space vector corresponding to the first spatial domain encoding vector.

[0097] A time-domain encoded vector processing unit is used to obtain a first key space vector and a first value space vector corresponding to the time-domain encoded vector;

[0098] The first attention weight calculation unit is used to calculate the first attention weight based on the first query space vector and the first key space vector.

[0099] The first decoding vector calculation unit is used to perform weighted calculation on the first value space vector according to the first attention weight to obtain the first decoding vector.

[0100] Optionally, the second attention calculation module 40 includes:

[0101] The second spatial domain encoding vector processing unit is used to obtain the second query space vector corresponding to the second spatial domain encoding vector;

[0102] The first decoding vector processing unit is used to obtain the second key space vector and the second value space vector corresponding to the first decoding vector;

[0103] The second attention weight calculation unit is used to calculate the second attention weight based on the second query space vector and the second key space vector;

[0104] The second decoding vector calculation unit is used to perform weighted calculation on the second value space vector according to the second attention weight to obtain the second decoding vector.

[0105] Optionally, the synthesized speech generation module 50 includes:

[0106] A speech Mel spectrum generation unit is used to generate a speech Mel spectrum based on the second decoding vector;

[0107] The speech synthesis unit is used to perform spectral conversion on the speech Mel spectrum to obtain the synthesized speech of the text to be synthesized.

[0108] Specific limitations regarding multimodal speech synthesis devices can be found in the limitations of multimodal speech synthesis methods described above, and will not be repeated here. Each module in the aforementioned multimodal speech synthesis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the corresponding operations of each module.

[0109] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes a readable storage medium and internal memory. The readable storage medium stores an operating system, computer-readable instructions, and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The database stores data related to a multimodal speech synthesis method. The network interface communicates with external terminals via a network connection. When the computer-readable instructions are executed by the processor, they implement a multimodal speech synthesis method. The readable storage medium provided in this embodiment includes both non-volatile and volatile readable storage media.

[0110] In one embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor performs the following steps when executing the computer-readable instructions:

[0111] The text to be synthesized is preprocessed to obtain input sequence information; the input sequence information includes character sequence information, character-level graph sequence information, and word-level graph sequence information.

[0112] The character sequence information is encoded to obtain a time-domain encoding vector; the character-level graph sequence information is encoded to obtain a first spatial domain encoding vector; and the word-level graph sequence information is encoded to obtain a second spatial domain encoding vector.

[0113] A first cross-modal attention calculation is performed on the temporal domain encoding vector and the first spatial domain encoding vector to obtain a first decoding vector;

[0114] A second cross-modal attention calculation is performed on the first decoding vector and the second spatial domain encoding vector to obtain the second decoding vector;

[0115] The speech spectrogram is obtained based on the second decoding vector to generate the synthesized speech of the text to be synthesized.

[0116] In one embodiment, one or more computer-readable storage media storing computer-readable instructions are provided. The readable storage media provided in this embodiment include non-volatile readable storage media and volatile readable storage media. The readable storage media stores computer-readable instructions, which, when executed by one or more processors, perform the following steps:

[0117] The text to be synthesized is preprocessed to obtain input sequence information; the input sequence information includes character sequence information, character-level graph sequence information, and word-level graph sequence information.

[0118] The character sequence information is encoded to obtain a time-domain encoding vector; the character-level graph sequence information is encoded to obtain a first spatial domain encoding vector; and the word-level graph sequence information is encoded to obtain a second spatial domain encoding vector.

[0119] A first cross-modal attention calculation is performed on the temporal domain encoding vector and the first spatial domain encoding vector to obtain a first decoding vector;

[0120] A second cross-modal attention calculation is performed on the first decoding vector and the second spatial domain encoding vector to obtain the second decoding vector;

[0121] The speech spectrogram is obtained based on the second decoding vector to generate the synthesized speech of the text to be synthesized.

[0122] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When executed, these computer-readable instructions can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0123] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0124] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A multimodal speech synthesis method, characterized in that, include: Preprocess the text to be synthesized to obtain the input sequence information; The input sequence information includes character sequence information, character-level graph sequence information, and word-level graph sequence information; The character sequence information is encoded to obtain a time-domain encoding vector; the character-level graph sequence information is encoded to obtain a first spatial domain encoding vector; and the word-level graph sequence information is encoded to obtain a second spatial domain encoding vector. A first cross-modal attention calculation is performed on the temporal domain encoding vector and the first spatial domain encoding vector to obtain a first decoding vector; A second cross-modal attention calculation is performed on the first decoding vector and the second spatial domain encoding vector to obtain the second decoding vector; The speech spectrogram is obtained based on the second decoding vector to generate the synthesized speech of the text to be synthesized; The step of performing a first cross-modal attention calculation on the temporal coding vector and the first spatial coding vector to obtain a first decoding vector includes: Obtain the first query space vector corresponding to the first spatial domain encoding vector; Obtain the first key space vector and the first value space vector corresponding to the time-domain encoding vector; Calculate the first attention weight based on the first query space vector and the first key space vector; The first value space vector is weighted according to the first attention weight to obtain the first decoding vector; The step of performing a second cross-modal attention calculation on the first decoding vector and the second spatial domain encoding vector to obtain the second decoding vector includes: Obtain the second query space vector corresponding to the second spatial domain encoding vector; Obtain the second key space vector and the second value space vector corresponding to the first decoding vector; Calculate the second attention weight based on the second query space vector and the second key space vector; The second value space vector is weighted according to the second attention weight to obtain the second decoding vector.

2. The multimodal speech synthesis method as described in claim 1, characterized in that, The preprocessing of the text to be synthesized to obtain input sequence information includes: Extract the character feature data of the text to be synthesized; Phoneme embedding is performed on the character feature data to generate character sequence information.

3. The multimodal speech synthesis method as described in claim 1, characterized in that, The preprocessing of the text to be synthesized to obtain input sequence information further includes: Extract the character feature data of the text to be synthesized; The character feature data is embedded into a graph to obtain character-level graph node information and character-level graph boundary information; The character-level graph sequence information is generated based on the character-level graph node information and the character-level graph boundary information.

4. The multimodal speech synthesis method as described in claim 1, characterized in that, The preprocessing of the text to be synthesized to obtain input sequence information further includes: Extract word feature data from the text to be synthesized; Graph embedding is performed on the word feature data to obtain word-level graph node information and word-level graph boundary information; The word-level graph sequence information is generated based on the word-level graph node information and the word-level graph boundary information.

5. The multimodal speech synthesis method as described in claim 1, characterized in that, The step of obtaining the speech spectrogram sequence based on the second decoding vector to generate the synthesized speech of the text to be synthesized includes: Generate a speech Mel spectrum based on the second decoding vector; The Mel spectrum of the speech is subjected to spectral conversion to obtain the synthesized speech of the text to be synthesized.

6. A speech synthesis device based on multimodal speech synthesis, characterized in that, include: The preprocessing module is used to preprocess the text to be synthesized to obtain the input sequence information; The input sequence information includes character sequence information, character-level graph sequence information, and word-level graph sequence information; The encoding processing module is used to encode the character sequence information to obtain a time-domain encoding vector; to encode the character-level graph sequence information to obtain a first spatial domain encoding vector; and to encode the word-level graph sequence information to obtain a second spatial domain encoding vector. The first attention calculation module is used to perform a first cross-modal attention calculation on the temporal domain encoding vector and the first spatial domain encoding vector to obtain a first decoding vector; The second attention calculation module is used to perform a second cross-modal attention calculation on the first decoding vector and the second spatial domain encoding vector to obtain the second decoding vector. A synthesized speech generation module is used to obtain a speech spectrogram based on the second decoding vector in order to generate synthesized speech of the text to be synthesized; The first attention calculation module includes: The first spatial domain encoding vector processing unit is used to obtain the first query space vector corresponding to the first spatial domain encoding vector. A time-domain encoded vector processing unit is used to obtain a first key space vector and a first value space vector corresponding to the time-domain encoded vector; The first attention weight calculation unit is used to calculate the first attention weight based on the first query space vector and the first key space vector. The first decoding vector calculation unit is used to perform weighted calculation on the first value space vector according to the first attention weight to obtain the first decoding vector; The second attention calculation module includes: The second spatial domain encoding vector processing unit is used to obtain the second query space vector corresponding to the second spatial domain encoding vector; The first decoding vector processing unit is used to obtain the second key space vector and the second value space vector corresponding to the first decoding vector; The second attention weight calculation unit is used to calculate the second attention weight based on the second query space vector and the second key space vector; The second decoding vector calculation unit is used to perform weighted calculation on the second value space vector according to the second attention weight to obtain the second decoding vector.

7. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that, When the processor executes the computer-readable instructions, it implements the multimodal speech synthesis method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing computer-readable instructions, characterized in that, When the computer-readable instructions are executed by one or more processors, the one or more processors cause the one or more processors to perform the multimodal speech synthesis method as described in any one of claims 1 to 5.

Images