Speech synthesis method and device

Abstract

The invention provides a voice synthesis method and device, and relates to the technical field of voice processing, and the method comprises the steps: obtaining a target image and a speaking manuscript, inputting the target image and the speaking manuscript into a pre-trained voice synthesis model, and obtaining a target synthesis voice; wherein the target image comprises a plurality of pieces of visual information, the target synthetic speech comprises a plurality of acoustic features, and one piece of visual information corresponds to at least one acoustic feature; the speech synthesis model is obtained by training based on a multi-modal data set, and the multi-modal data set is obtained by performing modal enhancement on a target data set. According to the method provided by the invention, modal enhancement is performed on the existing voice data set to construct the multi-modal data set, so that the problem of scarcity of the data set is solved; according to the one-to-one correspondence between the visual information in the static image and the acoustic features in the voice audio, based on the voice synthesis model obtained through multi-modal data set training, the synthesized target synthesis voice has a finer-grained voice style.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to a speech synthesis method and device. Background Art

[0002] In the era of artificial intelligence generated content (AIGC), AI technology is generating more and more multimedia content. For example, generating video content, a process that aims to transform static images centered on people into dynamic, talking animations. In order to enhance the vividness of the generated video, it is crucial to ensure that the visual information in the input image is in harmony with the speech features in the audio.

[0003] Existing studies mainly focus on leveraging facial information to infer basic speaker characteristics such as gender, age, and emotion, but they often ignore the abundant additional clues present in images and fail to generate audio with fine-grained speech style.

[0004] How to simulate and synthesize audio with finer-grained speech style through a given image is a technical problem that needs to be solved at present. Summary of the invention

[0005] The present invention provides a speech synthesis method and device to solve the defects in the prior art.

[0006] The present invention provides a speech synthesis method, comprising the following steps: Acquire a target image and a speech manuscript, and input the target image and the speech manuscript into a pre-trained speech synthesis model to obtain a target synthesized speech; Among them, the target image contains multiple visual information, the target synthesized speech contains multiple acoustic features, and one visual information corresponds to at least one acoustic feature; the speech synthesis model is trained based on a multimodal data set, and the multimodal data set is obtained by modally enhancing the target data set.

[0007] According to a speech synthesis method provided by the present invention, the speech synthesis model includes an image encoder, a speech decoder and a query converter Q-former; The image encoder is used to extract multiple visual information in the target image; The speech decoder is used to generate a plurality of acoustic features in the target synthesized speech; The query converter Q-former is used for modality interaction training.

[0008] According to a speech synthesis method provided by the present invention, the training process of the speech synthesis model includes: Performing a first-stage training on the speech decoder based on the multimodal data set, and obtaining a training loss value of the speech decoder; wherein the first-stage training is unsupervised speech style learning training; Performing a second-stage training on the query converter Q-former based on the multimodal dataset; wherein the second-stage training is a visual representation learning training related to speech style; The speech decoder after the first stage training and the query converter Q-former after the second stage training are connected, and the connected speech decoder and query converter Q-former are trained for the third stage based on the training loss value of the speech decoder; wherein the third stage training is speech style control training under visual conditions.

[0009] According to a speech synthesis method provided by the present invention, the step of acquiring a target image and a speech manuscript, and inputting the target image and the speech manuscript into a pre-trained speech synthesis model to obtain a target synthesized speech includes: Acquire a target image and a speech manuscript, and extract a plurality of key frames of the target image; wherein the key frames represent an image sequence centered on a speaker in the target image; Inputting the multiple key frames of the target image and the speech manuscript into a pre-trained speech synthesis model to obtain multiple speech audios; The target synthesized speech is synthesized based on the multiple speech audios.

[0010] According to a speech synthesis method provided by the present invention, before acquiring the target image and the speech manuscript, the method further includes: A target data set is acquired, and modality enhancement is performed on the target data set to obtain the multimodal data set; wherein the target data set is a natural language prompt speech data set.

[0011] According to a speech synthesis method provided by the present invention, the target data set includes a plurality of speech segments and a speech description corresponding to each speech segment; The performing modality enhancement on the target data set to obtain the multimodal data set includes: Input the speech description in the target data set into a pre-trained text modal conversion model to obtain a visual description corresponding to the speech description; wherein the speech description represents a text description corresponding to the speech feature of the target data set, and the visual description represents a picture description corresponding to the speech scene of the target data set; The visual description is input into a pre-trained image generation model to obtain a corresponding target image.

[0012] The present invention also provides a speech synthesis device, comprising the following modules: A speech synthesis module, used to obtain a target image and a speech manuscript, and input the target image and the speech manuscript into a pre-trained speech synthesis model to obtain a target synthesized speech; Among them, the target image contains multiple visual information, the target synthesized speech contains multiple acoustic features, and one visual information corresponds to at least one acoustic feature; the speech synthesis model is trained based on a multimodal data set, and the multimodal data set is obtained by modally enhancing the target data set.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned speech synthesis methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the speech synthesis method described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned speech synthesis methods.

[0016] The present invention provides a speech synthesis method and device, which obtains a target image and a speech manuscript, and inputs the target image and the speech manuscript into a pre-trained speech synthesis model to obtain a target synthesized speech; wherein the target image contains multiple visual information, and the target synthesized speech contains multiple acoustic features, and one visual information corresponds to at least one acoustic feature; the speech synthesis model is obtained by training based on a multimodal data set, and the multimodal data set is obtained by modally enhancing the target data set. It can be seen that the present invention solves the problem of data set scarcity by modally enhancing the existing speech data set to construct a multimodal data set; according to the one-to-one correspondence between the visual information in the static image and the acoustic features in the speech audio, based on the speech synthesis model obtained by training the multimodal data set, the synthesized target synthesized speech has a more fine-grained speech style. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of the speech synthesis method provided by the present invention.

[0019] Figure 2 It is an overall architecture diagram of the speech synthesis model provided by the present invention.

[0020] Figure 3 It is a schematic diagram of the intrinsic connection between visual information and acoustic features provided by the present invention.

[0021] Figure 4 It is a schematic diagram of the correlation strength between visual information and acoustic features provided by the present invention.

[0022] Figure 5 It is a schematic diagram of the speech synthesis model training process provided by the present invention.

[0023] Figure 6 It is a schematic diagram of the multimodal dataset construction process provided by the present invention.

[0024] Figure 7 It is a structural schematic diagram of the speech synthesis device provided by the present invention.

[0025] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] Combine the following Figure 1-Figure 8 A speech synthesis method and device of the present invention are described.

[0028] Figure 1 It is a flow chart of the speech synthesis method provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 100: Obtain a target image and a speech manuscript, and input the target image and the speech manuscript into a pre-trained speech synthesis model to obtain a target synthesized speech; Among them, the target image contains multiple visual information, the target synthesized speech contains multiple acoustic features, and one visual information corresponds to at least one acoustic feature; the speech synthesis model is trained based on a multimodal data set, and the multimodal data set is obtained by modally enhancing the target data set.

[0029] It should be noted that TTS, the full name of which is Text To Speech, is a technology that converts text information into natural and fluent speech output. The embodiment of the present invention proposes a visually improved TTS model that can predict the voice of the speaker and control the speaking style at the speech level according to the corresponding image sequence (key frame).

[0030] Specifically, Figure 2 This is the overall architecture diagram of the speech synthesis model provided by the present invention. Figure 2 As shown in Figure 1, the speech synthesis model with global visual insights located between the two data streams synthesizes audio sequences using key frames and corresponding sentences to organize into chapter-level speech. The model can implicitly extract visual information of the image listed on the far left, such as appearance and expression, and then compare it with the acoustic features shown on the far right, such as pitch and volume.

[0031] It should be noted that, in order to have a more comprehensive understanding of vision-based speech style control, the embodiment of the present invention systematically analyzes the relationship between visual information and acoustic features.

[0032] Specifically, this embodiment studies the relationship between personal appearance and timbre and finds that there is a significant negative correlation between audio and measurements of body shape and weight, and there is also a significant correlation between facial bone measurements and F0 and habitual frequency. Experiments show that the identity information provided by voice is equivalent to facial information, and people can match unfamiliar voices with static facial images. From a psychological perspective, speaking voices and other body behaviors are external manifestations of inner activities (such as emotions), so the two naturally form a close connection. Different directions and intentions of body sway convey different emotional states. Similarly, individuals with positive or negative emotions will show opposite head movements. In different communication environments, emotional functions and cognitive functions will be affected. In addition, different tones can trigger a range of emotional reactions. Previous studies have mainly focused on one or two aspects of the relationship between vision and vocalization. To have a more comprehensive understanding of vision-based voice style control, it is necessary to conduct a systematic analysis of the visual-audio association of human behavior based on human perception, which is currently lacking and urgently needed.

[0033] Specifically, Figure 3 is a schematic diagram of the intrinsic connection between visual information and acoustic features provided by the present invention, Figure 4 is a schematic diagram of the correlation strength between visual information and acoustic features provided by the present invention, combined with Figure 3 and Figure 4, the study of the correlation between sound and scene images by experimental psychology method provided in this embodiment is described. According to the brief blueprint of the correlation map and 20 human-centered conversation video samples, key frame images and corresponding audio clips are extracted, and a five-level correlation mean opinion score (RMOS) (N=10) is performed on each pair of visual features, which can be seen in Figure 4 For each visual-sound pairing of a sample, the average RMOS of all samples is taken as the correlation value of the pairing. For each visual-sound combination, the average RMOS of all samples is taken as the correlation value of the pairing. Then, the correlation matrix is ​​visualized as a Sankey diagram, which can be seen in Figure 3 , where only the 50% with the highest correlation are retained.

[0034] Specifically, we discover the systematic relationship between visual information and acoustic features from the perspective of human perception. Figure 3 , the visual information related to speech includes the speaker's appearance (hair, face, clothing, etc.), the speaker's facial expression (eyebrows, eyes, lips, etc.), the speaker's posture (head movement, body posture and gestures), the speaking scene (background) and the visual tone (artistic tone, image exposure, contrast, saturation, etc.). At the same time, the acoustic features related to vision include the speaker's timbre, speaking rhythm (speed pace), speaking volume, speaking emotion (emotion and intonation) and speaking topic. In addition to facial features, non-facial visual features such as the speaker's posture also have a relatively strong connection with various speech attributes such as speaking rhythm, speaking volume and speaking emotion. It is worth noting that the relationship between visual intonation and speaking emotion is also relatively strong, while the speaking scene context is mainly related to the speaking topic.

[0035] Furthermore, the overall cross-modal TTS framework provided by the embodiment of the present invention is described.

[0036] Specifically, the speech synthesis model includes an image encoder, a speech decoder and a query converter Q-former; The image encoder is used to extract multiple visual information in the target image; The speech decoder is used to generate a plurality of acoustic features in the target synthesized speech; The query converter Q-former is used for modality interaction training.

[0037] The training process of the speech synthesis model includes: Performing a first-stage training on the speech decoder based on the multimodal data set, and obtaining a training loss value of the speech decoder; wherein the first-stage training is unsupervised speech style learning training; Performing a second-stage training on the query converter Q-former based on the multimodal dataset; wherein the second-stage training is a visual representation learning training related to speech style; The speech decoder after the first stage training and the query converter Q-former after the second stage training are connected, and the connected speech decoder and query converter Q-former are trained for the third stage based on the training loss value of the speech decoder; wherein the third stage training is speech style control training under visual conditions.

[0038] In one embodiment, continue to see Figure 2 The TTS framework contains a CLIP image encoder, a query converter Q-former as a bridge network, and a VITS-based speech backbone network. The VITS-based speech backbone network is an end-to-end variational autoencoder, including a reference speech encoder and a speech decoder. Due to its end-to-end design, it shows strong capabilities in multi-speaker speech synthesis. The image encoder extracts global visual embeddings from the input image, while the Q-former extracts visual reproduction related to the speech style and provides conditions for the generation of the speech decoder. Since end-to-end image-audio alignment relies on a large amount of strongly correlated image-audio data, this embodiment adopts a pre-training-fine-tuning training method and designs a three-stage learning framework. Figure 5 is a schematic diagram of the speech synthesis model training process provided by the present invention, such as Figure 5 As shown, the three-stage learning includes: (1) unsupervised speech style learning; (2) learning of visual representations related to speech style; and (3) speech style control under visual conditions.

[0039] (1) Phase 1: Unsupervised speech style learning.

[0040] In order to give the VITS-based backbone system powerful speech modeling capabilities, the VITS-based speech backbone system was unsupervisedly trained on large-scale emotional speech data in the first stage. The speech encoder consists of a convolutional neural network layer and a gated recurrent unit (GRU) layer, and learns a global style embedding from the melt spectrogram of real audio. After normalization, the global style embedding is passed to the generator to supervise the training of various modules such as the post-encoder, pre-encoder, and duration predictor. This embodiment uses multiple discriminators for adversarial training. For simplicity, the training loss of the backbone is annotated as L vits .

[0041] (2) The second stage: learning visual representations related to speech style.

[0042] To effectively utilize high-quality relevant image and audio data, in the second stage, Q-former learns visual representations from images under the guidance of live audio and text descriptions that describe basic speech properties (including speech). Live audio provides rich fine-grained style information, but is more difficult to model. The style information provided by text descriptions is coarser, but easier to model. In order to utilize the style information in audio and descriptions, this embodiment uses multiple cross-modal contrastive learning as training objects.

[0043] It should be noted that the query transformer (Q-former) is initialized from a pre-trained BERT model, along with a set of randomly initialized trainable queries and a set of cross-attention layers injected into the transformer encoder layer. The Q-former plays three roles in the second stage: 1) The image encoder inputs the style query, which interacts with the visual features through the cross-attention layer to obtain the image embedding E i ; 2) The text encoder inputs the text description, extracts the rough speech style information, and obtains the text embedding E t ; 3) The bidirectional encoder inputs the merged style query and text description and outputs a fused representation. The style query and description can pay attention to each other in the forward process of the conversion layer. That is, two unimodal encoders and one bimodal encoder share a set of parameters in the self-attention and feedforward layers. This mechanism not only facilitates the style query to learn coarse-to-fine information from text description and audio, but also provides a way to fuse visual-text information. In the reasoning stage, the style query can replace the text description and audio information.

[0044] 1. Image-audio contrastive learning: The goal of this stage is to align the unimodal image embedding E from the Q-former i and the speech embedding E from the frozen speech encoder a In this embodiment, for every N pairs of images I K and Audio A K , the matching image-audio pairs are labeled as positive samples, and the mismatched image-audio pairs are labeled as negative samples. The dot product of the image embedding and the speech embedding is used as the classification vector for binary classification as the matching score. Specifically, for each image, the softmax function and the cross entropy loss function are applied to the score to obtain the image-audio contrast loss L iac-i , see formula (1).

[0045]

[0046] Similarly, for each audio, the same steps are applied to the scores to obtain the image-audio contrastive loss L iac-a , see formula (2). The total contrast loss should be L iac = Liac-i + L iac-a .

[0047]

[0048] In order to obtain a more refined image and audio alignment, this embodiment uses an additional classifier to fuse the image embedding E i and speech embedding E a , and pass an output linear layer to obtain two categories (positive / negative samples) classification vectors. The image-audio matching loss is L iam .

[0049] 2. Image-text contrastive learning: The text description provides pure speech attribute information in the second stage, which helps image-audio alignment. It can also be said that the participation of text description enables audio-visual alignment to be performed in text mode, thereby reducing the modeling complexity of Q-former. Similar to Image-Audio Contrastive Learning, cross-modal contrastive loss and matching loss are used as training objects. Contrastive loss L itc Single model output from Q-former, E i and E i , because in this case, the style query is prohibited from focusing on the text description. The matching loss L itm Bidirectional output from Q-former, in this case, the style query and text description are concatenated and allowed to follow each other.

[0050] 3. Text-audio contrastive learning: In the second stage, the text description provides pure speech attribute information, which is helpful for image-audio alignment. In order to simplify the text-audio contrastive learning, this embodiment only uses the contrast loss L atc .L atc The form of L iac resemblance.

[0051] The total training loss of Q-former in the second stage is L, which can be seen in formula (3).

[0052]

[0053] Considering the coarse-to-fine supervision of text description and speech audio, an annealing function is set for λia, and a fixed λia value will degrade the performance. ia The weight gradually increases and finally reaches 1.

[0054] (3) The third stage: speech style control under visual conditions.

[0055] Despite the efforts to align visual and speech representations, there is still a domain gap between the two modalities. To address this domain gap, a unified fine-tuning strategy is adopted in the third stage. The Q-former is directly connected to the VITS-based speech decoder and L vits As training objects for image-audio pairs. It is worth noting that at this stage, all parameters of the Q-former and speech decoder are trainable.

[0056] It should be noted that in terms of the speech backbone network, the design of VITS2 is adopted, but the random duration predictor is replaced by a normal duration predictor, and the gradient of the duration predictor for the global speech embedding is enabled during the backpropagation process. In order to improve the prosody, the text features in the phonemes and pre-trained BERT model are used as the input of the speech decoder, and multiple discriminators are used for the flow model and duration predictor. For the CLIP image encoder, the open source CLIP-VIT-H-14 is used. For the Q-former, XLM-Roberta-base is used for initialization, and 32 style queries with a hidden size of 768 are set. For the selection of negative sample pairs for cross-modal matching loss, a hard negative mining strategy is adopted. The hard negative mining strategy is an important strategy in the field of machine learning, especially when dealing with classification problems, to improve model performance. The core of this strategy is to enhance the model's discrimination and generalization capabilities by identifying and focusing on negative samples that are misclassified by the model (i.e., hard negative samples).

[0057] Based on the above-trained speech synthesis model, the target image and speech manuscript are obtained, and the target image and speech manuscript are input into the pre-trained speech synthesis model to obtain the target synthesized speech, including: Acquire a target image and a speech manuscript, and extract a plurality of key frames of the target image; wherein the key frames represent an image sequence centered on a speaker in the target image; Inputting the multiple key frames of the target image and the speech manuscript into a pre-trained speech synthesis model to obtain multiple speech audios; The target synthesized speech is synthesized based on the multiple speech audios.

[0058] The above is a description of the steps of the speech synthesis method provided by the present invention. From the description of the above steps, it can be seen that according to the speech synthesis method provided by the present invention, a target synthesized speech is obtained by acquiring a target image and a speech manuscript, and inputting the target image and the speech manuscript into a pre-trained speech synthesis model; wherein the target image contains multiple visual information, and the target synthesized speech contains multiple acoustic features, and one visual information corresponds to at least one acoustic feature; the speech synthesis model is obtained by training based on a multimodal dataset, and the multimodal dataset is obtained by modally enhancing the target dataset. It can be seen that the present invention solves the problem of dataset scarcity by modally enhancing the existing speech dataset to construct a multimodal dataset; according to the one-to-one correspondence between the visual information in the static image and the acoustic features in the speech audio, based on the speech synthesis model obtained by training the multimodal dataset, the synthesized target synthesized speech has a more fine-grained speech style.

[0059] Based on the above embodiment, in this embodiment, before step 100 of acquiring the target image and the speech manuscript, the method further includes: A target data set is acquired, and modality enhancement is performed on the target data set to obtain the multimodal data set; wherein the target data set is a natural language prompt speech data set.

[0060] The target data set includes a plurality of speech segments and a speech description corresponding to each speech segment; The performing modality enhancement on the target data set to obtain the multimodal data set includes: Input the speech description in the target data set into a pre-trained text modal conversion model to obtain a visual description corresponding to the speech description; wherein the speech description represents a text description corresponding to the speech feature of the target data set, and the visual description represents a picture description corresponding to the speech scene of the target data set; The visual description is input into a pre-trained image generation model to obtain a corresponding target image.

[0061] It should be noted that, based on the description of the above embodiments, body posture and speaking scene will have a significant impact on the timbre of speech, which puts forward requirements for training speech datasets: human-centered images should contain visual information, including not only facial expressions, but also head posture, gestures, and speaking context. The audio-visual datasets of the prior art are not suitable for this task for the following reasons: First, audio-visual datasets are generally designed for generating talking avatars. Most of them emphasize the accuracy of visual actions to capture the synchronization between audio signals and visual actions, but ignore the expressiveness of speech, which is crucial for TTS tasks. Second, in the traditional audio-visual dataset construction process, there is an inevitable trade-off between audio quality and scene diversity. Usually, audio-visual datasets use studio or network videos to extract image-audio pairs. Videos from recording studios have good voice quality but poor scene diversity. Videos captured from the Internet have diverse speaking scenes, but due to challenges such as the cocktail party effect, a lot of effort is required for data cleaning. The cocktail party effect is a special phenomenon in the human auditory system, which refers to the ability of people to focus on a person's conversation in a noisy environment and ignore other conversations or noise in the background.

[0062] Specifically, emotional speech datasets and natural language prompt speech datasets usually contain high-quality speech waves and detailed speech text descriptions, including gender, age, emotion, pitch, volume, etc. These rich text descriptions allow humans to naturally imagine the speaking scene, which can be directly converted into visual components through mature AIGC technology. Considering that the modality conversion should be consistent with human perception, based on the above embodiment, the modality of the existing speech dataset is selected to be expanded. Figure 6 It is a schematic diagram of the multimodal dataset construction process provided by the present invention. As shown in FIG6 , a novel text-audio-vision dataset is constructed, which is tailored for visually improved text-to-speech.

[0063] 1) Text-to-modal conversion: A prompt template is pre-set, and LLM converts the speech description (text description of speech features) into a visual description (picture description of the speech scene). The modal conversion is performed in the text domain.

[0064] 2) Visualization: Based on the visual description, the text-to-image model synthesizes images that capture the essence of the audio context and speaking style, completing the visual part of the text-audio-visual speech dataset.

[0065] In this embodiment, GPT-3.5 turbo is selected as the text modality converter, and DALL-E 3 is selected for image generation. This embodiment does not impose any special restrictions on this.

[0066] Furthermore, a bilingual text-visual-audio dataset was obtained based on the internal natural language prompt speech dataset. There are 28,929 pairs of image and audio, of which 28,063 samples are used for training segmentation and 866 samples are used for testing segmentation.

[0067] The speech synthesis method provided in this embodiment solves the problem of scarce data sets by performing modal enhancement on existing speech data sets to construct a multimodal data set.

[0068] In summary, the embodiment of the present invention systematically studies the relationship between human visual perception and sound perception through psychological experiments; and further proposes a multimodal contrastive learning framework, which demonstrates deep voice style control capabilities; in addition, using generative artificial intelligence, a voice-centric multimodal dataset is constructed through modal enhancement technology. The model trained on the multimodal dataset has generalization capabilities for real-world photos (including emoticons).

[0069] The speech synthesis device provided by the present invention is described below. The speech synthesis device described below and the speech synthesis method described above can be referred to each other.

[0070] Figure 7 is a structural schematic diagram of the speech synthesis device provided by the present invention, such as Figure 7 As shown, the speech synthesis device provided by the present invention comprises: The speech synthesis module 701 is used to obtain a target image and a speech manuscript, and input the target image and the speech manuscript into a pre-trained speech synthesis model to obtain a target synthesized speech; Among them, the target image contains multiple visual information, the target synthesized speech contains multiple acoustic features, and one visual information corresponds to at least one acoustic feature; the speech synthesis model is trained based on a multimodal data set, and the multimodal data set is obtained by modally enhancing the target data set.

[0071] The speech synthesis device provided by the present invention obtains a target synthesized speech by acquiring a target image and a speech manuscript, and inputting the target image and the speech manuscript into a pre-trained speech synthesis model; wherein the target image contains multiple visual information, and the target synthesized speech contains multiple acoustic features, and one visual information corresponds to at least one acoustic feature; the speech synthesis model is obtained by training based on a multimodal data set, and the multimodal data set is obtained by modally enhancing the target data set. It can be seen that the present invention solves the problem of data set scarcity by modally enhancing the existing speech data set to construct a multimodal data set; according to the one-to-one correspondence between the visual information in the static image and the acoustic features in the speech audio, based on the speech synthesis model obtained by training the multimodal data set, the synthesized target synthesized speech has a more fine-grained speech style.

[0072] Based on the above embodiment, in this embodiment, the speech synthesis model includes an image encoder, a speech decoder and a query converter Q-former; The image encoder is used to extract multiple visual information in the target image; The speech decoder is used to generate a plurality of acoustic features in the target synthesized speech; The query converter Q-former is used for modality interaction training.

[0073] Based on the above embodiment, in this embodiment, the device further includes a training module, which is specifically used for: Performing a first-stage training on the speech decoder based on the multimodal data set, and obtaining a training loss value of the speech decoder; wherein the first-stage training is unsupervised speech style learning training; Performing a second-stage training on the query converter Q-former based on the multimodal dataset; wherein the second-stage training is a visual representation learning training related to speech style; The speech decoder after the first stage training and the query converter Q-former after the second stage training are connected, and the connected speech decoder and query converter Q-former are trained for the third stage based on the training loss value of the speech decoder; wherein the third stage training is speech style control training under visual conditions.

[0074] Based on the above embodiment, in this embodiment, the speech synthesis module 701 is specifically used for: Acquire a target image and a speech manuscript, and extract a plurality of key frames of the target image; wherein the key frames represent an image sequence centered on a speaker in the target image; Inputting the multiple key frames of the target image and the speech manuscript into a pre-trained speech synthesis model to obtain multiple speech audios; The target synthesized speech is synthesized based on the multiple speech audios.

[0075] Based on the above embodiment, in this embodiment, the device further includes a modality enhancement module, which is specifically used to: Before acquiring the target image and speech manuscript, a target data set is acquired, and modality enhancement is performed on the target data set to obtain the multimodal data set; wherein the target data set is a natural language prompt speech data set.

[0076] Based on the above embodiment, in this embodiment, the target data set includes multiple voice segments and a voice description corresponding to each voice segment; The modal enhancement module is specifically used for: Input the speech description in the target data set into a pre-trained text modal conversion model to obtain a visual description corresponding to the speech description; wherein the speech description represents a text description corresponding to the speech feature of the target data set, and the visual description represents a picture description corresponding to the speech scene of the target data set; The visual description is input into a pre-trained image generation model to obtain a corresponding target image.

[0077] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the speech synthesis method, which includes: Acquire a target image and a speech manuscript, and input the target image and the speech manuscript into a pre-trained speech synthesis model to obtain a target synthesized speech; Among them, the target image contains multiple visual information, the target synthesized speech contains multiple acoustic features, and one visual information corresponds to at least one acoustic feature; the speech synthesis model is trained based on a multimodal data set, and the multimodal data set is obtained by modally enhancing the target data set.

[0078] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0079] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the speech synthesis method provided by the above methods, the method comprising: Acquire a target image and a speech manuscript, and input the target image and the speech manuscript into a pre-trained speech synthesis model to obtain a target synthesized speech; Among them, the target image contains multiple visual information, the target synthesized speech contains multiple acoustic features, and one visual information corresponds to at least one acoustic feature; the speech synthesis model is trained based on a multimodal data set, and the multimodal data set is obtained by modally enhancing the target data set.

[0080] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the speech synthesis method provided by the above methods, the method comprising: Acquire a target image and a speech manuscript, and input the target image and the speech manuscript into a pre-trained speech synthesis model to obtain a target synthesized speech; Among them, the target image contains multiple visual information, the target synthesized speech contains multiple acoustic features, and one visual information corresponds to at least one acoustic feature; the speech synthesis model is trained based on a multimodal data set, and the multimodal data set is obtained by modally enhancing the target data set.

[0081] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0082] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech synthesis method, characterized in that: include: Acquire a target image and a speech manuscript, and input the target image and the speech manuscript into a pre-trained speech synthesis model to obtain a target synthesized speech; Wherein, the target image includes multiple visual information, the target synthesized speech includes multiple acoustic features, and one visual information corresponds to at least one acoustic feature; The speech synthesis model is trained based on a multimodal dataset, and the multimodal dataset is obtained by modality enhancement of a target dataset.

2. The speech synthesis method according to claim 1, characterized in that: The speech synthesis model includes an image encoder, a speech decoder and a query converter Q-former; The image encoder is used to extract multiple visual information in the target image; The speech decoder is used to generate a plurality of acoustic features in the target synthesized speech; The query converter Q-former is used for modality interaction training.

3. The speech synthesis method according to claim 2, characterized in that: The training process of the speech synthesis model includes: Performing a first-stage training on the speech decoder based on the multimodal data set, and obtaining a training loss value of the speech decoder; wherein the first-stage training is unsupervised speech style learning training; Performing a second-stage training on the query converter Q-former based on the multimodal dataset; wherein the second-stage training is a visual representation learning training related to speech style; The speech decoder after the first stage training and the query converter Q-former after the second stage training are connected, and the connected speech decoder and query converter Q-former are trained for the third stage based on the training loss value of the speech decoder; wherein the third stage training is speech style control training under visual conditions.

4. The speech synthesis method according to claim 1, characterized in that: The step of acquiring a target image and a speech manuscript, and inputting the target image and the speech manuscript into a pre-trained speech synthesis model to obtain a target synthesized speech includes: Acquire a target image and a speech manuscript, and extract a plurality of key frames of the target image; wherein the key frames represent an image sequence centered on a speaker in the target image; Inputting the multiple key frames of the target image and the speech manuscript into a pre-trained speech synthesis model to obtain multiple speech audios; The target synthesized speech is synthesized based on the multiple speech audios.

5. The speech synthesis method according to claim 1, characterized in that: Before acquiring the target image and the speech manuscript, the method further includes: A target data set is acquired, and modality enhancement is performed on the target data set to obtain the multimodal data set; wherein the target data set is a natural language prompt speech data set.

6. The speech synthesis method according to claim 5, characterized in that: The target data set includes a plurality of speech segments and a speech description corresponding to each speech segment; The performing modality enhancement on the target data set to obtain the multimodal data set includes: Input the speech description in the target data set into a pre-trained text modal conversion model to obtain a visual description corresponding to the speech description; wherein the speech description represents a text description corresponding to the speech feature of the target data set, and the visual description represents a picture description corresponding to the speech scene of the target data set; The visual description is input into a pre-trained image generation model to obtain a corresponding target image.

7. A speech synthesis device, characterized in that: include: A speech synthesis module, used to obtain a target image and a speech manuscript, and input the target image and the speech manuscript into a pre-trained speech synthesis model to obtain a target synthesized speech; Wherein, the target image includes multiple visual information, the target synthesized speech includes multiple acoustic features, and one visual information corresponds to at least one acoustic feature; The speech synthesis model is trained based on a multimodal dataset, and the multimodal dataset is obtained by modality enhancement of a target dataset.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the speech synthesis method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the speech synthesis method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the speech synthesis method according to any one of claims 1 to 6 is implemented.

Images