Speech Synthesis Method, Device, Electronic Device and Medium Incorporating Visual Information

Abstract

The present application provides a speech synthesis method, device, electronic device and medium that integrate visual information, including: extracting text features of target text information based on a text encoder; performing feature extraction processing, vector quantization encoding processing and cross-attention processing on target video information to determine video features in the target video information; performing cross-attention processing on the text features and the video features to determine joint features; inputting the joint features into a pre-trained speech generation model, performing modeling processing on the joint features to generate video text features, and then performing random prosody prediction processing, feature augmentation processing and decoding processing on the video text features to generate a synthetic speech corresponding to the target text information. By integrating the information in the video, a more expressive speech is generated. This method not only solves the deficiencies of the prior art in terms of emotion control, but also can achieve high-quality speech synthesis using a small amount of sample data.

Description

Technical Field

[0001] This application relates to the field of speech synthesis technology, and in particular to a speech synthesis method, device, electronic device and medium that integrate visual information. Background Art

[0002] At the current stage, under the condition of collecting a large amount of data for training, the speech synthesis technology can already approach the level of real people. However, when training for specific person modeling, there are often problems such as difficulty in collecting data of this person and low data quality, and there is no other emotional speech in the collected data of specific persons. In such a situation, there are still great challenges in emotional speech synthesis technology. The existing speech synthesis technology relies on a large amount of high-quality data recorded in a recording studio, and different persons need to record a large amount of data of the corresponding person. In addition, if we want to achieve emotional speech synthesis of a specific person, we need to record a large amount of speech data of the specific person under different emotions in advance, so it will consume a lot of energy. Since the types of synthesized emotions are limited by the recorded data, in the case of only very few sample data of a specific person, it is difficult for the existing speech to model this person, and it is difficult to form a high-quality emotion-controllable speech synthesis ability. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a speech synthesis method, device, electronic device and medium that integrate visual information, and generate more expressive speech by integrating the information in the video. This method not only solves the deficiencies of the existing technology in terms of emotion control, but also can achieve high-quality speech synthesis by using a small amount of sample data.

[0004] An embodiment of this application provides a speech synthesis method that integrates visual information. The speech synthesis method includes:

[0005] Extracting text features of target text information based on a text encoder;

[0006] Performing feature extraction processing, vector quantization encoding processing and cross-attention processing on the target video information to determine video features in the target video information;

[0007] Performing cross-attention processing on the text features and the video features to determine joint features;

[0008] Inputting the joint features into a pre-trained speech generation model, performing modeling processing on the joint features to generate video text features, and then performing random prosody prediction processing, feature augmentation processing and decoding processing on the video text features to generate a synthesized speech corresponding to the target text information; wherein, the emotion feature of the synthesized speech is consistent with the emotion feature of the target video information.

[0009] In a possible implementation manner, the feature extraction processing, vector quantization encoding processing, and cross-attention processing of the target video information to determine the video features in the target video information include:

[0010] Based on the vector quantization encoder, mark the target video information, and add random noise to the marked target video information;

[0011] Input the target video information with added noise into the fusion diffusion model for sampling, and extract the first visual feature map of each video frame;

[0012] Based on the multi-modal neural network model, perform feature extraction processing on the target video information to obtain the visual feature map of each video frame, and perform multi-layer perceptron projection processing on the visual feature map of each video frame to generate a second visual feature map with the same dimension as the first visual feature map;

[0013] Perform combination processing and position encoding processing on the first visual feature map of each video frame and the corresponding second visual feature map to determine the initial video feature of each video frame;

[0014] Perform mean processing on multiple frames of the initial video features to generate the video features.

[0015] In a possible implementation manner, the modeling processing of the joint features to generate video text features, and then the random prosody prediction processing, feature augmentation processing, and decoding processing of the video text features to generate the synthetic speech corresponding to the target text information include:

[0016] Based on the self-attention mechanism, position encoding mechanism, and multi-head attention mechanism in the video text joint encoder of the speech generation model, perform modeling processing on the joint features to capture the cross-modal relationship between the text features and the video features, and generate video text features;

[0017] Input the video text features into the random prosody predictor of the speech generation model, perform random prosody prediction processing and feature augmentation processing on the video text features, and generate the augmented video text features;

[0018] Input the duration of phonemes, the target person's speech embedding vector, and the augmented video text features in the video text features into the decoder of the speech generation model for speech synthesis processing to generate the synthetic speech corresponding to the target text information.

[0019] In a possible implementation, the cross-modal relationship between the text feature and the video feature is captured by modeling the joint feature based on the self-attention mechanism, the position encoding mechanism, and the multi-head attention mechanism in the video-text joint encoder, and video-text features are generated, including:

[0020] Process the joint feature based on the self-attention mechanism to generate the joint feature after self-attention processing;

[0021] Perform position encoding processing on the position information of the time sequence information of the video frames and the target text information in the joint feature after self-attention processing to generate the joint feature after position encoding processing;

[0022] Perform multi-angle modeling processing on the joint feature after position encoding processing based on the multi-head attention mechanism to generate video-text features.

[0023] In a possible implementation, input the video-text features into the random rhythm predictor of the speech generation model to perform random rhythm prediction processing and high-dimensional feature expansion processing on the video-text features, and generate the expanded video-text features, including:

[0024] Predict the duration of each phoneme in the video-text features based on the emotion features and context relationships included in the video features in the video-text features;

[0025] Perform feature interpolation processing on the features corresponding to each phoneme in the video-text features according to the corresponding duration to generate the expanded video-text features.

[0026] In a possible implementation, the speech generation model is determined through the following steps:

[0027] Replace the text encoder in the original VITS model with a video-text joint encoder to obtain an initial speech generation model;

[0028] Input the sample joint features determined based on the sample video information and the sample text information into the initial speech generation model for speech synthesis processing to generate the predicted synthetic speech corresponding to the sample text information;

[0029] Determine the loss value of the initial speech generation model based on the predicted synthetic speech and the actual synthetic speech of the sample text information;

[0030] Iteratively train the initial speech generation model based on the loss value and the training samples to generate the speech generation model.

[0031] The embodiment of the present application also provides a speech synthesis device integrating visual information. The speech synthesis device includes:

[0032] A text feature extraction module, configured to extract text features of target text information based on a text encoder;

[0033] A video feature extraction module, configured to perform feature extraction processing, vector quantization coding processing, and cross-attention processing on target video information to determine video features in the target video information;

[0034] A joint module, configured to perform cross-attention processing on the text features and the video features to determine joint features;

[0035] A speech generation module, configured to input the joint features into a pre-trained speech generation model, perform modeling processing on the joint features to generate video text features, and then perform random prosody prediction processing, feature augmentation processing, and decoding processing on the video text features to generate a synthesized speech corresponding to the target text information; wherein, the emotional feature of the synthesized speech is consistent with the emotional feature of the target video information.

[0036] In a possible implementation manner, when the video feature extraction module is configured to perform feature extraction processing, vector quantization coding processing, and cross-attention processing on target video information to determine video features in the target video information, the video feature extraction module is further configured to:

[0037] Mark the target video information based on a vector quantization encoder, and add random noise to the marked target video information;

[0038] Input the target video information with added noise into a fusion diffusion model for sampling to extract a first visual feature map of each video frame;

[0039] Perform feature extraction processing on the target video information based on a multi-modal neural network model to obtain a visual feature map of each video frame, and perform multi-layer perceptron projection processing on the visual feature map of each video frame to generate a second visual feature map with the same dimension as the first visual feature map;

[0040] Perform combination processing and position encoding processing on the first visual feature map of each video frame and the corresponding second visual feature map to determine an initial video feature of each video frame;

[0041] Perform mean processing on multiple frames of the initial video features to generate the video features.

[0042] An embodiment of the present application further provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device runs, the processor communicates with the memory through the bus. When the machine-readable instructions are executed by the processor, the steps of the above-mentioned speech synthesis method for fusing visual information are executed.

[0043] An embodiment of the present application further provides a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is run by a processor, the steps of the above-mentioned speech synthesis method for fusing visual information are executed.

[0044] The speech synthesis method, device, electronic device, and medium for fusing visual information provided by the embodiments of the present application. The speech synthesis method includes: extracting text features of target text information based on a text encoder; performing feature extraction processing, vector quantization encoding processing, and cross-attention processing on target video information to determine video features in the target video information; performing cross-attention processing on the text features and the video features to determine joint features; inputting the joint features into a pre-trained speech generation model, performing modeling processing on the joint features to generate video text features, and then performing random prosody prediction processing, feature augmentation processing, and decoding processing on the video text features to generate a synthetic speech corresponding to the target text information; wherein, the emotional features of the synthetic speech are consistent with the emotional features of the target video information. By fusing the information in the video, a more expressive speech is generated. This method not only solves the deficiencies of the prior art in emotional control, but also can achieve high-quality speech synthesis using a small amount of sample data.

[0045] To make the above objects, features, and advantages of the present application more obvious and understandable, the following specifically enumerates preferred embodiments and cooperates with the accompanying drawings for detailed description as follows. Description of the Drawings

[0046] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0047] Figure 1 One of the flowcharts of a speech synthesis method for fusing visual information provided by the embodiments of the present application;

[0048] Figure 2 Another flowchart of a speech synthesis method for fusing visual information provided by the embodiments of the present application;

[0049] Figure 3 One of the schematic structural diagrams of a voice synthesis device integrating visual information provided by an embodiment of the present application;

[0050] Figure 4 Another schematic structural diagram of a voice synthesis device integrating visual information provided by an embodiment of the present application;

[0051] Figure 5 Schematic structural diagram of an electronic device provided by an embodiment of the present application. Specific implementation manners

[0052] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Usually, the components of the embodiments of the present application described and illustrated herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application to be protected, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, every other embodiment obtained by those skilled in the art without creative efforts belongs to the scope of protection of the present application.

[0053] First, the applicable application scenarios of the present application are introduced. The present application can be applied to the field of voice synthesis technology.

[0054] It has been found through research that at the present stage, the voice synthesis technology can be close to the level of real people under the condition of collecting a large amount of data for training. However, when training for a specific person's model, there are often problems such as difficulty in collecting the data of that person and low data quality, and there is no other emotional voice in the collected specific person's data. In such a situation, there are still great challenges in the emotional voice synthesis technology. The existing voice synthesis technology relies on a large amount of high-quality data recorded in a recording studio, and different people need to record a large amount of data of the corresponding person. In addition, if it is necessary to realize the emotional voice synthesis of a specific person, a large amount of voice data of the specific person under different emotions needs to be recorded in advance, so a lot of energy will be consumed. Since the types of synthesized emotions are limited by the recorded data, in the case of only a very small amount of sample data of a specific person, it is difficult for the existing voice to model that person and it is difficult to form a high-quality emotion-controllable voice synthesis ability.

[0055] Based on this, the embodiments of the present application provide a speech synthesis method that fuses visual information to generate more expressive speech by integrating information in the video. This method not only addresses the deficiencies of the prior art in terms of emotion control but also enables high-quality speech synthesis using a small amount of sample data.

[0056] Please refer to Figure 1 , Figure 1 which is one of the flowcharts of a speech synthesis method that fuses visual information provided by the embodiments of the present application. As Figure 1 shown in

[0057] S101: Extract the text features of the target text information based on a text encoder.

[0058] In this step, first convert the target text information into phoneme information, and use a text editor to perform text feature encoding processing on the phoneme information to obtain the text features of the target text information.

[0059] S102: Perform feature extraction processing, vector quantization encoding processing, and cross-attention processing on the target video information to determine the video features in the target video information.

[0060] In this step, perform feature extraction processing, vector quantization encoding processing, and cross-attention processing on the target video information to determine the video features in the target video information.

[0061] In a possible implementation, the performing feature extraction processing, vector quantization encoding processing, and cross-attention processing on the target video information to determine the video features in the target video information includes:

[0062] A: Mark the target video information based on a vector quantization encoder, and add random noise to the marked target video information.

[0063] Here, perform quantization marking on the target video information according to the vector quantization encoder to obtain discrete vectors, and add random noise to the quantized and marked target video information.

[0064] B: Input the target video information with added noise into a fusion diffusion model for sampling, and extract the first visual feature map of each video frame.

[0065] Here, input the target video information with added noise into a fusion diffusion model for sampling, and extract the first visual feature map of each video frame from the second upsampling module of the fusion diffusion model.

[0066] Among them, the fusion diffusion model gradually adds random noise to the data by defining a Markov chain of diffusion steps and then learns the inverse diffusion process to construct the required data samples from the noise. The fusion diffusion model is a latent variable model that uses a fixed Markov chain to map into the latent space and can generate corresponding video features from audio features through the diffusion process. By designing the fusion diffusion model, this system models the relationship between audio modality information and video modality information to achieve the purpose of generating corresponding video information from audio information.

[0067] C: Feature extraction processing is performed on the target video information based on the multimodal neural network model to obtain the visual feature map of each video frame, and multi-layer perceptron projection processing is performed on the visual feature map of each video frame to generate a second visual feature map consistent with the dimension of the first visual feature map.

[0068] Here, feature extraction processing is performed on the target video information based on the multimodal neural network model to obtain the visual feature map of each video frame, and multi-layer perceptron projection processing is performed on the visual feature map of each video frame to generate a second visual feature map consistent with the dimension of the first visual feature map.

[0069] Among them, the multimodal neural network model is the CLIP model.

[0070] D: The first visual feature map of each video frame and the corresponding second visual feature map are combined and position-encoded to determine the initial video feature of each video frame.

[0071] Here, the first visual feature map of each video frame and the corresponding second visual feature map are combined to obtain a combined feature map, and a set of learnable position encodings are added to the combined feature map to further enhance the positioning ability, obtaining the initial video feature of each video frame.

[0072] E: Mean processing is performed on multiple frames of the initial video features to generate the video features.

[0073] In a specific embodiment, Step 1: Train a fusion diffusion model based on a diffusion model, and retain and freeze the weights of the U-Net therein; Step 2: Use a pre-trained CLIP model and retain and freeze its parameters; Step 3: Tokenize the target video information through a vector quantization (VQ) encoder, add random noise, and input it into the U-Net model saved in Step 1. Extract the visual feature map from the second upsampling block in the U-Net; Step 4: Use the CLIP model in Step 2 to extract visual features, project them through a multi-layer perceptron, and keep the dimension consistent with the visual feature dimension extracted in Step 3. Finally, the CLIP visual feature map and the feature map extracted by the U-Net model are combined together and a set of learnable position encodings are added to further enhance the localization ability, finally forming video features. Step 5: Extract visual features for each video frame image of the input video through Steps 3 and 4, and finally average them to obtain the video features of the input video.

[0074] S103: Perform cross-attention processing on the text features and the video features to determine joint features.

[0075] In this step, cross-attention processing is performed on the text features and the video features to determine joint features.

[0076] Here, the joint features that fuse video features have high expressiveness and can generate emotion features such as sadness, happiness, and joy.

[0077] S104: Input the joint features into a pre-trained speech generation model, perform modeling processing on the joint features to generate video text features, and then perform random prosody prediction processing, feature augmentation processing, and decoding processing on the video text features to generate the synthetic speech corresponding to the target text information.

[0078] In this step, the joint features are input into the speech generation model, modeling processing is performed on the joint features to generate video text features, and then random prosody prediction processing, feature augmentation processing, and decoding processing are performed on the video text features to generate the synthetic speech corresponding to the target text information.

[0079] Among them, the text information of the synthetic speech is the target text information, and the emotion feature of the synthetic speech is the emotion feature of the target video information. Input the text to be inferred and the corresponding video information, and output the audio corresponding to the text to be inferred, and this audio has the information presented by the video, such as happy emotions and sad emotions.

[0080] Here, in the process of inputting the joint features into the pre-trained speech generation model, it is necessary to determine the speaker embedding vector of the synthesized speech, so as to perform speech synthesis generation on the target text information according to the emotional features of the voice features carried by the speaker embedding vector carrying the target video information. Among them, the speaker embedding vector can be screened in the speaker database.

[0081] In a possible implementation manner, the method of modeling the joint features to generate video text features, and then performing random prosody prediction processing, feature augmentation processing, and decoding processing on the video text features to generate the synthesized speech corresponding to the target text information includes:

[0082] (1): Based on the self-attention mechanism, position encoding mechanism, and multi-head attention mechanism in the video text joint encoder of the speech generation model, perform modeling processing on the joint features to capture the cross-modal relationship between the text features and the video features, and generate video text features.

[0083] Here, based on the self-attention mechanism, position encoding mechanism, and multi-head attention mechanism in the video text joint encoder of the speech generation model, perform modeling processing on the joint features to capture the cross-modal relationship between the text features and the video features, and generate video text features.

[0084] Among them, the video text joint encoder can be a transformer network layer.

[0085] In a possible implementation manner, the method of performing modeling processing on the joint features based on the self-attention mechanism, position encoding mechanism, and multi-head attention mechanism in the video text joint encoder to capture the cross-modal relationship between the text features and the video features, and generate video text features includes:

[0086] a: Process the joint features based on the self-attention mechanism to generate the joint features after self-attention processing.

[0087] Here, process the joint features according to the self-attention mechanism to generate the joint features after self-attention processing.

[0088] b: Perform position encoding processing on the time sequence information of the video frames and the position information of the target text information in the joint features after self-attention processing to generate the joint features after position encoding processing.

[0089] Here, perform position encoding processing on the time sequence information of the video frames and the position information of the target text information in the joint features after self-attention processing to generate the joint features after position encoding processing.

[0090] c: Perform multi - angle modeling processing on the joint features after processing the positional encoding based on the multi - head attention mechanism to generate video - text features.

[0091] Here, perform multi - angle modeling processing on the joint features after processing the positional encoding based on the multi - head attention mechanism to generate video - text features.

[0092] (2): Input the video - text features into the random rhythm predictor of the speech generation model, perform random rhythm prediction processing and feature augmentation processing on the video - text features to generate the augmented video - text features.

[0093] Here, input the video - text features into the random rhythm predictor of the speech generation model, perform random rhythm prediction processing and feature augmentation processing on the video - text features to generate the augmented video - text features.

[0094] In a possible implementation manner, input the video - text features into the random rhythm predictor of the speech generation model, perform random rhythm prediction processing and high - dimensional feature augmentation processing on the video - text features to generate the augmented video - text features, including:

[0095] i: Based on the emotion features and context relationships included in the video features in the video - text features, predict the duration of each phoneme in the video - text features.

[0096] Here, predict the duration of each phoneme in the video - text features according to the emotion features and context relationships included in the video features.

[0097] Here, the core of the random rhythm predictor is a regression model for predicting the duration of each phoneme. The specific steps are as follows: Perform non - linear transformation on the video - text features through a multi - layer perceptron (MLP) to obtain intermediate features, and perform probability distribution modeling on the duration of each phoneme of the intermediate features. Assume that the duration of each phoneme follows a normal distribution, and randomly sample from the normal distribution to obtain the specific duration. This process introduces randomness, making the generated speech have diverse rhythms. The random rhythm predictor not only depends on the text features but also combines the emotion features in the video features. The specific impact is as follows: The emotion information in the video features will affect the output of the random duration predictor. For example: If the person in the video shows "excitement", the predicted rhythm may be faster, resulting in a shorter duration of phonemes. If the person in the video shows "sadness", the predicted rhythm may be slower, resulting in a longer duration of phonemes.

[0098] ii: Interpolate the features corresponding to each phoneme in the video text features according to the corresponding duration to generate the augmented video text features.

[0099] Here, interpolate the features corresponding to each phoneme in the video text features according to the corresponding duration to generate the augmented video text features.

[0100] (3): Input the duration of the phoneme, the target person's speech embedding vector, and the augmented video text features in the video text features into the decoder of the speech generation model for speech synthesis processing to generate the synthetic speech corresponding to the target text information.

[0101] Here, input the duration of the phoneme, the target person's speech embedding vector, and the augmented video text features in the video text features into the decoder of the speech generation model for decoding discrete vectors to generate the synthetic speech corresponding to the target text information.

[0102] Among the specific embodiments, the video-text joint encoder plays a crucial role in this application. It fuses video features and text features to form a "joint feature" that combines the information of both, thereby guiding the subsequent speech synthesis process. The following is a detailed analysis of its specific working principle: Video features are extracted from the input video and contain expressive information such as emotions and tones. These features are extracted through pre-trained CLIP models and diffusion models, which can capture the emotional states (such as happiness and sadness) and tone characteristics in the video. Text features are extracted from the input text and represent the content and structural information of the text. First, the text is converted into a sequence of phonemes, and then a high-dimensional vector representation is generated through a text encoder to describe the meaning of the text. To effectively combine the video features and text features, a cross-attention module is used to obtain a joint feature that contains both the content information of the text and the emotional and tone information of the video. The obtained joint feature is further input into a multi-layer perceptron (MLP) for non-linear transformation to enhance the expressive power of the feature. The feature processed by the MLP is then input into the video-text joint encoder part of the speech generation model for processing to obtain video-text features. In the speech generation model, the duration of each phoneme is predicted based on the video-text features, thereby controlling the rhythm and intonation of the speech. The video-text features are extended to a higher-dimensional representation to match the resolution of the audio. The extended features are input into the decoder to generate the final audio waveform. In this application, the video-text joint encoder effectively fuses video features and text features through a cross-attention mechanism, forming a comprehensive joint feature. This joint feature guides the subsequent speech synthesis process, making the generated speech not only conform to the text content but also reflect the emotional and tone characteristics in the video. This method significantly improves the expressiveness and flexibility of speech synthesis.

[0103] In a possible implementation manner, the speech generation model is determined through the following steps:

[0104] I: Replace the text encoder in the original VITS model with a video-text joint encoder to obtain an initial speech generation model.

[0105] Here, VITS is a parallel end-to-end TTS method. In this application, to implement speech synthesis that integrates visual information, the text encoder in the original VITS model is replaced with a video-text joint encoder to obtain an initial speech generation model.

[0106] II: Input the sample joint feature determined based on the sample video information and sample text information into the initial speech generation model for speech synthesis processing to generate the predicted synthetic speech corresponding to the sample text information.

[0107] Here, the speech synthesis process in the initial speech generation model is consistent with the steps of the above-mentioned speech generation model for generating synthesized speech, and this part will not be elaborated here.

[0108] III: Determine the loss value of the initial speech generation model based on the predicted synthesized speech and the actual synthesized speech of the sample text information.

[0109] Here, calculate the predicted synthesized speech and the actual synthesized speech according to the loss function calculation formula to determine the loss value of the initial speech generation model.

[0110] IV: Iteratively train the initial speech generation model based on the loss value and the training samples to generate the speech generation model.

[0111] Here, if the loss value is less than or equal to the preset threshold, the initial speech generation model is used as the speech generation model. If the loss value is greater than the preset threshold, the network parameters of the initial speech generation model are changed, and the initial speech generation model with the changed network parameters is continuously trained using the training samples.

[0112] Further, please refer to Figure 2 , Figure 2 which is the second flowchart of a speech synthesis method integrating visual information provided by an embodiment of the present application. As Figure 2 shown, the joint feature constructed by the text feature and the audio feature is input into the video text joint encoder of the speech generation model for processing to obtain the video text feature. The video text feature is input into the random prosody predictor of the speech generation model to obtain the duration of the phonemes in the video text feature and the expanded video text feature. The duration of the phonemes in the video text feature, the target person's speech embedding vector, and the expanded video text feature are input into the decoder of the speech generation model for speech synthesis processing to generate the synthesized speech corresponding to the target text information.

[0113] In this application, a pre-trained video feature extractor is used to extract video features from a video. These video features include features such as emotion and tone, which are used to supervise the expressiveness of the generated audio. The text is first converted into phonemes and then passed through a text encoder to obtain text encoding features. The text encoding features and the video features pass through a cross-attention module to obtain video-text joint features. Finally, through a multilayer perceptron (MLP), they are input into the video-text joint encoder of the speech generation model to further model the features of the video and text. Then, the modeled video-text features are processed to obtain synthesized speech. By fusing the information in the video, more expressive speech is generated. This method not only solves the deficiencies of the prior art in terms of emotion control but also enables high-quality speech synthesis using a small amount of sample data.

[0114] A speech synthesis method for fusing visual information provided in an embodiment of this application, the speech synthesis method includes: extracting text features of target text information based on a text encoder; performing feature extraction processing, vector quantization encoding processing, and cross-attention processing on target video information to determine video features in the target video information; performing cross-attention processing on the text features and the video features to determine joint features; inputting the joint features into a pre-trained speech generation model to perform modeling processing on the joint features to generate video-text features, and then performing random prosody prediction processing, feature augmentation processing, and decoding processing on the video-text features to generate synthesized speech corresponding to the target text information; where the emotion features of the synthesized speech are consistent with the emotion features of the target video information. By fusing the information in the video, more expressive speech is generated. This method not only solves the deficiencies of the prior art in terms of emotion control but also enables high-quality speech synthesis using a small amount of sample data.

[0115] Please refer to Figure 3 、 Figure 4 , Figure 3 which is one of the structural schematic diagrams of a speech synthesis device for fusing visual information provided in an embodiment of this application; Figure 4 which is the second structural schematic diagram of a speech synthesis device for fusing visual information provided in an embodiment of this application. As Figure 3 shown in

[0116] A text feature extraction module 310, configured to extract text features of target text information based on a text encoder;

[0117] The video feature extraction module 320 is used to perform feature extraction processing, vector quantization encoding processing, and cross-attention processing on the target video information to determine the video features in the target video information;

[0118] The joint module 330 is used to perform cross-attention processing on the text features and the video features to determine the joint features;

[0119] The speech generation module 340 is used to input the joint features into a pre-trained speech generation model, perform modeling processing on the joint features to generate video text features, and then perform random prosody prediction processing, feature augmentation processing, and decoding processing on the video text features to generate the synthetic speech corresponding to the target text information; wherein, the emotional feature of the synthetic speech is consistent with the emotional feature of the target video information.

[0120] Further, when the video feature extraction module 320 is used to perform feature extraction processing, vector quantization encoding processing, and cross-attention processing on the target video information to determine the video features in the target video information, the video feature extraction module 320 is further used to:

[0121] Mark the target video information based on a vector quantization encoder, and add random noise to the marked target video information;

[0122] Input the target video information with added noise into a fusion diffusion model for sampling, and extract the first visual feature map of each video frame;

[0123] Perform feature extraction processing on the target video information based on a multi-modal neural network model to obtain the visual feature map of each video frame, and perform multi-layer perceptron projection processing on the visual feature map of each video frame to generate a second visual feature map with the same dimension as the first visual feature map;

[0124] Perform combination processing and position encoding processing on the first visual feature map of each video frame and the corresponding second visual feature map to determine the initial video feature of each video frame;

[0125] Perform mean processing on multiple frames of the initial video features to generate the video features.

[0126] Further, when the speech generation module 340 is used to capture the cross-modal relationship between the text features and the video features by performing modeling processing on the joint features based on the self-attention mechanism, position encoding mechanism, and multi-head attention mechanism in the video text joint encoder to generate video text features, the speech generation module 340 specifically uses:

[0127] Process the joint features based on the self-attention mechanism to generate the joint features after self-attention processing;

[0128] Perform position encoding processing on the position information of the time sequence information of the video frames and the target text information among the joint features after self-attention processing to generate the joint features after position encoding processing;

[0129] Perform multi-angle modeling processing on the joint features after position encoding processing based on the multi-head attention mechanism to generate video-text features.

[0130] Further, when the speech generation module 340 is used to input the video-text features into the random prosody predictor of the speech generation model to perform random prosody prediction processing and high-dimensional feature expansion processing on the video-text features to generate the expanded video-text features, the speech generation module 340 is specifically used for:

[0131] Predict the duration of each phoneme in the video-text features based on the emotion features and context relationships included in the video features in the video-text features;

[0132] Perform feature interpolation processing on the features corresponding to each phoneme in the video-text features according to the corresponding duration to generate the expanded video-text features.

[0133] Further, as Figure 4 shown, the speech synthesis device 300 integrating visual information further includes a model training module 350, and the model training module 350 determines the speech generation model through the following steps:

[0134] Replace the text encoder in the original VITS model with a video-text joint encoder to obtain an initial speech generation model;

[0135] Input the sample joint features determined based on the sample video information and the sample text information into the initial speech generation model for speech synthesis processing to generate the predicted synthesized speech corresponding to the sample text information;

[0136] Determine the loss value of the initial speech generation model based on the predicted synthesized speech and the actual synthesized speech of the sample text information;

[0137] Iteratively train the initial speech generation model based on the loss value and the training samples to generate the speech generation model.

[0138] A speech synthesis device integrating visual information provided by an embodiment of the present application. The speech synthesis device includes: a text feature extraction module for extracting text features of target text information based on a text encoder; a video feature extraction module for performing feature extraction processing, vector quantization encoding processing, and cross-attention processing on target video information to determine video features in the target video information; a joint module for performing cross-attention processing on the text features and the video features to determine joint features; a speech generation module for inputting the joint features into a pre-trained speech generation model, performing modeling processing on the joint features to generate video text features, and then performing random prosody prediction processing, feature augmentation processing, and decoding processing on the video text features to generate a synthetic speech corresponding to the target text information; wherein, the emotional feature of the synthetic speech is consistent with the emotional feature of the target video information. By integrating information in the video, more expressive speech is generated. This method not only solves the deficiencies of the prior art in terms of emotion control, but also enables high-quality speech synthesis using a small amount of sample data.

[0139] Please refer to Figure 5 , Figure 5 which is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As Figure 5 shown in the figure, the electronic device 500 includes a processor 510, a memory 520, and a bus 530.

[0140] The memory 520 stores machine-readable instructions executable by the processor 510. When the electronic device 500 runs, the processor 510 communicates with the memory 520 through the bus 530. When the machine-readable instructions are executed by the processor 510, the steps of the speech synthesis method integrating visual information in the method embodiments as described above Figure 1 and Figure 2 can be executed. The specific implementation manner can be referred to the method embodiments and will not be elaborated here.

[0141] An embodiment of the present application also provides a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is run by a processor, the steps of the speech synthesis method integrating visual information in the method embodiments as described above Figure 1 and Figure 2 can be executed. The specific implementation manner can be referred to the method embodiments and will not be elaborated here.

[0142] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0143] In several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical functional division, and there may be other division methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling, direct coupling, or communication connection to each other can be through some communication interfaces. The indirect coupling or communication connection of the devices or units can be in electrical, mechanical, or other forms.

[0144] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0145] In addition, in each embodiment of the present application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0146] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0147] Finally, it should be noted that the above-described embodiments are only specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, rather than limiting it. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that any person skilled in the art within the technical scope disclosed by the present application can still modify the technical solutions recorded in the foregoing embodiments or can easily think of changes, or perform equivalent replacements on some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A speech synthesis method integrating visual information, characterized in that The speech synthesis method includes: extracting text features of target text information based on a text encoder; performing feature extraction processing, vector quantization encoding processing, and cross-attention processing on target video information to determine video features in the target video information; performing cross-attention processing on the text features and the video features to determine joint features; inputting the joint features into a pre-trained speech generation model, performing modeling processing on the joint features to generate video text features, and then performing random prosody prediction processing, feature augmentation processing, and decoding processing on the video text features to generate a synthesized speech corresponding to the target text information; wherein, the emotion feature of the synthesized speech is consistent with the emotion feature of the target video information; The performing modeling processing on the joint features to generate video text features, and then performing random prosody prediction processing, feature augmentation processing, and decoding processing on the video text features to generate a synthesized speech corresponding to the target text information includes: performing modeling processing on the joint features based on the self-attention mechanism, position encoding mechanism, and multi-head attention mechanism in the video text joint encoder of the speech generation model to capture the cross-modal relationship between the text features and the video features, and generating video text features; inputting the video text features into the random prosody predictor of the speech generation model, and performing random prosody prediction processing and feature augmentation processing on the video text features to generate the augmented video text features; inputting the duration of phonemes, the target person's speech embedding vector, and the augmented video text features in the video text features into the decoder of the speech generation model for speech synthesis processing to generate a synthesized speech corresponding to the target text information.

2. The speech synthesis method according to claim 1, wherein The performing feature extraction processing, vector quantization encoding processing, and cross-attention processing on target video information to determine video features in the target video information includes: marking the target video information based on a vector quantization encoder, and adding random noise to the marked target video information; inputting the target video information with added noise into a fusion diffusion model for sampling to extract a first visual feature map of each video frame; performing feature extraction processing on the target video information based on a multi-modal neural network model to obtain a visual feature map of each video frame, and performing multi-layer perceptron projection processing on the visual feature map of each video frame to generate a second visual feature map with the same dimension as the first visual feature map; performing combination processing and position encoding processing on the first visual feature map of each video frame and the corresponding second visual feature map to determine an initial video feature of each video frame; performing mean processing on multiple frames of the initial video features to generate the video features.

3. The speech synthesis method according to claim 1, wherein The performing modeling processing on the joint features based on the self-attention mechanism, position encoding mechanism, and multi-head attention mechanism in the video text joint encoder to capture the cross-modal relationship between the text features and the video features, and generating video text features includes: Process the joint features based on the self-attention mechanism to generate the joint features after self-attention processing; Perform position encoding processing on the position information of the time sequence information of the video frames and the target text information among the joint features after self-attention processing to generate the joint features after position encoding processing; Perform multi-angle modeling processing on the joint features after position encoding processing based on the multi-head attention mechanism to generate video-text features.

4. The speech synthesis method according to claim 3, characterized in that Input the video-text features into the random prosody predictor of the speech generation model to perform random prosody prediction processing and high-dimensional feature expansion processing on the video-text features to generate the expanded video-text features, including: Predict the duration of each phoneme in the video-text features based on the emotion features and context relationships included in the video features in the video-text features; Perform feature interpolation processing on the features corresponding to each phoneme in the video-text features according to the corresponding duration to generate the expanded video-text features.

5. The speech synthesis method according to claim 1, wherein Determine the speech generation model through the following steps: Replace the text encoder in the original VITS model with a video-text joint encoder to obtain an initial speech generation model; Input the sample joint features determined based on the sample video information and the sample text information into the initial speech generation model for speech synthesis processing to generate the predicted synthesized speech corresponding to the sample text information; Determine the loss value of the initial speech generation model based on the predicted synthesized speech and the actual synthesized speech of the sample text information; Iteratively train the initial speech generation model based on the loss value and training samples to generate the speech generation model.

6. A speech synthesis device integrating visual information, characterized in that, The speech synthesis device includes: A text feature extraction module for extracting the text features of the target text information based on a text encoder; A video feature extraction module for performing feature extraction processing, vector quantization encoding processing, and cross-attention processing on the target video information to determine the video features in the target video information; A joint module for performing cross-attention processing on the text features and the video features to determine joint features; A speech generation module for inputting the joint features into a pre-trained speech generation model, performing modeling processing on the joint features to generate video-text features, and then performing random prosody prediction processing, feature expansion processing, and decoding processing on the video-text features to generate the synthesized speech corresponding to the target text information; wherein, the emotion features of the synthesized speech are consistent with the emotion features of the target video information; When the speech generation module is used for performing modeling processing on the joint features to generate video-text features, and then performing random prosody prediction processing, feature expansion processing, and decoding processing on the video-text features to generate the synthesized speech corresponding to the target text information, the speech generation module is specifically used for: Model the joint features through the self-attention mechanism, positional encoding mechanism, and multi-head attention mechanism in the video-text joint encoder based on the speech generation model to capture the cross-modal relationship between the text features and the video features, and generate video-text features; Input the video-text features into the random prosody predictor of the speech generation model, perform random prosody prediction processing and feature augmentation processing on the video-text features, and generate the augmented video-text features; Input the duration of phonemes, the target person's speech embedding vector, and the augmented video-text features in the video-text features into the decoder of the speech generation model for speech synthesis processing to generate the synthetic speech corresponding to the target text information.

7. The speech synthesis device according to claim 6, wherein When the video feature extraction module is used to perform feature extraction processing, vector quantization encoding processing, and cross-attention processing on the target video information to determine the video features in the target video information, the video feature extraction module is further used for: Mark the target video information based on the vector quantization encoder, and add random noise to the marked target video information; Input the target video information with added noise into the fusion diffusion model for sampling, and extract the first visual feature map of each video frame; Perform feature extraction processing on the target video information based on the multi-modal neural network model to obtain the visual feature map of each video frame, and perform multi-layer perceptron projection processing on the visual feature map of each video frame to generate a second visual feature map with the same dimension as the first visual feature map; Perform combination processing and positional encoding processing on the first visual feature map of each video frame and the corresponding second visual feature map to determine the initial video feature of each video frame; Perform mean processing on multiple frames of the initial video features to generate the video features.

8. An electronic device, characterized in that, Including: A processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device runs, the processor communicates with the memory through the bus. When the machine-readable instructions are run by the processor, the steps of the speech synthesis method for fusing visual information as described in any one of claims 1 to 5 are executed.

9. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium. When the computer program is run by the processor, the steps of the speech synthesis method for fusing visual information as described in any one of claims 1 to 5 are executed.

Images