Speech synthesis method, model training method and device, computer equipment and vehicle

Abstract

Provided are a speech synthesis method, a model training method and apparatus, a computer device, and a vehicle. The speech synthesis method includes: obtaining a target image and target text for speech synthesis, wherein the target image includes a target object as a virtual character; inputting the target image into a trained sound feature extraction model to obtain sound features of the target object output by the sound feature extraction model; and inputting the sound features and target text into the trained speech synthesis model to obtain a target speech corresponding to the target text output by the speech synthesis model.

Description

Technical Field

[0001] The present disclosure relates to the field of audio processing technology, and in particular to a speech synthesis method and apparatus, a training method and apparatus for a sound feature extraction model, a training method and apparatus for a text-to-speech conversion (TTS) model, a computer device, a vehicle, a computer-readable storage medium, and a computer program product. Background Art

[0002] Speech synthesis, also known as text-to-speech (TTS), refers to the technology of using electronic computers or some specialized devices to generate artificial speech. It can convert any text information into voice information and output it.

[0003] Speech synthesis technology has been widely used in human-computer interaction scenarios. For example, electronic devices such as in-vehicle devices, smart home devices, and mobile terminals can use speech synthesis technology to generate audio from text information and broadcast the generated audio to the user through speakers, thus enabling voice interaction between the user and the electronic device. Summary of the Invention

[0004] It would be advantageous to provide a mechanism that alleviates, mitigates, or even eliminates one or more of the above-mentioned problems.

[0005] According to one aspect of the present disclosure, a speech synthesis method is provided, comprising: obtaining a target image and a target text for speech synthesis, wherein the target image includes a target object as a virtual character image; inputting the target image into a trained sound feature extraction model to obtain the sound features of the target object output by the sound feature extraction model; and inputting the sound features and the target text into a trained speech synthesis model to obtain a target speech corresponding to the target text output by the speech synthesis model.

[0006] According to one aspect of the present disclosure, a training method for a sound feature extraction model is provided, comprising: obtaining a sample image and annotation data of the sample image, wherein the sample image includes a sample object as a virtual character image, and the annotation data includes sample sound features of the sample object; inputting the sample image into the sound feature extraction model to obtain predicted sound features output by the sound feature extraction model; determining a loss value of the sound feature extraction model based on the predicted sound features and the sample sound features; and adjusting parameters of the sound feature extraction model based on the loss value.

[0007] According to one aspect of the present disclosure, a training method for a text-to-speech (TTS) model is provided, wherein the TTS model includes a sound feature extraction model and a speech synthesis model, wherein the speech synthesis model includes a spectrum synthesis module and a vocoding module. The method includes: acquiring a sample image, sample text, and annotation data, wherein the sample image includes a sample object as a virtual character image, and the annotation data includes a sample speech of the sample object corresponding to the sample text; extracting sample speech spectrum features of the sample speech; jointly training the sound feature extraction model and the spectrum synthesis module based on the sample image, the sample text, and the sample speech spectrum features; and training the vocoding module based on the sample image, the sample text, the sample speech, and the trained sound feature extraction model and spectrum synthesis module.

[0008] According to one aspect of the present disclosure, a speech synthesis device is provided, comprising: an acquisition unit configured to acquire a target image and a target text for speech synthesis, wherein the target image includes a target object as a virtual character image; an extraction unit configured to input the target image into a trained sound feature extraction model to obtain the sound features of the target object output by the sound feature extraction model; and a synthesis unit configured to input the sound features and the target text into a trained speech synthesis model to obtain a target speech corresponding to the target text output by the speech synthesis model.

[0009] According to one aspect of the present disclosure, a training device for a sound feature extraction model is provided, comprising: an acquisition unit configured to acquire a sample image and annotation data of the sample image, wherein the sample image includes a sample object as a virtual character image, and the annotation data includes a sample sound feature of the sample object; an extraction unit configured to input the sample image into the sound feature extraction model to obtain a predicted sound feature output by the sound feature extraction model; a determination unit configured to determine a loss value of the sound feature extraction model based on the predicted sound feature and the sample sound feature; and an adjustment unit configured to adjust parameters of the sound feature extraction model based on the loss value.

[0010] According to one aspect of the present disclosure, a training device for a text-to-speech (TTS) model is provided, wherein the TTS model includes a sound feature extraction model and a speech synthesis model, wherein the speech synthesis model includes a spectrum synthesis module and a vocoding module. The method includes: an acquisition unit configured to acquire sample images, sample texts, and annotation data, wherein the sample images include sample objects as virtual character images, and the annotation data include sample speech of the sample objects corresponding to the sample texts; an extraction unit configured to extract sample speech spectrum features of the sample speech; a first training unit configured to jointly train the sound feature extraction model and the spectrum synthesis module based on the sample images, the sample texts, and the sample speech spectrum features; and a second training unit configured to train the vocoding module based on the sample images, the sample texts, the sample speech, and the trained sound feature extraction model and spectrum synthesis module.

[0011] According to one aspect of the present disclosure, a computer device is provided, comprising: at least one processor; and at least one memory having a computer program stored thereon. When the computer program is executed by the at least one processor, the at least one processor is caused to perform any of the above methods.

[0012] According to one aspect of the present disclosure, a vehicle is provided, comprising the apparatus of any one of the above aspects or the above computer device.

[0013] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor is caused to perform any of the above methods.

[0014] According to one aspect of the present disclosure, a computer program product is provided, comprising a computer program. When the computer program is executed by a processor, the processor is caused to perform any of the above methods.

[0015] According to the embodiments of the present disclosure, a trained sound feature extraction model is used to generate sound features corresponding to a virtual character image, and voice is generated based on the sound features, thereby achieving real-time and efficient customization of the virtual character's voice, meeting the user's personalized needs for the virtual character's voice, and improving the diversity and flexibility of human-computer voice interaction.

[0016] These and other aspects of the disclosure will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0018] Figure 1 is a schematic diagram illustrating an example system in which the various methods described herein may be implemented according to an exemplary embodiment;

[0019] Figure 2 is a flowchart illustrating a speech synthesis method according to an exemplary embodiment;

[0020] Figure 3 is a schematic block diagram illustrating a sound feature extraction model and a speech synthesis model according to an exemplary embodiment;

[0021] Figure 4 is a flow chart illustrating a method for training a sound feature extraction model according to an exemplary embodiment;

[0022] Figure 5 is a schematic diagram illustrating a training process of a sound feature extraction model according to an exemplary embodiment;

[0023] Figure 6 is a flowchart illustrating a method of training a TTS model according to an exemplary embodiment;

[0024] Figure 7 is a schematic block diagram illustrating a TTS model according to an exemplary embodiment;

[0025] Figure 8 is a schematic diagram illustrating a training process of a sound feature extraction model and a spectrum synthesis module according to an exemplary embodiment;

[0026] Figure 9 is a schematic diagram illustrating a training process of a vocoding module according to an exemplary embodiment;

[0027] Figure 10 is a schematic block diagram illustrating a speech synthesis apparatus according to an exemplary embodiment;

[0028] Figure 11 is an exemplary block diagram illustrating a training apparatus for a sound feature extraction model according to an exemplary embodiment;

[0029] Figure 12 is an exemplary block diagram illustrating a training apparatus for a TTS model according to an exemplary embodiment; and

[0030] Figure 13 is a block diagram illustrating an exemplary computer device that can be used with the exemplary embodiments. DETAILED DESCRIPTION

[0031] In this disclosure, unless otherwise specified, the use of terms such as "first" and "second" to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements. Such terms are only used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of the element, while in some cases, based on the context of the description, they may also refer to different instances.

[0032] The terms used in the description of the various examples described in this disclosure are for the purpose of describing specific examples only and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. As used herein, the term "plurality" means two or more, and the term "based on" should be interpreted as "based at least in part on". In addition, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations.

[0033] Before introducing exemplary embodiments of the present disclosure, several terms used herein are first explained.

[0034] 1. Speech Synthesis

[0035] Also known as Text To Speech (TTS), it refers to the technology of using electronic computers and some special devices to generate artificial speech, which can convert any text information into voice information and output it.

[0036] 2. Virtual Characters

[0037] The speech generated by speech synthesis technology is the speech of a simulated character, and the simulated character is called a virtual character.

[0038] In related technologies, TTS models typically provide users with a fixed set of virtual character voices (e.g., adult female, adult male, girl, boy, etc.). Users can only select one of these virtual character voices for speech synthesis, and cannot customize the virtual character's voice. This results in a monotonous and boring voice for human-computer voice interaction, reducing the user experience.

[0039] To address the above issues, embodiments of the present disclosure provide a speech synthesis method that enables real-time and efficient customization of virtual character voices, meeting users' personalized needs for virtual character voices and improving the diversity and flexibility of human-computer voice interaction. Exemplary embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0040] Figure 1 is a schematic diagram illustrating an example system 100 in which the various methods described herein may be implemented, according to an exemplary embodiment.

[0041] refer to Figure 1 The system 100 includes an in-vehicle system 110, a server 120, and a network 130 that communicatively couples the in-vehicle system 110 and the server 120.

[0042] The in-vehicle system 110 includes a display 114 and an application (APP) 112 that can be displayed via the display 114. The application 112 can be an application installed by default on the in-vehicle system 110 or downloaded and installed by the user 102, or a mini-program that is a lightweight application. If the application 112 is a mini-program, the user 102 can directly run the application 112 on the in-vehicle system 110 by searching for the application 112 in the host application (e.g., by the name of the application 112, etc.) or scanning a graphical code (e.g., a barcode, a QR code, etc.) of the application 112, without having to install the application 112. In some embodiments, the in-vehicle system 110 may include one or more processors and one or more memories (not shown), and the in-vehicle system 110 is implemented as an in-vehicle computer. In some embodiments, the in-vehicle system 110 may include more or fewer display screens 114 (e.g., no display screen 114), and / or one or more speakers or other human-computer interaction devices. In some embodiments, the in-vehicle system 110 may not communicate with the server 120.

[0043] Server 120 may represent a single server, a cluster of multiple servers, a distributed system, or a cloud server that provides basic cloud services (such as cloud database, cloud computing, cloud storage, cloud communication). Figure 1 1 , the server 120 is shown communicating with only one in-vehicle system 110 , but the server 120 can provide background services for multiple in-vehicle systems at the same time.

[0044] Network 130 allows for wireless communication and information exchange between vehicles and X (where "X" refers to a vehicle, road, pedestrian, or the Internet, etc.) according to agreed-upon communication protocols and data exchange standards. Examples of network 130 include a local area network (LAN), a wide area network (WAN), a personal area network (PAN), and / or a combination of communication networks such as the Internet. Network 130 can be a wired or wireless network. In one example, network 130 can be an in-vehicle network, an inter-vehicle network, and / or an in-vehicle mobile Internet.

[0045] For the purpose of the embodiments of this disclosure, Figure 1In the example, the application 112 may be an application with a voice broadcast function. For example, the application 112 may be a navigation application, which may broadcast the navigation planning route to the user. For another example, the application 112 may be an intelligent voice assistant application, which may receive the user's voice inquiry and broadcast the corresponding answer to the user. Correspondingly, the server 120 may be a server used together with the application 112. The server may provide an online speech synthesis service to the application 112 based on a deployed TTS model (such as the text-to-speech conversion TTS model of the embodiment of the present disclosure), for example, synthesizing the speech audio to be broadcast based on the corresponding text information, and sending the speech audio to the application 112 so that the application 112 broadcasts it to the user. Alternatively, the server 120 may also provide the TTS model to the in-vehicle system 110, and the application 112 running in the in-vehicle system 110 may provide local speech synthesis and speech broadcast services based on the TTS model.

[0046] Figure 2 2 is a flow chart illustrating a speech synthesis method 200 according to an exemplary embodiment. The method 200 may be implemented in a vehicle-mounted system (eg, Figure 1 , that is, the execution body of each step of the method 200 may be Figure 1 In some embodiments, the method 200 may be performed on a server (e.g., Figure 1 In some embodiments, method 200 may be performed by a combination of an in-vehicle system (e.g., in-vehicle system 110) and a server (e.g., server 120). Below, the steps of method 200 are described in detail, taking in-vehicle system 110 as an example.

[0047] refer to Figure 2 In step 210 , a target image and a target text for speech synthesis are obtained, where the target image includes a target object as a virtual character image.

[0048] In step 220 , the target image is input into the trained sound feature extraction model to obtain the sound features of the target object output by the sound feature extraction model.

[0049] In step 230 , the sound features and the target text are input into the trained speech synthesis model to obtain the target speech corresponding to the target text output by the speech synthesis model.

[0050] According to the embodiments of the present disclosure, a trained sound feature extraction model is used to generate sound features corresponding to a virtual character image, and voice is generated based on the sound features, thereby achieving real-time and efficient customization of the virtual character's voice, meeting the user's personalized needs for the virtual character's voice, and improving the diversity and flexibility of human-computer voice interaction.

[0051] According to some embodiments, the target image may be an image containing a virtual character image specified by the user (for example, a photo of a character taken by the user in real time, an image selected from stored images, etc.), or an image containing a virtual character image designed by an artist of the TTS model.

[0052] According to some embodiments, the avatar image in the target image may be a human image, such as an adult female image with short hair and yellow skin, etc. According to other embodiments, the avatar image in the target image may also be an anthropomorphic cartoon image, such as an anime character image, etc.

[0053] In the embodiments of the present disclosure, the target object refers to the object corresponding to the virtual character image, that is, the virtual character. For example, when the virtual character image included in the target image is an adult female with short hair and yellow skin, the target object is the short-haired, yellow-skinned adult female.

[0054] In an embodiment of the present disclosure, the input of the sound feature extraction model is an image, and the output is the sound features of the virtual character (target object) in the image. Accordingly, in step 220, by inputting the target image into the trained sound feature extraction model, the sound features of the target object output by the sound feature extraction model can be obtained. The sound feature is usually a vector of a fixed length, such as 128 dimensions, 256 dimensions, 1024 dimensions, etc. The sound feature can at least indicate the timbre of the target object. In some embodiments, in addition to the timbre, the sound feature can also indicate the pitch, speaking speed, etc. of the target object.

[0055] In the embodiment of the present disclosure, the input of the speech synthesis model is sound features and text, and the output is speech audio data corresponding to the text and having the sound features of the virtual character (target object) in the image. Accordingly, in step 230, by inputting the sound features of the target object and the target text of the speech to be synthesized into the trained speech synthesis model, the target speech with the sound features of the target object is output by the speech synthesis model.

[0056] It should be understood that the embodiments of the present disclosure do not limit the structures of the sound feature extraction model and the speech synthesis model.

[0057] According to some embodiments, a speech synthesis model is trained to: extract phoneme features of a target text; merge the phoneme features and the sound features (e.g., concatenate their vectors) to obtain merged features; convert the merged features into speech spectrum features; and convert the speech spectrum features into a target speech.

[0058] Figure 3 FIG2 is a schematic block diagram of a sound feature extraction model 310 and a speech synthesis model 320 according to an embodiment of the present disclosure. In an embodiment of the present disclosure, the sound feature extraction model 310 and the speech synthesis model 320 are combined to form a text-to-speech conversion TTS model (TTS model for short).

[0059] like Figure 3 As shown, the sound feature extraction model 310 takes the target image as input and outputs the sound features of the target object in the target image. The sound feature extraction model 310 can be, for example, a deep neural network (DNN) model.

[0060] The speech synthesis model 320 includes an encoding module 322 , a decoding module 324 , and a vocoding module 326 .

[0061] The Encoder module 322 takes the target text as input and preprocesses it, extracting its phoneme features. This is known as grapheme to phoneme conversion (G2P). Preprocessing includes, for example, converting Chinese characters into pinyin, removing punctuation marks (e.g., quotation marks, book title marks), and using specific delimiters to indicate pause durations between characters.

[0062] The decoding module (Decoder) 324 takes as input the combined features obtained by merging the sound features output by the sound feature extraction model 310 and the phoneme features output by the encoding module 322, and converts the combined features into speech spectral features. Typically, the decoding module 324 converts the combined features into speech spectral features based on an attention mechanism. The speech spectral features can be, for example, a Mel spectrogram.

[0063] The vocoder module 326 takes the speech spectrum features output by the decoding module 324 as input and is used to convert the speech spectrum features into a time-domain sound waveform, that is, into the target speech. The vocoder module 326 can be implemented as WaveNet, for example.

[0064] Figure 44 is a flow chart illustrating a method 400 for training a sound feature extraction model according to an exemplary embodiment. The sound feature extraction model may be, for example, a deep neural network (DNN) model.

[0065] The method 400 may be performed on a server (e.g., Figure 1 120) is executed, that is, the execution body of each step of method 400 may be Figure 1 In some embodiments, the method 400 may be performed on a vehicle-mounted system (e.g., Figure 1 In some embodiments, method 400 may be performed by a combination of an in-vehicle system (e.g., in-vehicle system 110) and a server (e.g., server 120). Below, the various steps of method 400 are described in detail, taking server 120 as an example.

[0066] refer to Figure 4 In step 410 , a sample image and annotation data of the sample image are obtained, where the sample image includes a sample object as a virtual character image, and the annotation data includes a sample sound feature of the sample object.

[0067] In step 420, the sample image is input into the sound feature extraction model to obtain the predicted sound features output by the sound feature extraction model.

[0068] In step 430 , a loss value of the sound feature extraction model is determined based on the predicted sound features and the sample sound features.

[0069] In step 440 , parameters of the sound feature extraction model are adjusted based on the loss value.

[0070] According to the embodiments of the present disclosure, a sound feature extraction model can be trained based on sample images and their annotated data.

[0071] According to some embodiments, the sample sound features are obtained using a trained speech feature extraction model, that is, method 400 also includes the following steps of obtaining the sample sound features through the trained speech feature extraction model: obtaining the sample speech corresponding to the sample object; and inputting the sample speech into the trained speech feature extraction model to obtain the sample sound features output by the speech feature extraction model.

[0072] For example, the speech of a sample subject (e.g., an adult female with short hair and yellow skin) can be used as the sample speech. This sample speech can be input into a trained speech feature extraction model to obtain sample sound features output by the speech feature extraction model. Furthermore, an image containing the sample subject (e.g., a photo of an adult female with short hair and yellow skin, a cartoon image, etc.) can be used as the sample image. Thus, the sample image and its annotated data (i.e., the sample sound features) in step 410 can be obtained.

[0073] According to the embodiments of the present disclosure, the trained speech feature extraction model can be used to automatically generate annotation data for sample images without manual annotation, thereby improving the efficiency of sample annotation and thus improving the training efficiency of the sound feature extraction model.

[0074] Furthermore, the feature extraction effect of the sound feature extraction model is highly correlated with the number and diversity of training samples. According to the embodiments of the present disclosure, a large number of training samples that comprehensively cover various scenarios can be automatically generated using the trained speech feature extraction model, thereby improving the feature extraction effect of the sound feature extraction model.

[0075] The speech feature extraction model is pre-trained before method 400 is executed. The speech feature extraction model can be, for example, an i-vector extraction model, such as a Gaussian Mixed Model (GMM), a neural network model, etc. It should be understood that the embodiments of the present disclosure do not limit the specific structure and training method of the speech feature extraction model.

[0076] In addition, it should be understood that the above steps 410-440 can be executed multiple times in a loop until the preset termination condition is met (for example, the loss value is less than the preset value, the number of loops reaches the preset maximum number of loops, etc.), and the model training process is ended to obtain a trained sound feature extraction model.

[0077] Figure 5 FIG. 5 is a schematic diagram showing a training process of a sound feature extraction model 500 according to an exemplary embodiment. Figure 5 As shown, a sample image is input into the sound feature extraction model 500, and the model 500 outputs the predicted sound features of the sample object in the sample image. Subsequently, a loss value of the model 500 is calculated based on the predicted sound features (predicted values) and the sample sound features (true values). Subsequently, based on the loss value, an algorithm such as backpropagation is used to adjust the parameters of the model 500.

[0078] By combining the sound feature extraction model trained according to the method 400 of the embodiment of the present disclosure with the trained speech synthesis model, the text-to-speech conversion TTS model of the embodiment of the present disclosure can be obtained.

[0079] It should be understood that based on method 400, a segmented training method for a TTS model can be provided, that is, different training samples are used to train the sound feature extraction model and the speech synthesis model respectively, and then the trained sound feature extraction model and the trained speech synthesis model are combined to form a TTS model.

[0080] Specifically, the speech synthesis model can be trained before the sound feature extraction model, or after the sound feature extraction model is trained, or trained in parallel with the sound feature extraction model. Parallel training of the sound feature extraction model and the speech synthesis model can improve the overall training efficiency of the TTS model.

[0081] Figure 6 6 is a flow chart illustrating a method 600 for training a text-to-speech (TTS) model according to an exemplary embodiment. Method 600 is an end-to-end (E2E) one-stop training method for a TTS model. The TTS model includes a sound feature extraction model and a speech synthesis model, which further includes a spectrum synthesis module and a vocoding module.

[0082] The method 600 may be performed on a server (e.g., Figure 1 120) is executed, that is, the execution body of each step of method 600 may be Figure 1 In some embodiments, the method 600 may be performed on a vehicle-mounted system (e.g., Figure 1 In some embodiments, method 600 may be performed by a combination of an in-vehicle system (e.g., in-vehicle system 110) and a server (e.g., server 120). Hereinafter, the steps of method 600 will be described in detail, taking server 120 as an example.

[0083] refer to Figure 6 In step 610 , a sample image, sample text, and annotation data are obtained. The sample image includes a sample object as a virtual character image, and the annotation data includes a sample voice of the sample object corresponding to the sample text.

[0084] In step 620, a sample speech spectrum feature of the sample speech is extracted. The sample speech spectrum feature may be, for example, a Mel spectrum of the sample speech.

[0085] In step 630, based on the sample image, sample text and sample speech spectrum features, a sound feature extraction model and a spectrum synthesis module are jointly trained.

[0086] At step 640 , a vocoding module is trained based on the sample images, sample texts, sample speech, and the trained sound feature extraction model and spectrum synthesis module.

[0087] According to the embodiments of the present disclosure, an end-to-end one-stop training method can be used to train the TTS model, ensuring the consistency of the sound feature extraction model and the speech synthesis model, thereby improving the speech synthesis effect of the TTS model.

[0088] Figure 7 FIG. 7 shows a schematic block diagram of a TTS model 700 according to an embodiment of the present disclosure. Figure 7 As shown, the TTS model 700 includes a sound feature extraction model 710 and a speech synthesis model 720. The speech synthesis model 720 further includes a spectrum synthesis module 722 and a vocoder module (Vocoder) 724. According to an embodiment of the present disclosure, the sound feature extraction model 710 and the spectrum synthesis module 722 are first jointly trained based on training samples (including sample images, sample text, and annotated data). Then, the vocoder module 724 is further trained based on the same training samples and the trained sound feature extraction model 710 and spectrum synthesis module 722. After the vocoder module 724 is trained, the trained TTS model 700 is obtained.

[0089] According to some embodiments, the sample image, sample text and annotated data (sample voice) in step 610 can be obtained according to the following steps: the voice spoken by the sample object (for example, an adult female with short hair and yellow skin) is used as the sample voice, the text corresponding to the sample voice is used as the sample text, and the image containing the sample object (for example, a photo of an adult female with short hair and yellow skin, a cartoon image, etc.) is used as the sample image.

[0090] According to some embodiments, the spectrum synthesis module further includes an encoding module (Encoder) and a decoding module (Decoder). Accordingly, the joint training of the sound feature extraction model and the spectrum synthesis module in step 630 further includes: inputting the sample image into the sound feature extraction model to obtain the first predicted sound feature output by the sound feature extraction model; inputting the sample text into the encoding module to obtain the predicted phoneme feature of the sample text output by the encoding module; merging the first predicted sound feature and the predicted phoneme feature and inputting them into the decoding module to obtain the first predicted speech spectrum feature output by the decoding module; determining a first loss value based on the first predicted speech spectrum feature and the sample speech spectrum feature; and adjusting the parameters of the sound feature extraction model and the spectrum synthesis module based on the first loss value.

[0091] Figure 8 FIG. 8 is a schematic diagram showing the training process of the sound feature extraction model 810 and the spectrum synthesis module 820 according to an embodiment of the present disclosure. Figure 8 As shown, the spectrum synthesis module 820 includes an encoding module 822 and a decoding module 824 .

[0092] The sample image is input into the sound feature extraction model 810, which outputs the first predicted sound feature of the sample object in the sample image. The sample text is input into the encoding module 822, which outputs the predicted phoneme feature of the sample text.

[0093] After the first predicted sound feature and the predicted phoneme feature are merged (for example, their vectors are concatenated), they are input into the decoding module 824, and the module 824 outputs the first predicted speech spectrum feature.

[0094] Subsequently, a first loss value is calculated based on the first predicted speech spectrum feature (predicted value) and the sample speech spectrum feature (true value). Subsequently, based on the first loss value, an algorithm such as backpropagation is used to adjust the parameters of the sound feature extraction model 810 and the spectrum synthesis module 820 (including the encoding module 822 and the decoding module 824).

[0095] According to some embodiments, the training of the vocoding module in step 640 further includes: inputting the sample image into a trained sound feature extraction model to obtain a second predicted sound feature output by the sound feature extraction model; inputting the sample text and the second predicted sound feature into a trained spectrum synthesis module to obtain a second predicted speech spectrum feature output by the spectrum synthesis module; inputting the second predicted speech spectrum feature into the vocoding module to obtain a predicted speech output by the vocoding module; determining a second loss value based on the predicted speech and the sample speech; and adjusting parameters of the vocoding module based on the second loss value.

[0096] Figure 9 FIG. 9 is a schematic diagram showing the training process of the vocoding module 930 according to an embodiment of the present disclosure. Figure 9 In the illustrated embodiment, the vocoding module 930 is trained based on the trained sound feature extraction model 910 and the spectrum synthesis module 920 .

[0097] like Figure 9 As shown, the sample image is input into the trained sound feature extraction model 910, and the model 910 outputs the second predicted sound feature of the sample object in the sample image.

[0098] The sample text and the second predicted sound feature are input into the trained spectrum synthesis module 920, and the module 920 outputs the second predicted speech spectrum feature.

[0099] The second predicted speech spectrum feature is input to the vocoding module 930, and the module 930 outputs the predicted speech.

[0100] Then, based on the predicted speech (predicted value) and the sample speech (true value), a second loss value is calculated. Then, based on the second loss value, an algorithm such as back propagation is used to adjust the parameters of the vocoding module 930.

[0101] Figure 10 1 is a schematic block diagram illustrating a speech synthesis apparatus 1000 according to an exemplary embodiment. Figure 10 As shown, the apparatus 1000 includes an acquisition unit 1010 , an extraction unit 1020 and a synthesis unit 1030 .

[0102] The acquisition unit 1010 is configured to acquire a target image and a target text for speech synthesis. The target image includes a target object as a virtual character image.

[0103] The extraction unit 1020 is configured to input the target image into the trained sound feature extraction model to obtain the sound features of the target object output by the sound feature extraction model.

[0104] The synthesis unit 1030 is configured to input the sound features and the target text into the trained speech synthesis model to obtain the target speech corresponding to the target text output by the speech synthesis model.

[0105] According to the embodiments of the present disclosure, a trained sound feature extraction model is used to generate sound features corresponding to a virtual character image, and voice is generated based on the sound features, thereby achieving real-time and efficient customization of the virtual character's voice, meeting the user's personalized needs for the virtual character's voice, and improving the diversity and flexibility of human-computer voice interaction.

[0106] Figure 11 FIG. 1 is an exemplary block diagram illustrating a training apparatus 1100 for a sound feature extraction model according to an exemplary embodiment. Figure 11 As shown, the apparatus 1100 includes an acquiring unit 1110 , an extracting unit 1120 , a determining unit 1130 , and an adjusting unit 1140 .

[0107] The acquisition unit 1110 is configured to acquire a sample image and annotation data of the sample image. The sample image includes a sample object as a virtual character image, and the annotation data includes a sample sound feature of the sample object.

[0108] The extraction unit 1120 is configured to input the sample image into the sound feature extraction model to obtain the predicted sound features output by the sound feature extraction model.

[0109] The determining unit 1130 is configured to determine a loss value of the sound feature extraction model based on the predicted sound feature and the sample sound feature.

[0110] The adjustment unit 1140 is configured to adjust parameters of the sound feature extraction model based on the loss value.

[0111] According to the embodiments of the present disclosure, a sound feature extraction model can be trained based on sample images and their annotated data.

[0112] Figure 12 1 is an exemplary block diagram illustrating a training apparatus 1200 for a TTS model according to an exemplary embodiment. The TTS model includes a sound feature extraction model and a speech synthesis model, and the speech synthesis model further includes a spectrum synthesis module and a vocoding module. Figure 12 As shown, the apparatus 1200 includes an acquisition unit 1210 , an extraction unit 1220 , a first training unit 1230 , and a second training unit 1240 .

[0113] The acquisition unit 1210 is configured to acquire a sample image, a sample text, and annotation data, wherein the sample image includes a sample object as a virtual character image, and the annotation data includes a sample voice of the sample object corresponding to the sample text;

[0114] The extraction unit 1220 is configured to extract sample speech spectrum features of the sample speech;

[0115] The first training unit 1230 is configured to jointly train a sound feature extraction model and a spectrum synthesis module based on sample images, sample texts, and sample speech spectrum features; and

[0116] The second training unit 1240 is configured to train the vocoding module based on the sample images, sample texts, sample speech and the trained sound feature extraction model and spectrum synthesis module.

[0117] According to the embodiments of the present disclosure, an end-to-end one-stop training method can be used to train the TTS model, ensuring the consistency of the sound feature extraction model and the speech synthesis model, thereby improving the speech synthesis effect of the TTS model.

[0118] It should be understood that Figure 10 The various units of the apparatus 1000 shown in FIG. 1 can be used in conjunction with the reference Figure 2 The steps in the method 200 described above correspond to the following. Figure 11 The various units of the apparatus 1100 shown in FIG. 1 can be used in conjunction with the reference Figure 4 The steps in the method 400 described above correspond to each other, and Figure 12 The various units of the apparatus 1200 shown in FIG. 1 can be used in conjunction with the reference Figure 6 The steps in the method 600 described above correspond to each other. Thus, the operations, features, and advantages described above for method 200 are also applicable to apparatus 1000 and its included units, the operations, features, and advantages described above for method 400 are also applicable to apparatus 1100 and its included units, and the operations, features, and advantages described above for method 600 are also applicable to apparatus 1200 and its included units. For the sake of brevity, some operations, features, and advantages are not described in detail here.

[0119] Although specific functions are discussed above with reference to specific units, it should be noted that the functions of the various units discussed herein can be divided into multiple units, and / or at least some functions of multiple units can be combined into a single unit. For example, the acquisition unit 1010 described above can, in some embodiments, be further divided into a first acquisition unit 1010a for acquiring a target image and a second acquisition unit 1010b for acquiring target text. For another example, the determination unit 1130 and the adjustment unit 1140 described above can, in some embodiments, be combined into a single unit.

[0120] It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. Figure 10-12 The various units described can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these units can be implemented as computer program code / instructions, which are configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these units can be implemented as hardware logic / circuits. For example, in some embodiments, one or more of units 1010-1240 can be implemented together in a system on chip (SoC). SoC can include an integrated circuit chip (which includes a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or one or more components in other circuits), and can optionally execute the received program code and / or include embedded firmware to perform functions.

[0121] According to one aspect of the present disclosure, a computer device is provided, comprising at least one memory, at least one processor, and a computer program stored in the at least one memory. The at least one processor is configured to execute the computer program to implement the steps of any of the method embodiments described above.

[0122] According to one aspect of the present disclosure, a vehicle is provided, which includes the apparatus or computer device as described above.

[0123] According to one aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any method embodiment described above are implemented.

[0124] According to one aspect of the present disclosure, a computer program product is provided, which includes a computer program. When the computer program is executed by a processor, the steps of any one of the method embodiments described above are implemented.

[0125] In the following, combined Figure 13 Illustrative examples of such a computer device, non-transitory computer-readable storage medium, and computer program product are described.

[0126] Figure 13 1 shows an example configuration of a computer device 1300 that can be used to implement the methods described herein. For example, Figure 1 The server 120 and / or the in-vehicle system 110 shown in FIG may include an architecture similar to the computer device 1300. The above-mentioned apparatuses 1000-1200 may also be implemented in whole or in part by the computer device 1300 or similar devices or systems.

[0127] The computer device 1300 may include at least one processor 1302, memory 1304, communication interface(s) 1306, a display device 1308, other input / output (I / O) devices 1310, and one or more mass storage devices 1312, all capable of communicating with one another, such as via a system bus 1314 or other appropriate connections.

[0128] The processor 1302 may be a single processing unit or multiple processing units, all of which may include a single or multiple computing units or multiple cores. The processor 1302 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operational instructions. Among other capabilities, the processor 1302 may be configured to retrieve and execute computer-readable instructions stored in the memory 1304, mass storage device 1312, or other computer-readable media, such as program code for an operating system 1316, program code for application programs 1318, program code for other programs 1320, and the like.

[0129] Memory 1304 and mass storage device 1312 are examples of computer-readable storage media for storing instructions that are executed by processor 1302 to implement the various functions described above. For example, memory 1304 may generally include both volatile memory and non-volatile memory (e.g., RAM, ROM, etc.). In addition, mass storage device 1312 may generally include a hard drive, a solid-state drive, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network attached storage, storage area networks, etc. Memory 1304 and mass storage device 1312 may be collectively referred to herein as memory or computer-readable storage media, and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code that may be executed by processor 1302 as a specific machine configured to implement the operations and functions described in the examples herein.

[0130] A plurality of programs may be stored on the mass storage device 1312. These programs include an operating system 1316, one or more application programs 1318, other programs 1320, and program data 1322, and they may be loaded into the memory 1304 for execution. Examples of such applications or program modules may include, for example, computer program logic (e.g., computer program code or instructions) for implementing the following components / functions: the speech synthesis device 1000 (including an acquisition unit 1010, an extraction unit 1020, and a synthesis unit 1030), the training device 1100 for the sound feature extraction model (including an acquisition unit 1110, an extraction unit 1120, a determination unit 1130, and an adjustment unit 1140), the training device 1200 for the TTS model (including an acquisition unit 1210, an extraction unit 1220, a first training unit 1230, and a second training unit 1240), the methods 200, 400, and / or the method 600 (including any suitable steps of the methods 200, 400, 600), and / or other embodiments described herein.

[0131] Although Figure 13 1304 of the computer device 1300, but modules 1316, 1318, 1320, and 1322, or portions thereof, may be implemented using any form of computer-readable media accessible by the computer device 1300. As used herein, "computer-readable media" includes at least two types of computer-readable media, namely, computer-readable storage media and communication media.

[0132] Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information, such as computer-readable instructions, data structures, program modules or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD), or other optical storage devices, magnetic cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other non-transmission media that can be used to store information for access by a computer device. In contrast, communication media can embody computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism. Computer-readable storage media as defined herein does not include communication media.

[0133] One or more communication interfaces 1306 are used to exchange data with other devices, such as through a network, a direct connection, etc. Such communication interfaces can be one or more of the following: any type of network interface (e.g., a network interface card (NIC)), a wired or wireless (such as IEEE 802.11 wireless LAN (WLAN)) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth TM The communication interface 1306 may include a wireless communication interface, a near field communication (NFC) interface, and the like. The communication interface 1306 may facilitate communication within a variety of network and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, and the like. The communication interface 1306 may also provide for communication with external storage devices (not shown) such as storage arrays, network attached storage, storage area networks, and the like.

[0134] In some examples, a display device 1308 such as a monitor may be included for displaying information and images to the user. Other I / O devices 1310 may be devices that receive various inputs from the user and provide various outputs to the user, and may include a touch input device, a gesture input device, a camera, a keyboard, a remote control, a mouse, a printer, an audio input / output device, and the like.

[0135] The technology described herein can be supported by these various configurations of computer device 1300 and is not limited to the specific examples of the technology described herein. For example, the functionality can also be implemented in whole or in part on a "cloud" by using a distributed system. The cloud includes and / or represents a platform for resources. The platform abstracts the underlying functionality of the hardware (e.g., servers) and software resources of the cloud. Resources can include applications and / or data that can be used when performing computing processing on a server away from computer device 1300. Resources can also include services provided over the Internet and / or through a subscriber network such as a cellular or Wi-Fi network. The platform can abstract resources and functionality to connect computer device 1300 to other computer devices. Therefore, the implementation of the functionality described herein can be distributed throughout the cloud. For example, functionality can be implemented partially on computer device 1300 and partially through a platform that abstracts the functionality of the cloud.

[0136] Some exemplary aspects of the disclosure are described below.

[0137] Aspect 1. A speech synthesis method, comprising:

[0138] Acquire a target image and target text for speech synthesis, wherein the target image includes a target object as a virtual character image;

[0139] Inputting the target image into a trained sound feature extraction model to obtain the sound features of the target object output by the sound feature extraction model; and

[0140] The sound features and the target text are input into a trained speech synthesis model to obtain a target speech corresponding to the target text output by the speech synthesis model.

[0141] Aspect 2. The method according to aspect 1, wherein the speech synthesis model is trained to:

[0142] Extracting phoneme features of the target text;

[0143] Merging the phoneme feature and the sound feature to obtain a merged feature;

[0144] Converting the combined features into speech spectrum features; and

[0145] The speech spectrum features are converted into the target speech.

[0146] Aspect 3. The method according to aspect 1, wherein the sound feature indicates at least the timbre of the target object.

[0147] Aspect 4. A method for training a sound feature extraction model, comprising:

[0148] Acquire a sample image and annotation data of the sample image, wherein the sample image includes a sample object as a virtual character image, and the annotation data includes a sample sound feature of the sample object;

[0149] Inputting the sample image into the sound feature extraction model to obtain the predicted sound features output by the sound feature extraction model;

[0150] determining a loss value of the sound feature extraction model based on the predicted sound feature and the sample sound feature; and

[0151] Based on the loss value, parameters of the sound feature extraction model are adjusted.

[0152] Aspect 5. The method according to aspect 4 further includes:

[0153] Obtaining a sample speech corresponding to the sample object; and

[0154] The sample speech is input into a trained speech feature extraction model to obtain the sample sound features output by the speech feature extraction model.

[0155] Aspect 6. A method for training a text-to-speech (TTS) model, wherein the TTS model includes a sound feature extraction model and a speech synthesis model, wherein the speech synthesis model includes a spectrum synthesis module and a vocoding module, the method comprising:

[0156] Acquire a sample image, sample text, and annotation data, wherein the sample image includes a sample object as a virtual character image, and the annotation data includes a sample voice of the sample object corresponding to the sample text;

[0157] Extracting sample speech spectrum features of the sample speech;

[0158] Based on the sample image, the sample text and the sample speech spectrum features, jointly training the sound feature extraction model and the spectrum synthesis module; and

[0159] The vocoding module is trained based on the sample image, the sample text, the sample speech, the trained sound feature extraction model and the trained spectrum synthesis module.

[0160] Aspect 7. The method according to aspect 6, wherein the spectrum synthesis module includes an encoding module and a decoding module, and wherein the joint training of the sound feature extraction model and the spectrum synthesis module includes:

[0161] Inputting the sample image into the sound feature extraction model to obtain a first predicted sound feature output by the sound feature extraction model;

[0162] Inputting the sample text into the encoding module to obtain the predicted phoneme features of the sample text output by the encoding module;

[0163] Combining the first predicted sound feature and the predicted phoneme feature and inputting the combined feature into the decoding module to obtain a first predicted speech spectrum feature output by the decoding module;

[0164] Determining a first loss value based on the first predicted speech spectrum feature and the sample speech spectrum feature; and

[0165] Based on the first loss value, parameters of the sound feature extraction model and the spectrum synthesis module are adjusted.

[0166] Aspect 8. The method according to aspect 6, wherein the training the vocoding module comprises:

[0167] Inputting the sample image into the trained sound feature extraction model to obtain a second predicted sound feature output by the sound feature extraction model;

[0168] Inputting the sample text and the second predicted sound feature into the trained spectrum synthesis module to obtain the second predicted speech spectrum feature output by the spectrum synthesis module;

[0169] Inputting the second predicted speech spectrum feature into the vocoding module to obtain the predicted speech output by the vocoding module;

[0170] determining a second loss value based on the predicted speech and the sample speech; and

[0171] Based on the second loss value, parameters of the vocoding module are adjusted.

[0172] Aspect 9. A speech synthesis device comprising:

[0173] an acquiring unit configured to acquire a target image and a target text for speech synthesis, wherein the target image includes a target object as a virtual character image;

[0174] an extraction unit configured to input the target image into a trained sound feature extraction model to obtain the sound features of the target object output by the sound feature extraction model; and

[0175] The synthesis unit is configured to input the sound features and the target text into a trained speech synthesis model to obtain a target speech corresponding to the target text output by the speech synthesis model.

[0176] Aspect 10. A training device for a sound feature extraction model, comprising:

[0177] an acquiring unit configured to acquire a sample image and annotation data of the sample image, wherein the sample image includes a sample object as a virtual character image, and the annotation data includes a sample sound feature of the sample object;

[0178] an extraction unit configured to input the sample image into the sound feature extraction model to obtain the predicted sound features output by the sound feature extraction model;

[0179] a determining unit configured to determine a loss value of the sound feature extraction model based on the predicted sound feature and the sample sound feature; and

[0180] An adjustment unit is configured to adjust parameters of the sound feature extraction model based on the loss value.

[0181] Aspect 11. A training device for a text-to-speech (TTS) model, wherein the TTS model includes a sound feature extraction model and a speech synthesis model, wherein the speech synthesis model includes a spectrum synthesis module and a vocoding module, wherein the method comprises:

[0182] an acquiring unit configured to acquire a sample image, a sample text, and annotation data, wherein the sample image includes a sample object as a virtual character image, and the annotation data includes a sample voice of the sample object corresponding to the sample text;

[0183] An extraction unit, configured to extract a sample speech spectrum feature of the sample speech;

[0184] A first training unit is configured to jointly train the sound feature extraction model and the spectrum synthesis module based on the sample image, the sample text and the sample speech spectrum features; and

[0185] The second training unit is configured to train the vocoding module based on the sample image, the sample text, the sample speech, the trained sound feature extraction model and the spectrum synthesis module.

[0186] Aspect 12. A computer device comprising:

[0187] at least one processor; and

[0188] at least one memory having a computer program stored thereon,

[0189] Wherein, when the computer program is executed by the at least one processor, the at least one processor performs the method described in any one of aspects 1-8.

[0190] Aspect 13. A vehicle comprising the apparatus according to any one of Aspects 9-11 or the computer device according to Aspect 12.

[0191] Aspect 14. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the processor is caused to perform the method according to any one of aspects 1-8.

[0192] Aspect 15. A computer program product, comprising a computer program, which, when executed by a processor, causes the processor to perform the method according to any one of aspects 1-8.

[0193] Although the present disclosure has been illustrated and described in detail in the drawings and the foregoing description, such illustration and description are to be considered illustrative and exemplary and not restrictive; the present disclosure is not limited to the disclosed embodiments. Variations to the disclosed embodiments will be understood and effected by those skilled in the art in practicing the claimed subject matter by studying the drawings, the disclosure and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps that are not listed, the indefinite article "a" or "an" does not exclude a plurality, the term "plurality" means two or more, and the term "based on" should be interpreted as "based at least in part on". The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. A method for training a text-to-speech (TTS) model, the TTS model comprising a sound feature extraction model and a speech synthesis model, the speech synthesis model comprising a spectrum synthesis module and a vocoding module, the spectrum synthesis module comprising an encoding module and a decoding module, the method comprising: Acquire a sample image, sample text, and annotation data, wherein the sample image includes a sample object as a virtual character image, and the annotation data includes a sample voice of the sample object corresponding to the sample text; Extracting sample speech spectrum features of the sample speech; Inputting the sample image into the sound feature extraction model to obtain a first predicted sound feature output by the sound feature extraction model; Inputting the sample text into the encoding module to obtain the predicted phoneme features of the sample text output by the encoding module; Combining the first predicted sound feature and the predicted phoneme feature and inputting the combined feature into the decoding module to obtain a first predicted speech spectrum feature output by the decoding module; Determining a first loss value based on the first predicted speech spectrum feature and the sample speech spectrum feature; Adjusting parameters of the sound feature extraction model and the spectrum synthesis module based on the first loss value; Inputting the sample image into the trained sound feature extraction model to obtain a second predicted sound feature output by the sound feature extraction model; Inputting the sample text and the second predicted sound feature into the trained spectrum synthesis module to obtain the second predicted speech spectrum feature output by the spectrum synthesis module; Inputting the second predicted speech spectrum feature into the vocoding module to obtain the predicted speech output by the vocoding module; determining a second loss value based on the predicted speech and the sample speech; and Based on the second loss value, parameters of the vocoding module are adjusted.

2. A speech synthesis method, comprising: Acquire a target image and target text for speech synthesis, wherein the target image includes a target object as a virtual character image; Inputting the target image into a trained sound feature extraction model to obtain the sound features of the target object output by the sound feature extraction model; and Inputting the sound features and the target text into a trained speech synthesis model to obtain a target speech corresponding to the target text output by the speech synthesis model, Wherein, the trained sound feature extraction model and the trained speech synthesis model are obtained through the training method described in claim 1.

3. The method according to claim 2, wherein: The speech synthesis model is trained to: Extracting phoneme features of the target text; Merging the phoneme feature and the sound feature to obtain a merged feature; Converting the combined features into speech spectrum features; as well as The speech spectrum features are converted into the target speech.

4. The method according to claim 2, wherein: The sound feature at least indicates the timbre of the target object.

5. A training device for a text-to-speech (TTS) model, the TTS model comprising a sound feature extraction model and a speech synthesis model, the speech synthesis model comprising a spectrum synthesis module and a vocoding module, the spectrum synthesis module comprising an encoding module and a decoding module, the device comprising: an acquiring unit configured to acquire a sample image, a sample text, and annotation data, wherein the sample image includes a sample object as a virtual character image, and the annotation data includes a sample voice of the sample object corresponding to the sample text; An extraction unit, configured to extract a sample speech spectrum feature of the sample speech; The first training unit is configured as follows: Inputting the sample image into the sound feature extraction model to obtain a first predicted sound feature output by the sound feature extraction model; Inputting the sample text into the encoding module to obtain the predicted phoneme features of the sample text output by the encoding module; Combining the first predicted sound feature and the predicted phoneme feature and inputting the combined feature into the decoding module to obtain a first predicted speech spectrum feature output by the decoding module; Determining a first loss value based on the first predicted speech spectrum feature and the sample speech spectrum feature; and Adjusting parameters of the sound feature extraction model and the spectrum synthesis module based on the first loss value; as well as The second training unit is configured as follows: Inputting the sample image into the trained sound feature extraction model to obtain a second predicted sound feature output by the sound feature extraction model; Inputting the sample text and the second predicted sound feature into the trained spectrum synthesis module to obtain the second predicted speech spectrum feature output by the spectrum synthesis module; Inputting the second predicted speech spectrum feature into the vocoding module to obtain the predicted speech output by the vocoding module; determining a second loss value based on the predicted speech and the sample speech; and Based on the second loss value, parameters of the vocoding module are adjusted.

6. A speech synthesis device comprising: an acquiring unit configured to acquire a target image and a target text for speech synthesis, wherein the target image includes a target object as a virtual character image; an extraction unit configured to input the target image into a trained sound feature extraction model to obtain the sound features of the target object output by the sound feature extraction model; and A synthesis unit is configured to input the sound feature and the target text into a trained speech synthesis model to obtain a target speech corresponding to the target text output by the speech synthesis model, Wherein, the trained sound feature extraction model and the trained speech synthesis model are obtained through the training device according to claim 5.

7. A computer device comprising: at least one processor; as well as at least one memory having a computer program stored thereon, When the computer program is executed by the at least one processor, the at least one processor is caused to perform the method according to any one of claims 1 to 4.

8. A vehicle comprising the apparatus according to claim 5 or 6 or the computer device according to claim 7.

9. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 4.

10. A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 4.

Images