Speech synthesis model training method, device, computer equipment and storage medium

Abstract

The embodiment of the present application discloses a speech synthesis model training method, device, computer equipment and storage medium, which belongs to the field of computer technology. The method includes: obtaining a first sample text information, a first sample speech and a second sample speech; calling a speech synthesis model, extracting a first predicted text feature corresponding to the first sample text information, extracting a first predicted timbre feature corresponding to the second sample speech, and synthesizing the first predicted speech based on the first predicted text feature and the first predicted timbre feature; calling a language classification model, performing language inversion classification on the first predicted timbre feature, and obtaining the first predicted language to which the second sample speech belongs; based on the first predicted language and the second language, as well as the first sample speech and the first predicted speech, training a language classification model and a speech synthesis model. The speech synthesis model trained by this method realizes cross-language speech synthesis, and the synthesized speech has a high degree of naturalness.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a speech synthesis model training method, a speech synthesis method, an apparatus, a computer device, and a storage medium. Background Art

[0002] Text-to-Speech (TTS) technology refers to the technology of converting text into high-quality, natural and fluent speech through computer equipment. In recent years, with the development of computer technology and the widespread application of deep learning, the development of speech synthesis technology has been promoted.

[0003] In the related art, in the process of training a speech synthesis model, it is necessary to use sample speech and sample text for training. Since the sample speech contains a certain language, after training, the speech synthesis model can synthesize speech in the language. However, the speech synthesis model can only synthesize speech in this language, which has limitations. Summary of the invention

[0004] The embodiments of the present application provide a speech synthesis model training method, apparatus, computer equipment and storage medium, which realize cross-language synthesis of the speech synthesis model, and the synthesized speech has a high degree of naturalness.

[0005] The technical solution is as follows:

[0006] On the one hand, a speech synthesis model training method is provided, the method comprising:

[0007] Acquire first sample text information, a first sample voice, and a second sample voice, wherein the first sample text information includes text content in a first language, the first sample voice and the second sample voice are voices of the same sample object, the first sample voice belongs to the first language and corresponds to the first sample text information, the second sample voice belongs to a second language, and the first language is different from the second language;

[0008] Calling a speech synthesis model, extracting a first predicted text feature corresponding to the first sample text information, extracting a first predicted timbre feature corresponding to the second sample speech, and synthesizing a first predicted speech based on the first predicted text feature and the first predicted timbre feature;

[0009] Calling a language classification model to perform language reversal classification on the first predicted timbre feature to obtain a first predicted language to which the second sample speech belongs, wherein the language reversal classification is used to make the first predicted language opposite to the language represented by the first predicted timbre feature;

[0010] Based on the first predicted language and the second language, as well as the first sample speech and the first predicted speech, the language classification model and the speech synthesis model are trained to reduce the difference between the first predicted language and the second language, and to reduce the difference between the first sample speech and the first predicted speech.

[0011] On the other hand, a speech synthesis model training device is provided, the device comprising:

[0012] a sample acquisition module, configured to acquire first sample text information, a first sample voice, and a second sample voice, wherein the first sample text information includes text content belonging to a first language, the first sample voice and the second sample voice are voices of the same sample object, the first sample voice belongs to the first language and corresponds to the first sample text information, the second sample voice belongs to a second language, and the first language is different from the second language;

[0013] A speech synthesis module, configured to call a speech synthesis model, extract a first predicted text feature corresponding to the first sample text information, extract a first predicted timbre feature corresponding to the second sample speech, and synthesize a first predicted speech based on the first predicted text feature and the first predicted timbre feature;

[0014] a language classification module, configured to call a language classification model, perform language reversal classification on the first predicted timbre feature, and obtain a first predicted language to which the second sample speech belongs, wherein the language reversal classification is used to make the first predicted language opposite to the language represented by the first predicted timbre feature;

[0015] A model training module is used to train the language classification model and the speech synthesis model based on the first predicted language and the second language, as well as the first sample speech and the first predicted speech, so as to reduce the difference between the first predicted language and the second language, and reduce the difference between the first sample speech and the first predicted speech.

[0016] In a possible implementation, the language classification module is used to:

[0017] The language classification model includes a first inversion layer and a first classification layer, calling the first inversion layer to perform feature inversion on the first predicted timbre feature to obtain a first inverted timbre feature, wherein the meaning represented by the first inverted timbre feature is opposite to the meaning represented by the first predicted timbre feature; and calling the first classification layer to classify the first inverted timbre feature to obtain the first predicted language; or

[0018] The language classification model includes a second inversion layer and a second classification layer. The second classification layer is called to classify the first predicted timbre feature into a language to obtain first classification information, wherein the first classification information indicates the language to which the second sample speech obtained by the current language classification belongs; the second inversion layer is called to invert the first classification information to obtain second classification information, and the first predicted language is determined based on the second classification information.

[0019] In another possible implementation manner, the sample object has a corresponding sample object identifier, and the apparatus further includes:

[0020] An object classification module, used for calling an object classification model to perform object classification on the first predicted timbre feature to obtain a predicted object identifier corresponding to the sample object;

[0021] The model training module is used to train the object classification model, the language classification model and the speech synthesis model based on the first predicted language and the second language, the first sample speech and the first predicted speech, and the sample object identifier and the predicted object identifier.

[0022] In another possible implementation, the speech synthesis model includes a text feature extraction network, a timbre feature extraction network and a speech synthesis network, and the speech synthesis module includes:

[0023] A text feature extraction unit, configured to call the text feature extraction network to extract the first predicted text feature corresponding to the first sample text information;

[0024] A timbre feature extraction unit, configured to call the timbre feature extraction network to extract the first predicted timbre feature corresponding to the second sample speech;

[0025] A speech synthesis unit is used to call the speech synthesis network to synthesize the first predicted speech based on the first predicted text feature and the first predicted timbre feature.

[0026] In another possible implementation, the speech synthesis model includes a text feature fusion network and at least two types of text feature extraction networks, and the text feature extraction unit is used to:

[0027] respectively calling the at least two types of text feature extraction networks to extract text features from the first sample text information to obtain at least two types of second predicted text features, wherein different types of text features describe different importance levels of different contents in the first sample text information;

[0028] The text feature fusion network is called to perform weighted processing on the at least two types of second predicted text features based on the first weights of the at least two types of text feature extraction networks, respectively, to obtain the first predicted text feature.

[0029] In another possible implementation, the text feature extraction network includes n groups of encoding layers, where n is an integer greater than 2; and the text feature extraction unit is used to:

[0030] Calling the first group of encoding layers to encode the first sample text information to obtain encoding features output by the first group of encoding layers;

[0031] In the case where the i-th group of coding layers includes at least two coding layers, the at least two coding layers are respectively called to encode the coding features output by the i-1-th group of coding layers to obtain the at least two coding features, and a fusion layer corresponding to the i-th group of coding layers is called to fuse the at least two coding features and the coding features output by the i-1-th group of coding layers, and the fused coding features are determined as the coding features output by the i-th group of coding layers; or

[0032] In a case where the i-th group of coding layers includes one coding layer, calling the i-th group of coding layers to encode the coding features output by the i-1-th group of coding layers to obtain the coding features output by the i-th group of coding layers;

[0033] Among them, the encoding features output by the last group of encoding layers are the first predicted text features, i is an integer greater than 1 and not greater than n, the first group of encoding layers includes one encoding layer, and among the second to nth groups of encoding layers, there is a group including at least two encoding layers.

[0034] In another possible implementation, the timbre feature extraction network includes a timbre feature extraction layer and a first attention layer, wherein the first attention layer includes reference timbre features of a plurality of reference objects; and the timbre feature extraction unit is used to:

[0035] Calling the timbre feature extraction layer to extract features of the second sample speech to obtain an initial predicted timbre feature of the sample object;

[0036] The first attention layer is called to determine the second weight of the initial predicted timbre feature corresponding to each reference object, wherein the second weight represents the degree of similarity between the reference timbre feature of the corresponding reference object and the initial predicted timbre feature, and based on multiple second weights, the reference timbre features of the multiple reference objects are weighted to obtain the first predicted timbre feature.

[0037] In another possible implementation, the speech synthesis network includes a second attention layer and a decoding layer, and the speech synthesis unit is used to:

[0038] Calling the second attention layer to fuse the first predicted text feature and the first predicted timbre feature to obtain a fused feature;

[0039] The decoding layer is called to decode the fused features to obtain predicted acoustic features, and the first predicted speech is determined based on the predicted acoustic features.

[0040] In another possible implementation, the device further includes:

[0041] An information acquisition module, used to acquire target text information and reference speech, wherein the reference speech belongs to any object and any language, and the target text information includes text content belonging to the target language;

[0042] The speech synthesis module is also used to call the speech synthesis model, extract the target text features corresponding to the target text information, extract the target timbre features corresponding to the reference speech, and synthesize the target speech corresponding to the target text information and belonging to the target language based on the target text features and the target timbre features.

[0043] In another possible implementation, the first data set includes a plurality of sample text information and a plurality of sample voices, and the plurality of sample voices in the first data set belong to at least two languages ​​and at least two sample objects; the first sample text information, the first sample voice, and the second sample voice belong to the first data set;

[0044] The second data set includes a plurality of sample text information and a plurality of sample speech, wherein the second data set includes text information in the first language, sample speech in the second language, and does not include sample speech in the first language, and the plurality of sample speech in the second data set belong to the same target sample object;

[0045] The device also includes:

[0046] The sample acquisition module is further configured to acquire second sample text information and third sample speech from the second data set after training the language classification model and the speech synthesis model based on the first data set, wherein the first sample text information includes text content belonging to the first language;

[0047] The speech synthesis module is further used to call the speech synthesis model, extract the third predicted text feature corresponding to the second sample text information, extract the second predicted timbre feature corresponding to the third sample speech, and synthesize the second predicted speech based on the third predicted text feature and the second predicted timbre feature;

[0048] The language classification module is further used to call the language classification model to perform language reversal classification on the second predicted timbre feature to obtain a second predicted language to which the third sample speech belongs, wherein the language reversal classification is used to make the second predicted language opposite to the language represented by the second predicted timbre feature;

[0049] The model training module is also used to train the language classification model and the speech synthesis model based on the second predicted language and the second language to reduce the difference between the second predicted language and the second language; and output the second predicted speech, obtain adjustment parameters based on the second predicted speech input, and train the speech synthesis model according to the adjustment parameters.

[0050] In another possible implementation, the sample acquisition module is further used to:

[0051] Acquire a first number and a second number respectively, wherein the first number is the number of sample speech in the second data set, and the second number is the number of sample speech in the first data set;

[0052] Determine a selection ratio based on a ratio between the first number and the second number, wherein the selection ratio refers to the ratio of the number of sample speech selected from the second data set for training the speech synthesis model to the first number, and the selection ratio is negatively correlated with the ratio;

[0053] Based on the selection ratio, a third sample speech is selected from the second data set for training the speech synthesis model.

[0054] On the other hand, a computer device is provided, which includes a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the operations performed by the speech synthesis model training method as described in the above aspects.

[0055] On the other hand, a computer-readable storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement the operations performed by the speech synthesis model training method as described in the above aspects.

[0056] On the other hand, a computer program product is provided, including a computer program, which, when executed by a processor, implements the operations performed by the speech synthesis model training method described in the above aspects.

[0057] The technical solution provided by the embodiment of the present application calls the speech synthesis model to be trained, synthesizes the first predicted speech based on the first sample text information including the text content belonging to the first language and the second sample speech belonging to the second language. In the process of synthesizing the speech, the first predicted timbre feature corresponding to the second sample speech is extracted. In the case that the speech synthesis model is not trained, the first predicted timbre feature will inevitably represent the second language. In order to make the first predicted timbre feature have no necessary connection with the second language, a language classification model is introduced, the language classification model is called, the first predicted timbre feature is language-reversed classified, and the first predicted language opposite to the second language is obtained. Then, when training the speech synthesis model, based on The idea of ​​adversarial training makes the first predicted language and the second language gradually approach the same, so that the first predicted timbre features will not be affected by the second language. Therefore, when performing speech synthesis based on the trained speech synthesis model, the timbre features extracted based on any text information belonging to any target language and the reference speech belonging to any language are irrelevant to the language to which the speech belongs. Instead, the speech will be synthesized according to the target language to which the text information belongs, thereby synthesizing a speech corresponding to the text information and belonging to the target language, realizing cross-language speech synthesis. Moreover, since the timbre of the synthesized speech is independent of the language, the timbre of the synthesized speech will not be affected by the language to which the reference speech belongs, thereby making the synthesized speech more natural. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0059] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0060] Figure 2 is a flow chart of a speech synthesis model training method provided in an embodiment of the present application;

[0061] Figure 3 is a flowchart of another speech synthesis model training method provided in an embodiment of the present application;

[0062] Figure 4 It is a structural diagram of a text feature extraction network provided in an embodiment of the present application;

[0063] Figure 5 is a structural schematic diagram of a timbre feature extraction layer provided in an embodiment of the present application;

[0064] Figure 6 It is a structural schematic diagram of a speech synthesis model provided in an embodiment of the present application;

[0065] Figure 7 is a flow chart of a model training process provided by an embodiment of the present application;

[0066] Figure 8 is a flow chart of a speech synthesis method provided by an embodiment of the present application;

[0067] Fig. 9 is a flowchart of another speech synthesis model training method provided in an embodiment of the present application;

[0068] Fig.10 It is a structural schematic diagram of a speech synthesis model training device provided in an embodiment of the present application;

[0069] Fig.11 It is a structural schematic diagram of another speech synthesis model training device provided in an embodiment of the present application;

[0070] Fig.12 is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0071] Fig.13 It is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0073] It is understood that the terms "first", "second", etc. used in this application can be used in this article to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of this application, the first arrangement order can be called the second arrangement order, and the second arrangement order can be called the first arrangement order.

[0074] The terms "at least one", "multiple", "each", "any" and the like used in this application include one, two or more than two, multiple include two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple. For example, multiple languages ​​include three languages, and each language refers to each of the three languages, and any refers to any one of the three languages, which can be the first, the second, or the third.

[0075] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0076] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0077] The key technologies of speech technology include automatic speech recognition (ASR), text-to-speech (TTS) and voiceprint recognition. Enabling computers to listen, see, speak and feel is the future development direction of human-computer interaction, among which speech has become one of the most promising human-computer interaction methods in the future.

[0078] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0079] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, automatic driving, smart transportation, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0080] The solution provided in the embodiment of the present application is based on artificial intelligence speech technology and machine learning technology to train a speech synthesis model, and use the trained speech synthesis model to synthesize speech belonging to any object and in any language.

[0081] The speech synthesis model training method provided in the embodiment of the present application is performed by a computer device. Optionally, the computer device is a terminal or a server. Optionally, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Server. Optionally, the terminal is a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, an intelligent voice interaction device, a smart home appliance, and a car terminal, etc., but is not limited to this.

[0082] Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application. Figure 1 The implementation environment includes a terminal 101 and a server 102. The terminal 101 and the server 102 are connected via a wireless or wired network, which is not limited in this application.

[0083] The terminal 101 collects multiple voices of any object, the multiple voices belong to at least two languages, that is, the multiple voices are voices of the same object in different languages, and the terminal 101 sends the collected voices to the server 102, and the server 102 is used to train a speech synthesis model based on text information and the collected voices. The text information is consistent with the content of some of the multiple voices.

[0084] In a possible implementation, a target application provided by server 102 is installed on terminal 101, and terminal 101 can implement functions such as voice collection and voice synthesis through the target application. Optionally, the target application is a target application in the operating system of terminal 101, or a target application provided by a third party, and a voice synthesis model can be configured in the target application. During the use of the target application, voice synthesis can be implemented based on the voice synthesis model, and synthesized voice can be played. Optionally, server 102 is a background server of the target application or a cloud server that provides cloud computing and cloud storage services.

[0085] The speech synthesis model trained by the speech synthesis model training method provided in the embodiment of the present application can be used in a variety of scenarios.

[0086] For example, it is used in the "listening to a book" scenario. The computer device adopts the speech synthesis model training method provided in the embodiment of the present application to train the speech synthesis model, and deploys the trained speech synthesis model in the application server corresponding to the reading application. When the user wants to listen to the book while reading an e-book based on the reading application, the user triggers the listening operation and sends a listening request for the e-book to the application server. The application server calls the speech synthesis model, synthesizes the target voice corresponding to the e-book based on the reference voice and the content of the e-book, and sends the target voice to the terminal. The terminal plays the target voice, so that the user can "listen to the book". Among them, the reference voice used by the application server can be the voice of a specific user pre-stored by the application server, or it can be the voice of the current user collected by the terminal.

[0087] Of course, the speech synthesis model trained by the embodiment of the present application can also be used in other scenarios where text needs to be converted into speech, for example, in smart phone customer service scenarios, game scenarios, etc.

[0088] Figure 2 1 is a flow chart of a speech synthesis model training method provided in an embodiment of the present application. The execution subject of the embodiment of the present application is a computer device. Figure 2 , the method comprises the following steps:

[0089] 201. A computer device obtains first sample text information, a first sample voice, and a second sample voice, where the first sample voice belongs to a first language and corresponds to the first sample text information, and the second sample voice belongs to a second language.

[0090] When a speech synthesis model in the related art synthesizes a new speech based on text information and a reference speech belonging to a certain language, the synthesized speech must be in the same language as the reference speech, which has limitations. However, the embodiment of the present application provides a speech synthesis model that can synthesize a speech containing the content represented by the text information and the timbre represented by the reference speech based on text information belonging to a certain language and a reference speech belonging to another different language.

[0091] In the embodiment of the present application, the first sample text information, the first sample speech, and the second sample speech are used to train the speech synthesis model, wherein the first sample speech can be regarded as a label of the first sample text information, and the second sample speech is a reference speech. The first sample text information and the second sample speech are inputs of the speech synthesis model, and the first sample speech is used to compare with the predicted speech synthesized by the speech synthesis model, so as to train the speech synthesis model.

[0092] The first sample text information includes text content in the first language, and the first sample voice corresponds to the first sample text information, that is, the voice content contained in the first sample voice is consistent with the text content in the first sample text information. For example, the first sample voice can be obtained by pronouncing the above object according to the text content in the first sample text information.

[0093] Furthermore, the first sample speech and the second sample speech are speech of the same sample object, and the first language is different from the second language, so the first sample speech and the second sample speech are speech of the same object in different languages. For example, the first language is Chinese, the second language is English, or the first language and the second language are other languages. In addition, the speech content contained in the second sample speech and the first sample speech can be the same or different.

[0094] 202. The computer device calls a speech synthesis model, extracts a first predicted text feature corresponding to the first sample text information, extracts a first predicted timbre feature corresponding to the second sample speech, and synthesizes a first predicted speech based on the first predicted text feature and the first predicted timbre feature.

[0095] The first predicted text feature is used to describe the text content. Since the text content belongs to the first language, the first predicted text feature will include features related to the first language.

[0096] The first predicted timbre feature represents the timbre of the sample object determined based on the second sample speech, and the first predicted speech is synthesized according to the timbre represented by the first predicted timbre feature and the content indicated by the first predicted text feature, so the first predicted speech can be regarded as the speech obtained by the above object uttering the above text content. Since the above text content belongs to the first language, the determined first predicted speech also belongs to the first language.

[0097] 203. The computer device calls the language classification model to perform language reversal classification on the first predicted timbre feature to obtain a first predicted language to which the second sample speech belongs. The language reversal classification is used to make the first predicted language opposite to the language represented by the first predicted timbre feature.

[0098] In an embodiment of the present application, given text information belonging to a certain language and a reference speech belonging to another different language, it is desired to generate a speech that contains the content represented by the text information and also contains the timbre of the reference speech, without being affected by the language to which the reference speech belongs.

[0099] However, the speech synthesis model will extract the timbre features of the reference speech, which are correlated with the language to which the reference speech belongs, and will inevitably be affected by the language to which the reference speech belongs. Therefore, in order to train a speech synthesis model that can be used across languages, it is also necessary to introduce a language classification model, and by jointly training the speech synthesis model with the language classification model, the correlation between the timbre features and the language to which the reference speech belongs can be weakened.

[0100] Since the timbre features extracted from the input speech will inevitably represent the language to which the speech belongs when the speech synthesis model has not been trained, the first predicted timbre features extracted based on the second sample speech will represent the second language to which the second sample speech belongs. When the language classification model is called to perform language reversal classification on the first predicted timbre features, if the first predicted timbre features represent the second language, then the first predicted language obtained by performing language reversal classification on the first predicted timbre features through the language classification model is opposite to the second language.

[0101] 204. The computer device trains a language classification model and a speech synthesis model based on the first predicted language and the second language, as well as the first sample speech and the first predicted speech, so as to reduce the difference between the first predicted language and the second language, and reduce the difference between the first sample speech and the first predicted speech.

[0102] When training the language classification model and the speech synthesis model, the difference between the first predicted language and the second language is reduced, that is, the second predicted language obtained by language reversal classification of the first predicted timbre features is gradually approached to be the same as the second language. This requires that the first predicted timbre features cannot represent the second language. Therefore, through the interaction between the language classification model and the speech synthesis model, the first predicted timbre features extracted by the speech synthesis model can be made less and less affected by the second language.

[0103] When training the speech synthesis model, the difference between the first sample speech and the first predicted speech is reduced, that is, the speech synthesis model is enabled to learn the ability to synthesize speech based on timbre features and text features. Moreover, since the second language to which the second sample speech input into the speech synthesis model belongs is different from the first language to which the synthesized speech belongs, that is, speech belonging to one language is input into the speech synthesis model, speech belonging to another language can be synthesized, so that the trained speech synthesis model has the ability of cross-language speech synthesis.

[0104] The method provided in the embodiment of the present application calls a speech synthesis model to be trained, synthesizes a first predicted speech based on first sample text information including text content belonging to a first language and a second sample speech belonging to a second language. In the process of synthesizing the speech, a first predicted timbre feature corresponding to the second sample speech is extracted. In the case where the speech synthesis model has not been trained, the first predicted timbre feature will inevitably represent the second language. In order to make the first predicted timbre feature have no necessary connection with the second language, a language classification model is introduced, and the language classification model is called to perform language inversion classification on the first predicted timbre feature to obtain a first predicted language opposite to the second language. Then, when training the speech synthesis model, based on the first predicted timbre feature, the first predicted timbre feature is classified into the second language. The anti-training idea makes the first predicted language and the second language gradually approach the same, so that the first predicted timbre features will not be affected by the second language. Therefore, when performing speech synthesis based on the trained speech synthesis model, the timbre features extracted based on any text information belonging to any target language and the reference speech belonging to any language are irrelevant to the language to which the speech belongs. Instead, the speech will be synthesized according to the target language to which the text information belongs, thereby synthesizing a speech corresponding to the text information and belonging to the target language, realizing cross-language speech synthesis. Moreover, since the timbre of the synthesized speech is independent of the language, the timbre of the synthesized speech will not be affected by the language to which the reference speech belongs, thereby making the synthesized speech more natural.

[0105] Above Figure 2 The embodiment shown briefly illustrates the training process of the speech synthesis model. Figure 3 The illustrated embodiment describes in detail the training process of the speech synthesis model.

[0106] Figure 3 1 is a flow chart of a speech synthesis model training method provided in an embodiment of the present application. The execution subject of the embodiment of the present application is a computer device. Figure 3 , the method comprises the following steps:

[0107] 301. A computer device obtains first sample text information, a first sample voice, and a second sample voice, where the first sample voice belongs to a first language and corresponds to the first sample text information, and the second sample voice belongs to a second language.

[0108] The first sample text information includes text content in the first language, and the first sample voice corresponds to the first sample text information, that is, the voice content contained in the first sample voice is consistent with the text content in the first sample text information. In addition, the first sample voice and the second sample voice are voices of the same sample object, and the first language is different from the second language. Therefore, the first sample voice and the second sample voice are voices of the same object in different languages, where the object refers to the speaker. In addition, the voice content contained in the second sample voice and the first sample voice can be the same or different.

[0109] In one possible implementation, a computer device stores a first data set, the first data set includes multiple sample text information and multiple sample texts, and the multiple sample voices in the first data set belong to at least two languages ​​and belong to at least two sample objects, wherein the multiple sample text information in the first data set all have corresponding sample voices, and for each sample object, the first data set at least includes sample voices in at least two languages ​​belonging to each sample object. From the first data set, the first sample text information, the first sample voice, and the second sample voice are obtained for this iterative training. The method of obtaining training samples in other iterative training processes is similar to this.

[0110] For example, the first data set includes 50 sample text information, 50 sample speech belonging to English, 50 sample speech belonging to Chinese, and 50 sample speech belonging to French. Among these 50 sample text information, 25 sample text information have consistent content and are sample speech belonging to Chinese, and the other 25 sample texts have corresponding sample speech belonging to French. These 150 sample speech belong to 10 sample objects. For one sample object, multiple sample speech belonging to the sample object belong to at least two languages.

[0111] Optionally, the first sample text information and the second sample speech are randomly selected from the first data set. For example, for a sample object m, there are N sample speech of the sample object in the first data, then a sample speech is randomly selected from the N sample speech as the second sample speech, that is:

[0112] y r =y Random(N)

[0113] Among them, y r represents the second sample speech, Random(N) represents any integer randomly determined from [1,N], r represents a randomly determined integer, y Random(N) That is, a sample speech is randomly selected from N sample speech.

[0114] In a possible implementation, the first sample text information also includes phonemes corresponding to the text content, and the phonemes are expressed in the first language, indicating how the text content is pronounced. When the first sample text information includes phonemes, it is possible to determine how the text content should sound in the subsequently synthesized speech. For example, if the text content is Chinese, the phonemes are initial consonants, finals, and tones; if the text content is English, the phonemes are English phonetic symbols.

[0115] 302. The computer device calls a text feature extraction network in the speech synthesis model to extract a first predicted text feature corresponding to the first sample text information.

[0116] The text feature extraction network is used to extract text features corresponding to the text information input into the speech synthesis model. The first predicted text feature is used to describe the text content in the first sample text information, and because the text content belongs to the first language, the first predicted text feature will contain features related to the first language, and the first predicted text feature indicates the pronunciation of the speech to be synthesized. The first predicted text feature is in matrix form, vector form or other form.

[0117] In a possible implementation, when the first sample text information includes text content and phonemes, the computer device calls a text feature extraction network, performs text feature extraction based on the text content and phonemes, and obtains a first predicted text feature. When the first sample text information includes text content but does not include phonemes, the computer device calls a text feature extraction network, queries the phonemes corresponding to the text content based on the text content, and then performs text feature extraction based on the text content and the queried phonemes to obtain a first predicted text feature.

[0118] In a possible implementation, the text feature extraction network includes n groups of coding layers, where n is an integer greater than 2. The computer device calls the first group of coding layers to encode the first sample text information to obtain the coding features output by the first group of coding layers. The first group of coding layers includes one coding layer, that is, the coding layer is called to encode the first sample text information to obtain the coding features output by the coding layer. Then the second group of coding layers is called to encode based on the coding features output by the first group of coding layers to obtain the coding features output by the second group of coding layers, and then the next group of coding layers is called until the last group of coding layers is called to encode the coding features output by the previous group of coding layers to obtain the coding features output by the last group of coding layers, and the coding features output by the last group of coding layers are used as the first predicted text features.

[0119] For any group of coding layers except the first group of coding layers, take the i-th group of coding layers as an example, i is an integer greater than 1 and not greater than n, and there is a group including at least two coding layers in the second to n-th groups of coding layers, that is, at least one group of coding layers in the second to n-th groups of coding layers includes at least two coding layers. In the case where the i-th group of coding layers includes at least two coding layers, at least two coding layers are respectively called to encode the coding features output by the i-1-th group of coding layers to obtain at least two coding features, and the fusion layer corresponding to the i-th group of coding layers is called to fuse the at least two coding features and the coding features output by the i-1-th group of coding layers, and the fused coding features are determined as the coding features output by the i-th group of coding layers. Alternatively, in the case where the i-th group of coding layers includes one coding layer, the i-th group of coding layers is called to encode the coding features output by the i-1-th group of coding layers to obtain the coding features output by the i-th group of coding layers.

[0120] Optionally, the network structures of multiple encoding layers in the text feature extraction network are the same, and the model parameters in the multiple encoding layers are different. Therefore, for different encoding layers, even if the same input encoding feature is encoded, the output encoding features are different.

[0121] Optionally, the computer device calls the fusion layer to fuse the at least two coding features and the coding features output by the i-1th group of coding layers, including: calling the fusion layer to concatenate or add the at least two coding features and the coding features output by the i-1th group of coding layers; or, calling the fusion layer to respectively determine the third weights corresponding to the at least two coding features and the coding features output by the i-1th group of coding layers, and then performing weighted processing on the at least two coding features and the coding features output by the i-1th group of coding layers based on multiple third weights.

[0122] For example, the following formula is used to represent the first predicted text feature extracted by the text feature extraction network:

[0123]

[0124] Among them, Z out represents the first predicted text feature, L out () indicates that the last group of coding layers encodes the input coding features, k indicates that there are k coding layers in the n-1th group of coding layers, and Z l represents the encoding features output by the n-2th group of encoding layers, S l (Z l ) represents the lth coding layer pair Z in the n-1th group of coding layers l to encode.

[0125] The text feature extraction network in the embodiment of the present application uses different coding layers to encode the same coding feature. The coding features obtained by encoding the same coding feature in different coding layers have different emphases. Therefore, by encoding the same coding feature through multiple parallel coding layers, the obtained coding feature can more accurately represent the first sample text information, thereby improving the accuracy of the extracted first predicted text feature.

[0126] For example, see Figure 4 The text feature extraction network shown in the figure has a coding layer of Transformer (a neural network based on the attention mechanism). In this case, the text feature extraction network can be called a multi-stream Transformer. The text feature extraction network includes three groups of coding layers, wherein the second group of coding layers includes two coding layers, and there is a fusion layer connected to the second group of coding layers. The first group of coding layers is the input layer in the text feature extraction network, and the third group of coding layers is the output layer in the text feature extraction network. The computer device calls the first group of coding layers to encode the first sample text information to obtain the coding features output by the first group of coding layers; calls the two coding layers in the second group of coding layers to encode the coding features output by the first group of coding layers respectively to obtain two coding features, and then calls the fusion layer to obtain the coding features output by the second group of coding layers from the two coding features and the coding features output by the first group of coding layers; and then calls the third group of coding layers to encode the coding features output by the second group of coding layers to obtain the first predicted text features.

[0127] Among them, for each Transformer, the computer device calls the Transformer, maps the input coding features to three feature spaces respectively to obtain the mapped coding features, and the three feature spaces correspond to the Q (query) dimension, K (key) dimension and V (value) dimension respectively, and then processes the mapped coding features based on the multi-head attention mechanism, and performs one-dimensional convolution processing on the processed coding features to obtain the coding features output by the coding layer.

[0128] In another possible implementation, the speech synthesis model includes a text feature fusion network and at least two types of text feature extraction networks. The computer device calls at least two types of text feature extraction networks respectively, performs text feature extraction on the first sample text information, and obtains at least two types of second predicted text features. Among them, different types of text feature extraction networks have different focuses when extracting features from the first sample text information, so the importance of different contents in the first sample text information described by different types of text features is different. For example, one type of text feature extraction network pays more attention to the connection between the context of the text content when extracting text features, and the first predicted text feature extracted by calling this type of text feature extraction network can accurately describe the global information of the text content, but may ignore the local detail information of the text content; another type of text feature extraction network pays more attention to the local details of the text content when extracting text features, such as paying attention to the meaning of certain words in the text content, and the first predicted text feature extracted by calling this type of text feature extraction network can accurately describe the local detail information of the text content, but may ignore the global information of the text content.

[0129] Then, the computer device calls the text feature fusion network, and performs weighted processing on at least two types of second predicted text features based on the first weights of at least two types of text feature extraction networks, to obtain a first predicted text feature. By fusing at least two types of second predicted text features, the obtained first predicted text feature can more accurately describe the first sample text information.

[0130] Among them, the first weight represents the importance of the text feature extracted by the corresponding type of text feature extraction network, and the sum of the first weights of the at least two types of text feature extraction networks is 1. Optionally, the first weight of each type of text feature extraction network is a preset weight. For example, there are two types of text feature extraction networks, and the first weight of each type of text feature extraction network is set to 0.5; or, the first weight is obtained based on attention mechanism learning, for example, an attention network is added between the text feature fusion network and at least two types of text feature extraction networks, and at least two types of second predicted text features are processed based on the attention network to determine the first weight corresponding to each type of second predicted text feature, that is, the first weight of each type of text feature extraction network.

[0131] For example, the text feature extraction network is a multi-stream Transformer, or a CBHG (1-D ConvolutionBank+Highway Network+Bidirectional Gated Recurrent Unit). Among them, CBHG includes multiple convolutional layers and a bidirectional Gated Recurrent Unit (GRU) network, wherein multiple convolutional layers can extract local detail information in text information, and the bidirectional GRU network can extract context information.

[0132] 303. The computer device calls a timbre feature extraction network in the speech synthesis model to extract a first predicted timbre feature corresponding to the second sample speech.

[0133] The first predicted timbre feature represents the timbre of the sample object determined based on the second sample speech, the timbre represented by the first predicted timbre feature is independent of the speech content of the second sample speech, and is related to the sample object that emits the second sample speech, and for any sample object, the sample object has a unique corresponding timbre feature. The first predicted timbre feature is in matrix form, vector form or other form.

[0134] In a possible implementation, the timbre feature extraction network includes a timbre feature extraction layer and a first attention layer. The computer device calls the timbre feature extraction layer to extract features of the second sample speech to obtain an initial predicted timbre feature of the sample object. The initial predicted timbre feature represents the timbre of the sample object.

[0135] Optionally, the timbre feature extraction layer is a reference encoder (Reference Encoder), which includes a multi-layer CNN (Convolutional Neural Network), a unidirectional GRU network and an activation layer, and the reference embedding vector (Reference Embedding) output by the reference encoder is determined as the initial predicted timbre feature. For example, see Figure 5 , the second sample speech is input into the 6-layer CNN to obtain the output coding features, and then the output coding features are input into the 128-unit GRU network (including 128 units of the GRU network), the features output by the 128-unit GRU network are input into the activation layer, and the features output by the activation layer are determined as the reference embedding vector. Of course, the timbre feature extraction layer can also be other network structures.

[0136] Although the initial predicted timbre features extracted above are already able to represent the timbre of the sample object, in order to improve the robustness of the timbre feature extraction network, a first attention layer is further introduced. The first attention layer includes reference timbre features of multiple reference objects. The multiple reference objects are any multiple objects, and the reference timbre features represent the timbre of the corresponding reference objects. The computer device calls the first attention layer to determine the second weight of the initial predicted timbre features corresponding to each reference object. The second weight represents the degree of similarity between the reference timbre features of the corresponding reference object and the initial predicted timbre features. The larger the second weight, the more similar the corresponding reference timbre features are to the initial predicted timbre features. Then, based on the multiple second weights, the reference timbre features of the multiple reference objects are weighted to obtain the first predicted timbre features, that is, the reference timbre features of the multiple reference objects are used to represent the timbre features of the sample object.

[0137] In another possible implementation, the computer device determines the initial predicted timbre feature obtained by calling the timbre feature extraction layer as the first predicted timbre feature.

[0138] It should be noted that the embodiment of the present application is only described by taking the example of first executing step 302 and then executing step 303. In another embodiment, step 303 may be executed first and then step 302, or step 302 and step 303 may be executed simultaneously. The embodiment of the present application does not limit this.

[0139] 304. The computer device calls a speech synthesis network to synthesize a first predicted speech based on the first predicted text feature and the first predicted timbre feature.

[0140] Among them, the first predicted speech is synthesized according to the timbre represented by the first predicted timbre feature and the content indicated by the first predicted text feature. The first predicted speech belongs to the first language, and the speech content is the text content in the first sample text information. Therefore, the first predicted speech can be regarded as the speech obtained by the above-mentioned object uttering the above-mentioned text content.

[0141] In one possible implementation, the speech synthesis network includes a second attention layer and a decoding layer. The computer device calls the second attention layer to fuse the first predicted text feature and the first predicted timbre feature to obtain a fused feature, which is used to describe the timbre and speech content of the speech to be synthesized; calls the decoding layer to decode the fused feature to obtain a predicted acoustic feature, which is used to describe the speech to be synthesized; and determines the first predicted speech based on the predicted acoustic feature.

[0142] Optionally, the computer device calls the second attention layer to respectively determine the weight corresponding to the first predicted text feature and the weight corresponding to the first predicted timbre feature, and then performs weighted processing on the first predicted text feature and the first predicted timbre feature based on the weight corresponding to the first predicted text feature and the weight corresponding to the first predicted timbre feature to obtain a fused feature.

[0143] 305. The computer device calls a language classification model to perform language reversal classification on the first predicted timbre feature to obtain a first predicted language to which the second sample speech belongs. The language reversal classification is used to make the first predicted language opposite to the language represented by the first predicted timbre feature.

[0144] Since the timbre features extracted from the input speech will inevitably represent the language to which the speech belongs when the speech synthesis model has not been trained, the first predicted timbre features extracted based on the second sample speech will represent the second language to which the second sample speech belongs. When the language classification model is called to perform language reversal classification on the first predicted timbre features, if the first predicted timbre features represent the second language, then the first predicted language obtained by performing language reversal classification on the first predicted timbre features by the language classification model is opposite to the second language. For example, Chinese is defined as the opposite of English, then when the second language is Chinese, the first predicted language is English.

[0145] The embodiment of the present application implements language reversal classification by any of the following two methods:

[0146] The first method is to invert the timbre features before classifying the language.

[0147] In this way, the language classification model includes a first inversion layer and a first classification layer. The computer device calls the first inversion layer to perform feature inversion on the first predicted timbre feature to obtain a first inverted timbre feature, the meaning of which is opposite to that of the first predicted timbre feature; and calls the first classification layer to classify the first inverted timbre feature to obtain a first predicted language.

[0148] Since the first predicted timbre feature will inevitably represent the second language when the speech synthesis model is not trained, the first inverted timbre feature obtained by inverting the first predicted timbre feature will not represent the second language. Then, the first predicted language obtained by classifying the first inverted timbre feature will not be the second language. Subsequently, based on the idea of ​​adversarial training, the difference between the first predicted language and the second language is reduced, that is, the first inverted timbre feature is made to represent the second language by training the speech synthesis model and the language classification model. Accordingly, even for the same speech, the predicted timbre feature extracted by the speech synthesis model will no longer represent the second language, thereby achieving the purpose of having no necessary connection between the timbre feature extracted from the speech and the language to which the speech belongs.

[0149] Optionally, inverting the first predicted timbre feature includes: adding a negative sign to each numerical value in the first predicted timbre feature, or when the first predicted timbre feature only contains 0 and 1, converting 0 in the first predicted timbre feature to 1 and converting 1 to 0; or, other methods can be used to invert the first predicted timbre feature, which is not limited to this embodiment of the present application.

[0150] The second method: classify the languages ​​first and then reverse the classification results.

[0151] In this way, the language classification model includes a second inversion layer and a second classification layer. The computer device calls the second classification layer to classify the first predicted timbre feature into a language to obtain first classification information, which indicates the language to which the second sample speech obtained by the current language classification belongs; calls the second inversion layer to invert the first classification information to obtain second classification information, and determines the first predicted language based on the second classification information.

[0152] Similar to the first method mentioned above, since the first predicted timbre feature will inevitably represent the second language when the speech synthesis model is not trained, the first predicted timbre feature is classified and the first classification information obtained will inevitably represent the second language. By inverting the first classification information, the second classification information obtained will not represent the second language. Subsequently, based on the idea of ​​adversarial training, the difference between the first predicted language and the second language is reduced, that is, the second classification information is made to represent the second language by training the speech synthesis model and the language classification model. Correspondingly, even for the same speech, the first classification information output by the language classification model will no longer represent the second language, thereby achieving the purpose of eliminating the necessary connection between the timbre feature extracted from the speech and the language to which the speech belongs.

[0153] 306. The computer device calls the object classification model to perform object classification on the first predicted timbre feature to obtain a predicted object identifier corresponding to the sample object.

[0154] In an embodiment of the present application, in order to ensure that the first predicted timbre feature extracted by the timbre feature extraction network is the timbre feature of the sample object, an object classification model is introduced, and the object classification model is called to perform object classification on the first predicted timbre feature to obtain a predicted object identifier, which is used to uniquely represent an object that can emit the timbre represented by the first predicted timbre feature.

[0155] In one possible implementation, a computer device calls an object classification model to perform object classification on the first predicted timbre feature, and obtains first probabilities corresponding to multiple object identifiers, respectively. The first probabilities represent the possibility that the corresponding object identifier is the identifier of the sample object, and the object identifier corresponding to the largest first probability is the predicted object identifier.

[0156] 307. The computer device trains an object classification model, a language classification model, and a speech synthesis model based on the first predicted language and the second language, the first sample speech and the first predicted speech, and the sample object identifier and the predicted object identifier, so that the difference between the first predicted language and the second language is reduced, the difference between the first sample speech and the first predicted speech is reduced, and the difference between the sample object identifier and the predicted object identifier is reduced.

[0157] When training the language classification model and the speech synthesis model, the difference between the first predicted language and the second language is reduced, that is, the second predicted language obtained by language reversal classification of the first predicted timbre features is gradually approached to be the same as the second language. This requires that the first predicted timbre features cannot represent the second language. Therefore, through the interaction between the language classification model and the speech synthesis model, the first predicted timbre features extracted by the speech synthesis model can be made less and less affected by the second language.

[0158] When training the object classification model and the speech synthesis model based on the predicted object identifier and the sample object identifier, the difference between the sample object identifier and the predicted object identifier is reduced, that is, the sample object identifier and the predicted object identifier are made to be close to the same. This requires that the first predicted timbre feature corresponds to the actual sample object to which it belongs and represents the timbre of the sample object.

[0159] In one possible implementation, for the predicted object identifier and the sample object identifier, in the case where the predicted object identifier corresponding to the predicted probability indication is used, the following cross entropy loss function is used to train the speech synthesis model and the object classification model:

[0160]

[0161] Among them, L spk represents the loss value, CE() represents the cross entropy loss, N represents the N second sample speech of the sample object, represents the predicted probability corresponding to the rth second sample speech, The sample object ID of the sample object.

[0162] When training a speech synthesis model based on the first sample speech and the first predicted speech, the difference between the first sample speech and the first predicted speech is reduced, that is, the speech synthesis model is enabled to learn the ability to synthesize speech based on timbre features and text features. Moreover, since the second language to which the second sample speech input into the speech synthesis model belongs is different from the first language to which the synthesized speech belongs, that is, when a speech belonging to one language is input into the speech synthesis model, a speech belonging to another language can be synthesized, so that the trained speech synthesis model has the ability of cross-language speech synthesis.

[0163] In addition, in one possible implementation, see Figure 6 The speech synthesis model shown. The first sample text information is input into multiple text feature extraction networks, which include at least a multi-stream Transformer and a CBHG; then the text features output by the multiple text feature extraction networks are input into a text feature fusion network to obtain a first predicted text feature output by the text feature fusion network, and the first predicted text feature is input into the second attention layer; the second sample speech is input into the timbre feature extraction layer, and the initial predicted timbre feature extracted by the timbre feature extraction layer is input into the first attention layer, Figure 6 Taking 4 reference objects as an example, based on the first attention layer, the weights corresponding to the reference timbre features of the 4 reference objects are determined to be 0.3, 0.4, 0.2 and 0.1 respectively, and the 4 reference timbre features are weighted based on the four weights to obtain a first predicted timbre feature, and the first predicted timbre feature is input into the object classification model and the language classification model for processing, and the first predicted timbre feature is input into the second attention layer at the same time, and the second attention layer is called to process the first predicted timbre feature and the first predicted text feature to obtain a fused feature, and then the decoding layer is called to decode the fused feature to obtain a first predicted speech. Among them, the language classification model includes a gradient reversal layer (Gradient ReversalLayer) and a language classification layer, and the gradient reversal layer is the second reversal layer in the above embodiment, which is used to reverse the first classification information output by the language classification layer.

[0164] It should be noted that the training process shown in step 301-step 307 is a training process of the speech synthesis model. After the speech synthesis model is trained based on the first sample text information, the first sample speech and the second sample speech, the speech synthesis model can continue to be trained. The training process is the same as the training process shown in the above embodiment, except that the training data used is different. In the process of continuing to train the speech synthesis model, sample text information and sample speech belonging to two languages ​​are obtained from the first data set. The sample text information and sample speech belonging to two languages ​​obtained this time are not completely the same as the first sample text information, the first sample speech and the second sample speech. For example, the speech synthesis model is trained based on the third sample text information, the fourth sample speech and the fifth sample speech belonging to the same other sample object, wherein the third sample text information includes text content belonging to a third language, the fourth sample speech belongs to a third language and corresponds to the third sample text information, the fifth sample speech belongs to a fourth language, the third language is different from the fourth language, the third language and the fourth language can be the same two languages ​​as the first language and the second language, or can be two languages ​​that are not completely the same. However, before the speech synthesis model training is completed, it is necessary to use sample speech belonging to at least two languages ​​and at least two sample objects to train the speech synthesis model, so as to train a multi-object and multi-language speech synthesis model.

[0165] The method provided in the embodiment of the present application calls a speech synthesis model to be trained, synthesizes a first predicted speech based on first sample text information including text content belonging to a first language and a second sample speech belonging to a second language. In the process of synthesizing the speech, a first predicted timbre feature corresponding to the second sample speech is extracted. In the case where the speech synthesis model has not been trained, the first predicted timbre feature will inevitably represent the second language. In order to make the first predicted timbre feature have no necessary connection with the second language, a language classification model is introduced, and the language classification model is called to perform language inversion classification on the first predicted timbre feature to obtain a first predicted language opposite to the second language. Then, when training the speech synthesis model, based on the first predicted timbre feature, the first predicted timbre feature is classified into the second language. The anti-training idea makes the first predicted language and the second language gradually approach the same, so that the first predicted timbre features will not be affected by the second language. Therefore, when performing speech synthesis based on the trained speech synthesis model, the timbre features extracted based on any text information belonging to any target language and the reference speech belonging to any language are irrelevant to the language to which the speech belongs. Instead, the speech will be synthesized according to the target language to which the text information belongs, thereby synthesizing a speech corresponding to the text information and belonging to the target language, realizing cross-language speech synthesis. Moreover, since the timbre of the synthesized speech is independent of the language, the timbre of the synthesized speech will not be affected by the language to which the reference speech belongs, thereby making the synthesized speech more natural.

[0166] Furthermore, in an embodiment of the present application, multiple text feature extraction networks are used to extract text features of the first sample text information, or a text feature extraction network including multiple groups of encoders, and at least one group of encoders includes multiple parallel encoders is used to extract text features of the first sample text information. In the process of extracting text features, the global information and local detail information of the first sample text information can be more fully and comprehensively considered, so that the extracted first predicted text features can more accurately describe the first sample text information, thereby improving the accuracy of the first predicted text features.

[0167] In addition, in an embodiment of the present application, an object classification model is also introduced. By classifying the first predicted timbre features, it is ensured that the timbre features extracted from the second sample speech are the only features corresponding to the sample object, so that the speech synthesis model can accurately distinguish the object to which the input speech belongs in order to synthesize the speech of the object.

[0168] Using the above Figure 3 The speech synthesis model trained in the illustrated implementation is a multi-object multi-language speech synthesis model. After the speech synthesis model is trained, on the one hand, the computer device can call the speech synthesis model to perform speech synthesis. On the other hand, in order to train a speech synthesis model for a specific object, after the speech synthesis model is trained based on the first data set, the speech synthesis model can continue to be trained based on the training data of the specific object. Figure 7 The computer device first obtains training data, then trains a multi-object multi-language speech synthesis model, and then trains a multi-language speech synthesis model for the target object based on the multi-object multi-language speech synthesis model.

[0169] Below through Figure 8 The illustrated embodiment illustrates the process of calling a multi-object multi-language speech synthesis model to perform speech synthesis.

[0170] Figure 8 is a flow chart of a speech synthesis method provided by an embodiment of the present application. The execution subject of the embodiment of the present application is a computer device. Figure 8 , the method comprises the following steps:

[0171] 801. A computer device obtains target text information and reference speech.

[0172] The reference speech belongs to any object and any language, and the target text information includes text content in the target language.

[0173] 802. The computer device calls the speech synthesis model, extracts the target text features corresponding to the target text information, extracts the target timbre features corresponding to the reference speech, and synthesizes the target speech corresponding to the target text information and belonging to the target language based on the target text features and the target timbre features.

[0174] The speech synthesis model includes a text feature extraction network, a timbre feature extraction network and a speech synthesis network. The computer device calls the text feature extraction network to extract text features corresponding to the target text information; calls the timbre feature extraction network to extract timbre features corresponding to the reference speech; and calls the speech synthesis network to synthesize the target speech based on the text features and timbre features.

[0175] The implementation of step 802 is the same as the implementation of steps 302 to 304 above, and will not be described in detail here.

[0176] The speech synthesis model provided in the embodiment of the present application introduces a language classification model during the training process, and conducts adversarial training on the language classification model and the speech synthesis model, so that the timbre features extracted by the trained speech synthesis model for the input reference speech will not be affected by the language to which the reference speech belongs. Therefore, when speech synthesis is performed based on the speech synthesis model, the timbre features extracted based on any text information belonging to any target language and a reference speech belonging to any language are irrelevant to the language to which the speech belongs, but the speech is synthesized according to the target language to which the text information belongs, thereby synthesizing a speech corresponding to the text information and belonging to the target language, thereby realizing cross-language speech synthesis, and since the timbre of the synthesized speech is independent of the language, the timbre of the synthesized speech will not be affected by the language to which the reference speech belongs, thereby making the synthesized speech have a higher degree of naturalness.

[0177] Below through Fig. 9 The illustrated embodiment illustrates the process of training a multi-language speech synthesis model for a target object.

[0178] Fig. 9 1 is a flowchart of another speech synthesis model training method provided in an embodiment of the present application. The execution subject of the embodiment of the present application is a computer device. Fig. 9 , the method comprises the following steps:

[0179] 901. A computer device obtains second sample text information and a third sample voice from a second data set, where the second sample text information includes text content in a first language.

[0180] Among them, the second data set includes multiple sample text information and multiple sample voices. The second data set includes text information belonging to the first language, includes sample voices belonging to the second language, and does not include sample voices belonging to the first language. The multiple sample voices in the second data set belong to the same target sample object, that is, the multiple sample voices are voices emitted by the same target sample object, and the languages ​​of the multiple sample voices can be the same or different.

[0181] In a possible implementation, it is necessary to iterate the training multiple times when training the speech synthesis model. During the multiple training processes, it is necessary to select multiple sample text information and multiple sample speech from the second data set. Figure 3 When training the speech synthesis model in the manner shown, the number of sample speech in the first data set used is different from the number of sample speech in the second data set used this time. In order to improve the stability of the trained speech synthesis model, it is necessary to ensure that the number of sample speech selected from the first data set for training the speech synthesis model is as close as possible to the number of sample speech selected from the second data set for training the speech synthesis model. Select in the following manner:

[0182] The computer device obtains a first quantity and a second quantity respectively, the first quantity being the quantity of sample speech in the second data set, and the second quantity being the quantity of sample speech in the first data set; based on the ratio between the first quantity and the second quantity, a selection ratio is determined, the selection ratio being the ratio of the quantity of sample speech selected from the second data set for training the speech synthesis model to the first quantity, and based on the selection ratio, a third sample speech for training the speech synthesis model is selected from the second data set. The selection ratio is negatively correlated with the ratio, that is, when the first quantity is greater than the second quantity, more sample speech needs to be selected from the second data set, and when the second quantity is greater than the first quantity, fewer sample speech needs to be selected from the second data set.

[0183] For example, the following formula is used to express the selection ratio:

[0184]

[0185] Among them, p represents the selection ratio, U o represents the first quantity, represents the second quantity, U t represents the number of sample speech of sample object t in the first data set, M represents the number of sample objects in the first data set, α is a preset parameter, and α is greater than or equal to 0 and less than or equal to 1.

[0186] It should be noted that the embodiment of the present application is only described by taking the example that the text information in the second data set belongs to the first language and the sample speech belongs to the second language. In another embodiment, the text information in the second data set may belong to the third language and the sample speech may belong to the fourth language. The embodiment of the present application does not limit this.

[0187] 902. The computer device calls a speech synthesis model, extracts a third predicted text feature corresponding to the second sample text information, extracts a second predicted timbre feature corresponding to the third sample speech, and synthesizes a second predicted speech based on the third predicted text feature and the second predicted timbre feature.

[0188] The implementation of step 902 is the same as the implementation of the above steps 302 to 304, and will not be repeated here.

[0189] 903. The computer device calls the language classification model to perform language reversal classification on the second predicted timbre feature to obtain the second predicted language to which the third sample speech belongs. The language reversal classification is used to make the second predicted language opposite to the language represented by the second predicted timbre feature.

[0190] The implementation of step 903 is the same as the implementation of step 305 above, and will not be described in detail here.

[0191] 904. The computer device trains a language classification model and a speech synthesis model based on the second predicted language and the second language to reduce the difference between the second predicted language and the second language; and outputs the second predicted speech, obtains adjustment parameters based on the second predicted speech input, and trains the speech synthesis model according to the adjustment parameters.

[0192] The implementation of training the language classification model and the speech synthesis model based on the second predicted language and the second language is the same as the implementation in the above step 307, which will not be repeated here.

[0193] The difference is that in the embodiment of the present application, since there is no sample speech belonging to the first language when training the speech synthesis model, the second predicted speech is output first after synthesizing the second predicted speech. The technician determines whether it is necessary to continue training the speech synthesis model based on the clarity and naturalness of the second predicted speech. When it is determined that the speech synthesis model needs to be continued to be trained, an adjustment parameter is input, and the adjustment parameter indicates the adjustment range of the model parameter in the speech synthesis model. For example, the adjustment parameter is 0.01%, and the model parameter is increased by 0.01%. The speech synthesis model is then adjusted according to the adjustment parameter. Of course, in another embodiment, other methods can also be used to determine the clarity and naturalness of the second predicted speech, and the embodiment of the present application does not limit this.

[0194] Optionally, when the computer device is a terminal, the terminal plays the second predicted speech, and correspondingly, the terminal obtains the input adjustment parameters and adjusts the speech synthesis model according to the adjustment parameters; when the computer device is a server, the server sends the second predicted speech to the terminal, the terminal plays the second predicted language, the terminal obtains the input adjustment parameters, and sends the adjustment parameters to the server, and the server adjusts the speech synthesis model according to the adjustment parameters.

[0195] In a possible implementation, the computer device calls the object classification model to perform object classification on the second predicted timbre feature to obtain a predicted object identifier corresponding to the target sample object. Then, the computer device trains the object classification model and the speech synthesis model based on the target sample object identifier and the corresponding predicted object identifier corresponding to the target sample object, so as to reduce the difference between the target sample object identifier and the predicted object identifier corresponding to the target sample object.

[0196] In addition, after training a multilingual speech synthesis model for a target object, the computer device can call the speech synthesis model to synthesize the speech of the target object. That is, the computer device obtains text information and a reference speech belonging to the target object, wherein the reference speech belongs to any language, and the target text information includes text content belonging to the target language. Then the speech synthesis model is called to extract text features corresponding to the text information, extract timbre features corresponding to the reference speech, and synthesize a target speech corresponding to the text information, belonging to the target language and belonging to the target object based on the text features and timbre features. The target object is the above-mentioned target sample object.

[0197] The method provided in the embodiment of the present application calls a trained multi-object multi-language speech synthesis model, synthesizes a second predicted speech based on the second sample text information and the third sample speech belonging to the target sample object, and introduces a language classification model in the process of synthesizing speech. The language classification model is called to perform language inversion classification on the second predicted timbre feature to obtain a second predicted language opposite to the second language. Then, when training the speech synthesis model, based on the adversarial training idea, the second predicted language is gradually approached to the same as the second language, so that the second predicted timbre feature will not be affected by the second language. Therefore, when performing speech synthesis based on the trained speech synthesis model, the timbre features extracted based on any text information belonging to any target language and the reference speech belonging to any language are independent of the language to which the speech belongs, but the speech is synthesized according to the target language to which the text information belongs, thereby synthesizing a speech corresponding to the text information, belonging to the target language and belonging to the target sample object, thereby realizing cross-language speech synthesis. Moreover, since the trained multi-object and multi-language speech synthesis model already has a certain degree of accuracy, in the process of training the multi-language speech synthesis model for the target object, an accurate speech synthesis model can be trained without using the sample speech corresponding to the speech to be synthesized, thus realizing the training of a zero-resource speech synthesis model.

[0198] In an embodiment of the present application, when training a multi-object multi-language speech synthesis model, a total of 120 hours of Chinese speech from 60 subjects and 40 hours of English speech from 50 subjects were used for training. Then, cross-language transfer training was performed for 1 male and 1 female (each subject had 400 sentences of Chinese speech and no English speech), and the model training results are shown in Table 1. In Table 1, MOS (Mean Opinion Score) of naturalness and similarity is used as the evaluation index, Baseline is the model trained by the mainstream cross-language transfer method in the prior art, and Proposed is the speech synthesis model trained in this application. As can be seen from Table 1, in terms of naturalness and similarity, the speech synthesized by the speech synthesis model in this application is significantly better than the speech synthesized by the speech synthesis model in the prior art.

[0199] Table 1

[0200]

[0201]

[0202] The data in Table 1 above are the performance of the speech synthesis model trained when there is no training data in the language of the speech to be synthesized, that is, in the case of zero resources. When there is a small amount of training data in the language of the speech to be synthesized, as shown in Table 2, MOS of naturalness is used as the evaluation indicator. It can be seen from Table 2 that in terms of naturalness, the speech synthesized by the speech synthesis model in the present application is also significantly better than the speech synthesized by the speech synthesis model in the prior art.

[0203] Table 2

[0204] Model female male Baseline 4.24±0.05 4.13±0.05 Proposed 4.38±0.05 4.30±0.06

[0205] Fig.10 is a structural diagram of a speech synthesis model training device provided in an embodiment of the present application. Fig.10 , the device comprises:

[0206] The sample acquisition module 1001 is used to acquire first sample text information, a first sample voice, and a second sample voice, wherein the first sample text information includes text content belonging to a first language, the first sample voice and the second sample voice are voices of the same sample object, the first sample voice belongs to the first language and corresponds to the first sample text information, and the second sample voice belongs to a second language, and the first language is different from the second language;

[0207] The speech synthesis module 1002 is used to call the speech synthesis model, extract the first predicted text feature corresponding to the first sample text information, extract the first predicted timbre feature corresponding to the second sample speech, and synthesize the first predicted speech based on the first predicted text feature and the first predicted timbre feature;

[0208] A language classification module 1003 is used to call a language classification model to perform language reversal classification on the first predicted timbre feature to obtain a first predicted language to which the second sample speech belongs, wherein the language reversal classification is used to make the first predicted language opposite to the language represented by the first predicted timbre feature;

[0209] The model training module 1004 is used to train the language classification model and the speech synthesis model based on the first predicted language and the second language, as well as the first sample speech and the first predicted speech, so as to reduce the difference between the first predicted language and the second language, and reduce the difference between the first sample speech and the first predicted speech.

[0210] The device provided in the embodiment of the present application calls the speech synthesis model to be trained, synthesizes the first predicted speech based on the first sample text information including the text content belonging to the first language and the second sample speech belonging to the second language. In the process of synthesizing the speech, the first predicted timbre feature corresponding to the second sample speech is extracted. In the case that the speech synthesis model is not trained, the first predicted timbre feature will inevitably represent the second language. In order to make the first predicted timbre feature have no necessary connection with the second language, a language classification model is introduced, the language classification model is called, the first predicted timbre feature is language-reversed classified, and the first predicted language opposite to the second language is obtained. Then, when training the speech synthesis model, the first predicted timbre feature is used as the first predicted language. Based on the idea of ​​adversarial training, the first predicted language and the second language are gradually approached to be the same, so that the first predicted timbre features no longer represent the second language. Therefore, when speech synthesis is performed based on the trained speech synthesis model, the timbre features extracted based on any text information belonging to any target language and a reference speech belonging to any language are irrelevant to the language to which the speech belongs. Instead, the speech will be synthesized according to the target language to which the text information belongs, thereby synthesizing a speech corresponding to the text information and belonging to the target language, realizing cross-language speech synthesis. Moreover, since the timbre of the synthesized speech is independent of the language, the timbre of the synthesized speech will not be affected by the language to which the reference speech belongs, thereby making the synthesized speech have a higher degree of naturalness.

[0211] In a possible implementation, the language classification module 1003 is used to:

[0212] The language classification model includes a first inversion layer and a first classification layer, calling the first inversion layer to perform feature inversion on the first predicted timbre feature to obtain a first inverted timbre feature, the meaning represented by the first inverted timbre feature is opposite to the meaning represented by the first predicted timbre feature; and calling the first classification layer to classify the first inverted timbre feature to obtain the first predicted language; or,

[0213] The language classification model includes a second inversion layer and a second classification layer. The second classification layer is called to classify the language of the first predicted timbre feature to obtain first classification information, and the first classification information indicates the language to which the second sample speech obtained by the current language classification belongs; the second inversion layer is called to invert the first classification information to obtain second classification information, and the first predicted language is determined based on the second classification information.

[0214] In another possible implementation, the sample object has a corresponding sample object identifier, see Fig.11 , the device further comprises:

[0215] The object classification module 1005 is used to call the object classification model to perform object classification on the first predicted timbre feature to obtain a predicted object identifier corresponding to the sample object;

[0216] The model training module 1004 is used to train the object classification model, the language classification model and the speech synthesis model based on the first predicted language and the second language, the first sample speech and the first predicted speech, and the sample object identifier and the predicted object identifier.

[0217] In another possible implementation, the speech synthesis model includes a text feature extraction network, a timbre feature extraction network and a speech synthesis network, see Fig.11 , the speech synthesis module 1002 comprises:

[0218] A text feature extraction unit 1012 is used to call the text feature extraction network to extract the first predicted text feature corresponding to the first sample text information;

[0219] The timbre feature extraction unit 1022 is used to call the timbre feature extraction network to extract the first predicted timbre feature corresponding to the second sample speech;

[0220] The speech synthesis unit 1032 is used to call the speech synthesis network to synthesize the first predicted speech based on the first predicted text feature and the first predicted timbre feature.

[0221] In another possible implementation, the speech synthesis model includes a text feature fusion network and at least two types of text feature extraction networks, see Fig.11 , the text feature extraction unit 1012 is used to:

[0222] The at least two types of text feature extraction networks are respectively called to extract text features from the first sample text information to obtain at least two types of second predicted text features, where different types of text features describe different importance levels of different contents in the first sample text information;

[0223] The text feature fusion network is called, and based on the first weights of the at least two types of text feature extraction networks, the at least two types of second predicted text features are weighted to obtain the first predicted text feature.

[0224] In another possible implementation, the text feature extraction network includes n sets of encoding layers, where n is an integer greater than 2; see Fig.11 , the text feature extraction unit 1012 is used to:

[0225] Calling the first group of encoding layers to encode the first sample text information to obtain encoding features output by the first group of encoding layers;

[0226] In the case where the i-th group of coding layers includes at least two coding layers, the at least two coding layers are respectively called to encode the coding features output by the i-1-th group of coding layers to obtain the at least two coding features, and a fusion layer corresponding to the i-th group of coding layers is called to fuse the at least two coding features and the coding features output by the i-1-th group of coding layers, and the fused coding features are determined as the coding features output by the i-th group of coding layers; or

[0227] In the case where the i-th group of coding layers includes one coding layer, calling the i-th group of coding layers to encode the coding features output by the i-1-th group of coding layers to obtain the coding features output by the i-th group of coding layers;

[0228] Among them, the coding feature output by the last group of coding layers is the first predicted text feature, i is an integer greater than 1 and not greater than n, the first group of coding layers includes one coding layer, and in the second to nth groups of coding layers, there is a group including at least two coding layers.

[0229] In another possible implementation, the timbre feature extraction network includes a timbre feature extraction layer and a first attention layer, wherein the first attention layer includes reference timbre features of multiple reference objects; see Fig.11 , the timbre feature extraction unit 1022 is used to:

[0230] Calling the timbre feature extraction layer to extract features of the second sample speech to obtain an initial predicted timbre feature of the sample object;

[0231] The first attention layer is called to determine the second weight of the initial predicted timbre feature corresponding to each reference object, the second weight representing the degree of similarity between the reference timbre feature of the corresponding reference object and the initial predicted timbre feature, and based on multiple second weights, the reference timbre features of the multiple reference objects are weighted to obtain the first predicted timbre feature.

[0232] In another possible implementation, the speech synthesis network includes a second attention layer and a decoding layer, see Fig.11 , the speech synthesis unit 1032 is used to:

[0233] Calling the second attention layer to fuse the first predicted text feature and the first predicted timbre feature to obtain a fused feature;

[0234] The decoding layer is called to decode the fused feature to obtain a predicted acoustic feature, and the first predicted speech is determined based on the predicted acoustic feature.

[0235] In another possible implementation, see Fig.11 , the device further comprises:

[0236] The information acquisition module 1006 is used to acquire target text information and reference speech, the reference speech belongs to any object and belongs to any language, and the target text information includes text content belonging to the target language;

[0237] The speech synthesis module 1002 is also used to call the speech synthesis model, extract the target text features corresponding to the target text information, extract the target timbre features corresponding to the reference speech, and synthesize the target speech corresponding to the target text information and belonging to the target language based on the target text features and the target timbre features.

[0238] In another possible implementation, the first data set includes a plurality of sample text information and a plurality of sample voices, and the plurality of sample voices in the first data set belong to at least two languages ​​and belong to at least two sample objects; the first sample text information, the first sample voice, and the second sample voice belong to the first data set;

[0239] The second data set includes a plurality of sample text information and a plurality of sample voices, the second data set includes the text information belonging to the first language, includes sample voices belonging to the second language, and does not include sample voices belonging to the first language, and the plurality of sample voices in the second data set belong to the same target sample object;

[0240] The device also includes:

[0241] The sample acquisition module 1001 is further used to acquire second sample text information and third sample speech from the second data set after training the language classification model and the speech synthesis model based on the first data set, wherein the first sample text information includes text content belonging to the first language;

[0242] The speech synthesis module 1002 is further used to call the speech synthesis model, extract the third predicted text feature corresponding to the second sample text information, extract the second predicted timbre feature corresponding to the third sample speech, and synthesize the second predicted speech based on the third predicted text feature and the second predicted timbre feature;

[0243] The language classification module 1003 is further used to call the language classification model to perform language reversal classification on the second predicted timbre feature to obtain the second predicted language to which the third sample speech belongs, wherein the language reversal classification is used to make the second predicted language opposite to the language represented by the second predicted timbre feature;

[0244] The model training module 1004 is also used to train the language classification model and the speech synthesis model based on the second predicted language and the second language to reduce the difference between the second predicted language and the second language; and to output the second predicted speech, obtain adjustment parameters based on the second predicted speech input, and train the speech synthesis model according to the adjustment parameters.

[0245] In another possible implementation, the sample acquisition module 1001 is further used to:

[0246] Obtain a first number and a second number respectively, the first number being the number of sample speech in the second data set, and the second number being the number of sample speech in the first data set;

[0247] Based on the ratio between the first number and the second number, determining a selection ratio, the selection ratio refers to the ratio of the number of sample speech selected from the second data set for training the speech synthesis model to the first number, and the selection ratio is negatively correlated with the ratio;

[0248] Based on the selection ratio, a third sample speech is selected from the second data set for training the speech synthesis model.

[0249] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0250] It should be noted that: the speech synthesis model training device provided in the above embodiment only uses the division of the above functional modules as an example when training the speech synthesis model. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the speech synthesis model training device provided in the above embodiment and the speech synthesis model training method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0251] An embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the operations performed by the speech synthesis model training method of the above embodiment.

[0252] Optionally, the computer device is provided as a terminal. Fig.12 1 is a schematic diagram of a terminal 1200 provided in an embodiment of the present application. The terminal 1200 includes: a processor 1201 and a memory 1202 .

[0253] The processor 1201 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1201 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1201 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1201 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1201 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0254] The memory 1202 may include one or more computer-readable storage media, which may be non-transitory. The memory 1202 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1202 is used to store at least one computer program, which is used to be executed by the processor 1201 to implement the speech synthesis model training method provided in the method embodiment of the present application.

[0255] In some embodiments, the terminal 1200 may further optionally include: a peripheral device interface 1203 and at least one peripheral device. The processor 1201, the memory 1202 and the peripheral device interface 1203 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1203 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1204, a display screen 1205, a camera assembly 1206, an audio circuit 1207 and a power supply 1208.

[0256] The peripheral device interface 1203 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1201 and the memory 1202. In some embodiments, the processor 1201, the memory 1202, and the peripheral device interface 1203 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1201, the memory 1202, and the peripheral device interface 1203 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0257] The radio frequency circuit 1204 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1204 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1204 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 1204 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 1204 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1204 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0258] The display screen 1205 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1205 is a touch display screen, the display screen 1205 also has the ability to collect touch signals on the surface or above the surface of the display screen 1205. The touch signal may be input as a control signal to the processor 1201 for processing.

[0259] The camera assembly 1206 is used to capture images or videos. Optionally, the camera assembly 1206 includes a front camera and a rear camera. The front camera is set on the front panel of the terminal, and the rear camera is set on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions.

[0260] The audio circuit 1207 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1201 for processing, or input them into the radio frequency circuit 1204 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 1200. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1201 or the radio frequency circuit 1204 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1207 may also include a headphone jack.

[0261] The power supply 1208 is used to power various components in the terminal 1200. The power supply 1208 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1208 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0262] Those skilled in the art will understand that Fig.12 The structure shown in the figure does not constitute a limitation on the terminal 1200, and the terminal 1200 may include more or fewer components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0263] Optionally, the computer device is provided as a server. Fig.131 is a schematic diagram of the structure of a server provided in an embodiment of the present application. The server 1300 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1301 and one or more memories 1302, wherein the memory 1302 stores at least one computer program, and the at least one computer program is loaded and executed by the processor 1301 to implement the methods provided in the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the server may also include other components for implementing device functions, which will not be described in detail here.

[0264] An embodiment of the present application also provides a computer-readable storage medium, which stores at least one computer program. The at least one computer program is loaded and executed by a processor to implement the operations performed by the speech synthesis model training method of the above embodiment.

[0265] An embodiment of the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the operations performed by the speech synthesis model training method of the above embodiment.

[0266] In some embodiments, the computer program involved in the embodiments of the present application may be deployed and executed on a computer device, or on multiple computer devices located at one location, or on multiple computer devices distributed at multiple locations and interconnected by a communication network. Multiple computer devices distributed at multiple locations and interconnected by a communication network may constitute a blockchain system.

[0267] It can be understood that in the specific implementation of the present application, related data such as user information (such as object identification, voice belonging to the object) is involved. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0268] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0269] The above are only optional embodiments of the embodiments of the present application and are not intended to limit the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the present application.

Claims

1. A speech synthesis model training method, characterized in that: The method comprises: Acquire first sample text information, a first sample voice, and a second sample voice, wherein the first sample text information includes text content in a first language, the first sample voice and the second sample voice are voices of the same sample object, the first sample voice belongs to the first language and corresponds to the first sample text information, the second sample voice belongs to a second language, and the first language is different from the second language; Calling a speech synthesis model, extracting a first predicted text feature corresponding to the first sample text information, extracting a first predicted timbre feature corresponding to the second sample speech, and synthesizing a first predicted speech based on the first predicted text feature and the first predicted timbre feature; Calling a language classification model to perform language reversal classification on the first predicted timbre feature to obtain a first predicted language to which the second sample speech belongs, wherein the language reversal classification is used to make the first predicted language opposite to the language represented by the first predicted timbre feature; Based on the first predicted language and the second language, as well as the first sample speech and the first predicted speech, training the language classification model and the speech synthesis model so that the difference between the first predicted language and the second language is reduced, and the difference between the first sample speech and the first predicted speech is reduced; Acquire target text information and reference speech, wherein the reference speech belongs to any object and any language, and the target text information includes text content belonging to the target language; The speech synthesis model is called to extract the target text features corresponding to the target text information, extract the target timbre features corresponding to the reference speech, and synthesize the target speech corresponding to the target text information and belonging to the target language based on the target text features and the target timbre features.

2. The method according to claim 1, characterized in that The calling of the language classification model to perform language inversion classification on the first predicted timbre feature to obtain the first predicted language to which the second sample speech belongs includes: The language classification model includes a first inversion layer and a first classification layer, calling the first inversion layer to perform feature inversion on the first predicted timbre feature to obtain a first inverted timbre feature, wherein the meaning represented by the first inverted timbre feature is opposite to the meaning represented by the first predicted timbre feature; and calling the first classification layer to classify the first inverted timbre feature to obtain the first predicted language; or The language classification model includes a second inversion layer and a second classification layer. The second classification layer is called to classify the first predicted timbre feature into a language to obtain first classification information, wherein the first classification information indicates the language to which the second sample speech obtained by the current language classification belongs; the second inversion layer is called to invert the first classification information to obtain second classification information, and the first predicted language is determined based on the second classification information.

3. The method according to claim 1, characterized in that The sample object has a corresponding sample object identifier. After extracting the first predicted timbre feature corresponding to the second sample speech, the method further includes: Calling an object classification model to perform object classification on the first predicted timbre feature to obtain a predicted object identifier corresponding to the sample object; The training of the language classification model and the speech synthesis model based on the first predicted language and the second language, as well as the first sample speech and the first predicted speech, comprises: The object classification model, the language classification model, and the speech synthesis model are trained based on the first predicted language and the second language, the first sample speech and the first predicted speech, and the sample object identifier and the predicted object identifier.

4. The method according to claim 1, characterized in that: The speech synthesis model includes a text feature extraction network, a timbre feature extraction network and a speech synthesis network. The calling of the speech synthesis model, extracting a first predicted text feature corresponding to the first sample text information, extracting a first predicted timbre feature corresponding to the second sample speech, and synthesizing a first predicted speech based on the first predicted text feature and the first predicted timbre feature, includes: Calling the text feature extraction network to extract the first predicted text feature corresponding to the first sample text information; Calling the timbre feature extraction network to extract the first predicted timbre feature corresponding to the second sample speech; The speech synthesis network is called to synthesize the first predicted speech based on the first predicted text feature and the first predicted timbre feature.

5. The method according to claim 4, characterized in that The speech synthesis model includes a text feature fusion network and at least two types of text feature extraction networks, and calling the text feature extraction network to extract the first predicted text feature corresponding to the first sample text information includes: respectively calling the at least two types of text feature extraction networks to extract text features from the first sample text information to obtain at least two types of second predicted text features, wherein different types of text features describe different importance levels of different contents in the first sample text information; The text feature fusion network is called to perform weighted processing on the at least two types of second predicted text features based on the first weights of the at least two types of text feature extraction networks, respectively, to obtain the first predicted text feature.

6. The method according to claim 4, characterized in that The text feature extraction network includes n groups of encoding layers, where n is an integer greater than 2; calling the text feature extraction network to extract the first predicted text feature corresponding to the first sample text information includes: Calling the first group of encoding layers to encode the first sample text information to obtain encoding features output by the first group of encoding layers; In the case where the i-th group of coding layers includes at least two coding layers, the at least two coding layers are respectively called to encode the coding features output by the i-1-th group of coding layers to obtain the at least two coding features, and a fusion layer corresponding to the i-th group of coding layers is called to fuse the at least two coding features and the coding features output by the i-1-th group of coding layers, and the fused coding features are determined as the coding features output by the i-th group of coding layers; or In a case where the i-th group of coding layers includes one coding layer, calling the i-th group of coding layers to encode the coding features output by the i-1-th group of coding layers to obtain the coding features output by the i-th group of coding layers; Among them, the encoding features output by the last group of encoding layers are the first predicted text features, i is an integer greater than 1 and not greater than n, the first group of encoding layers includes one encoding layer, and among the second to nth groups of encoding layers, there is a group including at least two encoding layers.

7. The method according to claim 4, characterized in that The timbre feature extraction network includes a timbre feature extraction layer and a first attention layer, wherein the first attention layer includes reference timbre features of a plurality of reference objects; calling the timbre feature extraction network to extract the first predicted timbre feature corresponding to the second sample speech includes: Calling the timbre feature extraction layer to extract features of the second sample speech to obtain an initial predicted timbre feature of the sample object; The first attention layer is called to determine the second weight of the initial predicted timbre feature corresponding to each reference object, wherein the second weight represents the degree of similarity between the reference timbre feature of the corresponding reference object and the initial predicted timbre feature, and based on multiple second weights, the reference timbre features of the multiple reference objects are weighted to obtain the first predicted timbre feature.

8. The method according to claim 4, characterized in that The speech synthesis network includes a second attention layer and a decoding layer, and calling the speech synthesis network to synthesize the first predicted speech based on the first predicted text feature and the first predicted timbre feature includes: Calling the second attention layer to fuse the first predicted text feature and the first predicted timbre feature to obtain a fused feature; The decoding layer is called to decode the fused features to obtain predicted acoustic features, and the first predicted speech is determined based on the predicted acoustic features.

9. The method according to claim 1, characterized in that: The first data set includes a plurality of sample text information and a plurality of sample voices, and the plurality of sample voices in the first data set belong to at least two languages ​​and at least two sample objects; the first sample text information, the first sample voice and the second sample voice belong to the first data set; The second data set includes a plurality of sample text information and a plurality of sample speech, wherein the second data set includes text information in the first language, sample speech in the second language, and does not include sample speech in the first language, and the plurality of sample speech in the second data set belong to the same target sample object; The method further comprises: After training the language classification model and the speech synthesis model based on the first data set, obtaining second sample text information and a third sample speech from the second data set, wherein the first sample text information includes text content belonging to the first language; Calling the speech synthesis model, extracting a third predicted text feature corresponding to the second sample text information, extracting a second predicted timbre feature corresponding to the third sample speech, and synthesizing a second predicted speech based on the third predicted text feature and the second predicted timbre feature; Calling the language classification model to perform language reversal classification on the second predicted timbre feature to obtain a second predicted language to which the third sample speech belongs, wherein the language reversal classification is used to make the second predicted language opposite to the language represented by the second predicted timbre feature; Based on the second predicted language and the second language, training the language classification model and the speech synthesis model to reduce the difference between the second predicted language and the second language; And, output the second predicted speech, obtain adjustment parameters based on the second predicted speech input, and train the speech synthesis model according to the adjustment parameters.

10. The method according to claim 9, characterized in that Acquiring a third sample speech from the second data set includes: Acquire a first number and a second number respectively, wherein the first number is the number of sample speech in the second data set, and the second number is the number of sample speech in the first data set; Determine a selection ratio based on a ratio between the first number and the second number, wherein the selection ratio refers to the ratio of the number of sample speech selected from the second data set for training the speech synthesis model to the first number, and the selection ratio is negatively correlated with the ratio; Based on the selection ratio, a third sample speech is selected from the second data set for training the speech synthesis model.

11. A speech synthesis model training device, characterized in that: The device comprises: a sample acquisition module, configured to acquire first sample text information, a first sample voice, and a second sample voice, wherein the first sample text information includes text content belonging to a first language, the first sample voice and the second sample voice are voices of the same sample object, the first sample voice belongs to the first language and corresponds to the first sample text information, the second sample voice belongs to a second language, and the first language is different from the second language; A speech synthesis module, configured to call a speech synthesis model, extract a first predicted text feature corresponding to the first sample text information, extract a first predicted timbre feature corresponding to the second sample speech, and synthesize a first predicted speech based on the first predicted text feature and the first predicted timbre feature; a language classification module, configured to call a language classification model, perform language reversal classification on the first predicted timbre feature, and obtain a first predicted language to which the second sample speech belongs, wherein the language reversal classification is used to make the first predicted language opposite to the language represented by the first predicted timbre feature; a model training module, configured to train the language classification model and the speech synthesis model based on the first predicted language and the second language, as well as the first sample speech and the first predicted speech, so as to reduce the difference between the first predicted language and the second language, and reduce the difference between the first sample speech and the first predicted speech; An information acquisition module, used to acquire target text information and reference speech, wherein the reference speech belongs to any object and any language, and the target text information includes text content belonging to the target language; The speech synthesis module is also used to call the speech synthesis model, extract the target text features corresponding to the target text information, extract the target timbre features corresponding to the reference speech, and synthesize the target speech corresponding to the target text information and belonging to the target language based on the target text features and the target timbre features.

12. The device according to claim 11, characterized in that The language classification module is used to: The language classification model includes a first inversion layer and a first classification layer, calling the first inversion layer to perform feature inversion on the first predicted timbre feature to obtain a first inverted timbre feature, wherein the meaning represented by the first inverted timbre feature is opposite to the meaning represented by the first predicted timbre feature; and calling the first classification layer to classify the first inverted timbre feature to obtain the first predicted language; or The language classification model includes a second inversion layer and a second classification layer, and the second classification layer is called to classify the first predicted timbre feature into a language to obtain first classification information, wherein the first classification information indicates the language to which the second sample speech obtained by the current language classification belongs; The second inversion layer is called to invert the first classification information to obtain second classification information, and the first predicted language is determined based on the second classification information.

13. The device according to claim 11, characterized in that The sample object has a corresponding sample object identifier, and the device further includes: An object classification module, used for calling an object classification model to perform object classification on the first predicted timbre feature to obtain a predicted object identifier corresponding to the sample object; The model training module is used to train the object classification model, the language classification model and the speech synthesis model based on the first predicted language and the second language, the first sample speech and the first predicted speech, and the sample object identifier and the predicted object identifier.

14. The device according to claim 11, characterized in that The speech synthesis model includes a text feature extraction network, a timbre feature extraction network and a speech synthesis network, and the speech synthesis module includes: A text feature extraction unit, configured to call the text feature extraction network to extract the first predicted text feature corresponding to the first sample text information; A timbre feature extraction unit, configured to call the timbre feature extraction network to extract the first predicted timbre feature corresponding to the second sample speech; A speech synthesis unit is used to call the speech synthesis network to synthesize the first predicted speech based on the first predicted text feature and the first predicted timbre feature.

15. The device according to claim 14, characterized in that The speech synthesis model includes a text feature fusion network and at least two types of text feature extraction networks, and the text feature extraction unit is used to: respectively calling the at least two types of text feature extraction networks to extract text features from the first sample text information to obtain at least two types of second predicted text features, wherein different types of text features describe different importance levels of different contents in the first sample text information; The text feature fusion network is called to perform weighted processing on the at least two types of second predicted text features based on the first weights of the at least two types of text feature extraction networks, respectively, to obtain the first predicted text feature.

16. The device according to claim 14, characterized in that The text feature extraction network includes n groups of encoding layers, where n is an integer greater than 2; and the text feature extraction unit is used to: Calling the first group of encoding layers to encode the first sample text information to obtain encoding features output by the first group of encoding layers; In the case where the i-th group of coding layers includes at least two coding layers, the at least two coding layers are respectively called to encode the coding features output by the i-1-th group of coding layers to obtain the at least two coding features, and a fusion layer corresponding to the i-th group of coding layers is called to fuse the at least two coding features and the coding features output by the i-1-th group of coding layers, and the fused coding features are determined as the coding features output by the i-th group of coding layers; or In a case where the i-th group of coding layers includes one coding layer, calling the i-th group of coding layers to encode the coding features output by the i-1-th group of coding layers to obtain the coding features output by the i-th group of coding layers; Among them, the encoding features output by the last group of encoding layers are the first predicted text features, i is an integer greater than 1 and not greater than n, the first group of encoding layers includes one encoding layer, and among the second to nth groups of encoding layers, there is a group including at least two encoding layers.

17. The device according to claim 14, characterized in that The timbre feature extraction network includes a timbre feature extraction layer and a first attention layer, wherein the first attention layer includes reference timbre features of a plurality of reference objects; the timbre feature extraction unit is used to: Calling the timbre feature extraction layer to extract features of the second sample speech to obtain an initial predicted timbre feature of the sample object; The first attention layer is called to determine the second weight of the initial predicted timbre feature corresponding to each reference object, wherein the second weight represents the degree of similarity between the reference timbre feature of the corresponding reference object and the initial predicted timbre feature, and based on multiple second weights, the reference timbre features of the multiple reference objects are weighted to obtain the first predicted timbre feature.

18. The device according to claim 14, characterized in that The speech synthesis network includes a second attention layer and a decoding layer, and the speech synthesis unit is used to: Calling the second attention layer to fuse the first predicted text feature and the first predicted timbre feature to obtain a fused feature; The decoding layer is called to decode the fused features to obtain predicted acoustic features, and the first predicted speech is determined based on the predicted acoustic features.

19. The device according to claim 11, characterized in that The first data set includes a plurality of sample text information and a plurality of sample voices, and the plurality of sample voices in the first data set belong to at least two languages ​​and at least two sample objects; the first sample text information, the first sample voice and the second sample voice belong to the first data set; The second data set includes a plurality of sample text information and a plurality of sample speech, wherein the second data set includes text information in the first language, sample speech in the second language, and does not include sample speech in the first language, and the plurality of sample speech in the second data set belong to the same target sample object; The device also includes: The sample acquisition module is further configured to acquire second sample text information and third sample speech from the second data set after training the language classification model and the speech synthesis model based on the first data set, wherein the first sample text information includes text content belonging to the first language; The speech synthesis module is further used to call the speech synthesis model, extract the third predicted text feature corresponding to the second sample text information, extract the second predicted timbre feature corresponding to the third sample speech, and synthesize the second predicted speech based on the third predicted text feature and the second predicted timbre feature; The language classification module is further used to call the language classification model to perform language reversal classification on the second predicted timbre feature to obtain a second predicted language to which the third sample speech belongs, wherein the language reversal classification is used to make the second predicted language opposite to the language represented by the second predicted timbre feature; The model training module is also used to train the language classification model and the speech synthesis model based on the second predicted language and the second language to reduce the difference between the second predicted language and the second language; and output the second predicted speech, obtain adjustment parameters based on the second predicted speech input, and train the speech synthesis model according to the adjustment parameters.

20. The device according to claim 19, characterized in that The sample acquisition module is further used for: Acquire a first number and a second number respectively, wherein the first number is the number of sample speech in the second data set, and the second number is the number of sample speech in the first data set; Determine a selection ratio based on a ratio between the first number and the second number, wherein the selection ratio refers to the ratio of the number of sample speech selected from the second data set for training the speech synthesis model to the first number, and the selection ratio is negatively correlated with the ratio; Based on the selection ratio, a third sample speech is selected from the second data set for training the speech synthesis model.

21. A computer device, characterized in that: The computer device includes a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the operations performed by the speech synthesis model training method as described in any one of claims 1 to 10.

22. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the operations performed by the speech synthesis model training method as described in any one of claims 1 to 10.

23. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the operations performed by the speech synthesis model training method described in any one of claims 1 to 10 are implemented.

Images