Speech synthesis method and device, computing device, storage medium and program product

Abstract

The present application provides a speech synthesis method and device. The method includes: obtaining a dialogue text to be synthesized and a dialogue text and dialogue speech of at least one reference dialogue associated with the dialogue text to be synthesized; extracting features of the text to be synthesized from the dialogue text to be synthesized; for each reference dialogue in at least one reference dialogue, extracting reference text features from the dialogue text of the reference dialogue, and extracting reference speech features from the dialogue speech of the reference dialogue; determining predicted speech features corresponding to the dialogue text to be synthesized based on the features of the text to be synthesized, at least one extracted reference text feature, and at least one extracted reference speech feature; generating a synthesized dialogue speech corresponding to the dialogue text to be synthesized based at least on the dialogue text to be synthesized and the predicted speech features. The method can be used to provide a more natural and smooth human-computer speech interaction experience.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular, to a speech synthesis method, a speech synthesis device, a computing device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Text to Speech (TTS) refers to the technology of converting text into speech. It is widely used in various application scenarios such as audio reading, news broadcasting, voice navigation, intelligent customer service, voice assistants, etc. In general, computing devices can convert text sequences into corresponding speech based on the text sequence in the text according to general pronunciation rules (such as Mandarin pronunciation rules, etc.). However, for application scenarios involving human-computer voice interaction, since the speech converted according to general pronunciation rules is relatively rigid and stiff, and lacks flexibility and variability in expression, it is easy to make users feel boring and abrupt, and it is difficult to provide users with a natural and smooth voice interaction experience. Summary of the invention

[0003] In view of this, the present application provides a speech synthesis method and apparatus, a computing device, a storage medium and a program product, which can alleviate, mitigate or even eliminate the above-mentioned problems.

[0004] According to one aspect of the present application, a speech synthesis method is provided, comprising: obtaining a dialogue text to be synthesized and a dialogue text and a dialogue speech of at least one reference dialogue associated with the dialogue text to be synthesized; extracting features of the text to be synthesized from the dialogue text to be synthesized; for each reference dialogue in at least one reference dialogue, extracting reference text features from the dialogue text of the reference dialogue, and extracting reference speech features from the dialogue speech of the reference dialogue; determining predicted speech features corresponding to the dialogue text to be synthesized based on the features of the text to be synthesized, the extracted at least one reference text feature, and the extracted at least one reference speech feature; and generating a synthesized dialogue speech corresponding to the dialogue text to be synthesized based at least on the dialogue text to be synthesized and the predicted speech features.

[0005] In some embodiments, determining the predicted speech feature corresponding to the dialogue text to be synthesized based on the features of the text to be synthesized, the at least one extracted reference text feature and the at least one extracted reference speech feature includes: determining at least one mixed reference feature corresponding to at least one reference dialogue based on the at least one extracted reference text feature and the at least one extracted reference speech feature, wherein each mixed reference feature includes a reference text feature and a reference speech feature corresponding to the corresponding reference dialogue; determining at least one dialogue interaction reference feature corresponding to at least one mixed reference feature based on the at least one mixed reference feature, wherein each dialogue interaction reference feature contains information associated with at least one other mixed reference feature other than the corresponding mixed reference feature; determining the predicted speech feature based on the features of the text to be synthesized, the at least one mixed reference feature and the at least one dialogue interaction reference feature.

[0006] In some embodiments, determining the predicted speech feature based on the feature of the text to be synthesized, at least one hybrid reference feature and at least one dialogue interaction reference feature includes: constructing at least one candidate reference feature according to the correspondence between at least one hybrid reference feature and at least one dialogue interaction reference feature, wherein each candidate reference feature includes a hybrid reference feature and a corresponding dialogue interaction reference feature; determining the predicted speech feature based on the feature of the text to be synthesized and the at least one candidate reference feature.

[0007] In some embodiments, determining the predicted speech feature based on the feature of the text to be synthesized and at least one candidate reference feature includes: determining the query feature based on the feature of the text to be synthesized; determining the weight corresponding to each candidate reference feature based on the similarity between the query feature and each candidate reference feature in at least one candidate reference feature; determining the weighted sum of at least one candidate reference feature based on the determined weight; and determining the predicted speech feature based on the query feature and the weighted sum.

[0008] In some embodiments, determining the predicted speech feature based on the query feature and the weighted sum includes: determining a synthesis vector based on the query feature and the weighted sum; and performing linear transformation and normalization on the synthesis vector to obtain the predicted speech feature.

[0009] In some embodiments, based on at least one hybrid reference feature, determining at least one conversation interaction reference feature corresponding to the at least one hybrid reference feature includes: constructing a conversation interaction graph with each hybrid reference feature in the at least one hybrid reference feature as a node, wherein, for each node, there are one or more adjacent nodes, and there are edges pointing from the one or more adjacent nodes to the node; for each node, aggregating the hybrid reference feature corresponding to the node and the hybrid reference features corresponding to one or more adjacent nodes of the node to obtain an aggregated reference feature corresponding to the node; and determining the conversation interaction reference feature corresponding to each node based on the aggregated reference feature corresponding to each node.

[0010] In some embodiments, each reference conversation also includes an interlocutor label, and constructing a conversation interaction graph with each of at least one hybrid reference feature as a node includes performing at least one of the following operations for each edge: determining the weight of the edge based on the similarity of the hybrid reference features of the two nodes associated with the edge; determining the relationship type of the edge based on the attributes of the two nodes associated with the edge, wherein the attributes include at least one of the following: the occurrence time of the reference conversation corresponding to the node and the interlocutor label of the reference conversation corresponding to the node.

[0011] In some embodiments, extracting reference speech features from the conversational speech of the reference conversation includes: determining a coding vector representing the conversational speech of the reference conversation; converting the determined coding vector into a fixed-length vector having a preset length; determining a similarity between the fixed-length vector and each pre-trained speech feature vector in a set of pre-trained speech feature vectors; and determining a reference speech feature corresponding to the conversational speech of the reference conversation based on the determined set of similarities.

[0012] In some embodiments, a set of pre-trained speech feature vectors is trained by the following process: obtaining the conversation text and conversation speech of multiple sample conversations; initializing a set of speech feature vectors to be trained; performing the following operations based on each sample conversation in the multiple sample conversations in turn: - determining a coding vector representing the conversation speech based on the conversation speech of the sample conversation, and extracting true interlocutor features corresponding to the conversation speech; - converting the determined coding vector into a fixed-length vector with a preset length; - determining the similarity between the fixed-length vector and each pre-trained speech feature vector in a set of speech feature vectors to be trained; - determining the sample speech features corresponding to the conversation speech based on the determined set of similarities; - extracting predicted interlocutor features corresponding to the conversation speech from the sample speech features; - generating reconstructed speech based on the sample speech features, a set of speech feature vectors to be trained and the conversation text of the sample conversation; - updating a set of speech feature vectors to be trained based on the difference between the conversation speech of the sample conversation and the reconstructed speech and the difference between the true interlocutor features and the predicted interlocutor features; and using the updated set of speech feature vectors to be trained as a set of pre-trained speech feature vectors.

[0013] In some embodiments, determining the coding vector representing the conversational speech of the reference conversation includes: dividing the conversational speech of the reference conversation into multiple audio frames; windowing each of the multiple audio frames to obtain multiple windowed audio frames; converting the multiple audio frames into an audio energy distribution through Fourier transform; passing the audio energy distribution through a filter group, and extracting the coding vector representing the conversational speech of the reference conversation based on the output of the filter group.

[0014] In some embodiments, generating a synthesized conversation speech corresponding to the conversation text to be synthesized based at least on the conversation text to be synthesized and the predicted speech features includes: generating a conversation phoneme sequence to be synthesized based on the conversation text to be synthesized and according to general pronunciation rules; encoding the conversation phoneme sequence to be synthesized to obtain an initial phoneme coding vector corresponding to the conversation text to be synthesized; and generating a synthesized conversation speech based on the determined initial phoneme coding vector and the predicted speech features.

[0015] In some embodiments, generating a synthesized conversational speech based on the determined initial phoneme coding vector and the predicted speech features includes: determining a weighted sum of a set of pre-trained speech feature vectors using the predicted speech features as weights as a predicted speech feature vector; determining a synthesized phoneme coding vector based on the initial phoneme coding vector and the speech feature vector; decoding the synthesized phoneme coding vector, and generating a synthesized conversational speech based on the decoded output.

[0016] In some embodiments, determining a synthesized phoneme coding vector based on an initial phoneme coding vector and a speech feature vector includes: obtaining a timbre feature vector associated with the dialogue text to be synthesized; obtaining a mixed phoneme coding vector based on the initial phoneme coding vector, the timbre feature vector associated with the dialogue text to be synthesized, and the speech feature vector; extracting phoneme features from the mixed phoneme coding vector, the phoneme features including at least one of a length feature, a pitch feature, and a volume feature; and determining a synthesized phoneme coding vector based on the extracted phoneme features.

[0017] In some embodiments, extracting features of the text to be synthesized from the dialogue text to be synthesized includes: determining a coding vector representing the dialogue text to be synthesized; and extracting features of the text to be synthesized based on the determined coding vector.

[0018] In some embodiments, determining the encoding vector representing the dialogue text to be synthesized includes: dividing the dialogue text to be synthesized into one or more words; converting each of the one or more words into a word vector; and determining the encoding vector based on the word vectors corresponding to the one or more words.

[0019] In some embodiments, extracting reference text features from the conversation text of the reference conversation includes: determining an encoding vector representing the conversation text of the reference conversation; and extracting reference text features corresponding to the conversation text of the reference conversation based on the determined encoding vector.

[0020] According to another aspect of the present application, a speech synthesis device is provided, comprising: an acquisition module configured to acquire a dialogue text to be synthesized and a dialogue text and a dialogue speech of at least one reference dialogue associated with the dialogue text to be synthesized; a first feature extraction module configured to extract features of the text to be synthesized from the dialogue text to be synthesized; a second feature extraction module configured to extract reference text features from the dialogue text of each reference dialogue in at least one reference dialogue, and to extract reference speech features from the dialogue speech of the reference dialogue; a speech feature prediction module configured to determine predicted speech features corresponding to the dialogue text to be synthesized based on features of the text to be synthesized, at least one extracted reference text feature, and at least one extracted reference speech feature; and a speech synthesis module configured to generate a synthesized dialogue speech corresponding to the dialogue text to be synthesized based at least on the dialogue text to be synthesized and the predicted speech features.

[0021] According to another aspect of the present application, a computing device is provided, comprising: a memory configured to store computer-executable instructions; and a processor configured to execute the speech synthesis method described in various embodiments according to the aforementioned aspects when the computer-executable instructions are executed by the processor.

[0022] According to another aspect of the present application, a computer-readable storage medium is provided, which stores computer-executable instructions. When the computer-executable instructions are executed, the speech synthesis method described in the various embodiments according to the aforementioned aspects is executed.

[0023] According to another aspect of the present application, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the speech synthesis method described in the various embodiments of the aforementioned aspects.

[0024] By using the speech synthesis method and device provided by the present application, in the process of generating synthesized conversation speech based on the conversation text to be synthesized, the conversation text and conversation speech of at least one reference conversation related thereto can be obtained. Subsequently, the text features to be synthesized can be extracted from the conversation text to be synthesized, and the corresponding reference text features and reference speech features can be extracted from the conversation text and speech text of each reference conversation. Then, the predicted speech features can be determined based on the extracted text features to be synthesized, the reference text features and the reference speech features. Finally, the synthesized conversation speech can be generated based on the conversation text to be synthesized and the determined predicted speech features. In this way, when generating synthesized conversational speech, the influence of the text information and audio information of the reference conversation (such as the user's historical conversation) related to the text to be synthesized on the synthesized conversational speech corresponding to the conversational text to be synthesized can be fully considered, especially the crucial role of the audio information of the reference conversation in natural spoken conversation in speech synthesis (such as the selection of expression), so as to help flexibly adjust the expression of the final synthesized conversational speech according to the reference conversation (especially the expression of the corresponding user reflected in the voice or audio information in the reference conversation or historical conversation), enhance the personalization and flexibility of its expression, and help make its expression more suitable for the current conversation scene. At the same time, this helps to provide a more natural and smooth human-computer voice interaction experience, thereby enhancing the personal experience of users in relevant application scenarios.

[0025] These and other aspects of the application will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Further details, features and advantages of the present application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0027] Figure 1 Schematically illustrates an example application scenario in which the speech synthesis solution provided according to some embodiments of the present application can be applied;

[0028] Figure 2 An example architecture of a speech synthesis system according to some embodiments of the present application is schematically shown;

[0029] Figure 3 An example flow chart of a speech synthesis method according to some embodiments of the present application is schematically shown;

[0030] Figure 4 Schematically illustrates the application of some embodiments of the present application. Figure 3 An example application architecture of the speech synthesis method shown;

[0031] Figure 5 An example flow chart of a method for extracting features of text to be synthesized according to some embodiments of the present application is schematically shown;

[0032] Figure 6 Schematically shows an example block diagram of a feature extraction architecture for text to be synthesized according to some embodiments of the present application;

[0033] Figure 7 An example flow chart of a reference speech feature extraction method according to some embodiments of the present application is schematically shown;

[0034] Figure 8 Schematically illustrates an example block diagram of a joint training architecture for reference speech feature extraction and conversational speech synthesis according to some embodiments of the present application;

[0035] Fig. 9 An example flow chart of a method for determining predicted speech features according to some embodiments of the present application is schematically shown;

[0036] Figures 10A-10C Schematically illustrates an example block diagram of a predicted speech feature determination architecture according to some embodiments of the present application;

[0037] Fig.11 An example flow chart of a method for generating synthetic conversational speech according to some embodiments of the present application is schematically shown;

[0038] Fig.12 Schematically illustrates an example block diagram of a synthetic conversational speech generation architecture according to some embodiments of the present application;

[0039] Fig.13 An example block diagram of a speech synthesis architecture according to some embodiments of the present application is schematically shown;

[0040] Fig.14 An example block diagram of a speech synthesis device according to some embodiments of the present application is schematically shown;

[0041] Fig.15 An example block diagram of a computing device according to some embodiments of the present application is schematically shown. DETAILED DESCRIPTION

[0042] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0043] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0044] The key technologies of speech technology include automatic speech recognition technology (ASR), text-to-speech technology (TTS) and voiceprint recognition technology. Enabling computers to listen, see, speak and feel is the future development direction of human-computer interaction, among which speech has become one of the most promising human-computer interaction methods in the future.

[0045] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0046] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0047] A feature can generally refer to the characteristics of a thing that are different from other things. In the present application, a feature can refer to data information extracted from raw data that can characterize the characteristics of the corresponding raw data and can be used by a model or algorithm, which can usually be expressed in the form of a vector. Thus, in the present application, a text feature can refer to information extracted from a text that can characterize the characteristics of the text and can be used by a model or algorithm, such as a text feature vector, and a speech feature can refer to information extracted from a speech that can characterize the characteristics of the speech and can be used by a model or algorithm, such as a speech feature vector.

[0048] The solutions provided in the embodiments of the present application involve artificial intelligence speech synthesis processing, natural language processing, machine learning and other technologies, which are specifically illustrated by the following embodiments.

[0049] Figure 1 An example application scenario 100 to which the speech synthesis solution provided according to some embodiments of the present application can be applied is schematically illustrated.

[0050] like Figure 1As shown, the scene 100 includes a server 110. The speech synthesis solution provided in the present application can be deployed on the server 110, and is used to generate a synthesized conversation speech corresponding to the conversation text to be synthesized based on the conversation text to be synthesized and the conversation text and conversation speech of at least one reference conversation associated with the conversation text to be synthesized. The server 110 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. In addition, it should be understood that the server 110 is shown only as an example. In fact, other devices or combinations of devices with computing power and storage capacity can also be used instead or in addition to provide corresponding services. The synthesized conversation speech generated on the server 110 can be sent to other devices, or can be stored locally or stored on an external storage device. Furthermore, the dialogue text to be synthesized and the dialogue text and dialogue speech of at least one reference dialogue associated with the dialogue text to be synthesized used at the server 110 may be stored locally, may be stored on an external storage device, or may be received from other devices.

[0051] Optionally, the server 110 may be linked to an external database 120, so as to obtain, for example, the dialogue text to be synthesized and / or the dialogue text and dialogue speech of at least one reference dialogue associated with the dialogue text to be synthesized from the database 120, and, for example, store the generated synthesized dialogue speech in the database 120, etc. Exemplarily, the database 120 may be an independent data storage device or device group, or may also be a backend data storage device or device group related to other online services (such as online services providing functions such as intelligent customer service and voice assistant).

[0052] Optionally, the user 140 can access the server 110 through the terminal device 130 via the network 150 to obtain the services provided by the server 110. For example, the user 140 can input instructions through the user interface provided by the terminal device 130, such as through relevant physical or virtual buttons, text, voice or gesture instructions, etc., to start the speech synthesis solution deployed on the server 110, receive the generated synthesized dialogue voice, etc. Exemplarily, the terminal device 130 includes but is not limited to a mobile phone, a computer, an intelligent voice interaction device, an intelligent home appliance, a car terminal, etc. It should be understood that although in Figure 1In the illustrated scenario 100, the terminal device 130 and the server 110 are presented separately, but in fact, the terminal device 130 and the server 110 can be integrated in the same device. That is, the speech synthesis solution provided in the present application can be directly deployed on a computing device, and the user can use the computing device to perform actions such as human-computer speech interaction.

[0053] In addition, in the present application, network 150 can be a wired network connected via cables, optical fibers, etc., or a wireless network such as 2G, 3G, 4G, 5G, Wi-Fi, Bluetooth, ZigBee, Li-Fi, etc., or an internal connection line of one or several devices, etc.

[0054] With the development of computer technology, research on speech synthesis technology has also been widely carried out. Through research, the applicant found that compared with traditional speech synthesis technology, the speech synthesis system based on neural network has significant advantages in the naturalness of synthesized speech. However, in the speech synthesis system based on neural network in the related technology, although the associated historical dialogue information is utilized, this utilization is often insufficient. This makes the synthesized speech, although it properly considers the influence of the specific dialogue scene, its expression is still likely to not meet the needs of the specific dialogue scene, which may make the user feel abrupt and unnatural, thereby affecting the overall human-computer interaction experience.

[0055] Furthermore, the applicant has found that the shortcomings of the speech synthesis systems based on neural networks in the related art are mainly caused by the following two reasons. First, these speech synthesis systems usually only consider the text information in the associated historical conversations, but do not consider other information such as audio information, which leads to insufficient utilization of the associated historical conversation information by these systems. Second, these speech synthesis systems usually only use a simple single-layer recurrent neural network to model the text information of the associated historical conversations. This approach is too simple and it is difficult to extract richer features from the associated historical conversation information, which also leads to insufficient utilization of the associated historical conversation information by these systems. Therefore, although these speech synthesis systems may take into account the associated historical conversation information, it is still difficult for them to generate satisfactory synthesized conversation speech.

[0056] Based on the above considerations, the applicant has proposed a new speech synthesis solution, which helps to make fuller use of the associated dialogue information to adjust the expression of the synthesized speech generated based on the dialogue text to be synthesized. Specifically, Figure 2 An exemplary overall architecture 200 of a speech synthesis system according to some embodiments of the present application is schematically shown.

[0057] like Figure 2As shown, the speech synthesis system 230 can obtain one or more reference dialogues 210 and the dialogue text to be synthesized 220 from other systems, devices or modules through the input interface, wherein the one or more reference dialogues 210 may include the text, voice and other optional information of the reference dialogue. Then, the speech synthesis system 230 can analyze the reference dialogue 210 and the dialogue text to be synthesized 220 through the dialogue analysis module 231, for example, extract the text features, voice features, etc., predict the voice features based on the dialogue analysis results through the voice feature prediction module 232, and generate the synthesized dialogue voice 240 based on the dialogue text to be synthesized 220 and the predicted voice features through the speech synthesis module 233. The synthesized dialogue voice 240 can be output to other systems, devices or modules through the output interface. Optionally, according to specific application requirements, the speech synthesis system 230 can be deployed on a server or terminal device, or can also be deployed on an integrated device. Optionally, according to specific application requirements, the speech synthesis system 230 can be deployed separately, can be deployed together with other systems, devices or modules, or can be deployed as part of other systems, devices or modules.

[0058] Further, Figure 3 The following schematically shows an example flow chart of a speech synthesis method 300 according to some embodiments of the present application. For example, the speech synthesis method 300 can be applied to Figure 2 The speech synthesis system 230 shown in FIG. 1 can be deployed in the form of an application program, etc. Figure 1 The server 110, the terminal device 130 or a combination of the two in the application scenario 100.

[0059] In step 310, the dialogue text to be synthesized and the dialogue text and dialogue speech of at least one reference dialogue associated with the dialogue text to be synthesized may be obtained. Exemplarily, the dialogue text and dialogue speech of at least one reference dialogue associated with the dialogue text to be synthesized may refer to all or part of the dialogue that occurred before the synthetic dialogue speech was generated based on the dialogue text to be synthesized in the current dialogue scene. The all or part of the dialogue may be a dialogue that occurs continuously, or may be two or more continuous dialogues with a time interval lower than a preset threshold. For example, assuming that a user has a continuous voice dialogue interaction with his terminal device (for example, with a voice assistant deployed on his terminal device) from time A, when his terminal device generates a synthetic dialogue speech based on a dialogue text to be synthesized at time B, the at least one reference dialogue associated with the dialogue text to be synthesized may include all or part of the dialogues that occurred between the user and his terminal device from time A to time B, such as a dialogue that occurred within a preset time period before time B. Therefore, relative to the moment when the synthetic dialogue speech was generated based on the dialogue text to be synthesized, the reference dialogue associated with the dialogue text to be synthesized may be considered to have occurred at a historical moment, and thus may also be referred to as a historical dialogue. In addition, according to actual needs, optionally, according to actual needs, the reference dialogue associated with the dialogue text to be synthesized may also include dialogues in other dialogue scenarios similar to the current dialogue scenario, and so on.

[0060] Optionally, the dialogue text to be synthesized may be text automatically generated by a system, device or module executing the speech synthesis method 300, such as dialogue text automatically generated based on an already occurred dialogue; may be dialogue text generated by other systems, devices or modules and provided to the system, device or module; may be manually input text, such as input by the current user or other users who are having a dialogue with the current user; or may be pre-stored text, such as preset dialogue text triggered based on keywords, etc. In addition, optionally, the dialogue text and dialogue speech of at least one reference dialogue associated with the dialogue text to be synthesized may be read from local storage or received from other devices. The present application does not specifically limit the method for obtaining the dialogue text to be synthesized and the associated reference dialogue.

[0061] In step 320, features of the text to be synthesized may be extracted from the dialogue text to be synthesized. Optionally, various types of neural networks may be used to extract features of the text to be synthesized from the dialogue text to be synthesized. For example, a single-layer or multi-layer convolutional neural network (CNN), a recurrent neural network (RNN), or a combination of the two may be used to implement the above process. Alternatively, the features of the text to be synthesized may also be obtained by non-machine learning methods.

[0062] In step 330, for each reference dialogue in at least one reference dialogue, reference text features may be extracted from the dialogue text of the reference dialogue, and reference voice features may be extracted from the dialogue speech of the reference dialogue. Similarly, various types of neural networks may be used to extract reference text features from the dialogue text of the reference dialogue, and various types of neural networks may be used to extract reference voice features from the dialogue speech of the reference dialogue. For example, a single-layer or multi-layer CNN, RNN, or a combination of the two may be used to implement the above process. In addition, optionally, the method of extracting reference text features from the dialogue text of the reference dialogue may be the same as the method of extracting text features to be synthesized from the dialogue text to be synthesized in step 320, but the two may also be different depending on actual needs.

[0063] In step 330, the predicted speech feature corresponding to the dialogue text to be synthesized can be determined based on the features of the text to be synthesized, the at least one reference text feature extracted, and the at least one reference speech feature extracted. Optionally, the predicted speech feature corresponding to the dialogue text to be synthesized can be obtained based on the extracted at least one reference speech feature according to the similarity between the features of the text to be synthesized and the at least one reference text feature extracted, for example, a reference speech feature is selected from the at least one reference speech feature extracted as the predicted speech feature, or the predicted speech feature is determined based on two or more reference speech features, etc. Alternatively, this step can be implemented by a machine learning model constructed based on a neural network. The machine learning model can be implemented based on CNN, RNN or other types of neural networks, or can also be implemented based on a combination of multiple neural networks. The machine learning model can be pre-trained through sample data so that the predicted speech feature corresponding to the dialogue text to be synthesized can be determined based on the features of the text to be synthesized, the at least one reference text feature extracted, and the at least one reference speech feature extracted.

[0064] In step 340, a synthesized conversational speech corresponding to the conversational text to be synthesized may be generated based at least on the conversational text to be synthesized and the predicted speech features. Optionally, the speech generated according to the speech synthesis method in the related art may be adjusted by predicting the speech features to obtain the final synthesized conversational speech. For example, the conversational text to be synthesized may be converted into a phoneme sequence according to a general pronunciation rule, and then the phoneme sequence may be adjusted by predicting the speech features, for example, the adjustment may be performed based on a pre-trained neural network model, and then the synthesized conversational speech may be generated based on the adjusted factor sequence; or, the conversational text to be synthesized and the predicted speech features may be directly used as inputs, and the corresponding synthesized conversational speech may be generated by a pre-trained neural network model; and so on. Optionally, in addition to the conversational text to be synthesized and the predicted speech features, the synthesized conversational speech may also be generated based on additional information, and the additional information may include, for example, preset timbre features, interlocutor label information, preset reference speech rate, preset reference volume, and the like.

[0065] pass Figure 3 The speech synthesis method 300 shown can make more full use of the reference dialogue associated with the text to be synthesized. Specifically, it not only considers the dialogue text of the associated reference dialogue, but also considers the influence of the dialogue speech of the associated reference dialogue on the final synthesized dialogue speech, especially considering the crucial role of the audio information of the reference dialogue in speech synthesis (such as the selection of expression mode) in natural spoken dialogue. This helps to flexibly adjust the expression mode of the final synthesized dialogue speech (such as changes in the tone, volume, speech speed, etc. related to the expression tone, emotion, etc.) according to the reference dialogue (especially the expression mode of the corresponding user reflected in the voice information of the reference dialogue), and increase the personalization and flexibility of the expression mode. In addition, compared with the solutions in the related art, since the dialogue speech of the reference dialogue is analyzed and the synthesized dialogue speech is adjusted based on it, the speech synthesis method 300 helps to make the expression mode of the synthesized dialogue speech more suitable for the corresponding dialogue scene, thereby helping to provide a more natural and smooth human-computer voice interaction experience, and then helping to improve the personal experience of users in related application scenarios.

[0066] Figure 3 The speech synthesis method 300 shown can be applied to any application scenario involving human-computer speech interaction, such as voice assistants, intelligent customer service, chat robots, etc. For example, Figure 4 The application architecture 400 to which the speech synthesis method 300 can be applied according to some embodiments of the present application is schematically shown. Figure 4 As shown, the application architecture 400 includes a human-machine dialogue speech synthesis system 430, which can be deployed on any suitable platform or device and can be provided as an independent function or service, or can be provided as part of a related function or service. Figure 4 As shown, the human-machine dialogue speech system 430 may have an input port to receive the dialogue text 410 to be synthesized and one or more reference dialogues 420, and an output port to output the generated synthesized dialogue speech 440. The reference dialogue 420 may include a dialogue text 421 and a dialogue speech 422, and may optionally include a dialoguer tag 423 and / or other information. The synthesized dialogue speech 440 may be generated according to the above-mentioned speech synthesis method 300 and various embodiments described in more detail below.

[0067] In some embodiments, Figure 3 Step 320 in may be accomplished by Figure 5 It should be understood that Figure 5 The flowchart shown is for illustrative purposes only. Figure 3 Step 320 in may also be implemented in other ways.

[0068] Specifically, in step 321, an encoding vector representing the dialogue text to be synthesized may be determined. This process may be implemented by various appropriate encoding methods. Exemplarily, various common character encoding methods may be used to convert the text character sequence of the dialogue to be synthesized into a series of character codes, such as ASCII (American Standard Code for Information Interchange), GB2312 (Chinese Character Coded Character Set for Information Interchange), GBK (National Standard Code), Unicode, etc. The converted series of character codes may be further processed, such as by concatenating, inputting into a pre-trained machine learning model, etc., to obtain an encoding vector representing the dialogue text to be synthesized. Exemplarily, step 321 may be implemented by Figure 5 The steps 3211-3213 shown are implemented, which helps to more conveniently determine the encoding vector representing the dialogue text to be synthesized.

[0069] In step 3211, the dialogue text to be synthesized may be segmented into one or more words. Exemplarily, the dialogue text to be synthesized may be segmented by using a segmentation tool to segment it into one or more words. The segmentation tool may be, for example, Jieba segmentation, HanLP, LTP, THULAC, NLPIR, etc. Alternatively, exemplarily, the segmentation process may also be implemented by other methods such as manual annotation, random segmentation, and complete segmentation into multiple single characters.

[0070] In step 3212, each of the one or more words may be converted into a word vector. The conversion process may be implemented by various existing or custom word vector tools, such as one-hot encoding, word2vec, Glove, Bert, etc.

[0071] In step 3213, a coding vector may be determined based on the word vectors corresponding to one or more words. For example, the word vectors corresponding to one or more words may be linearly or nonlinearly transformed to determine the one or more corresponding coding vectors; or, the word vectors corresponding to one or more words may be grouped, concatenated, summed, weighted summed, etc. to determine the one or more corresponding coding vectors; or, a pre-trained neural network model may be used to implement this step, and so on.

[0072] In step 322, features of the text to be synthesized may be extracted based on the determined encoding vector. For example, the features of the text to be synthesized may be extracted by a pre-trained neural network model, or the features of the text to be synthesized may be determined based on a series of preset encoding vectors by similarity between the determined encoding vector and a series of preset encoding vectors, and so on.

[0073] Through step 321 and step 322, the features of the text to be synthesized can be conveniently extracted from the dialogue text to be synthesized. Figure 6 The example block diagram of the structure 500 for extracting features of the synthesized text is shown.

[0074] like Figure 6As shown, the input of the text feature extraction architecture 500 to be synthesized can be a text character sequence 510 included in the dialogue text to be synthesized, and the corresponding sentence-level text feature 560 is output as the above-mentioned text feature to be synthesized. The input text character sequence 510 is first segmented by a word segmentation tool 520, thereby segmenting the text character sequence 510 into one or more words. The word segmentation tool 520 can be the stammering word segmentation tool mentioned above. Afterwards, the one or more words obtained by segmentation can be input into a word vector tool 530 to obtain a word vector sequence including one or more word vectors. The word vector tool 530 can be, for example, a gensim tool that supports multiple topic model algorithms such as TF-IDF (term frequency-inverse document frequency), LSA (Latent semantic analysis), LDA (Latent Dirichlet Allocation) and word2vec. Then, the word vector sequence can be input into the Pre-net layer, which can perform linear or nonlinear transformation on the input word vector to obtain the corresponding encoding vector. Optionally, the Pre-net layer can be composed of 2 fully connected layers, or other network structures can be adopted. Finally, the transformed encoding vector can be input into the CBHG network, and the output of the last step of the CBHG network can be taken as the output sentence-level text feature 560. The CBHG network can be composed of a series of one-dimensional convolutional filters, a highway network, and a bidirectional gated recurrent unit (GRU) RNN, which can effectively extract features from the encoding vector sequence. By using the Pre-net layer and the CBHG network, it helps to improve the generalization ability of the architecture 500.

[0075] In some embodiments, Figure 3 The operation of extracting reference text features from the dialogue text of the reference dialogue in step 330 shown in FIG. 1 may also be performed by a method similar to that of Figure 5 , Figure 6 The various embodiments described are implemented. For example, Figure 3 The step 330 shown may include: determining an encoding vector representing the dialogue text of the reference dialogue; extracting reference text features corresponding to the dialogue text of the reference dialogue based on the determined encoding vector. Furthermore, determining the encoding vector representing the dialogue text of the reference dialogue may include: segmenting the dialogue text of the reference dialogue into one or more words; converting each of the one or more words into a word vector; and determining the encoding vector representing the dialogue text of the reference dialogue based on the word vectors corresponding to the one or more words.

[0076] In some embodiments, Figure 3The operation of extracting reference speech features from the conversation speech of the reference conversation in step 330 can be performed by Figure 7 The example flow chart shown is used to implement this. Figure 7 The flowchart shown can conveniently extract reference speech features from the conversation speech of the reference conversation. It should be understood that Figure 7 The flowchart shown is for illustrative purposes only. Figure 3 Step 330 in may also be implemented in other ways.

[0077] Specifically, in step 331, a coding vector representing the conversational speech of the reference conversation may be determined. Exemplarily, the coding vector representing the conversational speech of the reference conversation may be determined based on various coding methods. For example, coding methods such as pulse code modulation (PCM) and adaptive delta modulation (ADM) may be used to convert the analog signal contained in the conversational speech into digital coding, and then obtain the corresponding coding vector. Alternatively, the corresponding coding vector may be determined based on the conversational speech by methods such as LPC (Linear Prediction Coefficients), PLP (Perceptual Linear Predictive), Fbank features (Filter bank) based on filter banks, LPCC (Linear Predictive Cepstral Coefficient), MFCC (Mel Frequency Cepstrum Coefficient), etc.

[0078] Exemplarily, step 331 may be implemented by the following process. First, the conversational speech of the reference conversation may be divided into a plurality of audio frames. Optionally, the conversational speech may be filtered before the division to filter out other audio information other than human voice. And optionally, the size of the divided audio frames may be set according to actual needs, for example, may be set to 50 ms or other values. Then, each of the plurality of audio frames may be windowed to obtain a plurality of windowed audio frames. Optionally, a window function such as a Hann window, a Hamming window, a rectangular window, etc. may be used to window the audio frames to facilitate subsequent Fourier transform. Then, the plurality of audio frames may be converted into an audio energy distribution by Fourier transform. For example, the plurality of audio frames may be converted into an audio energy distribution by short-time Fourier transform, etc. Finally, the audio energy distribution may be passed through a filter bank, and a coding vector representing the conversational speech of the reference conversation may be extracted based on the output of the filter bank. For example, the filter bank may be a Mel filter bank to convert the audio energy distribution into a Mel spectrum. Furthermore, the logarithm of the Mel spectrum can be taken to obtain the logarithmic Mel spectrum (log-Mel). Through the above process, the conversational speech can be converted into a coding vector that is convenient for speech feature extraction, wherein the use of the Mel spectrum or the reciprocal Mel spectrum can make the obtained coding vector more consistent with the auditory characteristics of the human ear, thereby facilitating more accurate extraction of appropriate reference speech features from the obtained coding vector.

[0079] In step 332, the determined code vector may be converted into a fixed-length vector having a preset length. Exemplarily, this step may be implemented by directly performing length shaping on the determined code vector; or, the determined code vector may be first subjected to a series of linear or nonlinear transformations and then subjected to length shaping; or, feature extraction may be further performed while performing length shaping; and so on.

[0080] In step 333, the similarity between the fixed-length vector and each pre-trained speech feature vector in a set of pre-trained speech feature vectors may be determined. Optionally, the similarity may be measured based on cosine similarity, Euclidean distance, Manhattan distance, etc.

[0081] In step 334, reference speech features corresponding to the conversational speech of the reference conversation may be determined based on the determined set of similarities. Exemplarily, the determined set of similarities may be used as reference speech features corresponding to the conversational speech of the reference conversation, or the determined set of similarities may be further linearly or nonlinearly transformed to obtain reference speech features corresponding to the conversational speech of the reference conversation, and so on.

[0082] For example, steps 332-334 may be performed by Figure 8The speech feature extraction branch 610 in the illustrated architecture 600 is implemented. Figure 6 As shown, the speech feature extraction branch 610 includes a reference encoder 611 and a GST (Global Style Token) attention network 612. Exemplarily, the reference encoder 611 may include a plurality of convolutional layers (e.g., 6 convolutional layers) and a GRU layer, and may be used to perform step 332. The output of the GRU layer may be input to the GST attention network 612, which may be used to perform steps 333 and 334.

[0083] In some embodiments, a set of pre-trained speech feature vectors used in step 333 may be trained by the following process. For example, Figure 8 An example block diagram of a training architecture 600 that can be used to obtain a pre-trained speech feature vector according to some embodiments of the present application is schematically shown.

[0084] First, the conversation text 620 and conversation speech 630 of multiple sample conversations can be obtained, and a set of speech feature vectors to be trained can be initialized, for example Figure 8The GST features to be trained 640 shown in the figure can be initialized based on preset parameters or can be performed randomly. Then, the following operations can be performed in turn based on each sample conversation in the multiple sample conversations: based on the conversation speech 620 of the sample conversation, determine the encoding vector that represents the conversation speech, which can be implemented based on the step 331 described above, and extract the true value interlocutor feature 650 corresponding to the conversation speech, which can be obtained using a pre-trained known true value interlocutor feature extractor 660; convert the determined encoding vector into a fixed-length vector with a preset length, which can be implemented based on the step 332 described above (for example, by referring to the encoder 611); determine the fixed-length vector and each pre-trained vector in the group of speech feature vectors to be trained. The similarity of the trained speech feature vector can be achieved based on step 333 described above; based on the determined set of similarities, the sample speech features (such as GST weights 642) corresponding to the conversation speech are determined, which can be achieved based on step 334 described above; from the sample speech features, the predicted interlocutor features corresponding to the conversation speech are extracted, which can be achieved through the interlocutor feature prediction branch 670; based on the sample speech features, the speech feature vector to be trained and the conversation text of the sample conversation, the reconstructed speech 680 is generated, which can be achieved through the speech synthesis branch 690, and the speech synthesis branch can include FastSpeech 2 encoder 691, variance adapter 692 and FastSpeech 2 decoder, which will be described in detail below; based on the difference between the conversation speech 620 and the reconstructed speech 680 of the sample conversation and the difference between the true interlocutor feature 650 and the predicted interlocutor feature 673, a set of speech feature vectors to be trained is updated. In other words, a loss function can be constructed based on the difference between the conversation speech 620 and the reconstructed speech 680 of the sample conversation and the difference between the true interlocutor feature 650 and the predicted interlocutor feature 673. During the training process, various network parameters (including the speech feature vectors to be trained) need to be adjusted to minimize the loss function. Finally, the updated set of speech feature vectors to be trained can be used as a set of pre-trained speech feature vectors.

[0085] Through the above training process, a set of pre-trained speech feature vectors suitable for characterizing speech features can be automatically determined. In addition, the interlocutor feature prediction branch 670 can be set as an adversarial learning branch, which can include a gradient flip layer 671 and an interlocutor feature extraction network 672, and is used to predict interlocutor features based on sample speech features such as GST weights 642. Thus, during the training process, the branch 670 can form an adversarial relationship with the speech feature extraction branch 610, so that the speech feature extraction branch 610 can be as independent of the interlocutor information as possible, which makes the extracted speech features not limited by the interlocutors in the sample set, and can be applied to different interlocutors.

[0086] In some embodiments, Figure 3 Step 340 shown may be performed by Fig. 9 The example flow chart shown is used to implement this. Fig. 9 The flowchart shown can take into account the influence between and within the interlocutors in the reference conversation, so that the determined predicted speech features can be more suitable for the associated conversation scene. It should be understood that Fig. 9 The flowchart shown is for illustrative purposes only. Figure 3 Step 340 in may also be implemented in other ways.

[0087] Specifically, in step 341, at least one mixed reference feature corresponding to at least one reference conversation can be determined based on at least one extracted reference text feature and at least one extracted reference voice feature. Each mixed reference feature can include a reference text feature and a reference voice feature corresponding to the corresponding reference conversation. In the present application, "respectively corresponding" can be understood as a one-to-one correspondence, for example, "at least one mixed reference feature corresponding to at least one reference conversation" can be understood as at least one reference conversation corresponding to at least one mixed reference feature in a one-to-one correspondence. Exemplarily, for reference text features and reference voice features corresponding to the same reference conversation, they can be combined into a mixed reference feature by operations such as splicing and addition.

[0088] Schematically, step 341 can be performed by Fig. 10A The example architecture 700A shown in FIG. Fig. 10A As shown, at least one reference conversation, such as reference conversation 1 to reference conversation N, can be acquired in advance, wherein each reference conversation can include conversation text and conversation speech. The conversation text of each reference conversation can be input into a text feature encoder to obtain a corresponding reference text feature. The text feature encoder can be similarly implemented according to the various embodiments of step 330 or 320 described above. The conversation speech of each reference conversation can be input into a speech feature encoder to obtain a corresponding reference speech feature. The speech feature encoder can be implemented according to the various embodiments of step 330 described above. The reference text features and reference speech features corresponding to the same reference conversation can, for example, be spliced ​​into a mixed reference feature. Thus, mixed reference features g_1 to mixed reference features g_N corresponding to reference conversations 1 to reference conversations N, respectively, can be obtained.

[0089] At step 342, at least one conversation interaction reference feature corresponding to the at least one hybrid reference feature may be determined based on the at least one hybrid reference feature. Each conversation interaction reference feature contains information associated with at least one other hybrid reference feature other than the corresponding hybrid reference feature. Exemplarily, for each hybrid reference feature, the corresponding conversation interaction reference feature may be obtained by aggregating it with several adjacent hybrid reference features, wherein two adjacent hybrid reference features may refer to the occurrence time difference of the reference conversations corresponding to the two hybrid reference features being within a preset time threshold, or there being only a threshold number of other reference conversations between the reference conversations corresponding to the two hybrid reference features, etc.

[0090] In some embodiments, step 342 may be implemented by the following operations. First, each hybrid reference feature in at least one hybrid reference feature may be used as a node to construct a conversation interaction graph, wherein for each node, there are one or more adjacent nodes, and there are edges pointing from the one or more adjacent nodes to the node. Then, for each node, the hybrid reference feature corresponding to the node and the hybrid reference features corresponding to one or more adjacent nodes of the node may be aggregated to obtain an aggregated reference feature corresponding to the node. Finally, based on the aggregated reference features corresponding to each node, the conversation interaction reference features corresponding to each node may be determined.

[0091] In some embodiments, each reference conversation may further include a conversation partner label. In such embodiments, when constructing a conversation interaction graph, for each edge, at least one of the following operations may be performed: determining the weight of the edge based on the similarity of the mixed reference features of the two nodes associated with the edge; determining the relationship type of the edge based on the attributes of the two nodes associated with the edge, wherein the attributes include at least one of the following: the occurrence time of the reference conversation corresponding to the node and the conversation partner label of the reference conversation corresponding to the node.

[0092] For example, step 342 may be performed by Fig. 10BThe example architecture 700B shown is implemented. The left part of the graph model area 710 in the architecture 700B shows a conversation interaction graph constructed based on the hybrid reference feature, in which N nodes can be used to represent N reference conversations, and the vector corresponding to each node is set to the corresponding hybrid reference feature (g_1 to g_N). The corresponding N nodes can be arranged according to the order of the occurrence time of the reference conversation. For each node, there can be an edge pointing to itself, as well as edges pointing to other previous or subsequent nodes. Optionally, for each node, there can be an edge pointing from the node to all other nodes in the conversation interaction graph; or, in order to save computing and storage resources and improve computing efficiency, a time window of a certain length can be set, and the time window length can be set to m+n, for example, so that for each node, there are at most m edges from the node to the previous node and n edges from the node to the subsequent node. Through this conversation interaction graph, the influence of adjacent conversations on the expression of the current conversation can be fully considered, which helps to improve the naturalness of the final synthesized speech.

[0093] Optionally, the weight of each edge can be determined based on the similarity of the vectors corresponding to the two associated nodes, that is, based on the similarity of the mixed reference features corresponding to the two associated nodes. Exemplarily, for each node, the similarity of the nodes associated with all incoming edges can be determined, and these similarities can be normalized, such as by softmax, to obtain the weight corresponding to each incoming edge, that is, the sum of the weights of all incoming edges of each node can be guaranteed to be 1. Here, the incoming edge of a node refers to the edge pointing to the node from other nodes.

[0094] In the case where the reference conversation includes a talker label, as presented in the legend area 720, the N nodes can be divided into different types according to the different talker labels, such as talker 1 to talker M shown in the figure. Correspondingly, each edge can also be divided into different types according to the talker labels of the corresponding reference conversations of the two associated nodes, such as from talker 1 to talker 1, from talker 2 to talker 2, from talker 1 to talker 2, from talker 2 to talker 1, etc. as shown in the figure. Additionally or alternatively, each edge can also be divided into different types according to the time sequence of the corresponding reference conversations of the two associated nodes, such as from the future to the past and from the past to the future as shown in the figure. Optionally, the edge pointing from a node to the node itself can be regarded as from the future to the past, or it can also be regarded as from the past to the future. When the relationship type of each edge is determined based on both the interlocutor label and the occurrence time of the reference conversation corresponding to the node, the relationship type of the edge may include pointing from interlocutor 1 to interlocutor 1 and from the future to the past, from interlocutor 1 to interlocutor 2 and from the future to the past, from interlocutor 1 to interlocutor 2 and from the past to the future, etc. The conversation interaction graph constructed in this way can take into account human dependence and intra-person dependence, wherein human dependence can be understood as the influence between different interlocutors, for example, a conversation of a certain interlocutor may be affected by one or more conversations of other interlocutors, and may affect one or more conversations of other interlocutors; intra-person dependence can be understood as the influence of the same interlocutor on itself, for example, a conversation of a certain interlocutor may be affected by one or more other conversations of itself, and may affect one or more other conversations of itself. Therefore, in the process of determining the predicted speech features, the correlation between different interlocutors and different conversations of the same interlocutor can be fully considered, so that the predicted speech features are more suitable for the associated conversation scenes and enhance the naturalness of the final synthesized conversation speech.

[0095] The constructed dialogue interaction graph can be iteratively transformed to obtain dialogue interaction reference features h_1 to h_N containing richer information. The dialogue interaction reference features h_1 to h_N may contain information related to the time when the dialogue occurred and the interpersonal / intrapersonal dependencies of the interlocutors. Exemplarily, this process can be implemented by DialogueGCN processing. For example, the above iterative transformation can be performed in two steps. In the first step, the information of the neighboring nodes of each node can be aggregated, which can be achieved, for example, by the following formula:

[0096]

[0097] in, is the activation function, such as ReLU (linear rectification function), etc., i and j represent the node numbers, , represents the mixed reference feature of the corresponding node, , represents the edge weight, is the regularization constant, which can be preset or automatically determined during the training process, Indicates the relationship type on the edge The set of neighbor nodes of node i in the following example, R is the set of relationship types of edges in the dialogue interaction graph, and is the learning parameter determined during the training process. Then, in the second step, based on the output of the first step, the information of the neighboring nodes of each node can be aggregated again, which can be achieved, for example, by the following formula:

[0098]

[0099] in, is the activation function, and is the learning parameter determined during the training process. It can be used as the dialogue interaction reference feature mentioned above. This transformation process realizes the effective aggregation of neighbor node information and ensures that the inter-interlocutor dependency and the interlocutor self-dependence can affect the dialogue interaction reference feature.

[0100] In step 343, the predicted speech feature may be determined based on the feature of the text to be synthesized, at least one mixed reference feature and at least one dialogue interaction reference feature. This process may be implemented based on various types of neural networks, such as CNN, RNN, etc.

[0101] In some embodiments, step 343 can be implemented by the following operations. First, at least one candidate reference feature can be constructed based on the correspondence between at least one mixed reference feature and at least one conversation interaction reference feature, wherein each candidate reference feature includes a mixed reference feature and a corresponding conversation interaction reference feature. Exemplarily, the construction process can be implemented by performing operations such as concatenation and addition on the mixed reference feature and the corresponding conversation interaction reference feature. Then, the predicted speech feature can be determined based on the text feature to be synthesized and the at least one candidate reference feature.

[0102] Exemplarily, the predicted speech feature can be determined based on the feature of the text to be synthesized and at least one candidate reference feature by the following operations. First, the query feature can be determined based on the feature of the text to be synthesized. Exemplarily, the feature of the text to be synthesized can be used as the query feature, and the query feature can be obtained by performing linear or nonlinear transformation on the feature of the text to be synthesized, or the query feature can be obtained based on the feature of the text to be synthesized and other features, etc. The above-mentioned other features can, for example, include a conversationalist tag corresponding to the dialogue text to be synthesized, which is usually defaulted to the tag corresponding to the local device. Then, based on the similarity between the query feature and each candidate reference feature in at least one candidate reference feature, a weight corresponding to each candidate reference feature can be determined. For example, the similarity can be used as the weight, or the determined similarity can be normalized to determine the weight. Subsequently, a weighted sum of at least one candidate reference feature can be determined based on the determined weight. Finally, the predicted speech feature can be determined based on the query feature and the weighted sum. For example, the determined weighted sum can be used as the predicted speech feature, or the weighted sum can be further processed to obtain the predicted speech feature, etc. Exemplarily, a synthesis vector may be determined based on the query feature and the weighted sum, and then the synthesis vector may be linearly transformed and normalized to obtain the predicted speech feature.

[0103] For example, step 343 may be performed by Fig. 10C As shown in the figure, the mixed reference features g_1 to g_N can be concatenated with the corresponding dialogue interaction reference features h_1 to h_N to obtain N candidate reference features. The N candidate reference features can be input into the attention network. The dialogue text to be synthesized can be input into the text feature encoder to obtain the corresponding text features to be synthesized. The text feature encoder can be connected to Fig. 10A The same as the text feature encoder in . Subsequently, the text features to be synthesized can be concatenated with the interlocutor labels to obtain a query vector. The query vector can be input to the attention network. The attention network can perform the weight determination process described above. The determined weights can be concatenated with the query vector (i.e., the text features to be synthesized and the interlocutor labels), and linearly transformed and normalized to obtain the final predicted speech features.

[0104] Optionally, Figures 10A-10C The architecture shown can be used with Figure 5 The architecture shown is trained together to ensure that Figure 5 The text features extracted by the architecture shown are similar to those extracted by Figures 10A-10CThe predicted speech features determined by the architecture shown are related. Exemplarily, multiple sample conversations can be divided into multiple rounds of sample conversation groups, and each round of sample conversation group can include k sample conversations, where k is, for example, 6, 7, 8, etc. In each round of sample conversation group, the first k-1 sample conversations can be used as reference conversations, the conversation text of the last sample conversation can be used as the conversation text to be synthesized, and the conversation speech of the last sample conversation can be used as the true value of the synthesized conversation speech generated based on the conversation text to be synthesized. During the training process, the parameters of various networks or models can be continuously updated so that the difference between the synthesized conversation speech generated based on the conversation text to be synthesized and the corresponding true value is minimized as much as possible.

[0105] In some embodiments, Figure 3 Step 350 in can be Fig.11 It should be understood that Fig.11 The flowchart shown is for illustrative purposes only. Figure 3 Step 350 in may also be implemented in other ways.

[0106] Specifically, in step 351, based on the dialogue text to be synthesized, according to general pronunciation rules, a dialogue phoneme sequence to be synthesized can be generated. For example, according to Mandarin pronunciation rules, the dialogue text to be synthesized can be converted into a dialogue phoneme sequence to be synthesized for subsequent processing.

[0107] In step 352, the dialogue phoneme sequence to be synthesized may be encoded to obtain an initial phoneme encoding vector corresponding to the dialogue text to be synthesized. Alternatively, the dialogue factor sequence to be synthesized may be encoded into an initial phoneme encoding vector according to various encoding rules.

[0108] In step 353, a synthesized conversational speech may be generated based on the determined initial phoneme encoding vector and the predicted speech feature. Exemplarily, step 353 may be implemented by the following operations. First, a weighted sum of a set of pre-trained speech feature vectors may be determined with the predicted speech feature as a weight, as the predicted speech feature vector. The set of pre-trained speech feature vectors may be based on the initial phoneme encoding vector and the predicted speech feature. Figure 8The training process described in the specification can be obtained. Then, based on the initial phoneme coding vector and the speech feature vector, the synthetic phoneme coding vector can be determined. Optionally, the synthetic phoneme coding vector can be determined by performing operations such as splicing and adding the initial phoneme coding vector and the speech feature vector, and optionally performing a series of transformation operations. Or, optionally, the synthetic phoneme coding vector can be determined based on the initial phoneme coding vector and the speech feature vector and other additional features. Finally, the synthetic phoneme coding vector can be decoded, and synthetic conversational speech can be generated based on the decoded output. The decoding process can correspond to the aforementioned encoding process, that is, it can be implemented using an encoder and a decoder that follow the same rules.

[0109] Exemplarily, in the process of determining the synthesized phoneme coding vector based on the initial phoneme coding vector and the speech feature vector, the timbre feature vector associated with the dialogue text to be synthesized can be obtained, and a mixed phoneme coding vector can be obtained based on the initial phoneme coding vector, the timbre feature vector associated with the dialogue text to be synthesized, and the speech feature vector (for example, by concatenating or adding the three). Then, the phoneme features can be extracted from the mixed phoneme coding vector, and the phoneme features can include one or more of the length features, the pitch features, and the volume features. Finally, the synthesized phoneme coding vector can be determined based on the extracted phoneme features. Thus, the length features, the pitch features, the volume features, etc. of the synthesized dialogue speech can be accurately controlled based on the initial phoneme coding vector, the timbre feature vector associated with the dialogue text to be synthesized, and the speech feature vector, thereby effectively controlling the expression of the synthesized dialogue speech.

[0110] For example, step 350 may be performed by Fig.12 The example architecture 800 shown is implemented as shown in FIG. Figure 8 The architecture 800 is similar to the speech synthesis branch 690 shown in the figure. As shown in the figure, the lower side of the architecture 800 includes three branches. The left branch corresponds to the acquired timbre feature vector associated with the dialogue text to be synthesized, which can be preset by the user or can be a default value. The middle branch corresponds to the process of obtaining an initial phoneme encoding vector based on the text to be synthesized, wherein the dialogue text to be synthesized can first be converted into a dialogue phoneme sequence to be synthesized according to general pronunciation rules, and then encoded by FastSpeech 2 to obtain an initial phoneme encoding vector. Optionally, the FastSpeech 2 encoder can also be replaced by other types of encoders. The right branch corresponds to the process of determining the predicted speech features based on a reference dialogue, and the reference dialogue can be input into the multimodal dialogue interaction graph module, which can be based on information about Figures 3 to 10CThe predicted speech feature can be multiplied with a set of pre-trained speech features to obtain a corresponding speech feature vector. The timbre feature vector, the initial phoneme encoding vector, and the speech feature vector associated with the dialogue text to be synthesized can be concatenated and input into the variance adapter to obtain a synthesized phoneme encoding vector. Exemplarily, the variance adapter can include multiple predictors, such as a sound length predictor, a pitch predictor, a volume predictor, etc., which can be composed of a neural network and Figure 8 The synthesized phoneme encoding vector can be pre-trained in the structure shown in the figure. Finally, the synthesized phoneme encoding vector can be decoded (for example, by FastSpeech 2 decoder) into the corresponding phoneme sequence, and the synthesized conversational speech can be generated based on the decoded phoneme sequence. Through this architecture, the synthesized conversational speech can be generated quickly and accurately.

[0111] For ease of understanding, Fig.13 A speech synthesis architecture 900 is schematically shown, which can be used to implement all or part of the embodiments of the speech synthesis method 300 described above. Each module and related steps in the architecture 900 have been described in detail above and will not be repeated here.

[0112] Fig.14 An example block diagram of a speech synthesis device 1000 according to some embodiments of the present application is schematically shown. Fig.14 As shown, the speech synthesis device 1000 includes an acquisition module 1010, a first feature extraction module 1020, a second feature extraction module 1030, a speech feature prediction module 1040 and a speech synthesis module 1050. Exemplarily, the speech synthesis device 1000 can be deployed in Figure 1 On the server 110, terminal device 130 or a combination of the two as shown.

[0113] Specifically, the acquisition module 1010 can be configured to acquire the dialogue text to be synthesized and the dialogue text and dialogue speech of at least one reference dialogue associated with the dialogue text to be synthesized; the first feature extraction module 1020 can be configured to extract the features of the text to be synthesized from the dialogue text to be synthesized; the second feature extraction module 1030 can be configured to extract reference text features from the dialogue text of each reference dialogue in at least one reference dialogue, and extract reference speech features from the dialogue speech of the reference dialogue; the speech feature prediction module 1040 can be configured to determine the predicted speech features corresponding to the dialogue text to be synthesized based on the features of the text to be synthesized, the extracted at least one reference text feature and the extracted at least one reference speech feature; the speech synthesis module 1050 can be configured to generate a synthesized dialogue speech corresponding to the dialogue text to be synthesized based at least on the dialogue text to be synthesized and the predicted speech features.

[0114] It should be understood that the apparatus 1000 can be implemented in software, hardware, or a combination of software and hardware. Multiple different modules can be implemented in the same software or hardware structure, or one module can be implemented by multiple different software or hardware structures.

[0115] In addition, the apparatus 1000 can be used to implement the method 300 described above, and the relevant details have been described in detail above, and for the sake of brevity, they will not be repeated here. The apparatus 1000 can have the same features and advantages as described with respect to the above method.

[0116] Fig.11 An example block diagram of a computing device 1010 according to some embodiments of the present application is schematically shown. Figure 1 The server 110 or other types of computing devices that can be used to deploy the apparatus 1000 provided in the present application.

[0117] As shown, the example computing device 1100 includes a processing system 1101, one or more computer-readable media 1102, and one or more I / O interfaces 1103 that are communicatively coupled to each other. Although not shown, the computing device 1100 may also include a system bus or other data and command transmission system that couples the various components to each other. The system bus may include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any of a variety of bus architectures, or may also include control and data lines.

[0118] Processing system 1101 represents the functionality of performing one or more operations using hardware. Thus, processing system 1101 is illustrated as including hardware elements 1104 that may be configured as processors, functional blocks, etc. This may include implementing application specific integrated circuits in hardware or other logic devices formed using one or more semiconductors. Hardware elements 1104 are not limited by the materials from which they are formed or the processing mechanisms employed therein. For example, a processor may be composed of (multiple) semiconductors and / or transistors (e.g., electronic integrated circuits (ICs)). In such a context, processor executable instructions may be electronic executable instructions.

[0119] The computer-readable medium 1102 is illustrated as including a memory / storage device 1105. The memory / storage device 1105 represents a memory / storage device associated with one or more computer-readable media. The memory / storage device 1105 may include a volatile storage medium (such as a random access memory (RAM)) and / or a non-volatile storage medium (such as a read-only memory (ROM), flash memory, an optical disk, a magnetic disk, etc.). The memory / storage device 1105 may include fixed media (e.g., RAM, ROM, a fixed hard drive, etc.) and removable media (e.g., flash memory, a removable hard drive, an optical disk, etc.). Exemplarily, the memory / storage device 1105 may be used to store the conversation text, conversation voice, etc. of the reference conversation mentioned in the above embodiments. The computer-readable medium 1102 may be configured in various other ways as further described below.

[0120] One or more input / output interfaces 1103 represent functions that allow a user to enter commands and information into the computing device 1100 and also allow the information to be presented to the user and / or sent to other components or devices using various input / output devices. Examples of input devices include keyboards, cursor control devices (e.g., mice), microphones (e.g., for voice input), scanners, touch functions (e.g., capacitive or other sensors configured to detect physical touch), cameras (e.g., visible or invisible wavelengths (such as infrared frequencies) can be used to detect motion that does not involve touch as gestures), network cards, receivers, and the like. Examples of output devices include display devices, speakers, printers, tactile response devices, network cards, transmitters, and the like. Exemplarily, in the embodiments described above, the user can be allowed to enter various instructions, perform voice interactions, and the like through input devices and output devices.

[0121] The computing device 1100 also includes a speech synthesis application 1106. The speech synthesis application 1106 may be stored as a computer program instruction in the memory / storage device 1105. The road network data generation application 1106 may implement all functions of the various modules of the device 1000 described with respect to FIG. 10 together with the processing system 1101 and the like.

[0122] Various techniques may be described herein in the general context of software, hardware, elements, or program modules. Typically, these modules include routines, programs, objects, elements, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The terms "module," "function," etc., as used herein, generally represent software, firmware, hardware, or a combination thereof. The features of the techniques described herein are platform-independent, meaning that these techniques can be implemented on a variety of computing platforms with a variety of processors.

[0123] An implementation of the described modules and techniques may be stored on or transmitted across some form of computer-readable media. Computer-readable media may include various media accessible by computing device 1100. By way of example and not limitation, computer-readable media may include “computer-readable storage media” and “computer-readable signal media”.

[0124] As opposed to a simple signal transmission, carrier wave, or signal itself, "computer-readable storage medium" refers to a medium and / or device capable of persistently storing information, and / or a tangible storage device. Therefore, computer-readable storage media refers to non-signal-bearing media. Computer-readable storage media include hardware such as volatile and non-volatile, removable and non-removable media and / or storage devices implemented in a method or technology suitable for storing information (such as computer-readable instructions, data structures, program modules, logic elements / circuits, or other data). Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, hard disk, cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or other storage device, tangible medium, or article of manufacture suitable for storing the desired information and accessible by a computer.

[0125] "Computer readable signal media" refers to signal bearing media that is configured to send instructions to the hardware of the computing device 1100, such as via a network. Signal media typically can embody computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, data signal, or other transmission mechanism. Signal media also includes any information transfer media. By way of example and not limitation, signal media include wired media such as a wired network or direct connection and wireless media such as acoustic, RF, infrared, and other wireless media.

[0126] As previously described, hardware elements 1101 and computer-readable media 1102 represent instructions, modules, programmable device logic, and / or fixed device logic implemented in hardware form, which in some embodiments can be used to implement at least some aspects of the technology described herein. Hardware elements may include integrated circuits or systems on chips, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and other implementations in silicon or components of other hardware devices. In this context, hardware elements can be used as processing devices that perform program tasks defined by instructions, modules, and / or logic embodied by hardware elements, as well as hardware devices for storing instructions for execution, such as the computer-readable storage media described previously.

[0127] The aforementioned combination may also be used to implement the various techniques and modules described herein. Therefore, software, hardware or program modules and other program modules may be implemented as one or more instructions and / or logic embodied on some form of computer-readable storage medium and / or by one or more hardware elements 1101. Computing device 1100 may be configured to implement specific instructions and / or functions corresponding to software and / or hardware modules. Therefore, for example, by using a computer-readable storage medium and / or hardware element 1101 of a processing system, a module may be implemented as a module executable by computing device 1100 as software, at least in part, in hardware. Instructions and / or functions may be executed / operable by, for example, one or more computing devices 1100 and / or processing systems 1101 to implement the techniques, modules and examples described herein.

[0128] The techniques described herein may be supported by these various configurations of computing device 1100 and are not limited to the specific examples of the techniques described herein.

[0129] It should be understood that, for the sake of clarity, embodiments of the present application are described with reference to different functional units. However, it will be apparent that, without departing from the present application, the functionality of each functional unit can be implemented in a single unit, in multiple units or as a part of other functional units. For example, the functionality that is described as being performed by a single unit can be performed by multiple different units. Therefore, reference to a specific functional unit is only considered as a reference to the appropriate unit for providing the described functionality, rather than indicating a strict logical or physical structure or organization. Therefore, the present application can be implemented in a single unit, or can be physically and functionally distributed between different units and circuits.

[0130] The present application provides a computer-readable storage medium on which computer-readable instructions are stored. When the computer-readable instructions are executed, the above-mentioned speech synthesis method is implemented.

[0131] The present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computing device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computing device executes the speech synthesis method provided in the above various embodiments.

[0132] Variations to the disclosed embodiments will be understood and effected by those skilled in the art in practicing the claimed subject matter by studying the drawings, the disclosure and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and "a" or "an" does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0133] It is understandable that in the specific implementation of this application, user historical conversation data, etc. are involved. When the above embodiments of this application involving such data are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

Claims

1. A speech synthesis method, comprising: Acquire a dialogue text to be synthesized and a dialogue text and a dialogue voice of at least one reference dialogue associated with the dialogue text to be synthesized; Extracting features of the text to be synthesized from the dialogue text to be synthesized; For each reference dialogue in the at least one reference dialogue, extracting reference text features from the dialogue text of the reference dialogue, and extracting reference voice features from the dialogue voice of the reference dialogue; Determining a predicted speech feature corresponding to the dialogue text to be synthesized based on the text feature to be synthesized, the at least one reference text feature extracted, and the at least one reference speech feature extracted; generating a synthesized dialogue speech corresponding to the dialogue text to be synthesized based at least on the dialogue text to be synthesized and the predicted speech features, Wherein, determining the predicted speech feature corresponding to the dialogue text to be synthesized based on the text feature to be synthesized, the at least one reference text feature extracted, and the at least one reference speech feature extracted comprises: Determining at least one mixed reference feature corresponding to the at least one reference dialogue based on the extracted at least one reference text feature and the extracted at least one reference voice feature, wherein each mixed reference feature includes a reference text feature and a reference voice feature corresponding to a corresponding reference dialogue; Based on the at least one hybrid reference feature, determining at least one conversation interaction reference feature corresponding to the at least one hybrid reference feature, wherein each conversation interaction reference feature includes information associated with at least one other hybrid reference feature other than the corresponding hybrid reference feature; The predicted speech feature is determined based on the to-be-synthesized text feature, the at least one mixed reference feature, and the at least one dialogue interaction reference feature.

2. The speech synthesis method according to claim 1, wherein: The determining the predicted speech feature based on the to-be-synthesized text feature, the at least one mixed reference feature, and the at least one dialogue interaction reference feature comprises: constructing at least one candidate reference feature according to the correspondence between the at least one hybrid reference feature and the at least one conversation interaction reference feature, wherein each candidate reference feature includes a hybrid reference feature and a corresponding conversation interaction reference feature; The predicted speech feature is determined based on the to-be-synthesized text feature and the at least one candidate reference feature.

3. The speech synthesis method according to claim 2, wherein: The determining the predicted speech feature based on the to-be-synthesized text feature and the at least one candidate reference feature comprises: Determining query features based on the features of the text to be synthesized; Determining a weight corresponding to each candidate reference feature based on a similarity between the query feature and each candidate reference feature of the at least one candidate reference feature; determining a weighted sum of the at least one candidate reference feature based on the determined weight; The predicted speech feature is determined based on the query feature and the weighted sum.

4. The speech synthesis method according to claim 3, wherein: The determining the predicted speech feature based on the query feature and the weighted sum includes: Determining a composite vector based on the query feature and the weighted sum; The synthesized vector is linearly transformed and normalized to obtain the predicted speech feature.

5. The speech synthesis method according to claim 1, wherein: The determining, based on the at least one hybrid reference feature, at least one dialog interaction reference feature corresponding to the at least one hybrid reference feature comprises: Using each of the at least one hybrid reference feature as a node, constructing a dialog interaction graph, wherein for each node, there are one or more adjacent nodes, and there are edges pointing from the one or more adjacent nodes to the node; For each node, aggregating the mixed reference feature corresponding to the node and the mixed reference features corresponding to one or more adjacent nodes of the node to obtain an aggregated reference feature corresponding to the node; Based on the aggregated reference features corresponding to each node, a conversation interaction reference feature corresponding to each node is determined.

6. The speech synthesis method according to claim 5, wherein: Each reference conversation further includes a conversation partner label, and constructing a conversation interaction graph using each of the at least one hybrid reference feature as a node includes performing at least one of the following operations for each edge: Determining a weight of the edge based on the similarity of the mixed reference features of the two nodes associated with the edge; The relationship type of the edge is determined based on attributes of two nodes associated with the edge, wherein the attributes include at least one of the following: an occurrence time of a reference conversation corresponding to the node and an interlocutor label of the reference conversation corresponding to the node.

7. The speech synthesis method according to claim 1, wherein: Extracting reference speech features from the conversation speech of the reference conversation includes: Determining a coding vector representing the conversation speech of the reference conversation; Converting the determined encoding vector into a fixed-length vector having a preset length; Determining a similarity between the fixed-length vector and each pre-trained speech feature vector in a set of pre-trained speech feature vectors; Based on the determined set of similarities, reference speech features corresponding to the conversation speech of the reference conversation are determined.

8. The speech synthesis method according to claim 7, wherein: The set of pre-trained speech feature vectors is obtained by training through the following process: Obtaining conversation texts and conversation voices of multiple sample conversations; Initialize a set of speech feature vectors to be trained; Based on each of the multiple sample conversations, perform the following operations: - based on the conversational speech of the sample conversation, determining a coding vector representing the conversational speech, and extracting true value interlocutor features corresponding to the conversational speech; -converting the determined encoding vector into a fixed-length vector having a preset length; -determining the similarity between the fixed-length vector and each pre-trained speech feature vector in the set of speech feature vectors to be trained; -determining sample speech features corresponding to the conversation speech based on the determined set of similarities; - extracting predicted interlocutor features corresponding to the conversation speech from the sample speech features; - generating a reconstructed speech based on the sample speech features, the set of speech feature vectors to be trained and the conversation text of the sample conversation; -updating the set of speech feature vectors to be trained based on the difference between the conversation speech of the sample conversation and the reconstructed speech and the difference between the true interlocutor features and the predicted interlocutor features; The updated set of speech feature vectors to be trained is used as the set of pre-trained speech feature vectors.

9. The speech synthesis method according to claim 7, wherein: Determining the encoding vector representing the conversation speech of the reference conversation includes: dividing the conversation speech of the reference conversation into a plurality of audio frames; Windowing each audio frame in the multiple audio frames to obtain multiple windowed audio frames; Converting the plurality of audio frames into an audio energy distribution by Fourier transform; The audio energy distribution is passed through a filter bank, and a coding vector representing the conversation speech of the reference conversation is extracted based on the output of the filter bank.

10. The speech synthesis method according to claim 1, wherein: The step of generating a synthesized dialogue speech corresponding to the dialogue text to be synthesized based at least on the dialogue text to be synthesized and the predicted speech feature comprises: Based on the dialogue text to be synthesized, according to general pronunciation rules, a dialogue phoneme sequence to be synthesized is generated; Encoding the dialogue phoneme sequence to be synthesized to obtain an initial phoneme encoding vector corresponding to the dialogue text to be synthesized; The synthesized conversational speech is generated based on the determined initial phoneme encoding vector and the predicted speech feature.

11. The speech synthesis method according to claim 10, wherein: The generating the synthesized dialogue speech based on the determined initial phoneme encoding vector and the predicted speech feature comprises: Using the predicted speech feature as a weight, determining a weighted sum of a set of pre-trained speech feature vectors as a predicted speech feature vector; Determine a synthesized phoneme coding vector based on the initial phoneme coding vector and the speech feature vector; The synthesized phoneme encoding vector is decoded, and the synthesized conversational speech is generated based on the decoded output.

12. The speech synthesis method according to claim 11, wherein: The determining of the synthesized phoneme coding vector based on the initial phoneme coding vector and the speech feature vector comprises: Obtaining a timbre feature vector associated with the dialogue text to be synthesized; Based on the initial phoneme coding vector, the timbre feature vector associated with the dialogue text to be synthesized and the speech feature vector, a mixed phoneme coding vector is obtained; Extracting phoneme features from the mixed phoneme encoding vector, wherein the phoneme features include at least one of a sound length feature, a pitch feature, and a volume feature; Based on the extracted phoneme features, the synthesized phoneme encoding vector is determined.

13. The speech synthesis method according to claim 1, wherein: The extracting the features of the text to be synthesized from the dialogue text to be synthesized comprises: Determining a coding vector representing the dialogue text to be synthesized; Based on the determined encoding vector, the features of the text to be synthesized are extracted.

14. The speech synthesis method according to claim 13, wherein: The determining of the encoding vector representing the dialogue text to be synthesized comprises: Segmenting the dialogue text to be synthesized into one or more words; Convert each of the one or more words into a word vector; The encoding vector is determined based on the word vectors corresponding to the one or more words.

15. The speech synthesis method according to claim 1, wherein: The extracting of reference text features from the dialogue text of the reference dialogue comprises: Determining an encoding vector representing the conversation text of the reference conversation; Based on the determined encoding vector, reference text features corresponding to the dialogue text of the reference dialogue are extracted.

16. A speech synthesis device, comprising: An acquisition module, configured to acquire the dialogue text to be synthesized and the dialogue text and dialogue voice of at least one reference dialogue associated with the dialogue text to be synthesized; A first feature extraction module is configured to extract features of the text to be synthesized from the dialogue text to be synthesized; A second feature extraction module is configured to extract, for each reference dialogue in the at least one reference dialogue, reference text features from the dialogue text of the reference dialogue, and reference voice features from the dialogue voice of the reference dialogue; A speech feature prediction module is configured to determine the predicted speech feature corresponding to the dialogue text to be synthesized based on the text feature to be synthesized, the at least one reference text feature extracted and the at least one reference speech feature extracted, wherein the determining the predicted speech feature corresponding to the dialogue text to be synthesized based on the text feature to be synthesized, the at least one reference text feature extracted and the at least one reference speech feature extracted comprises: determining at least one mixed reference feature corresponding to the at least one reference dialogue based on the at least one reference text feature extracted and the at least one reference speech feature extracted, wherein each mixed reference feature comprises a reference text feature and a reference speech feature corresponding to the corresponding reference dialogue; determining at least one dialogue interaction reference feature corresponding to the at least one mixed reference feature based on the at least one mixed reference feature, wherein each dialogue interaction reference feature comprises information associated with at least one other mixed reference feature other than the corresponding mixed reference feature; determining the predicted speech feature based on the text feature to be synthesized, the at least one mixed reference feature and the at least one dialogue interaction reference feature; The speech synthesis module is configured to generate a synthesized dialogue speech corresponding to the dialogue text to be synthesized based at least on the dialogue text to be synthesized and the predicted speech features.

17. A computing device comprising: a memory configured to store computer-executable instructions; A processor configured to perform the method according to any one of claims 1 to 15 when the computer executable instructions are executed by the processor.

18. A computer-readable storage medium storing computer-executable instructions, which, when executed, perform the method according to any one of claims 1 to 15.

19. A computer program product comprising computer instructions, which when executed by a processor implement the steps of the method of any one of claims 1 to 15.

Images