Method, device and equipment for generating composite style speech and storage medium

Abstract

The application belongs to the field of artificial intelligence and the field of medical health, and relates to a composite style voice generation method, comprising: obtaining a text sequence, using a text encoder to perform feature extraction on the text sequence to obtain text semantic information, wherein the text semantic information comprises a semantic feature vector and a prosody feature vector; performing style conversion on the text semantic information through each style expert network of a style expert module to obtain multiple style voice features; obtaining a target style condition, processing the text semantic information and the target style condition through a gating network to obtain a style weight, performing weighted summation operation on the multiple style voice features according to the style weight to obtain a composite style feature; and converting the composite style feature through a voice synthesizer to obtain a composite style voice of the text sequence. In addition, the application also relates to blockchain technology, and a target page can be stored in the blockchain. The application can improve the fluency of the composite style voice.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and the field of medical health, in particular to a method for generating a composite style voice. BACKGROUND

[0002] In the field of speech synthesis technology, with the rapid development of artificial intelligence and machine learning technology, multi-style speech synthesis has become one of the current research hotspots. Multi-style speech synthesis only generates a composite style voice with multiple style characteristics according to the input text to meet the diversified needs in different application scenarios. For example, in the medical field, multi-style speech synthesis technology can greatly improve the quality and efficiency of medical services, especially in the communication between doctors and patients. For example, when a doctor explains a treatment plan to a child patient through a voice assistant, a gentle and kind voice style can be fused to reduce the child's fear and anxiety, so that the child can more easily accept the treatment plan.

[0003] Traditional multi-style speech synthesis methods mainly rely on manual specification or simple linear interpolation to realize style fusion.

[0004] However, the manual specification of style parameters is not only time-consuming and laborious, but also difficult to accurately capture subtle semantic changes in the input text, resulting in a disconnection between the style of the synthesized voice and the content of the text. Although simple linear interpolation can achieve preliminary fusion between styles, its static processing method cannot dynamically adapt to the diversified semantic needs of the input text, resulting in a lack of naturalness and expressiveness in the synthesis result.

[0005] It can be seen that the existing multi-style speech synthesis method cannot automatically and dynamically fuse multiple style characteristics according to the semantic content of the input text and generate a natural and smooth composite style voice. SUMMARY

[0006] The purpose of the embodiments of the present application is to propose a method for generating a composite style voice, an apparatus, a device and a storage medium, which mainly aims to improve the accuracy and fluency of the composite style voice.

[0007] In order to solve the above technical problems, in a first aspect, the embodiments of the present application provide a method for generating a composite style voice, which adopts the following technical solution:

[0008] Obtain a text sequence, use a text encoder to extract features of the text sequence to obtain text semantic information, wherein the text semantic information includes a semantic feature vector and a prosody feature vector;

[0009] Convert the text semantic information through each style expert network of a style expert module to obtain multiple style voice features;

[0010] The target style condition is obtained, the text semantic information and the target style condition are processed through a gating network to obtain a style weight, and a plurality of style speech features are weighted and summed according to the style weight to obtain a composite style feature.

[0011] The composite style feature is converted through a speech synthesizer to obtain a composite style speech of the text sequence.

[0012] To solve the above technical problems, in a second aspect, the embodiments of the present application also provide a composite style speech generation device, which adopts the technical scheme as follows:

[0013] The encoding module is configured to obtain a text sequence, perform feature extraction on the text sequence by using a text encoder, and obtain text semantic information, wherein the text semantic information includes a semantic feature vector and a prosody feature vector.

[0014] The conversion module is configured to perform style conversion on the text semantic information through each style expert network of the style expert module to obtain a plurality of style speech features.

[0015] The weighting module is configured to obtain a target style condition, process the text semantic information and the target style condition through a gating network to obtain a style weight, and perform weighted summation operation on the plurality of style speech features according to the style weight to obtain a composite style feature.

[0016] The speech generation module is configured to convert the composite style feature through a speech synthesizer to obtain a composite style speech of the text sequence.

[0017] To solve the above technical problems, in a third aspect, the embodiments of the present application also provide a computer device, which includes at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the composite style speech generation method as described above.

[0018] To solve the above technical problems, in a fourth aspect, the embodiments of the present application also provide a computer-readable storage medium storing a computer program, and the computer program is executed by a processor to implement the composite style speech generation method as described above.

[0019] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0020] The text sequence is processed by the text encoder to predict the prosody feature vector of the text sequence, and the text encoder can also capture the context information and semantic relationship in the text sequence, thereby generating a high-quality semantic feature vector; combining the semantic feature vector and the prosody feature vector helps the subsequent model to generate natural and fluent composite style speech.

[0021] The text semantic information is respectively converted into multiple style speech features by using multiple style expert networks of the style expert module, providing a basis for subsequent generation of composite style speech; the text semantic information based on the text sequence is converted, improving the fluency of the subsequent composite style speech.

[0022] The text semantic information and the target style condition are processed by the gating network to obtain a style weight, and the multiple style speech features are weighted and summed according to the style weight to obtain a composite style feature, avoiding manual specification of style parameters and improving the generation efficiency of the subsequent composite speech; and the diversified semantic requirements of the text sequence are dynamically adapted, improving the fluency of the subsequently generated composite style speech.

[0023] The composite style feature is converted by the speech synthesizer to obtain the composite style speech of the text sequence, improving the fluency of the composite style speech; and the composite style feature is processed by the speech synthesizer, improving the naturalness and clarity of the composite speech. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the scheme in the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0025] Figure 1 is an exemplary system architecture diagram to which the present application can be applied;

[0026] Figure 2 is a flowchart of one embodiment of the composite style speech generation method according to the present application;

[0027] Figure 3 is a structural schematic diagram of one embodiment of the composite style speech generation device according to the present application;

[0028] Figure 4 is a structural schematic diagram of one embodiment of the device according to the present application. DETAILED DESCRIPTION

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terms used in the specification are intended to describe the particular embodiments and are not intended to limit the application; the terms "include" and "have" and their any variations used in the specification and the claims and the above description of drawings are intended to cover the non-exclusive inclusion; the terms "first", "second" and the like used in the specification and the claims and the above description of drawings are intended to distinguish different objects, not to describe a particular order.

[0030] Reference herein to "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another. It is expressly understood that the embodiments described herein are merely examples and are not intended to limit the scope of the application.

[0031] In order to make the person skilled in the art better understand the scheme of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings below.

[0032] As shown in Figure 1 The system architecture 100 can include a terminal device 101, a network 102 and a server 103, and the terminal device 101 can be a notebook computer 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103. The network 102 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0033] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0034] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing, in addition to the notebook computer 1011, the tablet computer 1012 or the mobile phone 1013, the terminal device 101 can also be an electronic book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer and a desktop computer, etc.

[0035] The server 103 can be a server providing various services, for example, a background server providing support for a page displayed on the terminal device 101.

[0036] It should be noted that the method for generating composite style speech provided by the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the device for generating composite style speech is generally arranged in a server / terminal device.

[0037] It should be understood that, Figure 1 The number of terminal devices, networks and servers in

[0038] With reference to Figure 2 , a flow chart of one embodiment of the method for generating composite style speech according to the present application is shown. The order of the steps in the flow chart can be changed according to different needs, and some steps can be omitted. The method for generating composite style speech provided by the embodiments of the present application can be applied to any scene requiring generation of composite style speech, and then the method for generating composite style speech can be applied to products in these scenes. The method for generating composite style speech comprises the following steps:

[0039] In step S201, a text sequence is obtained, and a text encoder is used to extract features of the text sequence to obtain text semantic information, wherein the text semantic information comprises a semantic feature vector and a prosody feature vector.

[0040] In the present embodiment, the electronic device (for example Figure 1The text sequence can be obtained by wired connection or wireless connection (e.g., the server / terminal device shown). It should be noted that the wireless connection can include, but is not limited to, 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other now known or future developed wireless connection. The execution subject of the present application can be a speech synthesis model, or simply a model. The text sequence usually has the same format as the text sequence training data used in the training of the speech synthesis model, but the training data contains other noise, which is determined according to the situation. In the business scenario of generating a composite style voice of a text sequence, the text sequence can include a text sequence automatically generated by artificial intelligence (AI) or a manually written text sequence.

[0041] In the present embodiment, the speech synthesis model includes a text encoder, a style expert module, a gating network, and a speech synthesizer. The style expert module includes multiple independent style expert networks, each of which is responsible for generating a style feature. The text sequence is a text sequence arranged in a certain order by words or characters in a sentence, such as a paragraph of text (e.g., Good night, have a good dream). The text encoder can use a sequence-to-sequence model based on a self-attention mechanism, such as a Transformer model. Since the prosodic feature vector of the text sequence also needs to be extracted, a prosody prediction module is introduced in the encoder of the model to achieve this. The prosody prediction module can use a neural network, such as a convolutional neural network.

[0042] The text sequence of the speech to be synthesized is obtained, and the text sequence is processed by the text encoder to extract the text semantic information of the text sequence, wherein the text semantic information includes a semantic feature vector and a prosodic feature vector. Specifically, the text sequence is converted into a vector representation by a vector conversion technique, such as word embedding. The semantic feature vector of the text sequence is extracted by the multi-layer self-attention mechanism and the feedforward neural network of the text encoder. In order to ensure that the semantic information of the text sequence is fully captured, the dimension of the semantic feature vector is high, such as 512 dimensions or 1024 dimensions, etc. The prosody prediction module is introduced in the encoder of the model, and the semantic feature vector is processed by the prosody prediction module to predict the prosodic feature vector of each character or syllable, including but not limited to stress, duration, rhythm, etc.

[0043] In the medical field, for example, a doctor writes down the illness description and treatment plan in words, such as "Your illness is mild pneumonia, you need to take antibiotics on time, and pay attention to rest"; the text sequence of the text is taken as the text sequence to be synthesized, the text semantic information of the text sequence is extracted, specifically, the text sequence is converted into a vector representation, and the prosodic feature vector of each character in the text is predicted.

[0044] In this embodiment, the text sequence is processed by the text encoder to predict the prosodic feature vector of the text sequence, and the context information and semantic relationship in the text sequence can also be captured by the text encoder, thereby generating a high-quality semantic feature vector; combining the semantic feature vector and the prosodic feature vector helps the subsequent model to generate natural and fluent composite style speech.

[0045] Step S202, style conversion is performed on the text semantic information by each style expert network of the style expert module to obtain multiple style speech features.

[0046] In this embodiment, it is explained above that the style expert module includes multiple independent style expert networks, each of which is responsible for generating a style feature, such as gentle, sad, happy, angry, etc., and the style expert network adopts a neural network architecture, such as a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0047] In this embodiment, the text semantic information is input into each style expert network of the style expert module, and the text semantic information is adjusted and converted by the style feature of each style expert network to generate multiple style speech features, wherein the content of the multiple style speech features is the same, but the styles are different, such as the same text sequence content.

[0048] In the medical field, for example, the text semantic information extracted from the above "Your illness is mild pneumonia, you need to take antibiotics on time, and pay attention to rest" is input into multiple style expert networks, and the style features corresponding to "Your illness is mild pneumonia, you need to take antibiotics on time, and pay attention to rest" are generated by the style expert networks, such as serious style features, gentle style features, etc.

[0049] In this embodiment, the text semantic information is converted by using multiple style expert networks of the style expert module to generate multiple style speech features, which provides a basis for subsequent generation of composite style speech; the text semantic information based on the text sequence is converted to improve the fluency of the subsequent composite style speech.

[0050] In step S203, the target style condition is obtained, the text semantic information and the target style condition are processed through a gating network to obtain a style weight, and the plurality of style speech features are weighted and summed according to the style weight to obtain a composite style feature.

[0051] In the embodiment, the target style condition includes an explicit style label or an implicit style embedding. The explicit style label is a style label predefined by a user. For example, if the user wants to generate a composite speech that is gentle and pleasant, the user inputs the style labels “gentle” and “pleasant” as the target style condition. The implicit style embedding is obtained by analyzing the text semantic information through a style encoder to obtain a corresponding style label. The gating network includes an input layer, a feature extraction layer, a weight generation layer, and an output layer. The text semantic information and the target style condition are output to the gating network through the input layer of the gating network. The style features of the target style condition are extracted through the feature extraction layer. The weight of each style feature is calculated through the weight generation layer to obtain the style weight. In the output layer, the plurality of style speech features are weighted and summed according to the style weight to obtain the composite style feature, and the composite style feature is output.

[0052] In the medical field, for example, a doctor selects target style features according to the characteristics of a patient (such as a child, an adult, etc.). The selected target style features are calculated through the gating network to obtain the composite style feature. For example, a child can select two target style features “gentle” and “affectionate”.

[0053] In the embodiment, the text semantic information and the target style condition are processed through the gating network to obtain the style weight, and the plurality of style speech features are weighted and summed according to the style weight to obtain the composite style feature, which avoids manually specifying the style parameters and improves the generation efficiency of the subsequent composite speech. The dynamic adaptation to the diversified semantic requirements of the text sequence is realized, and the fluency of the generated composite style speech is improved.

[0054] In step S204, the composite style feature is converted through a speech synthesizer to obtain a composite style speech of the text sequence.

[0055] In the embodiment, the speech synthesizer adopts a model based on a generative adversarial network (GAN), such as HiFi-GAN, to improve the naturalness and clarity of the composite style speech. The speech synthesizer includes a generator and a discriminator. The generator converts the composite style feature into a high-quality composite speech waveform, and the discriminator is used to determine whether the generated composite style speech is similar to the real speech.

[0056] In this embodiment, the composite style feature is converted by the speech synthesizer to obtain the composite style speech of the text sequence.

[0057] Specifically, the generator of the speech synthesizer adopts a multi-scale convolutional neural network structure, which gradually converts the low-dimensional composite style feature into a high-resolution speech waveform through stepwise upsampling and skip connection. In each upsampling step, the generator uses a transpose convolutional layer to increase the resolution of the feature map, while retaining the information of the previous layer through residual connection. This multi-scale structure can effectively capture local and global features in the speech signal to generate a speech waveform of the composite style speech. The discriminator adopts a multi-scale spectral normalization convolutional neural network (SNGAN), which discriminates different scale feature maps to improve the stability and robustness of the discriminator. Finally, the high-resolution speech waveform is output as a playable composite style speech.

[0058] In the medical field, for example, the composite style feature generated by the doctor according to the characteristics of the patient (such as children, adults, etc.) is used to convert the text sequence "Your condition is mild pneumonia, you need to take antibiotics on time, and pay attention to rest" to obtain a composite style speech with "gentle" and "affectionate" style. The composite style speech is used to broadcast "Your condition is mild pneumonia, you need to take antibiotics on time, and pay attention to rest".

[0059] In this embodiment, the composite style feature is converted by the speech synthesizer to obtain the composite style speech of the text sequence, which improves the fluency of the composite style speech. The composite style feature is processed by the speech synthesizer to improve the naturalness and clarity of the composite speech.

[0060] In one embodiment, the text encoder is used to extract features from the text sequence to obtain text semantic information, wherein the text semantic information includes a semantic feature vector and a prosody feature vector, which includes:

[0061] The text sequence is converted into a vector representation by a vector conversion technique to obtain a text sequence vector.

[0062] The semantic feature vector is obtained by extracting features from the text sequence vector using the encoder of the text encoder.

[0063] The prosody feature vector is obtained by performing feature extraction on the semantic feature vector using the convolutional layer of the prosody prediction module.

[0064] In the embodiment, each character of the text sequence is converted into a vector representation by a vector conversion technology (such as word embedding, etc.), and all character vectors are collected to obtain a text sequence vector. Since the Transformer model cannot capture the position information in the text sequence, additional position encoding is needed to represent the position information, which can be fixed (such as generated by sine and cosine functions). These position encodings are added to the text sequence vector to provide position information of the characters in the sequence; the text sequence after adding the position encoding is processed by a self-attention mechanism to calculate attention weights, and the context information of the text sequence is obtained through the attention weights

[0065] After the self-attention mechanism, the representation of each word or character is further processed by a feedforward neural network, which usually includes two linear transformations and an activation function (such as ReLU). After processing by the self-attention mechanism and the feedforward neural network, the semantic feature vector is finally obtained.

[0066] In the above step S201, it is mentioned that the text encoder is introduced with a prosody prediction module; the semantic feature vector is input into the prosody prediction module, and the semantic feature vector is convolved by the multiple convolution layers of the prosody prediction module to extract the local features of the semantic feature vector. The dimension of the local feature is reduced by the pooling layer, and the key feature is extracted from the local feature, which is a pre-defined feature during training, such as stress, etc. Finally, the speech feature vector is output by the fully connected layer.

[0067] In the embodiment, the text semantic information of the text sequence is extracted by the text encoder, which provides a basis for subsequent generation of composite style speech and improves the naturalness and fluency of the subsequent composite style speech.

[0068] In one embodiment, each style expert network of the style expert module performs style conversion on the text semantic information to obtain multiple style speech features, including:

[0069] The text semantic information is input into the style expert module, and the prosody feature vector of the text semantic information is extracted based on each style expert network;

[0070] The prosody feature vector is adjusted according to the style feature of the style expert network to obtain a target style vector;

[0071] The target style vector is fused with the semantic feature vector to obtain a style speech feature output by each style expert network;

[0072] All the style speech features are collected to obtain multiple style speech features.

[0073] In this embodiment, the prosodic feature vector in the text semantic information is obtained, the prosodic feature vector is adjusted according to the target style vector (such as speech speed, intonation, stress, etc.) captured in each style expert network of the style expert module, to obtain the target style vector, and then the target style vector is fused with the semantic feature vector to obtain the style speech feature corresponding to each style expert network. For example, the text semantic information is input into the "gentle" style expert network, and the generated style speech feature has a soft and gentle intonation. If the text semantic information is input into the "angry" style expert network, the generated style speech feature has a sharp and powerful intonation.

[0074] In this embodiment, by inputting the text semantic information into the style expert module, multiple style speech features are obtained, which provides a basis for subsequent generation of composite style speech; by adjusting the text semantic information through each independent style sub-network, the emotional features under different styles can be more accurately captured and expressed, thereby enhancing the authenticity and appeal of speech expression and improving the fluency of the composite style speech.

[0075] In one embodiment, the target style condition is obtained, comprising:

[0076] determining whether there is a user-uploaded style label;

[0077] if there is, verifying the style label with a predefined style label set, and when the verification is passed, taking the style label as the target style condition;

[0078] if there is not, analyzing the semantic feature vector using a style encoder, identifying the style label corresponding to the semantic feature vector, and taking the style label as the target style condition.

[0079] In this embodiment, in the generation process of the composite style speech, it is determined whether the user has uploaded a style label. If the user needs to generate a composite style speech with a specified style label, the user-uploaded style label is obtained, it is determined whether the style label is in the predefined style label set, if the style label is in the predefined style label set, the style label is taken as the target style condition; if there is not, the predefined style label set is fed back to the user in the form of a table to prompt the user to upload the correct style label according to the table, and the style label is taken as the target style condition. The style label includes at least one. When the style label is one, the speech of a single style is generated. When the style label is multiple, the composite style speech is generated.

[0080] If not, the semantic feature vector is analyzed by a style encoder to identify the style label corresponding to the semantic feature vector, and the style label is taken as the target style condition; the above-mentioned style encoder refers to a convolutional neural network with a classifier (such as a softmax classifier), extracts a style representation vector of the semantic feature vector through a feature extraction layer (such as a convolutional layer and a pooling layer) of the neural network, maps the style representation vector to a pre-defined style label set, outputs at least one style label that is most likely, and takes at least one style label as the target style condition.

[0081] In the embodiment, by allowing the user to upload self-style labels, the user can more directly specify the speech style they want to generate, improving the flexibility and user satisfaction of the system; by analyzing the semantic feature vector through the style encoder to obtain the corresponding style label, the accuracy of extracting the style label is improved.

[0082] In one embodiment, the processing of the text semantic information and the target style condition through the gating network to obtain a style weight comprises:

[0083] extracting a style label vector of each style label in the target style condition;

[0084] calculating a distance value of a semantic feature vector in the semantic information and the style label vector;

[0085] calculating the style weight according to the distance value.

[0086] In the embodiment, the style label is converted into a vector representation through vector conversion technology (such as Word2Vec, GloVe or BERT, etc.), to obtain a style label vector, wherein the dimension of the style label vector is the same as or compatible with the dimension of the semantic feature vector; a distance value of a semantic feature vector in the semantic information and the style label vector is calculated through a pre-set distance calculation method, such as Euclidean distance, cosine similarity, Manhattan distance, etc., and the style weight corresponding to each style label is calculated through the distance value, such as taking a number to make the style label with smaller distance obtain a larger weight, or taking a positive number to make the style label with higher similarity obtain a larger weight.

[0087] In the embodiment, the distance value is calculated according to the pre-set distance value calculation method, and the style weight is calculated according to the distance value, improving the accuracy of weight allocation and the accuracy of the subsequently generated composite style speech.

[0088] In one embodiment, the conversion of the composite style feature through the speech synthesizer to obtain the composite style speech of the text sequence comprises:

[0089] extracting a feature map of the composite style voice by a generator of the voice synthesizer;

[0090] iterating the feature map through multiple convolution layers of the generator to obtain a final feature map;

[0091] converting the final feature map into the composite style voice of the text sequence.

[0092] In this embodiment, the composite style feature is input into the voice synthesizer, and the generator of the voice synthesizer uses a multi-scale convolutional neural network structure to generate a feature map of the composite style voice by processing the input data layer by layer.

[0093] Specifically, in the first layer of the generator, the composite style feature is converted into a feature map, and the generator gradually extracts local information in the feature map by layer-by-layer convolution processing and generates a new feature map. Each convolution layer calculates a new feature map according to the feature map output by the previous layer and the parameters of the convolution kernel. In the subsequent layers of the generator, in order to generate a high-resolution voice waveform, an up-sampling operation needs to be performed on the latest feature map. The up-sampling process usually uses a transpose convolution layer (also known as an inverse convolution layer) to increase the resolution of the feature map. In the up-sampling process, the size of the feature map is expanded, and the number of channels may be adjusted. After layer-by-layer convolution processing and up-sampling processing, finally, the voice synthesizer converts the final feature map into the composite style voice of the text sequence.

[0094] In this embodiment, the voice synthesizer using a multi-scale convolutional neural network structure can efficiently extract and convert composite style features to generate high-resolution, high-quality, stable and robust composite style voices, improving the fluency and accuracy of the composite style voice.

[0095] In one embodiment, before the feature extraction of the text sequence by the text encoder, the method further comprises:

[0096] obtaining a first training set, training each of the style specialist networks of the style specialist module using the first training set to obtain a plurality of trained style specialist networks;

[0097] constructing an initial voice synthesis model, the initial voice synthesis model comprising an initial text encoder, an initial style specialist module and an initial voice synthesizer, wherein the initial specialist network comprises a plurality of trained style specialist networks and an initial gating network;

[0098] obtaining a second training set, training the initial voice synthesis model using the second training set to obtain the voice synthesis model.

[0099] In the embodiment, the first training set refers to a single-style speech training set and a corresponding style label; and the second training set refers to a text sequence training set and a corresponding composite style speech training set.

[0100] For each style expert network in the style expert module, a training sample with a specific style label in the single-style speech training set is used for training, each style expert network is responsible for learning and extracting a feature vector related to the corresponding style thereof, a style reconstruction loss function is designed for evaluating whether the extracted feature vector can accurately reconstruct the style of the original speech, the style reconstruction loss function includes a mean square error (MSE), a feature matching loss (such as a perceptual loss), an adversarial loss (such as a discriminator loss in a GAN), etc.; the parameters of the style expert network are continuously optimized by calculating the style reconstruction loss, and the parameters of the style expert network are updated using an optimization algorithm (such as Adam, SGD, etc.) to minimize the style reconstruction loss, according to the gradient information of the loss function, the weight and bias parameters of the network are adjusted, the above steps are repeated, and the style expert network is iteratively trained multiple times until the style reconstruction loss converges to a stable value, and multiple trained style expert networks are obtained.

[0101] An initial speech synthesis model is constructed, and the initial speech synthesis model includes an initial text encoder, an initial style expert module, and an initial speech synthesizer, wherein the initial expert network includes multiple trained style expert networks and an initial gating network.

[0102] Specifically, the text sequence training data and the style labels in the composite style speech training set are input into the initial speech synthesis model, the text sequence is taken as the input text of the model, and the style label is taken as the target style condition, the input text sequence is converted into text semantic information by the initial text encoder, the outputs of the style expert networks are selected and weightedly fused according to the semantic feature vector and the target style condition by the gating network in the style expert module, the composite speech is output by the speech synthesizer, and the parameters of the initial speech synthesis model are adjusted by a preset loss function and an optimization method until the loss converges to a stable value, and a trained speech synthesis model is obtained.

[0103] In the embodiment, by training each style expert network in the style expert module separately, each network can focus on learning and extracting a feature vector related to the corresponding style thereof. This modular training method can reduce the complexity of model training and improve the training efficiency; based on the multiple trained style expert networks, an initial speech synthesis model is constructed, and the speech synthesis model is trained, which can reduce the influence of the style expert network and improve the overall performance of the speech synthesis model.

[0104] It needs to be emphasized that, in order to further ensure the privacy and security of the generation result of the composite style voice and the like, the generation result of the composite style voice and the like can also be stored in a node of a block chain.

[0105] The block chain referred to in the present application is a new application mode of distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm and other computer technologies. The block chain is essentially a decentralized database, which is a series of data blocks associated using cryptographic methods, each data block containing information of a batch of network transactions, for verifying the validity of the information (anti-fake) and generating the next block. The block chain can include a block chain underlying platform, a platform product service layer, and an application service layer, etc.

[0106] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the use of digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Theory, method, technology and application system.

[0107] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The software technology of artificial intelligence mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc. Several major directions.

[0108] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by computer readable instructions instructing related hardware, and the computer readable instructions can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of each method. Among them, the storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) and other non-volatile storage media, or a random access memory (RAM) and the like.

[0109] It should be understood that although each step in the flowchart of the accompanying drawings is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other sequences. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence is not necessarily sequential, but can be alternately executed with at least part of other steps or sub-steps or stages of other steps.

[0110] Further referring to Figure 3 , as an implementation of the method shown in the above Figure 2 , the present application provides an embodiment of a composite style voice generation device, which corresponds to the method embodiment shown in Figure 2 , and the device can be specifically applied to various computer devices.

[0111] As shown in Figure 3 , the composite style voice generation device 300 described in the embodiment includes an encoding module 301, a conversion module 302, a weighting module 303, and a voice generation module 304.

[0112] Among them:

[0113] The encoding module 301 is configured to obtain a text sequence, extract features of the text sequence by using a text encoder, and obtain text semantic information, wherein the text semantic information includes a semantic feature vector and a prosody feature vector.

[0114] In one embodiment, the encoding module 301 includes:

[0115] The conversion sub-module is configured to convert the text sequence into a vector representation by using a vector conversion technology to obtain a text sequence vector.

[0116] The first feature extraction sub-module is configured to extract features of the text sequence vector by using an encoder of the text encoder to obtain the semantic feature vector.

[0117] The second feature extraction sub-module is configured to perform feature extraction operations on the semantic feature vector by using a convolution layer of the prosody prediction module to obtain the prosody feature vector.

[0118] In one embodiment, the device further includes:

[0119] The first training module is configured to obtain a first training set, train each of the style expert networks of the style expert module by using the first training set, and obtain a plurality of trained style expert networks.

[0120] The construction module is configured to construct an initial speech synthesis model, where the initial speech synthesis model comprises an initial text encoder, an initial style expert module, and an initial speech synthesizer, and the initial style expert module comprises the plurality of trained style expert networks and an initial gating network.

[0121] The second training module is configured to obtain a second training set, train the initial speech synthesis model by using the second training set, and obtain the speech synthesis model.

[0122] The conversion module 302 is configured to perform style conversion on the text semantic information by each of the style expert networks of the style expert module, and obtain a plurality of style speech features.

[0123] In an embodiment, the conversion module 302 comprises:

[0124] The input submodule is configured to input the text semantic information into the style expert module, and extract the prosody feature vector of the text semantic information based on each of the style expert networks.

[0125] The adjustment submodule is configured to adjust the prosody feature vector according to the style feature of the style expert network, and obtain a target style vector.

[0126] The fusion submodule is configured to fuse the target style vector and the semantic feature vector, and obtain a style speech feature output by each of the style expert networks.

[0127] The collection submodule is configured to collect all the style speech features, and obtain a plurality of style speech features.

[0128] The weighting module 303 is configured to obtain a target style condition, process the text semantic information and the target style condition by a gating network, obtain a style weight, perform a weighted summation operation on the plurality of style speech features according to the style weight, and obtain a composite style feature.

[0129] In an embodiment, the weighting module 303 comprises:

[0130] The judgment submodule is configured to determine whether a style label uploaded by a user exists.

[0131] The verification submodule is configured to, if the style label exists, verify the style label with a predefined style label set, and use the style label as the target style condition when the verification is passed.

[0132] an analysis submodule configured to, if the semantic feature vector does not exist, analyze the semantic feature vector by using a style encoder, identify the style label corresponding to the semantic feature vector, and take the style label as the target style condition.

[0133] In an embodiment, the weighting module 303 comprises:

[0134] a lifting submodule configured to extract a style label vector of each style label in the target style condition;

[0135] a first calculation submodule configured to calculate a distance value of a semantic feature vector in the semantic information and the style label vector;

[0136] a second calculation submodule configured to calculate the style weight according to the distance value.

[0137] a speech generation module 304 configured to convert the composite style feature by using a speech synthesizer to obtain a composite style speech of the text sequence.

[0138] In an embodiment, the speech generation module 304 comprises:

[0139] a second extraction submodule configured to extract a feature map of the composite style speech by using a generator of the speech synthesizer;

[0140] an iteration submodule configured to iterate the feature map by using a plurality of convolutional layers of the generator to obtain a final feature map;

[0141] a second conversion submodule configured to convert the final feature map into the composite style speech of the text sequence.

[0142] In this embodiment, the text sequence is processed by using the text encoder to predict a prosody feature vector of the text sequence, and the text encoder can also capture context information and semantic relationships in the text sequence, thereby generating a high-quality semantic feature vector. Combining the semantic feature vector and the prosody feature vector is helpful for the subsequent model to generate natural and fluent composite style speech.

[0143] The text semantic information is converted by using a plurality of style expert networks of the style expert module to generate a plurality of style speech features, thereby providing a basis for subsequent generation of composite style speech. The text semantic information of the text sequence is converted in style, thereby improving the fluency of the subsequent composite style speech.

[0144] The text semantic information and the target style condition are processed through a gating network to obtain a style weight, the multiple style speech features are weighted and summed according to the style weight to obtain a composite style feature, manual specification of a style parameter is avoided, and the generation efficiency of subsequent composite speech is improved; and dynamic adaptation to diversified semantic requirements of the text sequence is achieved, and the fluency of the subsequently generated composite style speech is improved.

[0145] The composite style feature is converted through a speech synthesizer to obtain the composite style speech of the text sequence, the fluency of the composite style speech is improved; and the composite style feature is processed through the speech synthesizer, the naturalness and the clarity of the composite speech are improved.

[0146] To solve the above technical problems, the embodiments of the present application further provide a device (computer device). For details, please refer to Figure 4 , Figure 4 The basic structure block diagram of the computer device of the present embodiment is shown in the figure.

[0147] The computer device 4 includes a memory 41, a processor 42 and a network interface 43 which are connected to each other through a system bus. It should be pointed out that only the computer device 4 with the memory 41, the processor 42 and the network interface 43 is shown in the figure, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented. Among them, the computer device herein is a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0148] The computer device can be a desktop computer, a notebook computer, a palm computer and a cloud server, etc. The computer device can interact with the user through a keyboard, a mouse, a remote controller, a touchpad or a sound control device, etc.

[0149] The memory 41 includes at least one type of readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or a memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit and the external storage device of the computer device 4. In this embodiment, the memory 41 is generally used to store an operating system and various application software installed on the computer device 4, such as computer readable instructions of the composite style voice generation method, etc. In addition, the memory 41 can also be used to temporarily store various data that have been output or will be output.

[0150] The processor 42 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer readable instructions or process data stored in the memory 41, such as computer readable instructions of the composite style voice generation method.

[0151] The network interface 43 can include a wireless network interface or a wired network interface, and the network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0152] In the implementation of the electronic device, the text sequence is processed by the text encoder to predict the prosodic feature vector of the text sequence, and the context information and semantic relationship in the text sequence can also be captured by the text encoder to generate a high-quality semantic feature vector. Combining the semantic feature vector and the prosodic feature vector is helpful for the subsequent model to generate natural and fluent composite style voice.

[0153] The text semantic information is respectively converted in style by using multiple style expert networks of the style expert module to generate multiple style speech features, thereby providing a basis for subsequent generation of the composite style speech.

[0154] The text semantic information and the target style condition are processed by the gating network to obtain a style weight, and the multiple style speech features are weighted and summed according to the style weight to obtain a composite style feature, thereby avoiding manual specification of a style parameter, improving the generation efficiency of the subsequent composite speech, and dynamically adapting to diversified semantic requirements of the text sequence to improve the fluency of the subsequently generated composite style speech.

[0155] The composite style feature is converted by the speech synthesizer to obtain the composite style speech of the text sequence, thereby improving the fluency of the composite style speech, and the composite style feature is processed by the speech synthesizer to improve the naturalness and clarity of the composite speech.

[0156] The application also provides another implementation, that is, to provide a storage medium (computer readable storage medium) which stores computer readable instructions, the computer readable instructions can be executed by at least one processor to make the at least one processor execute the steps of the composite style speech generation method as described above.

[0157] In the implementation of the computer readable storage medium of the application, the text sequence is processed by the text encoder to predict the prosody feature vector of the text sequence, and the context information and semantic relationship in the text sequence can also be captured by the text encoder, thereby generating a high-quality semantic feature vector; in combination with the semantic feature vector and the prosody feature vector, it is helpful for the subsequent model to generate natural and fluent composite style speech.

[0158] The text semantic information is respectively converted in style by using multiple style expert networks of the style expert module to generate multiple style speech features, thereby providing a basis for subsequent generation of the composite style speech.

[0159] The text semantic information and the target style condition are processed by the gating network to obtain a style weight, and the multiple style speech features are weighted and summed according to the style weight to obtain a composite style feature, thereby avoiding manual specification of a style parameter, improving the generation efficiency of the subsequent composite speech, and dynamically adapting to diversified semantic requirements of the text sequence to improve the fluency of the subsequently generated composite style speech.

[0160] The composite style feature is converted by the voice synthesizer to obtain a composite style voice of the text sequence, and the fluency of the composite style voice is improved.

[0161] The non-company software tools or components appearing in the embodiments of the present application are only illustrative and do not represent actual use.

[0162] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and a general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, an optical disk), and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the methods described in the various embodiments of the present application.

[0163] Obviously, the above-described embodiments are only some of the embodiments of the present application, not all the embodiments, and the preferred embodiments of the present application are given in the drawings, but do not limit the patent scope of the present application. The present application can be implemented in many different forms, and conversely, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing specific embodiments, or equivalently replace some technical features. Any equivalent structure made by using the contents of the specification and drawings, directly or indirectly applied to other related technical fields, is also within the scope of the patent protection of the present application.

Claims

1. A method for generating a composite style speech, the method comprising: The method comprises the following steps: obtaining a text sequence, extracting features of the text sequence by using a text encoder to obtain text semantic information, wherein the text semantic information comprises a semantic feature vector and a prosody feature vector; performing style conversion on the text semantic information by each style expert network of a style expert module to obtain multiple style speech features; obtaining a target style condition, processing the text semantic information and the target style condition by a gating network to obtain a style weight, performing weighted sum operation on the multiple style speech features according to the style weight to obtain a composite style feature; performing conversion on the composite style feature by a speech synthesizer to obtain a composite style speech of the text sequence; the step of performing style conversion on the text semantic information by each style expert network of the style expert module to obtain multiple style speech features comprises: inputting the text semantic information into the style expert module, extracting the prosody feature vector of the text semantic information based on each style expert network; adjusting the prosody feature vector according to the style feature of the style expert network to obtain a target style vector; fusing the target style vector and the semantic feature vector to obtain a style speech feature output by each style expert network; collecting all the style speech features to obtain multiple style speech features; wherein the content of the multiple style speech features is the same, and the styles of the multiple style speech features are different.

2. The method of claim 1, wherein the composite style voice is generated by using a voice of a user as a base voice and changing a style of the base voice. the step of extracting features of the text sequence by using a text encoder to obtain text semantic information, wherein the text semantic information comprises a semantic feature vector and a prosody feature vector, comprises: converting the text sequence into a vector representation by a vector conversion technology to obtain a text sequence vector; extracting features of the text sequence vector by an encoder of the text encoder to obtain the semantic feature vector; performing feature extraction operation on the semantic feature vector by a convolution layer of a prosody prediction module to obtain the prosody feature vector.

3. The method of claim 1, wherein the composite style voice is generated by using a voice of a user as a base voice and changing a style of the base voice. the step of obtaining a target style condition comprises: determining whether there is a style label uploaded by a user; if there is, verifying the style label with a predefined set of style labels, and when the verification is passed, taking the style label as the target style condition; if there is not, analyzing the semantic feature vector by using a style encoder to identify the style label corresponding to the semantic feature vector, and taking the style label as the target style condition.

4. The method of claim 1, wherein the composite style voice is generated by using a voice of a user as a base voice and changing a style of the base voice. the step of processing the text semantic information and the target style condition by a gating network to obtain a style weight comprises: extracting a style label vector of each style label in the target style condition; calculating a distance value of the semantic feature vector in the semantic information and the style label vector; calculating the style weight according to the distance value.

5. The method of claim 1, wherein the composite style voice is generated by using a voice of a user as a base voice and changing a style of the base voice. the step of performing conversion on the composite style feature by a speech synthesizer to obtain a composite style speech of the text sequence comprises: extracting a feature map of the composite style speech by using a generator of the speech synthesizer; The feature map is iterated through a plurality of convolutional layers of the generator to obtain a final feature map; The final feature map is converted into a composite style voice of the text sequence.

6. The method of claim 5, wherein the composite style voice is generated by using a voice of a user as a base voice and changing a style of the base voice. Before the text sequence is subjected to feature extraction by the text encoder, the method further comprises: A first training set is obtained, and each style expert network of the style expert module is trained using the first training set to obtain a plurality of trained style expert networks; An initial speech synthesis model is constructed, the initial speech synthesis model comprising an initial text encoder, an initial style expert module, and an initial speech synthesizer, wherein the initial style expert module comprises a plurality of trained style expert networks and an initial gating network; A second training set is obtained, and the initial speech synthesis model is trained using the second training set to obtain the speech synthesis model.

7. An apparatus for generating a composite style voice, the apparatus comprising: The device implements the method for generating a composite style voice according to any one of claims 1 to 6, and the device comprises: An encoding module configured to obtain a text sequence, and perform feature extraction on the text sequence using a text encoder to obtain text semantic information, wherein the text semantic information comprises a semantic feature vector and a prosody feature vector; A conversion module configured to perform style conversion on the text semantic information through each style expert network of a style expert module to obtain a plurality of style voice features; A weighting module configured to obtain a target style condition, process the text semantic information and the target style condition through a gating network to obtain a style weight, and perform weighted summation operation on the plurality of style voice features according to the style weight to obtain a composite style feature; A speech generation module configured to convert the composite style feature through a speech synthesizer to obtain a composite style voice of the text sequence.

8. A computer device, comprising: The computer device comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method for generating a composite style voice according to any one of claims 1 to 6.

9. A computer readable storage medium storing a computer program, characterized in that, The computer program, when executed by a processor, implements the method for generating a composite style voice according to any one of claims 1 to 6.

Images