Composite style voice generation method and device, equipment and storage medium

Abstract

The invention belongs to the artificial intelligence field and the medical health field, and relates to a composite style voice generation method comprising the following steps: obtaining a text sequence, using a text encoder to carry out feature extraction on the text sequence to obtain text semantic information, the text semantic information comprising a semantic feature vector and a rhythm feature vector; performing style conversion on the text semantic information through each style expert network of a style expert module to obtain a plurality of style voice features; obtaining a target style condition, processing the text semantic information and the target style condition through a gating network to obtain a style weight, and carrying out weighted summation operation on the plurality of style voice features according to the style weight to obtain a composite style feature; and converting the composite style features through a voice synthesizer to obtain a composite style voice of the text sequence. In addition, the invention also relates to a block chain technology, and the target page and the like can be stored in a block chain. The fluency of the composite style voice can be improved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and medical health, and in particular to a method for generating composite-style speech. Background Art

[0002] In the field of speech synthesis technology, with the rapid development of artificial intelligence and machine learning technology, multi-style speech synthesis has become one of the current research hotspots. Multi-style speech synthesis only generates compound style speech with multiple style features based on the input text to meet the diverse needs of different application scenarios. For example, in the medical field, multi-style speech synthesis technology can greatly improve the quality and efficiency of medical services, especially in the communication between doctors and patients. For example, when a doctor explains the treatment plan to a child patient through a voice assistant, a gentle and friendly voice style can be integrated to reduce the child's fear and anxiety, making it easier for the child to accept the treatment plan.

[0003] Traditional multi-style speech synthesis methods mainly rely on manual specification or simple linear interpolation to achieve style fusion.

[0004] However, manually specifying style parameters is not only time-consuming and laborious, but also difficult to accurately capture subtle semantic changes in the input text, resulting in a disconnect between the style of the synthesized speech and the text content. Although simple linear interpolation can achieve a preliminary fusion of styles, its static processing method cannot dynamically adapt to the diverse semantic requirements of the input text, making the synthesis result lack naturalness and expressiveness.

[0005] It can be seen that the existing multi-style speech synthesis methods cannot automatically and dynamically integrate multiple style features according to the semantic content of the input text and generate natural and fluent composite style speech. Summary of the invention

[0006] The purpose of the embodiments of the present application is to provide a method, device, equipment and storage medium for generating compound-style speech, the main purpose of which is to improve the accuracy and fluency of compound-style speech.

[0007] In order to solve the above technical problems, in the first aspect, the embodiment of the present application provides a method for generating a composite style speech, which adopts the following technical solution:

[0008] Acquire a text sequence, and use a text encoder to perform feature extraction on the text sequence to obtain text semantic information, wherein the text semantic information includes a semantic feature vector and a prosodic feature vector;

[0009] Performing style conversion on the text semantic information through each style expert network of the style expert module to obtain multiple style speech features;

[0010] Obtaining a target style condition, processing the text semantic information and the target style condition through a gating network to obtain a style weight, and performing a weighted sum operation on a plurality of the style speech features according to the style weight to obtain a composite style feature;

[0011] The composite style feature is converted by a speech synthesizer to obtain the composite style speech of the text sequence.

[0012] In order to solve the above technical problems, in a second aspect, the embodiment of the present application further provides a device for generating a composite style speech, which adopts the following technical solution:

[0013] An encoding module, used for acquiring a text sequence, performing feature extraction on the text sequence using a text encoder, and obtaining text semantic information, wherein the text semantic information includes a semantic feature vector and a prosodic feature vector;

[0014] A conversion module, configured to perform style conversion on the text semantic information through each style expert network of the style expert module to obtain a plurality of style speech features;

[0015] A weighting module, used to obtain a target style condition, process the text semantic information and the target style condition through a gating network to obtain a style weight, and perform a weighted sum operation on a plurality of the style speech features according to the style weight to obtain a composite style feature;

[0016] The speech generation module is used to convert the composite style features through a speech synthesizer to obtain the composite style speech of the text sequence.

[0017] In order to solve the above technical problems, in a third aspect, an embodiment of the present application also provides a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for generating a composite style speech as described above.

[0018] In order to solve the above technical problems, in a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method for generating a composite style speech as described above.

[0019] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0020] The text sequence is processed by the text encoder to predict the prosodic feature vector of the text sequence. The text encoder can also capture the contextual information and semantic relationships in the text sequence to generate a high-quality semantic feature vector. Combining the semantic feature vector and the prosodic feature vector helps the subsequent model generate natural and fluent composite style speech.

[0021] By using multiple style expert networks of the style expert module to perform style conversion on text semantic information respectively, multiple style speech features are generated, which provides a basis for the subsequent generation of composite style speech; style conversion is performed based on the text semantic information of the text sequence, which improves the fluency of the subsequent composite style speech.

[0022] The text semantic information and the target style condition are processed through a gated network to obtain a style weight, and a weighted sum is performed on multiple style speech features according to the style weight to obtain a composite style feature, thereby avoiding manual specification of style parameters and improving the efficiency of subsequent composite speech generation; and dynamically adapting to the diverse semantic needs of text sequences is achieved, thereby improving the fluency of the composite style speech generated subsequently.

[0023] The composite style features are converted by a speech synthesizer to obtain the composite style speech of the text sequence, thereby improving the fluency of the composite style speech; and the composite style features are processed by a speech synthesizer to improve the naturalness and clarity of the composite speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0026] Figure 2 A flowchart of an embodiment of a method for generating a composite style speech according to the present application;

[0027] Figure 3 is a schematic structural diagram of an embodiment of a device for generating composite style speech according to the present application;

[0028] Figure 4 It is a structural schematic diagram of an embodiment of a device according to the present application. DETAILED DESCRIPTION

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0030] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0031] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0032] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0033] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0034] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0035] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0036] It should be noted that the method for generating the composite-style speech provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the apparatus for generating the composite-style speech is generally arranged in the server / terminal device.

[0037] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0038] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for generating a composite style speech according to the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The method for generating a composite style speech provided in the embodiment of the present application can be applied to any scenario where the generation of a composite style speech is required, and the method for generating a composite style speech can be applied to products in these scenarios. The method for generating a composite style speech comprises the following steps:

[0039] Step S201: Acquire a text sequence, and use a text encoder to perform feature extraction on the text sequence to obtain text semantic information, wherein the text semantic information includes a semantic feature vector and a prosodic feature vector.

[0040] In this embodiment, the method for generating a composite style speech is executed on an electronic device (eg Figure 1The server / terminal device shown in the figure) can obtain the text sequence through a wired connection or a wireless connection. It should be noted that the above-mentioned wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or developed in the future. The executor of the present application may be a speech synthesis model, or simply referred to as a model. The above-mentioned text sequence usually has the same format as the text sequence training data used in the training of the speech synthesis model, but the training data contains other noise, depending on the specific situation. Among them, in the business scenario of generating a composite style speech of a text sequence, the text sequence may include a text sequence automatically generated by artificial intelligence (AI) or a manually written text sequence.

[0041] In this embodiment, the speech synthesis model is composed of a text encoder, a style expert module (StyleExperts), a gating network (Gating Network) and a speech synthesizer, wherein the style expert module includes multiple independent style expert networks, each of which is responsible for generating a style feature; the text sequence is a text sequence in which words or characters in a sentence are arranged in a certain order, such as a paragraph of text (such as good night, have a sweet dream); the text encoder can adopt a sequence-to-sequence model based on a self-attention mechanism, such as a Transformer model. Since it is also necessary to extract the rhythm feature vector of the text sequence, a rhythm prediction module is introduced into the encoder of the model to achieve it. The rhythm prediction module can adopt a neural network, such as a convolutional neural network.

[0042] A text sequence of speech to be synthesized is obtained, and the text sequence is processed through a text encoder to extract text semantic information of the text sequence, wherein the text semantic information includes a semantic feature vector and a prosodic feature vector. Specifically, the text sequence is converted into a vector representation through a vector conversion technology, such as word embedding technology, and the semantic feature vector of the text sequence is extracted through the multi-layer self-attention mechanism and feedforward neural network of the text encoder. In order to ensure that the semantic information of the text sequence is fully captured, the dimension of the semantic feature vector is relatively high, such as 512 dimensions or 1024 dimensions. The above mentioned that a prosodic prediction module is introduced into the encoder of the model, and the semantic feature vector is processed through the prosodic prediction module to predict the prosodic feature vector of each character or syllable, including but not limited to stress, duration, rhythm, etc.

[0043] In the medical field, for example, a doctor writes down the condition description and treatment plan in words, such as "Your condition is mild pneumonia. You need to take antibiotics on time and rest carefully"; this paragraph of text is used as a text sequence for speech to be synthesized, and the text semantic information of the text sequence is extracted. Specifically, the text sequence is converted into a vector representation, and the rhythmic feature vector of each character in the paragraph of text is predicted.

[0044] In this embodiment, the text sequence is processed by a text encoder to predict the prosodic feature vector of the text sequence. The text encoder can also capture the contextual information and semantic relationship in the text sequence, thereby generating a high-quality semantic feature vector. Combining the semantic feature vector and the prosodic feature vector helps the subsequent model to generate natural and fluent composite style speech.

[0045] Step S202: Performing style conversion on the text semantic information through each style expert network of the style expert module to obtain a plurality of style speech features.

[0046] In this embodiment, it is described above that the style expert module includes multiple independent style expert networks, each of which is responsible for generating a style feature, such as gentleness, sadness, joy, anger, etc. The style expert network adopts a neural network architecture, such as a convolutional neural network (CNN) and a recurrent neural network (RNN).

[0047] In this embodiment, the text semantic information is respectively input into each style expert network of the style expert module, and the text semantic information is adjusted and converted through the style features of each style expert network to generate multiple style speech features, wherein the contents of the multiple style speech features are the same but the styles are different, such as the same text sequence content.

[0048] In the medical field, for example, the text semantic information extracted from the sentence "Your condition is mild pneumonia. You need to take antibiotics on time and pay attention to rest" is input into multiple style expert networks, and the style expert network generates corresponding style features for "Your condition is mild pneumonia. You need to take antibiotics on time and pay attention to rest", such as serious style features, gentle style features, etc.

[0049] In this embodiment, style conversion is performed on text semantic information by using multiple style expert networks of the style expert module to generate multiple style speech features, which provides a basis for the subsequent generation of composite style speech; style conversion is performed based on text semantic information of the text sequence, thereby improving the fluency of the subsequent composite style speech.

[0050] Step S203, obtaining a target style condition, processing the text semantic information and the target style condition through a gated network to obtain a style weight, performing a weighted sum operation on a plurality of the style speech features according to the style weight, and obtaining a composite style feature.

[0051] In this embodiment, the target style condition includes an explicit style label or an implicit style embedding, wherein the explicit style label refers to a style label predefined by the user. For example, if the user wants to generate a gentle and pleasant composite speech, the style labels of "gentle" and "pleasant" are input as the target style condition. The implicit style embedding is obtained by analyzing the text semantic information through a style encoder to obtain a corresponding style label. The gating network includes an input layer, a feature extraction layer, a weight generation layer and an output layer. The text semantic information and the target style condition are output to the gating network through the input layer of the gating network. The style features of the target style condition are extracted through the feature extraction layer. The weight corresponding to each style feature is calculated through the weight generation layer to obtain the style weight. In the output layer, a weighted sum operation is performed on multiple style speech features according to the style weight to obtain a composite style feature, and the composite style feature is output.

[0052] In the medical field, for example, doctors select target style features based on the characteristics of the patient (such as children, adults, etc.), calculate the weights of the selected target style features through a gated network, and then perform weighted summation based on the style weights corresponding to each target style feature to obtain a composite style feature. For example, children can choose the two target style features of "gentle" and "kind".

[0053] In this embodiment, the text semantic information and the target style condition are processed through a gated network to obtain a style weight, and a weighted sum is performed on multiple style speech features according to the style weight to obtain a composite style feature, thereby avoiding manual specification of style parameters and improving the efficiency of subsequent composite speech generation; and achieving dynamic adaptation to the diverse semantic requirements of text sequences and improving the fluency of the subsequently generated composite style speech.

[0054] Step S204: convert the composite style features through a speech synthesizer to obtain composite style speech of the text sequence.

[0055] In this embodiment, the above-mentioned speech synthesizer adopts a model based on a generative adversarial network (GAN), such as HiFi-GAN, to improve the naturalness and clarity of the composite style speech; wherein the above-mentioned speech synthesizer includes a generator and a discriminator, the generator converts the composite style features into a high-quality composite speech waveform, and the discriminator is used to determine whether the generated composite style speech is similar to the real speech.

[0056] In this embodiment, the composite style features are converted by a speech synthesizer to obtain the composite style speech of the text sequence.

[0057] Specifically, the generator of the speech synthesizer adopts a multi-scale convolutional neural network structure, which gradually converts low-dimensional composite style features into high-resolution speech waveforms through step-by-step upsampling and skip connections. In each upsampling step, the generator uses a transposed convolution layer to increase the resolution of the feature map, while retaining the information of the previous layer through residual connections. This multi-scale structure can effectively capture local and global features in speech signals and generate speech waveforms of composite style speech; the discriminator adopts a multi-scale spectral normalized convolutional neural network (SNGAN), which improves the stability and robustness of the discriminator by discriminating feature maps of different scales; and finally outputs the high-resolution speech waveform as a playable composite style speech.

[0058] In the medical field, for example, the composite style features generated by the above-mentioned doctor according to the characteristics of the patient (such as children, adults, etc.) are converted into the text sequence "Your condition is mild pneumonia, you need to take antibiotics on time and pay attention to rest" to obtain a composite style voice with a "gentle" and "kind" style, and the composite style voice is used to broadcast "Your condition is mild pneumonia, you need to take antibiotics on time and pay attention to rest".

[0059] In this embodiment, the composite style features are converted by a speech synthesizer to obtain the composite style speech of the text sequence, thereby improving the fluency of the composite style speech; and the composite style features are processed by a speech synthesizer to improve the naturalness and clarity of the composite speech.

[0060] In one embodiment, the text sequence is subjected to feature extraction by a text encoder to obtain text semantic information, wherein the text semantic information includes a semantic feature vector and a prosodic feature vector, including:

[0061] Convert the text sequence into a vector representation by using a vector conversion technology to obtain a text sequence vector;

[0062] Performing feature extraction on the text sequence vector by the encoder of the text encoder to obtain the semantic feature vector;

[0063] The prosody feature vector is obtained by performing a feature extraction operation on the semantic feature vector through the convolution layer of the prosody prediction module.

[0064] In this embodiment, each character of the text sequence is converted into a vector representation through vector conversion technology (such as word embedding, etc.), and all character vectors are collected to obtain a text sequence vector. Since the Transformer model cannot capture the position information in the text sequence, it is necessary to add additional position codes to represent its position information. The position codes can be fixed (such as those generated by sine and cosine functions). These position codes are added to the text sequence vector to provide the position information of the characters in the sequence. The text sequence after adding the position codes is processed through the self-attention mechanism, and the attention weight is calculated. The context information of the text sequence is obtained through the attention weight.

[0065] After the self-attention mechanism, the representation of each word or character is further processed by a feedforward neural network, which usually contains two linear transformations and an activation function (such as ReLU). After being processed by the self-attention mechanism and the feedforward neural network, a semantic feature vector is finally obtained.

[0066] In the above step S201, it is mentioned that the text encoder introduces a prosody prediction module; the semantic feature vector is input into the prosody prediction module, and the semantic feature vector is convolved through the multi-layer convolutional layer of the prosody prediction module to extract the local features of the semantic feature vector, and then the dimension of the local features is reduced through the pooling layer, and key features are extracted from the local features. The key features are predefined features during training, such as stress, etc. Finally, the speech feature vector is output through the fully connected layer.

[0067] In this embodiment, the text semantic information of the text sequence is extracted through the text encoder, which provides a basis for the subsequent generation of compound style speech and improves the naturalness and fluency of the subsequent compound style speech.

[0068] In one embodiment, each style expert network of the style expert module performs style conversion on the text semantic information to obtain multiple style speech features, including:

[0069] Inputting the text semantic information into the style expert module, and extracting the prosodic feature vector of the text semantic information based on each of the style expert networks;

[0070] According to the style features of the style expert network, the prosody feature vector is adjusted to obtain a target style vector;

[0071] The target style vector is merged with the semantic feature vector to obtain the style speech feature output by each style expert network;

[0072] All of the style speech features are collected to obtain a plurality of style speech features.

[0073] In this embodiment, a prosodic feature vector in the text semantic information is obtained, and based on each style expert network of the style expert module, the prosodic feature vector is adjusted according to the target style vector (such as speaking speed, intonation, stress, etc.) captured in the style expert network to obtain a target style vector, and then the target style vector is fused with the semantic feature vector to obtain the style speech feature corresponding to each style expert network. For example, if the text semantic information is input into the "gentle" style expert network, the generated style speech feature has a soft and gentle intonation; if the text semantic information is input into the "angry" style expert network, the generated style speech feature has a sharp and powerful intonation.

[0074] In this embodiment, by inputting text semantic information into the style expert module, multiple style speech features are obtained, which provide a basis for the subsequent generation of composite style speech; by adjusting the text semantic information through each independent style sub-network, the emotional characteristics under different styles can be more accurately captured and expressed, thereby enhancing the authenticity and appeal of the speech expression and improving the fluency of the composite style speech.

[0075] In one embodiment, the step of obtaining the target style condition includes:

[0076] Determine whether there is a style tag uploaded by the user;

[0077] If it exists, verify the style tag with a predefined style tag set, and when the verification passes, use the style tag as a target style condition;

[0078] If not, the semantic feature vector is analyzed by using a style encoder to identify the style label corresponding to the semantic feature vector, and the style label is used as the target style condition.

[0079] In this embodiment, during the generation process of the composite style speech, it is determined whether the user has uploaded the style tag. If the user needs to generate a composite style speech with a specified style tag, the style tag uploaded by the user is obtained, and it is determined whether the style tag is in a predefined style tag set. If the style tag is in the predefined style tag set, the style tag is used as the target style condition; if not, the predefined style tag set is fed back to the user in a table to prompt the user to upload the correct style tag according to the table, and the style tag is used as the target style condition, wherein the style tag includes at least one. When there is one style tag, a single-style speech is generated, and when there are multiple style tags, a composite style speech is generated.

[0080] If it does not exist, the semantic feature vector is analyzed by a style encoder to identify the style label corresponding to the semantic feature vector, and the style label is used as the target style condition; the above-mentioned style encoder refers to a convolutional neural network with a classifier (such as a softmax classifier), which extracts the style representation vector of the semantic feature vector through the feature extraction layer (such as a convolutional layer, a pooling layer) of the neural network, maps the style representation vector to a predefined style label set, outputs at least one most likely style label, and uses at least one style label as the target style condition.

[0081] In this embodiment, by allowing users to upload their own style labels, users can more directly specify the speech style they want to generate, thereby improving the flexibility of the system and user satisfaction; the style label is obtained by analyzing the semantic feature vector through the style encoder, thereby improving the accuracy of extracting the style label.

[0082] In one embodiment, the processing of the text semantic information and the target style condition by a gating network to obtain a style weight includes:

[0083] Extracting a style label vector for each style label in the target style condition;

[0084] Calculating a distance value between a semantic feature vector in the semantic information and the style label vector;

[0085] The style weight is calculated according to the distance value.

[0086] In this embodiment, the style label is converted into a vector representation through a vector conversion technology (such as Word2Vec, GloVe or BERT, etc.) to obtain a style label vector, wherein the dimension of the style label vector is the same as or compatible with the dimension of the semantic feature vector; the distance value between the semantic feature vector in the semantic information and the style label vector is calculated by a preset distance calculation method, such as Euclidean distance, cosine similarity, Manhattan distance, etc., and the style weight corresponding to each style label is calculated by the distance value, such as taking a number so that the style label with a smaller distance obtains a larger weight, or taking a positive number so that the style label with a higher similarity obtains a larger weight.

[0087] In this embodiment, the distance value is calculated according to a preset distance value calculation method, and the style weight is calculated according to the distance value, which improves the accuracy of weight allocation and the accuracy of the subsequently generated composite style speech.

[0088] In one embodiment, converting the composite style feature by a speech synthesizer to obtain the composite style speech of the text sequence includes:

[0089] Extracting a feature graph of the composite style speech using a generator of the speech synthesizer;

[0090] Iterating the feature map through the multi-layer convolutional layers of the generator to obtain a final feature map;

[0091] The final feature map is converted into a composite style speech of the text sequence.

[0092] In this embodiment, the composite style feature is input into a speech synthesizer, and a multi-scale convolutional neural network structure is used by a generator of the speech synthesizer to generate a feature map of the composite style speech by processing the input data layer by layer;

[0093] Specifically, in the first layer of the generator, the composite style features are converted into feature maps. The generator gradually extracts local information in the feature maps through layer-by-layer convolution processing and generates new feature maps. Each convolution layer calculates a new feature map based on the feature map output by the previous layer and the parameters of the convolution kernel. In the subsequent layers of the generator, in order to generate a high-resolution speech waveform, it is necessary to upsample the latest feature map. The upsampling process usually uses a transposed convolution layer (or deconvolution layer) to increase the resolution of the feature map. During the upsampling process, the size of the feature map will be expanded, and the number of channels may be adjusted. After layer-by-layer convolution processing and upsampling processing, the speech synthesizer finally converts the final feature map into the composite style speech of the text sequence.

[0094] In this embodiment, the speech synthesizer using a multi-scale convolutional neural network structure can efficiently extract and convert complex style features, generate high-resolution, high-quality, stable and robust complex style speech, and improve the fluency and accuracy of the complex style speech.

[0095] In one embodiment, before extracting features from the text sequence using a text encoder, the method further comprises:

[0096] Acquire a first training set, and use the first training set to train each of the style expert networks of the style expert module to obtain a plurality of trained style expert networks;

[0097] Constructing an initial speech synthesis model, wherein the initial speech synthesis model includes an initial text encoder, an initial style expert module and an initial speech synthesizer, wherein the initial expert network includes a plurality of trained style expert networks and an initial gating network;

[0098] A second training set is obtained, and the initial speech synthesis model is trained using the second training set to obtain the speech synthesis model.

[0099] In this embodiment, the first training set refers to a single-style speech training set and a corresponding style label; the second training set refers to a text sequence training set and a corresponding compound-style speech training set;

[0100] For each style expert network in the style expert module, training samples with specific style labels in a single-style speech training set are used for training. Each style expert network is responsible for learning and extracting feature vectors related to its corresponding style. A style reconstruction loss function is designed to evaluate whether the extracted feature vectors can accurately reconstruct the style of the original speech. The style reconstruction loss function includes mean square error (MSE), feature matching loss (such as perceptual loss), adversarial loss (such as discriminator loss in GAN), etc.; the parameters of the style expert network are continuously optimized by calculating the style reconstruction loss, and the parameters of the style expert network are updated using an optimization algorithm (such as Adam, SGD, etc.) to minimize the style reconstruction loss. According to the gradient information of the loss function, the weights and bias parameters of the network are adjusted. The above steps are repeated, and the style expert network is iteratively trained for multiple times until the style reconstruction loss converges to a stable value, thereby obtaining multiple trained style expert networks.

[0101] Constructing an initial speech synthesis model, wherein the initial speech synthesis model includes an initial text encoder, an initial style expert module and an initial speech synthesizer, wherein the initial expert network includes a plurality of trained style expert networks and an initial gating network;

[0102] Specifically, the text sequence training data and the style labels in the composite style speech training set are input into the initial speech synthesis model, and the text sequence is used as the input text of the model, and the style label is used as the target style condition. The input text sequence is converted into text semantic information through the initial text encoder, and the output of the style expert network is selected and weighted according to the semantic feature vector and the target style condition through the gating network in the style expert module. The composite speech is output through the speech synthesizer, and the parameters of the initial speech synthesis model are adjusted through the preset loss function and optimization method until the loss converges to a stable value, so as to obtain a trained speech synthesis model.

[0103] In this embodiment, by first training each style expert network in the style expert module separately, each network can be focused on learning and extracting feature vectors related to its corresponding style. This modular training method can reduce the complexity of model training and improve training efficiency; based on the trained multiple style expert networks, an initial speech synthesis model is constructed and trained, which can reduce the influence of the style expert network and improve the overall performance of the speech synthesis model.

[0104] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned compound style speech generation results, the above-mentioned compound style speech generation results, etc. can also be stored in a blockchain node.

[0105] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0106] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0107] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0108] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0109] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0110] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a device for generating a composite style speech, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the apparatus can be specifically applied to various computer devices.

[0111] like Figure 3 As shown, the composite style speech generation device 300 described in this embodiment includes: an encoding module 301 , a conversion module 302 , a weighting module 303 , and a speech generation module 304 .

[0112] in:

[0113] The encoding module 301 is used to obtain a text sequence, and use a text encoder to perform feature extraction on the text sequence to obtain text semantic information, wherein the text semantic information includes a semantic feature vector and a prosodic feature vector.

[0114] In one embodiment, the encoding module 301 includes:

[0115] A conversion submodule, used to convert the text sequence into a vector representation by using a vector conversion technology to obtain a text sequence vector;

[0116] A first feature extraction submodule, configured to extract features from the text sequence vector through an encoder of the text encoder to obtain the semantic feature vector;

[0117] The second feature extraction submodule is used to perform a feature extraction operation on the semantic feature vector through the convolution layer of the prosody prediction module to obtain the prosody feature vector.

[0118] In one embodiment, the apparatus further comprises:

[0119] A first training module, used for acquiring a first training set, and using the first training set to train each of the style expert networks of the style expert module to obtain a plurality of trained style expert networks;

[0120] A construction module, used to construct an initial speech synthesis model, wherein the initial speech synthesis model includes an initial text encoder, an initial style expert module and an initial speech synthesizer, wherein the initial expert network includes a plurality of trained style expert networks and an initial gating network;

[0121] The second training module is used to obtain a second training set, and use the second training set to train the initial speech synthesis model to obtain the speech synthesis model.

[0122] A conversion module 302, configured to perform style conversion on the text semantic information through each style expert network of the style expert module to obtain a plurality of style speech features;

[0123] In one embodiment, the conversion module 302 includes:

[0124] An input submodule, used for inputting the text semantic information into the style expert module, and extracting the prosodic feature vector of the text semantic information based on each of the style expert networks;

[0125] An adjustment submodule, used for adjusting the prosody feature vector according to the style features of the style expert network to obtain a target style vector;

[0126] A fusion submodule, used to fuse the target style vector with the semantic feature vector to obtain the style speech feature output by each style expert network;

[0127] The collection submodule is used to collect all the style speech features to obtain multiple style speech features.

[0128] A weighting module 303 is used to obtain a target style condition, process the text semantic information and the target style condition through a gating network to obtain a style weight, and perform a weighted sum operation on a plurality of the style speech features according to the style weight to obtain a composite style feature;

[0129] In one embodiment, the weighting module 303 includes:

[0130] The judgment submodule is used to judge whether there is a style tag uploaded by the user;

[0131] A verification submodule, for verifying the style tag with a predefined style tag set if it exists, and when the verification passes, using the style tag as a target style condition;

[0132] The analysis submodule is used for analyzing the semantic feature vector using a style encoder if it does not exist, identifying the style label corresponding to the semantic feature vector, and using the style label as the target style condition.

[0133] In one embodiment, the weighting module 303 includes:

[0134] A submodule is provided for extracting a style label vector of each style label in the target style condition;

[0135] A first calculation submodule, used for calculating the distance value between the semantic feature vector in the semantic information and the style label vector;

[0136] The second calculation submodule is used to calculate the style weight according to the distance value.

[0137] The speech generation module 304 is used to convert the composite style features through a speech synthesizer to obtain the composite style speech of the text sequence.

[0138] In one embodiment, the speech generation module 304 includes:

[0139] A second extraction submodule, configured to extract a feature graph of the composite style speech using a generator of the speech synthesizer;

[0140] An iterator module, used for iterating the feature map through the multi-layer convolutional layers of the generator to obtain a final feature map;

[0141] The second conversion submodule is used to convert the final feature map into the composite style speech of the text sequence.

[0142] In this embodiment, the text sequence is processed by a text encoder to predict the prosodic feature vector of the text sequence. The text encoder can also capture the contextual information and semantic relationship in the text sequence, thereby generating a high-quality semantic feature vector. Combining the semantic feature vector and the prosodic feature vector helps the subsequent model to generate natural and fluent composite style speech.

[0143] By using multiple style expert networks of the style expert module to perform style conversion on text semantic information respectively, multiple style speech features are generated, which provides a basis for the subsequent generation of composite style speech; style conversion is performed based on the text semantic information of the text sequence, which improves the fluency of the subsequent composite style speech.

[0144] The text semantic information and the target style condition are processed through a gated network to obtain a style weight, and a weighted sum is performed on multiple style speech features according to the style weight to obtain a composite style feature, thereby avoiding manual specification of style parameters and improving the efficiency of subsequent composite speech generation; and dynamically adapting to the diverse semantic needs of text sequences is achieved, thereby improving the fluency of the composite style speech generated subsequently.

[0145] The composite style features are converted by a speech synthesizer to obtain the composite style speech of the text sequence, thereby improving the fluency of the composite style speech; and the composite style features are processed by a speech synthesizer to improve the naturalness and clarity of the composite speech.

[0146] In order to solve the above technical problems, the present application embodiment also provides a device (computer device). Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0147] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 4 with a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0148] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0149] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the method for generating a composite style speech, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0150] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions or process data stored in the memory 41, such as computer-readable instructions for running the method for generating the composite style speech.

[0151] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0152] During the implementation of the electronic device of the present application, a text sequence is processed through a text encoder to predict the prosodic feature vector of the text sequence. The text encoder can also capture the contextual information and semantic relationship in the text sequence, thereby generating a high-quality semantic feature vector. Combining the semantic feature vector and the prosodic feature vector helps the subsequent model to generate natural and fluent composite style speech.

[0153] By using multiple style expert networks of the style expert module to perform style conversion on text semantic information respectively, multiple style speech features are generated, which provides a basis for the subsequent generation of composite style speech; style conversion is performed based on the text semantic information of the text sequence, which improves the fluency of the subsequent composite style speech.

[0154] The text semantic information and the target style condition are processed through a gated network to obtain a style weight, and a weighted sum is performed on multiple style speech features according to the style weight to obtain a composite style feature, thereby avoiding manual specification of style parameters and improving the efficiency of subsequent composite speech generation; and dynamically adapting to the diverse semantic needs of text sequences is achieved, thereby improving the fluency of the composite style speech generated subsequently.

[0155] The composite style features are converted by a speech synthesizer to obtain the composite style speech of the text sequence, thereby improving the fluency of the composite style speech; and the composite style features are processed by a speech synthesizer to improve the naturalness and clarity of the composite speech.

[0156] The present application also provides another implementation, namely, providing a storage medium (computer-readable storage medium), wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the method for generating composite style speech as described above.

[0157] During the implementation of the computer-readable storage medium of the present application, a text sequence is processed by a text encoder to predict the prosodic feature vector of the text sequence. The text encoder can also capture the contextual information and semantic relationship in the text sequence, thereby generating a high-quality semantic feature vector. Combining the semantic feature vector and the prosodic feature vector helps the subsequent model to generate natural and fluent composite style speech.

[0158] By using multiple style expert networks of the style expert module to perform style conversion on text semantic information respectively, multiple style speech features are generated, which provides a basis for the subsequent generation of composite style speech; style conversion is performed based on the text semantic information of the text sequence, which improves the fluency of the subsequent composite style speech.

[0159] The text semantic information and the target style condition are processed through a gated network to obtain a style weight, and a weighted sum is performed on multiple style speech features according to the style weight to obtain a composite style feature, thereby avoiding manual specification of style parameters and improving the efficiency of subsequent composite speech generation; and dynamically adapting to the diverse semantic needs of text sequences is achieved, thereby improving the fluency of the composite style speech generated subsequently.

[0160] The composite style features are converted by a speech synthesizer to obtain the composite style speech of the text sequence, thereby improving the fluency of the composite style speech; and the composite style features are processed by a speech synthesizer to improve the naturalness and clarity of the composite speech.

[0161] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

[0162] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0163] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for generating a composite style speech, characterized in that: The steps include: Acquire a text sequence, and use a text encoder to perform feature extraction on the text sequence to obtain text semantic information, wherein the text semantic information includes a semantic feature vector and a prosodic feature vector; Performing style conversion on the text semantic information through each style expert network of the style expert module to obtain multiple style speech features; Obtaining a target style condition, processing the text semantic information and the target style condition through a gating network to obtain a style weight, and performing a weighted sum operation on a plurality of the style speech features according to the style weight to obtain a composite style feature; The composite style feature is converted by a speech synthesizer to obtain the composite style speech of the text sequence.

2. The method for generating a composite style speech according to claim 1, wherein: The method of extracting features from the text sequence using a text encoder to obtain text semantic information, wherein the text semantic information includes a semantic feature vector and a prosodic feature vector, includes: Convert the text sequence into a vector representation by using a vector conversion technology to obtain a text sequence vector; Performing feature extraction on the text sequence vector by the encoder of the text encoder to obtain the semantic feature vector; The prosody feature vector is obtained by performing a feature extraction operation on the semantic feature vector through the convolution layer of the prosody prediction module.

3. The method for generating a composite style speech according to claim 1, wherein: Each style expert network of the style expert module performs style conversion on the text semantic information to obtain multiple style speech features, including: Inputting the text semantic information into the style expert module, and extracting the prosodic feature vector of the text semantic information based on each of the style expert networks; According to the style features of the style expert network, the prosody feature vector is adjusted to obtain a target style vector; The target style vector is merged with the semantic feature vector to obtain the style speech feature output by each style expert network; All of the style speech features are collected to obtain a plurality of style speech features.

4. The method for generating a composite style speech according to claim 1, wherein: The obtaining of the target style condition includes: Determine whether there is a style tag uploaded by the user; If it exists, verify the style tag with a predefined style tag set, and when the verification passes, use the style tag as a target style condition; If not, the semantic feature vector is analyzed by using a style encoder to identify the style label corresponding to the semantic feature vector, and the style label is used as the target style condition.

5. The method for generating a composite style speech according to claim 1, wherein: The process of processing the text semantic information and the target style condition through a gating network to obtain a style weight includes: Extracting a style label vector for each style label in the target style condition; Calculating a distance value between a semantic feature vector in the semantic information and the style label vector; The style weight is calculated according to the distance value.

6. The method for generating a composite style speech according to claim 1, wherein: The converting the composite style feature by a speech synthesizer to obtain the composite style speech of the text sequence includes: Extracting a feature graph of the composite style speech using a generator of the speech synthesizer; Iterating the feature map through the multi-layer convolutional layers of the generator to obtain a final feature map; The final feature map is converted into a composite style speech of the text sequence.

7. The method for generating a composite style speech according to claim 6, wherein: Before extracting features from the text sequence using a text encoder, the method further comprises: Acquire a first training set, and use the first training set to train each of the style expert networks of the style expert module to obtain a plurality of trained style expert networks; Constructing an initial speech synthesis model, wherein the initial speech synthesis model includes an initial text encoder, an initial style expert module and an initial speech synthesizer, wherein the initial style expert module includes a plurality of trained style expert networks and an initial gating network; A second training set is obtained, and the initial speech synthesis model is trained using the second training set to obtain the speech synthesis model.

8. A device for generating a composite style speech, characterized in that: The device comprises: An encoding module, used for acquiring a text sequence, performing feature extraction on the text sequence using a text encoder, and obtaining text semantic information, wherein the text semantic information includes a semantic feature vector and a prosodic feature vector; A conversion module, configured to perform style conversion on the text semantic information through each style expert network of the style expert module to obtain a plurality of style speech features; A weighting module, used to obtain a target style condition, process the text semantic information and the target style condition through a gating network to obtain a style weight, and perform a weighted sum operation on a plurality of the style speech features according to the style weight to obtain a composite style feature; The speech generation module is used to convert the composite style features through a speech synthesizer to obtain the composite style speech of the text sequence.

9. A computer device, characterized in that: The computer device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for generating a composite style speech according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for generating a composite style speech according to any one of claims 1 to 7 is implemented.

Images